Prompt Caching

Question 1

What does prompt caching actually store on the server?

Accepted Answer

The KV-cache of the prompt's prefix — Prompt caching preserves the attention states of the prefix. On a hit the model skips prefill and jumps straight to the dynamic tail.

Question 2

Which content should sit at the VERY TOP of a cache-friendly prompt?

Accepted Answer

The system prompt and tool schemas — Cache hits require an identical byte-prefix. Put invariant content first so the shared prefix is as long as possible.

Question 3

You add a single whitespace character halfway through your system prompt. What happens on the next call?

Accepted Answer

Everything after it must re-prefill — Caching is byte-exact and prefix-only. The prefix up to the change may still hit; everything after must be recomputed.

Question 4

Anthropic's ephemeral prompt cache TTL is roughly 5 minutes. For which workload does this hurt the most?

Accepted Answer

Low-QPS bursty traffic with inter-request gaps exceeding the TTL — Low-QPS workloads see most prefixes expire before reuse. Solutions: send periodic keep-alive calls or accept the write penalty on the first user after a lull.

Question 5

What is the single biggest anti-pattern that silently disables prompt caching?

Accepted Answer

Injecting a per-request timestamp or user ID at the top of the prompt — A dynamic token in the prefix makes every request unique. Move dynamic values to the tail (or into a tool call) instead.

Question 6

Anthropic charges roughly 1.25× on the first write and ~0.10× on reads. Under what read/write ratio does caching become a net loss?

Accepted Answer

Below ~0.3 cache reads per write so rewrites outnumber reads — Write is priced above baseline (1.25×); reads are far below (0.10×). Solving 1.25 + 0.10r = 1 + r gives a break-even around r ≈ 0.28 reads per write — below that, you're paying to cache something you almost never read back.

Question 7

You have segments A, B, C, each with a slightly different system prompt. How do you avoid fragmenting the cache into three separate prefixes?

Accepted Answer

Put shared instructions first, then segment-specific content — Structure the prompt as [shared] + [segment-specific] + [dynamic]. Only the segment-specific block breaks reuse between segments.

Question 8

For a large stable knowledge base (200K tokens) reused across sessions, which provider feature fits best?

Accepted Answer

Gemini's persistent cachedContent resource with configurable expiry — Gemini's cached content is built for large corpora with predictable reuse windows; explicit TTL avoids the low-QPS expiry problem.

Question 9

Tool schemas typically go in the system section. What impact does reordering them between calls have?

Accepted Answer

It breaks the cache prefix — Byte-identical prefixes require deterministic ordering. Serialize tool schemas in a stable order.

Question 10

How does prompt caching compose with a RAG pipeline?

Accepted Answer

RAG chunks go after the cached prefix — You cache what does not change. RAG chunks are dynamic per query and belong in the tail; the stable frame caches once and reads on every call.

Key insight

Lay out a cacheable prompt

Test your knowledge

What does prompt caching actually store on the server?

The short version

In more depth

Prompt caching

The golden rule

Provider differences

What breaks the cache

Composes with everything else

Interview Questions

References & Further Reading

Related topics

Love this visualization?