Cutting Anthropic Token Costs and Latency with Prompt Caching
Automatic caching vs. explicit breakpoints, the TTL math by access pattern, and the trap where caching costs more than not caching at all.
If you’re building agents with Anthropic models, you’ve seen the pattern: token prices keep falling, and your invoice keeps climbing. The reason is structural. Multi-step, tool-using agents re-send their entire context - system prompt, tool definitions, conversation history - on every turn. An agent that takes 20 turns processes its prefix 20 times. Token volume compounds with every step, and so does time-to-first-token, because the model has to re-run its expensive prefill pass over all of it before emitting a single output token.
Prefill and KV Cache 101
Transformers generate text in two phases. In the prefill stage, the model ingests the whole prompt at once: every token computes attention against every token before it, and each layer stores the keys and values it computed for every token. That stored state is the Key Value (KV) Cache. Prefill is the expensive part - the attention work grows quadratically with prompt length - and it has to finish before the first output token appears. It’s where both your input-token cost and your time-to-first-token live.
In the decode (or generation) stage, the model generates one token at a time. Each new token recomputes nothing over the prompt: it reads the KV cache, attends to it, appends its own entry, and repeats. Decode is cheap per step precisely because prefill already did the heavy lifting - which means re-running prefill over a prefix the model has already processed, byte for byte, is pure waste.
Prompt caching removes that redundancy: the provider persists the prefix’s KV cache and reuses it across requests. It’s a config change rather than a re-architecture: cached tokens are billed at 10% of the base input price, and the cached portion skips prefill entirely. But it fails silently when misconfigured, and one specific misconfiguration costs more than not caching at all.
One scoping note up front: everything here uses Anthropic’s pricing model and API, where caching is opt-in and cache writes carry a premium. OpenAI’s automatic prefix caching has no write surcharge (and therefore can’t go net-negative), and Gemini’s context caching sits somewhere in between - the prefix-structure principles transfer, the math doesn’t.
What caching actually does
The reuse condition is strict: a cache hit requires the beginning of your prompt to be byte-identical to a previous request. On Anthropic’s API, the multipliers against base input price are:
A 5-minute cache write pays for itself on the first reuse - every hit after that is a 90% discount on that portion of the prompt. (The 1-hour write is 2× and needs a couple of reuses to break even; the math is worked out below.) The effect shows up in three places:
Cost. In an agent loop, the system prompt, tool definitions, and growing history ride along on every turn. A 20-turn run over a 10K-token prefix pays for ~200K input tokens uncached, the cost-equivalent of ~30K with caching.
Latency. A cache hit skips prefill over the cached portion. On long contexts this is the difference between a snappy first token and a multi-second stall - same model, same prompt, different shape.
Rate limits. On the Claude API, cache reads aren’t deducted against your input-token rate limits. At real traffic volumes this often matters more than the invoice: caching effectively raises your throughput ceiling without a sales conversation.
Turning it on: two modes, four breakpoints
Anthropic gives you two ways in, and they compose.
Automatic caching is one field at the top level of the request. The server places the cache point on the last cacheable block and moves it forward as the conversation grows - each turn reads the previous prefix from cache and writes the new tail:
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
cache_control={"type": "ephemeral"}, # that's it
system=SYSTEM_PROMPT,
messages=conversation_history,
)This is the right default for multi-turn conversations: zero bookkeeping, and the breakpoint management (which used to be the fiddly part) is the server’s problem. One platform caveat: Bedrock doesn’t support it - there you fall back to explicit breakpoints.
Explicit breakpoints put cache_control on individual content blocks - up to 4 per request. You want these when different sections of your prompt change at different frequencies: tool definitions change on deploy, a knowledge-base document changes daily, the conversation changes every turn. Each breakpoint marks the end of an independently reusable prefix. Place one on the last block of each section you want cached:
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=[
search_flights_tool,
book_flight_tool,
{
"name": "cancel_flight",
"description": "Cancel an existing booking.",
"input_schema": {...},
# last tool: changes on deploy
"cache_control": {"type": "ephemeral", "ttl": "1h"},
},
],
system=[{
"type": "text",
"text": LONG_STATIC_INSTRUCTIONS,
"cache_control": {"type": "ephemeral"}, # changes rarely
}],
messages=[{"role": "user", "content": user_message}], # changes every request
)Two rules govern the layout. First, the cache hierarchy is fixed: tools, then system, then messages, and a change at any level invalidates that level and everything after it. Edit a tool description and your entire cache is cold. Second, breakpoints themselves are free - you pay only for what’s written and read - so there’s no cost to using all four.
Cache mode composition. Explicit breakpoints pin the static prefix (tools, system instructions); the top-level automatic flag handles the moving conversation tail. The automatic breakpoint consumes one of the four slots; the rest are yours.
One mechanic worth knowing before it bites you: cache reads work by looking backward from your breakpoint for a prefix some earlier request already wrote - but only up to 20 blocks back. In a fast-growing conversation where a single turn adds more than 20 blocks (agents with many parallel tool calls do this), the lookback can walk right past your last write and miss it. The fix is boring: a second breakpoint earlier in the message history, so a write accumulates where the lookback can find it.
The TTL math, by access pattern
Caching comes in two lifetimes - 5 minutes and 1 hour - and picking wrong is not a rounding error. Here’s the property that drives everything: a cache hit refreshes the entry at no additional cost, and the TTL is a minimum lifetime, not a hard expiry. As long as the gap between consecutive requests stays under the TTL, one write keeps serving hits indefinitely. The moment a gap exceeds the TTL, the entry expires and the next request pays a full write again.
So the decision variable isn’t your total traffic - it’s your worst-case gap between requests touching the same prefix. Let’s make it concrete: a 50K-token stable prefix on a model with $3/MTok base input. Per request, that prefix costs:
Pattern 1 - steady interactive traffic (gaps under 5 minutes). A live chat session, an agent loop making back-to-back calls. The 5m cache never expires: 1 write + N hits. Twenty requests: $0.1875 + 19 × $0.015 = $0.47 versus $3.00 uncached - an 84% cut. The 1h TTL buys you nothing here except a more expensive first write. Use 5m.
Pattern 2 - sporadic traffic (gaps between 5 minutes and 1 hour). A scheduled job every 20 minutes, a support conversation where the human replies whenever they reply, a long-running agent whose individual steps take 10 minutes. This is where the 5m cache turns actively harmful: every request arrives to a cold cache, pays the 1.25× write premium, and nothing ever reads it. Seventy-two requests over a day, 20 minutes apart:
The 1h line is 1 write + 71 refreshing hits, because every 20-minute gap is comfortably under the hour. Note the middle row: caching with the wrong TTL costs more than not caching at all. (Again, this failure mode is specific to pricing models with write premiums - it can’t happen on OpenAI’s automatic caching.) If you see cache_creation_input_tokens on every request and reads on none, you’re in that row.
Pattern 3 - batch and fan-out workloads. This is where longer TTLs are the structurally right choice. Batch-shaped work - process 500 documents against the same system prompt, run an evaluation suite, fan an agent out over a work queue - has three properties that all favor 1h:
Gaps are irregular by design. Queue depth varies, some items take 90 seconds and some take 12 minutes, workers pause and resume. A 5m TTL turns every slow item into a cache miss for the next one; 1h absorbs the variance. The write premium is a one-time 2× on the prefix, amortized across the whole batch - on our 500-document run, $0.30 once versus $0.015 × 499 for everything after: $7.79 total against $75 uncached.
Parallelism needs a warm-up step. A cache entry only becomes available once the first response begins. Fire 50 concurrent requests against a cold cache and all 50 miss - and all 50 pay the write premium. The right sequence is: send one request (or a max_tokens: 0 pre-warm, which bills zero output tokens), wait for it, then fan out. With a 1h TTL, one pre-warm covers the whole run even if the fan-out takes 40 minutes.
The countercase proves the rule. Anthropic’s async Message Batches API rejects pre-warming outright, precisely because batch items may execute long after a short-lived cache has expired. When you orchestrate the batch yourself, the TTL is the knob that keeps the cache alive across the run’s actual duration - so size it to the run, not to the default.
You can also mix TTLs in one request - the constraint is that longer TTLs must come first in the prompt. The layout falls out naturally: tools and system prompt on 1h (they’re shared across every user and every batch item), conversation tail on 5m or automatic (it’s specific to one fast-moving session).
Break-even, if you want the general rule instead of worked examples. A 5m write (1.25×) beats uncached from the first reuse: 1.35× total against 2× for two uncached requests. A 1h write (2×) is a wash at one reuse (2.1× vs. 2.0×) and wins from the second reuse onward - so it needs three total requests within the hour to clearly pay off against no caching at all. Against the 5m strategy, 1h wins whenever your typical gap exceeds five minutes, because the comparison is one 2× write against an endless series of 1.25× writes. And if your prefix is reused less than once an hour - don’t cache it. Not everything should be.
Failure modes
In production, the failures are rarely in the happy path above. They show up here:
Misplacing cache breakpoints. The most common mistake: putting cache_control on the final block of the request - the one with the incoming user message. The cache key is a hash of everything up to and including the breakpoint; that block changes every request, so you pay a fresh write every time and never get a read. Automatic caching walks into the same trap if your last block is per-request context. Put the explicit breakpoint on the last block that stays identical across the requests you want sharing a cache.
Silent failure below the minimum. Prompts under the minimum cacheable length (512–4,096 tokens depending on the model) are processed without caching and without an error. The response tells you what actually happened:
u = response.usage
# total input = cache reads + cache writes + uncached tokens after the breakpoint
print(u.cache_read_input_tokens, u.cache_creation_input_tokens, u.input_tokens)If both counters are zero, you’re not caching. If you see writes on every request and reads on none, your “stable” prefix isn’t stable. If your prefix falls just short of the minimum, padding it up to the threshold is often cheaper than leaving it uncached - reads at 0.1× beat full price quickly.
Non-obvious invalidators. Exact match means exact. Things that break caches without touching a single word of the prompt: toggling a server-side tool like web search (it silently edits the system prompt), changing tool_choice, adding an image mid-conversation, and - less obviously - a runtime that serializes tool-definition JSON with unstable key ordering, producing a byte-different prefix on every single call. The FAQ at the end has the full invalidation table; when in doubt, diff two consecutive raw requests.
Your framework has knobs for this - verify what it sends
Most agent frameworks now expose all of the above declaratively. Pydantic AI is a good example of the shape to look for: independent switches for the conversation, the instructions, and the tool definitions, each with its own TTL:
agent = Agent(
"anthropic:claude-sonnet-4-6",
instructions="Detailed static instructions...",
model_settings=AnthropicModelSettings(
anthropic_cache=True, # auto-cache the growing conversation
anthropic_cache_instructions=True, # pin the system prompt (5m)
anthropic_cache_tool_definitions="1h", # tools change on deploy: longer TTL
),
)Notice that the TTL-per-section layout maps exactly onto the change-frequency argument from the math section - the framework is just spelling it for you. The good ones also handle details you’d otherwise get wrong: placing the cache boundary after static instructions but before dynamic ones (the ones interpolating today’s date or the user’s name), enforcing the 4-breakpoint budget by dropping the oldest markers first, and falling back gracefully on platforms without automatic caching. LangChain, LiteLLM, and the gateways have equivalents.
But frameworks add convenience, not correctness. A dynamic instruction in the wrong place breaks the cache no matter who placed the marker, and an abstraction layer is one more place where key ordering or block layout can silently shift. The verification loop is the same regardless of stack: read cache_write_tokens and cache_read_tokens off the result, and confirm the ratio matches your mental model of what’s stable.
What Exactly Invalidates the Cache?
The hierarchy is tools, then system, then messages, and a change at any level invalidates that level plus everything after it:
Summary
Prompt caching reduces to three decisions:
Where does the stable prefix end? Put the breakpoint there - on the last block that stays byte-identical across the requests you want sharing a cache.
What is the worst-case gap between requests? Pick the TTL that exceeds it: 5m for interactive traffic, 1h for sporadic and batch workloads.
Is the prefix reused at least once per TTL window? If not, don’t cache it - with a write premium, an unused cache entry costs more than no cache.
Then verify. cache_read_input_tokens and cache_creation_input_tokens on every response tell you which of the three decisions is wrong: no reads and no writes means the prefix is below the minimum length; writes on every request means the prefix isn’t as stable as assumed. The invalidation table above accounts for most of the surprises.