If you’re building agents with Anthropic models, you’ve seen the pattern: token prices keep falling, and your invoice keeps climbing. The reason is structural. Multi-step, tool-using agents re-send their entire context - system prompt, tool definitions, conversation history - on every turn. An agent that takes 20 turns processes its prefix 20 times. Token volume compounds with every step, and so does time-to-first-token, because the model has to re-run its expensive prefill pass over all of it before emitting a single output token.

Prefill and KV Cache 101

Transformers generate text in two phases. In the prefill stage, the model ingests the whole prompt at once: every token computes attention against every token before it, and each layer stores the keys and values it computed for every token. That stored state is the Key Value (KV) Cache. Prefill is the expensive part - the attention work grows quadratically with prompt length - and it has to finish before the first output token appears. It’s where both your input-token cost and your time-to-first-token live.

In the decode (or generation) stage, the model generates one token at a time. Each new token recomputes nothing over the prompt: it reads the KV cache, attends to it, appends its own entry, and repeats. Decode is cheap per step precisely because prefill already did the heavy lifting - which means re-running prefill over a prefix the model has already processed, byte for byte, is pure waste.

Prompt caching removes that redundancy: the provider persists the prefix’s KV cache and reuses it across requests. It’s a config change rather than a re-architecture: cached tokens are billed at 10% of the base input price, and the cached portion skips prefill entirely. But it fails silently when misconfigured, and one specific misconfiguration costs more than not caching at all.

One scoping note up front: everything here uses Anthropic’s pricing model and API, where caching is opt-in and cache writes carry a premium. OpenAI’s automatic prefix caching has no write surcharge (and therefore can’t go net-negative), and Gemini’s context caching sits somewhere in between - the prefix-structure principles transfer, the math doesn’t.

What caching actually does

The reuse condition is strict: a cache hit requires the beginning of your prompt to be byte-identical to a previous request. On Anthropic’s API, the multipliers against base input price are:

A 5-minute cache write pays for itself on the first reuse - every hit after that is a 90% discount on that portion of the prompt. (The 1-hour write is 2× and needs a couple of reuses to break even; the math is worked out below.) The effect shows up in three places:

1. Cost. In an agent loop, the system prompt, tool definitions, and growing history ride along on every turn. A 20-turn run over a 10K-token prefix pays for ~200K input tokens uncached, the cost-equivalent of ~30K with caching.

2. Latency. A cache hit skips prefill over the cached portion. On long contexts this is the difference between a snappy first token and a multi-second stall - same model, same prompt, different shape.

3. Rate limits. On the Claude API, cache reads aren’t deducted against your input-token rate limits. At real traffic volumes this often matters more than the invoice: caching effectively raises your throughput ceiling without a sales conversation.

Turning it on: two modes, four breakpoints

Anthropic gives you two ways in, and they compose.

Automatic caching is one field at the top level of the request. The server places the cache point on the last cacheable block and moves it forward as the conversation grows - each turn reads the previous prefix from cache and writes the new tail:

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},  # that's it
    system=SYSTEM_PROMPT,
    messages=conversation_history,
)

This is the right default for multi-turn conversations: zero bookkeeping, and the breakpoint management (which used to be the fiddly part) is the server’s problem. One platform caveat: Bedrock doesn’t support it - there you fall back to explicit breakpoints.

Explicit breakpoints put cache_control on individual content blocks - up to 4 per request. You want these when different sections of your prompt change at different frequencies: tool definitions change on deploy, a knowledge-base document changes daily, the conversation changes every turn. Each breakpoint marks the end of an independently reusable prefix. Place one on the last block of each section you want cached:

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    tools=[
        search_flights_tool,
        book_flight_tool,
        {
            "name": "cancel_flight",
            "description": "Cancel an existing booking.",
            "input_schema": {...},
            # last tool: changes on deploy
            "cache_control": {"type": "ephemeral", "ttl": "1h"},
        },
    ],
    system=[{
        "type": "text",
        "text": LONG_STATIC_INSTRUCTIONS,
        "cache_control": {"type": "ephemeral"},               # changes rarely
    }],
    messages=[{"role": "user", "content": user_message}],     # changes every request
)

Two rules govern the layout. First, the cache hierarchy is fixed: tools, then system, then messages, and a change at any level invalidates that level and everything after it. Edit a tool description and your entire cache is cold. Second, breakpoints themselves are free - you pay only for what’s written and read - so there’s no cost to using all four.

Cache mode composition. Explicit breakpoints pin the static prefix (tools, system instructions); the top-level automatic flag handles the moving conversation tail. The automatic breakpoint consumes one of the four slots; the rest are yours.

One mechanic worth knowing before it bites you: cache reads work by looking backward from your breakpoint for a prefix some earlier request already wrote - but only up to 20 blocks back. In a fast-growing conversation where a single turn adds more than 20 blocks (agents with many parallel tool calls do this), the lookback can walk right past your last write and miss it. The fix is boring: a second breakpoint earlier in the message history, so a write accumulates where the lookback can find it.

The TTL math, by access pattern

Caching comes in two lifetimes - 5 minutes and 1 hour - and picking wrong is not a rounding error. Here’s the property that drives everything: a cache hit refreshes the entry at no additional cost, and the TTL is a minimum lifetime, not a hard expiry. As long as the gap between consecutive requests stays under the TTL, one write keeps serving hits indefinitely. The moment a gap exceeds the TTL, the entry expires and the next request pays a full write again.

So the decision variable isn’t your total traffic - it’s your worst-case gap between requests touching the same prefix. Let’s make it concrete: a 50K-token stable prefix on a model with $3/MTok base input. Per request, that prefix costs:

Pattern 1 - steady interactive traffic (gaps under 5 minutes). A live chat session, an agent loop making back-to-back calls. The 5m cache never expires: 1 write + N hits. Twenty requests: $0.1875 + 19 × $0.015 = $0.47 versus $3.00 uncached - an 84% cut. The 1h TTL buys you nothing here except a more expensive first write. Use 5m.

Pattern 2 - sporadic traffic (gaps between 5 minutes and 1 hour). A scheduled job every 20 minutes, a support conversation where the human replies whenever they reply, a long-running agent whose individual steps take 10 minutes. This is where the 5m cache turns actively harmful: every request arrives to a cold cache, pays the 1.25× write premium, and nothing ever reads it. Seventy-two requests over a day, 20 minutes apart:

The 1h line is 1 write + 71 refreshing hits, because every 20-minute gap is comfortably under the hour. Note the middle row: caching with the wrong TTL costs more than not caching at all. (Again, this failure mode is specific to pricing models with write premiums - it can’t happen on OpenAI’s automatic caching.) If you see cache_creation_input_tokens on every request and reads on none, you’re in that row.

Pattern 3 - batch and fan-out workloads. This is where longer TTLs are the structurally right choice. Batch-shaped work - process 500 documents against the same system prompt, run an evaluation suite, fan an agent out over a work queue - has three properties that all favor 1h:

• Gaps are irregular by design. Queue depth varies, some items take 90 seconds and some take 12 minutes, workers pause and resume. A 5m TTL turns every slow item into a cache miss for the next one; 1h absorbs the variance. The write premium is a one-time 2× on the prefix, amortized across the whole batch - on our 500-document run, $0.30 once versus $0.015 × 499 for everything after: $7.79 total against $75 uncached.

• Parallelism needs a warm-up step. A cache entry only becomes available once the first response begins. Fire 50 concurrent requests against a cold cache and all 50 miss - and all 50 pay the write premium. The right sequence is: send one request (or a max_tokens: 0 pre-warm, which bills zero output tokens), wait for it, then fan out. With a 1h TTL, one pre-warm covers the whole run even if the fan-out takes 40 minutes.

• The countercase proves the rule. Anthropic’s async Message Batches API rejects pre-warming outright, precisely because batch items may execute long after a short-lived cache has expired. When you orchestrate the batch yourself, the TTL is the knob that keeps the cache alive across the run’s actual duration - so size it to the run, not to the default.

You can also mix TTLs in one request - the constraint is that longer TTLs must come first in the prompt. The layout falls out naturally: tools and system prompt on 1h (they’re shared across every user and every batch item), conversation tail on 5m or automatic (it’s specific to one fast-moving session).

Break-even, if you want the general rule instead of worked examples. A 5m write (1.25×) beats uncached from the first reuse: 1.35× total against 2× for two uncached requests. A 1h write (2×) is a wash at one reuse (2.1× vs. 2.0×) and wins from the second reuse onward - so it needs three total requests within the hour to clearly pay off against no caching at all. Against the 5m strategy, 1h wins whenever your typical gap exceeds five minutes, because the comparison is one 2× write against an endless series of 1.25× writes. And if your prefix is reused less than once an hour - don’t cache it. Not everything should be.

Failure modes

In production, the failures are rarely in the happy path above. They show up here:

Misplacing cache breakpoints. The most common mistake: putting cache_control on the final block of the request - the one with the incoming user message. The cache key is a hash of everything up to and including the breakpoint; that block changes every request, so you pay a fresh write every time and never get a read. Automatic caching walks into the same trap if your last block is per-request context. Put the explicit breakpoint on the last block that stays identical across the requests you want sharing a cache.

Silent failure below the minimum. Prompts under the minimum cacheable length (512–4,096 tokens depending on the model) are processed without caching and without an error. The response tells you what actually happened:

u = response.usage
# total input = cache reads + cache writes + uncached tokens after the breakpoint
print(u.cache_read_input_tokens, u.cache_creation_input_tokens, u.input_tokens)

If both counters are zero, you’re not caching. If you see writes on every request and reads on none, your “stable” prefix isn’t stable. If your prefix falls just short of the minimum, padding it up to the threshold is often cheaper than leaving it uncached - reads at 0.1× beat full price quickly.

Non-obvious invalidators. Exact match means exact. Things that break caches without touching a single word of the prompt: toggling a server-side tool like web search (it silently edits the system prompt), changing tool_choice, adding an image mid-conversation, and - less obviously - a runtime that serializes tool-definition JSON with unstable key ordering, producing a byte-different prefix on every single call. The FAQ at the end has the full invalidation table; when in doubt, diff two consecutive raw requests.

Your framework has knobs for this - verify what it sends

Most agent frameworks now expose all of the above declaratively. Pydantic AI is a good example of the shape to look for: independent switches for the conversation, the instructions, and the tool definitions, each with its own TTL:

agent = Agent(
    "anthropic:claude-sonnet-4-6",
    instructions="Detailed static instructions...",
    model_settings=AnthropicModelSettings(
        anthropic_cache=True,                   # auto-cache the growing conversation
        anthropic_cache_instructions=True,      # pin the system prompt (5m)
        anthropic_cache_tool_definitions="1h",  # tools change on deploy: longer TTL
    ),
)

Notice that the TTL-per-section layout maps exactly onto the change-frequency argument from the math section - the framework is just spelling it for you. The good ones also handle details you’d otherwise get wrong: placing the cache boundary after static instructions but before dynamic ones (the ones interpolating today’s date or the user’s name), enforcing the 4-breakpoint budget by dropping the oldest markers first, and falling back gracefully on platforms without automatic caching. LangChain, LiteLLM, and the gateways have equivalents.

But frameworks add convenience, not correctness. A dynamic instruction in the wrong place breaks the cache no matter who placed the marker, and an abstraction layer is one more place where key ordering or block layout can silently shift. The verification loop is the same regardless of stack: read cache_write_tokens and cache_read_tokens off the result, and confirm the ratio matches your mental model of what’s stable.

What Exactly Invalidates the Cache?

The hierarchy is tools, then system, then messages, and a change at any level invalidates that level plus everything after it:

Summary

Prompt caching reduces to three decisions:

1. Where does the stable prefix end? Put the breakpoint there - on the last block that stays byte-identical across the requests you want sharing a cache.

2. What is the worst-case gap between requests? Pick the TTL that exceeds it: 5m for interactive traffic, 1h for sporadic and batch workloads.

3. Is the prefix reused at least once per TTL window? If not, don’t cache it - with a write premium, an unused cache entry costs more than no cache.

Then verify. cache_read_input_tokens and cache_creation_input_tokens on every response tell you which of the three decisions is wrong: no reads and no writes means the prefix is below the minimum length; writes on every request means the prefix isn’t as stable as assumed. The invalidation table above accounts for most of the surprises.

Nowadays, most people seem to over-engineer their RAG stack. They jump straight to embeddings, vector databases, and reranking pipelines. Meanwhile, their users just want to find the doc that says “How to reset my password.”

In engineering, there’s always the right tool for the right problem. In AI Retrieval Systems it’s not different.

The Decision Factors

Before we dive into recipes, let’s establish when you should use each approach. The key factors are:

1. Data Freshness Requirements - Real-time updates (news, social media) favor approaches with easy re-indexing. Daily or weekly updates work well with hybrid approaches. A stable corpus (monthly or quarterly updates) makes pre-embedding sensible.

2. Corpus Characteristics - High churn (more than 10% changes daily) means you should avoid full pre-embedding. Stable documents work fine with pre-embedding. Long-tail distribution (90% never accessed) means on-the-fly wins.

3. Query Patterns - Keyword-heavy queries should start with full-text search. Semantic or conversational queries benefit from embeddings. Mixed patterns need hybrid approaches.

4. Scale & Performance - Less than 1000 queries per day means simple approaches are sufficient. 1K to 10K queries per day requires selective optimization. More than 10K queries per day justifies full optimization.

5. Team Capabilities - No ML expertise means stay with full-text plus query rewriting. Some ML experience makes hybrid search manageable. Having an ML team available makes advanced approaches viable.

Now, let’s look at the recipe book. Start at the top. Move down only when you have data proving you need to.

Recipe 1: The MVP – Full-Text Search Only

What it is

Good old BM25. Elasticsearch. Postgres full-text search. The stuff that existed before “embedding” became a verb.

When to use

You’re just starting out. Your users write keyword-style queries (”pandas merge dataframe”). Exact matches matter (”invoice #12345”). You want zero ML complexity. Your corpus has proprietary terminology (more on this later).

Pros

Zero API costs. Fast (under 10ms). Easy to debug (you can see exactly why a document matched). Surprisingly effective (handles many use cases). No chunking strategy needed – works with full documents. No evaluation complexity – easy to test and validate. No model deprecation risk (BM25 doesn’t change).

Cons

Misses synonyms (”car” vs “automobile”). Fails on semantic queries (”How do I...?”). Can’t understand intent beyond keywords.

Real talk

In my experience, this handles a significant portion of use cases. Don’t skip this step. You might be surprised how far you can get.

Why this is underrated

When you jump straight to embeddings, you immediately face questions like: What chunk size? (512 tokens? 1024?) What overlap? (50 tokens? 100?) Semantic chunking or fixed-size? How do I evaluate if my chunking is good?

With full-text search, you skip all of this. Your documents are your documents. Search just works.

Recipe 2: Agentic Query Rewriting

What it is

Use an LLM to transform messy user queries into clean keyword searches.

The insight

Most “semantic search” problems are actually query formulation problems.

When to use

Users ask questions conversationally. Vocabulary mismatch (users say “fix bugs”, docs say “debugging”). You have internal jargon (your framework called “Atlas”). You want flexibility to iterate quickly on query strategies.

Cost

~$0.001 per query (using GPT-4o-mini for query rewriting)

The magic

An LLM can remove stopwords (”how do I” becomes nothing). It can add synonyms (”car” becomes “car automobile vehicle”). It can translate domain terms (”speed up code” becomes “optimize performance”). It can decompose complex queries (”read CSV and plot” becomes [”read CSV”, “plot data”]). It can learn from your glossary (via system prompt).

Why this is more flexible than embeddings

With embeddings, if results aren’t good, you need to adjust chunking strategy, re-embed entire corpus, run regression tests on your eval set, and hope it improved.

With query rewriting, if results aren’t good, you adjust the system prompt. That’s it. Test immediately.

Multi-turn agentic rewriting

Even better, you can create a loop:

def agentic_search(query, max_iterations=3):
    for i in range(max_iterations):
        # Rewrite query
        optimized = query_rewriter.rewrite(query, iteration=i)
        
        # Search
        results = bm25_search(optimized)
        
        # Evaluate quality
        quality = evaluate_results(results, query)
        
        if quality > threshold:
            return results
        
        # Agent learns and tries again
        query = refine_based_on_feedback(query, results, quality)
    
    return results

The agent can iterate, learn, and adapt – all without re-embedding anything.

Example: The Proprietary Terminology Problem

Say your company has a Python framework called “Atlas.” If you use general-purpose embeddings:

General embedding model (trained on internet):
“Atlas” = [vectors pointing toward: Greek mythology, maps, geography]
Your actual Atlas docs = [vectors about data processing]
Similarity score: 0.15 (terrible!)

The model has no idea your “Atlas” exists. It falls back to what it learned in training. But with query rewriting:

system_prompt = """
  Domain-specific terms (NEVER modify these, use as exact keywords):
    - Atlas: our internal data processing framework
    - Mercury: our messaging system
    - Zeus: our auth service

    Preserve these terms exactly and optimize the rest of the query.
"""

# User: "How do I use Atlas for batch jobs?"
# Agent: "Atlas batch jobs data processing pipeline"
# BM25: Perfect match on "Atlas" ✓

For proprietary terms, exact keyword matching beats semantic understanding.

Recipe 3: Hybrid Search (Sparse + Dense Reranking)

What it is

Use BM25 to get candidates (top 50-100), then rerank with embeddings (top 10).

Why this works

BM25 is fast and great at keyword matching. Embeddings are good at semantic understanding. Together, they cover each other’s weaknesses.

When to use

Users ask semantic questions (”find alternatives to X”). BM25 plus query rewriting alone isn’t cutting it (you have data proving this). You can tolerate 100-500ms latency. Your corpus is relatively stable (not changing every minute).

The pipeline

Cost considerations

Let’s do the math with current pricing (OpenAI text-embedding-3-small at $0.02 per 1M tokens):

• Embedding 50 docs per query (avg 500 tokens each) means 50 docs × 500 tokens = 25,000 tokens

• Cost: 25,000 × $0.00002 = ~$0.0005 per query. At 1,000 queries per day × 30 days = ~$15 per month.

Actually pretty reasonable. But there’s a catch: latency.

Embedding 50 documents on-the-fly adds 200-500ms per query. For user-facing search, that’s noticeable. This is where the real trade-off lives – not cost, but speed.

Important consideration: The chunking problem returns

When you introduce embeddings, you need to decide how to chunk your documents (fixed-size? semantic? by section?). You need to determine what chunk size and overlap to use. You need to handle chunks that span important context.

This adds complexity that pure full-text search avoids.

Recipe 4: On-The-Fly Embedding (The Fresh Data Play)

The insight

If your data changes frequently, why pay to re-embed everything?

What it is

When to use

High document churn (more than 10% of docs updated daily). Real-time content (news, social media, live updates). You’re experimenting with embedding models (no re-indexing needed). Data freshness is critical (documents must be up-to-date). Small K for reranking (20-50 docs).

Math time

On-the-fly / online (1000 queries/day, 50 docs/query):
- Embedding cost: ~$15/month (ongoing)
- Storage: $0 (just store text)
- Latency: 200-500ms per query
- Freshness: Perfect (always current)
- Model switching: Easy (just change the API call)

The model deprecation benefit

Here’s something people don’t talk about enough: embedding models get deprecated.

OpenAI deprecated text-embedding-ada-002 in favor of text-embedding-3. If you pre-embedded 10 million documents with the old model, you now need to re-embed all 10 million documents with the new model, update your vector database, run regression tests on your evaluation set, validate that quality didn’t degrade, handle the cutover period, and deal with any API changes.

With on-the-fly / online embedding

You literally just change one line of code. Done.

The downside

Latency. You’re embedding documents on every query. This is only viable if you’re okay with 200-500ms latency, K is small (reranking 20-50 docs, not 500), and your use case favors freshness over speed.

Recipe 5: Pre-Embedding with Hot/Cold Tiers (The Pragmatic Play)

What it is

Pre-embed frequently accessed documents (”hot tier”), embed rarely-accessed documents on-the-fly (”cold tier”).

The insight

Access patterns follow Pareto distribution. 20% of docs get 80% of traffic.

# Track access patterns
access_counts = Counter()

def adaptive_search(query):
    # BM25 to get candidates
    candidates = bm25_search(query, top_k=100)
    
    # Separate hot and cold
    hot = [d for d in candidates if d.id in hot_tier]
    cold = [d for d in candidates if d.id not in hot_tier]
    
    # Hot docs: use pre-computed embeddings (fast)
    hot_scores = vector_db.similarity_search(query_emb, hot)
    
    # Cold docs: embed on-the-fly (slower, but rare)
    cold_scores = embed_and_score(cold, query_emb)
    
    return merge_and_rank(hot_scores, cold_scores)

## Periodically promote frequently accessed docs to hot tier
def update_tiers_weekly():
    frequently_accessed = [doc_id for doc_id, count 
                          in access_counts.items() 
                          if count > threshold]
    
    # Only re-embed the new hot docs
    newly_hot = set(frequently_accessed) - set(hot_tier)
    embed_and_index(newly_hot)

When to use

Clear access patterns (some docs are accessed way more than others). Medium-to-large corpus (more than 100K documents). Mix of stable and changing content. Need good latency for common queries. Want to minimize re-embedding on model updates.

Benefits

Fast for 80% of queries (hit pre-embedded cache). Fresh for rarely-accessed docs. Only re-embed hot tier when switching models (20% of corpus). Adapts to changing access patterns. Best latency/cost/flexibility trade-off.

The model update story

When your embedding model gets deprecated:

Full pre-embedding: Re-embed 1M docs × $0.01 = $10,000 + downtime
Hot/cold tiers: Re-embed 200K docs × $0.01 = $2,000 + minimal downtime
On-the-fly: Change one line of code = $0 + zero downtime

Recipe 6: Full Pre-Embedding (The Scale Play)

What it is

Embed everything upfront. Store in vector database. Search with ANN (approximate nearest neighbors).

When to use

Very high query volume (more than 10K queries per day). Need under 50ms latency. Very stable corpus (under 5% churn per month). Access pattern is broad (no long tail). You have ML team to manage infrastructure.

Cost breakdown

Pre-embedding (1M docs):
- One-time embedding: 1M docs × 500 tokens × $0.00002 = $10
- Storage: 1M × 1536 dims × 4 bytes = 6GB (~$10-30/month)
- Search latency: under 50ms (blazing fast!)
- Freshness: Only as fresh as last re-index

When NOT to use

Documents change frequently (more than 10% per week). You’re experimenting with embedding models. Low query volume (under 1K queries per day). You haven’t tried simpler approaches first.

The model deprecation nightmare

This is where full pre-embedding hurts the most. When you need to switch models, you face downtime (your search is degraded while re-embedding), compute cost (re-embedding millions of documents), testing burden (full regression test suite on new embeddings), chunking reevaluation (maybe new model works better with different chunk sizes?), and risk (what if the new model is worse for your domain?).

This is overkill for most systems. I’ve seen teams spend months optimizing their vector database setup when query rewriting would have solved 90% of their problems.

But if you’re Pinterest, Shopify, or handling massive scale with a stable corpus, this is where you end up.

The Multi-Intent Query Problem

Here’s where things get spicy. We’ve been discussing single-intent queries: “How do I merge dataframes?”

But real users ask stuff like: “How do I read a CSV file, clean missing data, and plot the results?”

That’s three separate intents. Searching for this as one query is like trying to find a restaurant that serves pizza, sushi, and tacos. Good luck.

The Perplexity Playbook

Modern agentic RAG systems (Perplexity, ChatGPT search) handle this elegantly:

Query Understanding Agent

Break down the query.

# Input: "read CSV, clean data, plot results"

# Agent output:

{
  "query_type": "complex",
  "sub_queries": [
    "pandas read csv file",
    "pandas clean missing data",
    "matplotlib plot dataframe"
  ],
  "dependencies": ["read > clean > plot"]
}

Parallel Adaptive Processing

Route each sub-query optimally

Sub-query 1 (simple):
  "pandas read csv"
  Stopwords + lemma, then BM25
  Cost: $0, Latency: 15ms

Sub-query 2 (moderate):
  "pandas clean missing data"
  Synonym expansion, then BM25
  Cost: $0, Latency: 20ms

Sub-query 3 (complex):
  "matplotlib plot dataframe"
  LLM rewrite, then Multi-search
  Cost: $0.001, Latency: 250ms

Total (parallel): $0.002, 250ms (not 285ms!)

Synthesize

Combine results into coherent answer

Here’s a complete workflow:

1. Reading CSV Files
   [relevant docs from sub-query 1]
   
2. Cleaning Missing Data
   [relevant docs from sub-query 2]
   
3. Plotting Results
   [relevant docs from sub-query 3]

[Code example combining all three steps]

Why this works

Each sub-query is focused and precise, leading to better retrieval. Parallel execution means lower latency (max, not sum). Adaptive routing results in lower cost (only complex queries pay for LLM). Structured output provides better UX.

Cost comparison

Without decomposition

• LLM rewriting entire complex query: $0.005

• Embedding 50 docs: $0.025

• Total: $0.03

With decomposition

• Decompose: $0.001

• Sub-query 1 (simple): $0

• Sub-query 2 (simple): $0

• Sub-query 3 (complex): $0.001

• Total: $0.002

15x cheaper, better quality.

This is where agentic retrieval really shines. The agent can intelligently decide which sub-queries need expensive processing (embeddings) and which can be handled with cheap methods (simple preprocessing + BM25).

The Decision Tree (Or: When to Use What)

Okay, you’ve read this far. You just want to know: “What should I build?”

Start here: Do you have search at all? If not, build BM25 first. Seriously. Stop reading and build it. If you do have search, continue.

Measure your baseline. Run your current search for 2-4 weeks and collect user feedback. Are users happy with the results? If yes, stop. You’re done. Go ship features. If no, continue.

What’s the main complaint?

If users say “Can’t find docs that clearly exist,” try query rewriting first. At $0.001 per query with zero re-indexing, it’s worth testing. Run an A/B test for 2 weeks. If you see good improvement, keep it and you’re done. If it’s not enough, continue.

If users say “Results are okay but not great,” A/B test hybrid search (sparse plus embedding rerank). Is the added latency worth it? If yes, decide on implementation. If your data changes frequently, use on-the-fly embedding. If you have clear hot docs, use hot/cold tiers. If you have a stable corpus and high scale, use full pre-embedding. If the latency isn’t worth it, optimize query rewriting further instead.

If users say “Need better semantic understanding,” use hybrid search and choose your approach based on your situation. High churn (more than 10% per day) means on-the-fly. Medium scale with clear patterns means hot/cold tiers. Massive scale with stable data means full pre-embedding.

Key decision factors:

Full-text with query rewriting offers perfect data freshness with low setup complexity and query latency under 50ms. Model switching is trivial, no chunking is needed, and it works for most use cases.

On-the-fly embedding provides perfect data freshness with low setup complexity but higher query latency of 200-500ms. Model switching is trivial, chunking is needed, and it’s best for high churn scenarios.

Hot/cold tiers provide mixed data freshness with medium setup complexity and query latency of 50-100ms. Model switching is easy, chunking is needed, and it offers balanced performance for varied needs.

Full pre-embedding has stale data until reindex with high setup complexity but query latency under 50ms. Model switching is painful, chunking is needed, and it’s designed for massive scale operations.

The 80/20 rule: 60% of systems should stop at full-text plus query rewriting. 25% need hybrid with on-the-fly or hot/cold. 10% need full pre-embedding. 5% need custom solutions.

Bottomline: Don’t be the person who builds the 5% solution for a 60% problem.

The page loaded. That should have been the end of it.

I’d just shipped a feature in the Next.js agent evaluation suite I’ve been building. The button did the thing the button was supposed to do. A spinner spun. A row appeared in the database. By every measure I’d asked for, the work was done.

I opened DevTools anyway, the way a mechanic listens to an engine after a repair, and clicked the button once.

The endpoint fired four times.

I want to describe the feeling, because I think the feeling is the whole point. It wasn’t alarm. It wasn’t even surprise. It was the small, almost pleasant click of a suspicion being confirmed before you knew you were suspicious. Four requests, stacked neatly in the panel, identical payloads, identical responses, all triggered by one human gesture.

The feature worked. The feature was also, in some quiet, expensive way, broken.

Nothing in the world had been going to tell me that. Claude Code had done exactly what I asked and nothing more — it even told me I was absolutely right many times. The page, dutiful and unbothered, had loaded.

This is what I mean when I say vibe coders have a reading problem. Not literacy in the schoolyard sense. Reading as in: looking at a thing that appears to be working and asking it to prove that it is. Reading as in the network tab, the rendered DOM, the actual query Drizzle produced, the diff you didn’t ask for.

Coding assistants optimize for what you ask it to optimize for. They don’t optimize for what you don’t think to measure.

And measuring is reading, and reading is slow, and slowness, right now, feels like the one luxury we’ve all agreed we no longer have.

The Network tab

The bug, when I tracked it down, was the kind of thing that gets flagged in a careful code review but that no model I’ve used has flagged unprompted: a useEffect watching a config object recreated inline on every render. New object identity, new effect run, new fetch, new state, new render, new object. A re-render storm.

The code looked fine. It was wrong in the specific way that only running code, watched closely, makes obvious — the Network tab was a waterfall the moment the page loaded.

I fixed it in two lines — wrapping the object in a useMemo with an honest dependency array. The fix took eleven minutes. Finding it took instinct I’m not sure I’d have if I’d grown up prompting instead of debugging.

I’ve started doing something else, too, and I want to be careful here, because it sounds like a productivity tip and I don’t mean it as one.

Hats

“You’re totally right!”, until you’re not

Every couple of features — every time the codebase crosses some invisible threshold of accumulation — I stop adding things and ask the agent to take off the implementer hat and put on an adversary’s. Architect’s hat. Performance reviewer’s hat. Security auditor’s hat. Read this codebase like you’re being paid to find what’s wrong with it.

The default mode of these tools is yes-and — extend, comply, ship. It’s a beautiful disposition for getting things done and a slightly dangerous one for noticing that the things you’re doing have started to rhyme with a mistake.

The first time I ran one of these passes, the reply came back as a tidy list of things I had been ignoring for a week. Not bugs, exactly. Drifts. An import that had reached across a layer it shouldn’t. A handler that caught an error and swallowed it. A type that had been quietly load-bearing in a way no one signed off on.

I read it twice. The feeling wasn’t quite embarrassment, though there was some of that; it was closer to the warmth of being shown the room you’ve been living in with the lights finally on. I’d seen all of it. I had not been looking.

The boring refactor that pays you three times

What surfaces on these passes is rarely dramatic. A 600-line component that wants to be five smaller ones. A prop drilled through four layers when context would have done. I split things up. I rename. I move a file.

The codebase gets a little more legible to me, and here is the part I didn’t expect: it gets a lot more legible to the coding assistant, too. Smaller files mean less unrelated context loaded per task, sharper boundaries, fewer wrong turns on the next request — Claude Code burns fewer tokens and makes fewer mistakes against a well-modularized repo.

A 600-line component is a kind of tax. You pay it every time the agent loads it into context, every time the runtime re-renders it, every time the bundler ships it.

But that’s only half of it, and the other half is the part I keep not seeing written down. A 600-line component is a kind of tax. You pay it every time the agent loads it into context, every time the runtime re-renders it, every time the bundler ships it.

In Next.js, the same boundaries that shrink your prompt context shrink your client bundles. A 600-line client component drags every one of its imports across the server/client boundary with it; split into focused pieces, most of them turn out not to need to be client components at all. The parts that genuinely need interactivity stay on the client. The rest move to the server, where they cost zero kilobytes to ship. Route-level code splitting suddenly has something to split. The bundle shrinks. The page gets faster.

None of this required “doing performance work” - it required reading the codebase honestly. Architecture, which I used to think of as a gift to my future human self, has quietly become part of the prompt, part of the render, and part of the page weight. Same discipline, three bills it pays.

The other thing that happens, on these review passes, is that I learn. I learn a TypeScript pattern I’d been working around for weeks. I learn why my generic was inferring unknown and what to do about it.

I have, in the last year, picked up more idiomatic TS from these conversations than from any book I’ve half-finished. Which, given that I’m coming to TypeScript from a long stretch of Python, is not a low bar.

The learning happens almost exclusively when I cast the model as a teacher and not a hand. It is the best tutor I’ve ever had. It is also, in implementer mode, perfectly content to let me stay exactly as ignorant as I was when I started.

The question

Which brings me to the third thing, and the one I find hardest, because it requires me to be smaller in the conversation than I instinctively want to be.

I’ve been trying to stop instructing. I’ve been trying to ask.

“Implement X this way” is a sentence I used to type a hundred times a day. It feels efficient. It feels like leadership. What it actually does is cap the ceiling of the output at the ceiling of my own knowledge on a Tuesday afternoon.

“I’m thinking of doing X — what are your thoughts?” costs the same tokens. Sometimes fewer. The reply is different in a way I find hard to describe except to say there is room in it.

Room for the model to mention the thing I didn’t know to ask about. Room for a tradeoff I hadn’t considered. Room, occasionally, for a flat “I wouldn’t do that, and here’s why,” which is the sentence I most need and most rarely get when I’ve already announced my answer in the question.

The problem is not you, Claude Code. The problem is me.

The conventional worry about AI and engineers is that we’ll get lazy. I don’t think that’s quite right. Lazy people, in my experience, know they’re being lazy.

The thing I’m watching happen, in myself on bad days and in others on most days, is subtler and stranger. It is the feeling of productivity uncoupled from the practice of attention. It is shipping without reading. It is the page loading and the laptop closing and the network tab, unwatched, firing four times into a void no one has agreed to look at.

Parting words

I should say this plainly. I am writing from inside this, not from some clean perch above it.

I have shipped code I didn’t fully read. I have accepted diffs because the tests were green and the hour was late and the difference between green and understood felt, at the time, like a difference I could afford to defer.

What I’m trying to hold onto is reading. Not as discipline — as the kind of curiosity that makes a person open a panel they didn’t have to open, on a feature that already worked.

That curiosity is, increasingly, the part of the job no one can do for me. It might be the part that’s still mine.

Let me tell you about my relationship with linear algebra.

Back in college, I sat through endless lectures about eigenvectors, matrix decompositions, and vector spaces. I dutifully memorized formulas for exams, promptly forgot them, and thought: “When will I ever use this?”

Fast forward to 2025. I’m building a RAG system, knee-deep in semantic search, and suddenly I’m computing dot products like my career depends on it. Because, well, it kind of does.

The punchline? That “useless” linear algebra is now the foundation of how we search through billions of documents, understand meaning, and make AI systems actually useful.

So here’s my redemption arc: Linear algebra isn’t boring. We just taught it wrong.

Why Everyone Suddenly Cares About Vectors

If you’ve built anything with RAG (Retrieval-Augmented Generation) in the last two years, you’ve heard the pitch: “Turn your text into embeddings, then search semantically!”

But what ARE embeddings, really?

Think of embeddings as coordinates in meaning-space. Every word, sentence, or document gets converted into a list of numbers (a vector) that captures its semantic essence.

“cat” = [0.2, 0.8, 0.1, 0.4, ...]  (hundreds or thousands of numbers)
“kitten” = [0.19, 0.79, 0.09, 0.41, ...]  (very close!)
“car” = [0.7, 0.1, 0.9, 0.2, ...]  (far away)

Words with similar meanings get similar coordinates. It’s like GPS, but for concepts.

Why this matters for RAG

Traditional keyword search is dumb. If you search for “automobile” and the document says “car,” you get nothing. Zero. Nada.

Embeddings are smart. They know “automobile” and “car” point to nearly the same spot in meaning-space. So when you search, you find what you meant, not just what you said.

This is why every RAG system worth its salt uses embeddings. They’re the difference between “404 Not Found” and “Here’s exactly what you need.”

· · ·

The Tale of Two Similarity Metrics

So you’ve got your vectors. Now you need to answer: “How similar are these two things?”

Enter our protagonists: Cosine Similarity and Dot Product.

If you remember high school math class (barely), the dot product is that thing where you multiply corresponding elements and add them up. Simple enough.

But cosine similarity? That’s where things get interesting.

Cosine Similarity

Cosine similarity measures the angle between two vectors. If they point in the same direction, similarity = 1. Opposite directions = -1. Perpendicular = 0.

cosine_similarity(A, B) = (A · B) / (||A|| × ||B||)

It’s called “cosine” because this formula literally computes the cosine of the angle between vectors. (That’s right, we’re doing trigonometry in high-dimensional space. College you would be proud.)

Why it’s famous: It ignores magnitude and focuses purely on direction. A short document and a long document about the same topic get the same similarity score. Fair and intuitive.

Dot Product

The dot product is simpler: just multiply corresponding elements and sum them up.

dot_product(A, B) = (a₁ × b₁) + (a₂ × b₂) + ... + (aₙ × bₙ)

No division. No square roots. Just multiply and add.

The catch: It cares about magnitude. Bigger vectors naturally get bigger scores, which can be... problematic.

· · ·

The Key Difference (And Why It Matters)

Here’s where it gets interesting.

Cosine similarity normalizes by magnitude. Think of it like this:

A · B = ||A|| × ||B|| × cos(θ)

The dot product equals: (magnitude of A) × (magnitude of B) × (cosine of angle).

Cosine similarity divides this by both magnitudes, isolating just the angle:

cos(θ) = (A · B) / (||A|| × ||B||)

In plain English

• Dot product = “How much do these vectors point in the same direction, weighted by their size?”

• Cosine similarity = “Do these vectors point in the same direction, period?”

Why This Distinction Matters

Imagine two documents:

Doc 1: “Cats are cute”
embedding magnitude = 0.8

Doc 2: “Cats are cute and fluffy and wonderful and purr and have whiskers...”
embedding magnitude = 2.3

Both are about cute cats. They should be considered similar.

Using dot product (unnormalized): Doc 2 gets a way higher score just because it’s longer. Unfair!

Using cosine similarity: Both get the same score (≈ 1.0) because they point in the same direction. Fair!

This is why cosine similarity became the gold standard for semantic search. It’s magnitude-invariant – it doesn’t penalize short documents or favor long ones.

Which Approach Should You Use?

Before we dive deeper, here’s the cheat sheet:

Full-text with rewriting costs about $0.001 per query with latency under 50ms. This works for about 60% of systems and offers zero re-indexing.

Hybrid on-the-fly approaches cost around $0.015 per query with 200-500ms latency. These work well for high churn data and provide perfect freshness.

Hot/cold tiers balance cost at $0.005 per query with 50-100ms latency. This offers the best trade-off for balanced needs.

Full pre-embedding is the cheapest at $0.0005 per query with latency under 50ms. This approach works for massive scale and provides maximum speed.

Keep this in mind as we explore the theory behind why these trade-offs exist.

The Plot Twist: Cosine Similarity Might Be Overkill

Here’s where our story takes a turn.

Most modern embedding APIs – OpenAI’s text-embedding-3, Cohere, Voyage AI – return normalized embeddings by default. (Always check your model’s documentation, but chances are, they’re normalized.)

What does “normalized” mean? Every vector has a magnitude of exactly 1.

And when both vectors have magnitude = 1:

cosine_similarity(A, B) = (A · B) / (1 × 1) = A · B

The dot product and cosine similarity are IDENTICAL.

So all that careful normalization we’ve been doing? That division by magnitudes? We’re dividing by 1. It’s like wearing a belt with suspenders.

Why This Changes Everything

Speed: Dot product is faster. No square roots, no division. Just multiply and add.

In high-scale systems (millions of documents, thousands of queries per second), this matters. Vector databases optimize for dot product similarity when working with normalized vectors for exactly this reason.

Simplicity: Less code, fewer bugs. One operation instead of three.

Performance: Every millisecond counts when you’re paying for compute at scale.

So Is Cosine Similarity Dead?

Not quite.

Use cosine similarity when: Your embeddings are NOT pre-normalized. You’re dealing with unnormalized vectors (TF-IDF, custom embeddings). Document length shouldn’t affect similarity. You need interpretable scores (0-1 range).

Use dot product when: Your embeddings ARE pre-normalized (most modern models). You need maximum speed. You’re using a vector database with normalized vectors. You’re processing millions of comparisons.

The trick: Check if your model normalizes embeddings. If yes, use dot product. If no, normalize once at index time, then use dot product forever.

Cosine similarity isn’t dead. It’s just... optimized away.