Freeze Your Prompt Prefix, Cut Your LLM Bill

Drew Breunig's "The State of Context Engineering" is on HN's front page this morning, and its core premise should feel familiar to anyone who has written C: the context window is memory, and memory needs management. The folklore era of prompt-tweaking is consolidating into a real engineering discipline.

Most teams still treat context like a shopping bag — stuff everything in, resend all of it every turn. Meanwhile another front-page post today notes OpenAI and Anthropic cut prices by up to 80% in a few months. Cheap tokens don't help much when your agent resends 40,000 of them per tool call.

Why this matters

Agents made context a quadratic cost. A 40-step run that appends tool output to history re-bills turn one forty times, turn two thirty-nine times, and so on. The same bloat dominates time-to-first-token, so it is also your latency budget. And models measurably degrade when stuffed with irrelevant context — the "lost in the middle" effect. Prefix caching attacks cost and latency; pruning attacks quality. You want both.

How it works

Providers cache the computed key-value states behind your prompt. If the next request starts with byte-identical tokens, that prefix is served from cache instead of being re-processed. OpenAI does this automatically for prefixes over roughly 1,024 tokens. Anthropic requires an explicit breakpoint — a cache_control marker — and bills cached reads at a small fraction of normal input price.

The design rule that falls out: static first, dynamic last. System prompt, tool definitions, and few-shot examples at the top, frozen. User input, retrieved documents, and tool results at the bottom. One edited token invalidates everything below it, so anything that changes must sit after everything that doesn't. Then cap growth: compact old turns into a rolling summary, and truncate tool output before it enters history — a raw log dump gets re-billed and re-read on every later turn.

Where this helps

• Coding agents: sessions run 50+ turns; a frozen instruction prefix turns each tool loop into a cache hit.
• Support bots: tone rules and product docs are identical across users; only the ticket goes at the end.
• RAG pipelines: retrieved chunks are per-request — keep them last so shared scaffolding stays cached.
• Batch jobs: thousands of requests sharing one prefix let the cache do most of the prefill.

Watch out

• A timestamp in your system prompt — "Current date: ..." — quietly zeroes your hit rate. Move volatile bits to the end.
• Anthropic charges extra for cache writes and enforces a minimum prefix length (around 1k tokens); small prompts don't qualify.
• Cache TTLs are short — about five minutes on Anthropic, refreshed on hits. Low-traffic endpoints will eat misses.
• Caching makes wrong instructions cheap too. A stale cached prefix still answers confidently; hit rate is not a quality metric.

Try it yourself

Paste a few thousand characters of static instructions into the system block, run this twice within five minutes, and compare the usage fields. The second call shows cache_read_input_tokens — same work, fraction of the price.

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 64,
    "system": [{
      "type": "text",
      "text": "PASTE_YOUR_STATIC_INSTRUCTIONS_HERE",
      "cache_control": { "type": "ephemeral" }
    }],
    "messages": [
      { "role": "user", "content": "Review: const x = y ?? 0;" }
    ]
  }'

TL;DR

• What changed: context engineering is now a named discipline — the context window is managed like memory, not a shopping bag.
• Why it matters: a frozen prompt prefix is served from cache at a fraction of the price and latency, on every single turn.
• Try today: reorder one prompt static-first, add a cache_control breakpoint, and watch the cache-read tokens appear.