Stop Re-paying for Your System Prompt

The big LLM APIs cache your prompt prefix — if you ask. One JSON field turns every repeated system prompt into a near-free cache read.

Dafallah Ibrahim

— 2 min read
Share

Every call your app makes re-sends the same 6,000 tokens of system prompt, tool schemas, and few-shot examples. The provider bills you full price for them every time — even though the model has read those exact tokens thousands of times today.

Here's what most teams miss: the major LLM APIs will cache that prefix for you. No new library. On Anthropic it's one JSON field. On OpenAI it happens automatically once your prompt is large enough. Most code I review never checks whether it's hitting.

Why this matters

Cached input tokens are billed at a small fraction of the normal rate, and the provider skips recomputing attention state for them. Two wins at once: the bill shrinks on prompt-heavy workloads, and time-to-first-token drops because generation starts sooner.

This is where the money actually is. Agents re-send their entire tool list and history on every step of every loop — a 20-step run re-processes its own prefix 20 times. Multiply by thousands of runs and you've found the line item finance keeps asking about.

How it works

Under the hood, inference builds a KV cache — per-token attention state — while reading your prompt. Providers can persist that state server-side and reuse it when a later request begins with the exact same token sequence.

The rules are simple. Only prefixes cache, so the start of your prompt must be byte-for-byte identical across calls. TTLs are short — minutes, refreshed on each hit — so this rewards steady traffic. And the API tells you what happened: Anthropic returns cache_read_input_tokens and cache_creation_input_tokens in usage; OpenAI reports cached_tokens inside prompt_tokens_details.

Activation differs: Anthropic wants an explicit breakpoint via cache_control; OpenAI caches automatically above a minimum prompt length; Gemini has explicit context-cache objects you reference by ID.

Where this helps

  • Agent loops — tool schemas and instructions stay fixed while the conversation grows after the cached prefix.
  • RAG apps — one fat system prompt of rules, one small variable chunk per query.
  • Eval harnesses — same fixture prompt, many candidate outputs under test.
  • Support bots — the company handbook goes in the cache; the user's question goes last.

Watch out

Exact prefix match or nothing. A timestamp, request ID, or Hello {{name}} at the top of the prompt invalidates the whole cache on every call. Put volatile content at the end of the message list.

Cache writes cost slightly more than regular input, so a breakpoint in front of two volatile tokens is a net loss. Short TTLs mean your three-calls-a-day internal tool will never hit. Caches are also per model and per account — bump the model version and you start cold. Check your provider's data-retention terms too: cached prefixes can outlive the request.

Try it yourself

Run this twice within a few minutes and compare the usage blocks:

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 32,
    "system": [{
      "type": "text",
      "text": "You are a terse code reviewer. [paste real instructions until this block is a few thousand tokens]",
      "cache_control": {"type": "ephemeral"}
    }],
    "messages": [{"role": "user", "content": "ping"}]
  }' | jq '.usage'

First call: cache_creation_input_tokens climbs. Second call: cache_read_input_tokens takes over. That difference is your discount.

TL;DR

  • What: LLM APIs cache identical prompt prefixes — Anthropic via cache_control, OpenAI automatically, Gemini explicitly.
  • Why: cached reads cost a fraction of full price and cut time-to-first-token on exactly the workloads agents and RAG apps generate.
  • Try: add one breakpoint, run the curl twice, and read the usage fields to confirm the hit.

Read more