Your LLM API Has a Cache You're Not Using

One parameter cuts token costs up to 90% for repeated context. Most developers haven't enabled it yet.

Dafallah Ibrahim

— 2 min read
Share

Every time your agent loop fires, it re-sends the same system prompt and tool definitions — thousands of tokens the model already processed seconds ago. You pay full price for every single one of them.

Both Anthropic and OpenAI now support prompt caching. The savings aren't subtle, and most developers haven't turned it on.

Why this matters

Agentic loops are the worst offender. A typical agent sends a 4,000-token system prompt and tool spec on every iteration. Twenty iterations means 80,000 tokens of pure redundancy. At Sonnet pricing, that's real money for zero new information.

Prompt caching lets the model skip reprocessing that repeated prefix. Input costs drop up to 90%. Time-to-first-token drops too, because the model doesn't re-derive attention over cached tokens.

How it works

Anthropic's API uses explicit cache breakpoints. You add a cache_control marker to any content block — system prompt, messages, or tool definitions. The API caches everything up to that breakpoint with a 5-minute TTL.

The economics are simple. The first call pays a 25% write premium on cached tokens. Every subsequent call within the TTL pays 10% of the normal input rate. Each cache hit refreshes the window.

OpenAI's implementation is automatic — no API changes needed. It kicks in after 1024 tokens and gives a 50% discount on cached prefixes. Less control, but zero configuration.

Where this helps

  • Agent loops: System prompt and tool spec stay constant across 10–50 iterations. Cache once, pay nearly nothing for the rest.
  • RAG pipelines: Large document chunks sent repeatedly with different query suffixes. Cache the document, swap only the question.
  • Code analysis: Full repository context stays fixed while you ask different questions about it.
  • Multi-turn chat: Earlier conversation history never changes. Mark a breakpoint after the stable prefix.

Watch out

Cache hits require exact prefix matches. Change one character in your system prompt and you invalidate the entire cache. Version your prompts deliberately.

Anthropic requires a minimum cacheable prefix: 1,024 tokens for Sonnet and Haiku, 2,048 for Opus. Short prompts won't benefit. You also get a maximum of 4 breakpoints per request — plan where they go.

The 5-minute TTL means low-traffic apps will frequently miss the cache. If your service averages one request every 10 minutes, you'll pay the write premium on every call.

Try it yourself

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a senior code reviewer. ",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Review the auth module."}
    ]
)

# First call:  pays 1.25x on cached tokens (write)
# Next call:   pays 0.10x on cached tokens (read)
# Check usage.cache_read_input_tokens to verify hits

TL;DR

  • What changed: LLM APIs now cache repeated input prefixes — Anthropic via explicit markers, OpenAI automatically.
  • Why it matters: Agent and RAG workflows with large static context see up to 90% input cost reduction plus faster responses.
  • What to try today: Add cache_control to your biggest static prompt block and inspect cache_read_input_tokens in the response.

Read more