Caching for AI Agents: What Prompt, Semantic and Plan Caching Actually Save, and Where They Stop.
Prompt caching, semantic caching, and plan caching each save a slice of an agent run. A Compiled Agent Model reuses the repeat, and falls back to your current model when the task is new.
Caching for AI agents saves a slice of the bill, and then it stops. Prompt caching makes a repeated prefix cheaper and faster to the first token. A response or semantic cache is instant when the whole answer can be reused. Plan caching reuses the route. None of them learns a task so a later run with new values can skip the work. A Compiled Agent Model does. Repeat work runs near-instantly at a fraction of the cost, and anything new falls back to the model you already use.
What does prompt caching save at Anthropic, OpenAI, and Google?
Prompt caching saves money and time on input the provider has already processed. Cached input is cheaper and faster to the first token. Output and reasoning tokens are still paid, and every step is still a frontier call. A whole run is typically 20-50% cheaper. Treat that range as an estimate, not a rate these providers publish.
Anthropic's pricing prices a cache hit as a fraction of the base input price. A cache hit costs 10% of the standard input price. Some current models are lower: 0.05x base input on Claude Opus 5.5 and Claude Sonnet 5.5, and 0.025x on Claude Fable 5.1 and Claude Mythos 5.1. A cache write costs 1.25x for five minutes and 2x for an hour. Output tokens stay on their own rate. The page also says prompt caching reduces latency, which is the shorter wait before the first token.
OpenAI's prompt caching guide says reused input is discounted up to 95%, and that caching reduces the time spent processing input before the response starts. For GPT-5.6 and later, a cache write is 1.25x the uncached input rate. A later read is 0.1x on most of those models, and 0.05x on GPT-6.1 Sol. The API pricing page matches those reads: gpt-5.6-sol lists input at $4.00 per million tokens and cached input at $0.40, and gpt-6.1-sol lists input at $2.00 and cached input at $0.10. Output is its own column, so a cache hit does not discount what the model writes. Older models on that page save less. gpt-4o lists cached input at $1.25 against $2.50 input.
Google's Gemini API pricing lists context caching below the input price, and lists output, including thinking tokens, on its own line. On the standard paid tier, Gemini 4 Argon lists input at $2.00 per million tokens and context caching at $0.10 through November 30, 2026, with both rates scheduled to double on December 1, 2026. Caching on that model is implicit only. Gemini 3.8 Flash lists input at $0.75 per million tokens and context caching at $0.075 through December 31, 2026, and that row also lists a storage price. Google's optimization guide lists caching as faster time to first token.
Mostly cached instructions land toward the high end of the 20-50% estimate. Mostly reasoning and tool calls land lower, because output, reasoning, and each frontier step are still on the bill.
What do response and semantic caches save?
A response cache saves the whole answer, and it is instant when that answer is the one you wanted again.
An exact cache stores the reply and returns it when the same request shows up. GPTCache does that by default, and it can also return a stored answer for a prompt it treats as similar. Redis does this in its semantic cache: a later prompt that is similar to a cached one can receive the stored response instead of calling the model. Gateways in front of a model often keep the exact version. Same request in, stored reply out, no model call.
It fits when the whole answer is reusable, such as one stable reply or a lookup whose arguments did not change. The wait is gone, and so is the model bill for that call.
Agents that see new values each run do not get that. Today's invoice is not yesterday's: the vendor, the amount, and the date changed. An exact cache misses. A semantic cache can still hit, because the prompt looks familiar, and then it returns the stored answer, including the previous amount. Similar is not the same job. On a repeat whose values change, this cache either misses or hands back an answer from a different run.
What does plan and workflow caching save?
Plan and workflow caching saves the plan. It still pays the big model for every step.
You keep the route that worked: which tool came first, which check came next, where the result was written. The next run follows that route instead of inventing it. The steps are still frontier calls. Reading the new invoice, matching it, and writing the result each go back to the big model. The plan was reused. The work inside it was not. On a task with several tool calls, the steps are the bill.
Decision models for agents sit in this same layer. They answer a typed question inside a step, such as which tool to call or whether the call should proceed, and your code branches on that answer. They do not store the whole task, and they do not take the frontier model off the steps that still need it. A plan cache and a decision model can be used together. Neither one is the repeat run.
What does a Compiled Agent Model save that caching does not?
A Compiled Agent Model saves the repeat itself. Work it has learned runs near-instantly, at a fraction of the cost, and anything new falls back to your current model.
The first run uses the model you already have, and that completed run is what gets learned. Next time, the values are new and the plan is not. A different vendor or amount still belongs to the learned task. The run does not pay a frontier model to rediscover the route or to carry out each familiar step.
Fallback is automatic. A request that does not fit goes to your current model.
Here is one measured case, with the label we use for it. Invoice reconciliation, one run per model: 1.2s and $0.000229 vs Opus 5.5 at 21.3s and $0.105. That case is ~450x at cost. The label is one run of this task for each model, not a claim about every workflow.
How do these approaches compare?
| Approach | What it saves | Where it stops |
|---|---|---|
| Prompt caching | Cheaper cached input, and a faster first token | Output, reasoning, and a frontier call on every step |
| Response and semantic caching | An instant reply when the whole answer is reusable | A miss, or the wrong answer, when the values are new |
| Plan and workflow caching | The plan, so the route is not invented again | The big model is still paid for every step |
| Decision models for agents | A typed choice inside a step | The rest of the step still runs |
| Compiled Agent Model | A near-instant repeat, at a fraction of the cost | Anything new falls back to your current model |
FAQ
Does prompt caching make a repeat agent run cheap? It discounts cached input and can shorten the wait before the first token. Output, reasoning, and a frontier call on every step are still paid. Treat 20-50% off a whole run as an estimate.
When does a semantic cache return the wrong answer? When a new run only looks like an old one, and the cache returns the stored reply. A new amount on an invoice is a different job. An exact cache misses it. A similar match can answer it with yesterday's result.
Does plan caching remove the frontier model from a step? No. It reuses the plan and still pays the big model to carry out each step.
What does a Compiled Agent Model do with work it has not learned? It falls back to your current model. The learned path is for the repeat. Anything new takes the fallback.
Where can I read the longer account?
The longer account of the repeat is Don't forget what worked. For how decision models for agents differ from a Compiled Agent Model, see decision models and a Compiled Agent Model.