InferenceIndex

Blog

The Economics of Repeat Agent Work

Where agent spend goes, which work comes back with new values, and what each cost lever saves before it stops.

An agent bill is input, output, and reasoning, paid on every step of a tool-call loop, plus the wait and any retry. Much of that work is the same task shape with new values. Prompt caching, response and semantic caching, decision models for agents, and model routing each remove a slice and then stop. A Compiled Agent Model runs learned work near-instantly, at a fraction of the cost, and falls back to your current model when the task is new.

What are the findings?

  1. Agent spend is input tokens, output tokens, and reasoning tokens, multiplied across every step of a tool-call loop, plus the time spent waiting and the cost of trying a failed step again.
  2. A large part of agent work is the same task shape with new values, as in invoices, support triage, scheduling, and data entry. How large that share is has to be counted in your own workload.
  3. Prompt caching, response and semantic caching, decision models for agents, and model routing each remove a slice of the bill, and each one stops before a learned task is reused.
  4. A Compiled Agent Model runs work it has learned near-instantly, at a fraction of the cost, and falls back automatically to your current model when the task is new.
  5. Invoice reconciliation, one run per model: 1.2s and $0.000229 vs Opus 5.5 at 21.3s and $0.105 (~450x lower cost, at cost).

Where does agent spend go?

Three token piles sit under every later lever.

Input tokens are what the model reads: instructions, tool definitions, the document or ticket, the thread so far, and the last tool result. Providers price this pile per million tokens. A cache hit discounts this pile only.

Output tokens are what the model writes: the reply, the tool arguments, the extracted fields. Output is usually priced higher than input, and a discount on input does not touch it.

Reasoning tokens are the work between the read and the write, when the provider bills that work. On Google's Gemini API pricing, output is listed including thinking tokens, separate from input and from context caching. A harder plan costs more even when the visible answer is one label.

A tool-call loop multiplies all three. The agent reads, chooses a tool, waits, reads the result, and chooses again. Each lap sends the growing thread back as input, spends more reasoning, and writes the next call. On an invoice, that is: read the invoice, fetch the purchase order, match vendor and lines and totals, write the exception. A total that does not parse starts the lap again, and the failed lap stays on the bill. Support triage, scheduling, and data entry are the same chain with different values.

Latency is the same spend in time. Time to first token is only the wait before anything appears. Each tool adds its own wait after that. A lever that shortens the first wait leaves the rest of the loop alone.

Retries pay the loop again. A tool error or an empty field leads to a second call, and the first attempt is not refunded. Cost per call therefore understates cost per successful task, because the failures sit inside the bill for the success. The published price per million tokens is a rate card for one pile. It is not the cost of the job.

How much agent work repeats?

This essay does not quote a market-wide rate, and it does not invent one. Of the runs you already pay for, how many are the same task shape with new values?

The shape is the tools, the order of the checks, and the fields that have to come back. The values are the names, amounts, dates, and sentences. When the shape holds and the values move, the run is repeat work, even though no two prompts match.

Invoices are the clearest case. The vendor, the amount, and the date change. The job does not: read the invoice, find the purchase order, match the lines, write the exception or the approval. Support triage changes the words and keeps the queues and the policy: which queue, how urgent, whether a person should see it. Scheduling changes the people and the times and keeps the constraints, then checks calendars and writes a hold. Data entry changes the document and keeps the fields.

You can count this in the jobs you already run. Group a month of runs by the tools they called and the order of the checks. If what changed inside the group is a name, a number, a date, or a sentence, that group is the share that repeats. If what changed is a new tool, a new policy, or a job you have not done, that group stays on your current model.

A repeated prefix helps both groups a little. A stored answer helps only when that answer can be reused. Reusing the task helps the first group and should leave the second alone. The share moves when the work changes, so recount the month.

What does prompt caching save, and where does it stop?

Prompt caching discounts input the provider has already processed, and it can shorten the wait before the first token. It does not discount output, it does not discount reasoning, and it does not remove the model call on the next step.

Anthropic's pricing prices a cache hit as a fraction of the base input price. A cache hit is 0.1x base input on most models. Some current models are lower: 0.05x on Claude Opus 5.5 and Claude Sonnet 5.5, and 0.025x on Claude Fable 5.2, Claude Mythos 5.2, Claude Fable 5.1, and Claude Mythos 5.1. Cache-hit prices per million tokens on that page are $0.20 for Claude Opus 5.5, Claude Fable 5.2, and Claude Mythos 5.2, $0.10 for Claude Sonnet 5.5, and $0.25 for Claude Fable 5.1 and Claude Mythos 5.1. A five-minute cache write costs 1.25x base input, and a one-hour write costs 2x. Output stays on its own rate. The page also says prompt caching reduces latency.

OpenAI's prompt caching guide says reused input is discounted up to 95%, and that caching reduces the time spent processing input before the response starts. For GPT-5.6 and later, a 30-minute cache write is 1.25x the uncached input rate, and a 12-hour write is 2.5x. A later read is 0.1x on most of those models, and 0.05x on GPT-6.1 Sol and GPT-6.1 Terra. The guide also says the minimum cacheable prompt length is 1,024 tokens for GPT-5.6 and later. The API pricing page lists short-context rates per million tokens: gpt-5.6-sol input at $4.00 and cached input at $0.40, and gpt-6.1-sol input at $2.00 and cached input at $0.10. Output is a separate column. gpt-4o, an older model on that page, lists cached input at $1.25 against input at $2.50.

Google's Gemini API pricing lists context caching below the input price, and lists output, including thinking tokens, on its own line. On the standard paid tier, Gemini 4 Argon lists input at $2.00 per million tokens and context caching at $0.10 through November 30, 2026, and both of those rates are listed at double that from December 1, 2026. Caching on that model is implicit only. Gemini 3.8 Flash lists input at $0.75 per million tokens and context caching at $0.075 through December 31, 2026, with a higher rate listed after that, and that row also lists a storage price. Google's optimization guide lists caching as faster time to first token.

Those pages price input. A stable prefix keeps more of the run on the discounted rate. Reasoning, tool results, and new values do not, and output stays full price. A write that is never read can cost more than uncached input, at 1.25x, 2x, or 2.5x. The prefix has to match from the start of the request. On a tool-call loop, a hit on the instructions still leaves every step as a model call.

What do response and semantic caches save, and where do they stop?

A response cache returns a stored answer. It is instant when that answer is the one you wanted again.

An exact cache stores the reply and returns it for the same request. GPTCache does that by default, and it can also return a stored answer for a prompt it treats as similar. Redis does this in its semantic cache: a later prompt that is similar to a cached one can receive the stored response instead of calling the model. Same request in, stored reply out, no model call. That fits a stable reply or a lookup whose arguments did not change. The wait is gone, and so is the model bill for that call.

It stops when the values are new. Today's invoice is not yesterday's: the vendor, the amount, and the date changed. An exact cache misses, which is the safe failure. A semantic cache can still hit, because the prompt looks familiar, and then it returns the stored answer, including the previous amount. Similar is not the same job. The cost of that hit is a confident answer from a different run. Use this cache where the whole answer is reusable.

What do decision models for agents save, and where do they stop?

Decision models for agents answer a typed question inside a step. They are complementary to a Compiled Agent Model, and you can use them together. They do not store a solved task, and they do not take the model off the steps that still need it.

Jev is TypeSafe's hosted model. A call sends a state and a set of typed questions, and it returns one structured answer per question. The quick start documents questions that choose an option from a list, score a state on a rubric, or check whether a statement is true. Intent routing uses a choice question to decide the next action. The actions can be a function in your app, an LLM, or a person. Your code branches on the answer. Jev does not write the reply.

Laya is an open-source decision model from Convai Innovations. You can self-host it, or call it through Laya Studio, a hosted service that is not affiliated with Convai Innovations or TypeSafe. A request is a state plus typed questions: pick one option, rate on a scale, or answer yes or no. It returns a probability and does not write the reply. The agent tool routing page uses those questions to pick a tool, a model tier, or a handoff, and says the reasoning model still does the work and fills in the tool arguments. Laya Guard checks a tool call before it runs and returns allow, ask, or block.

The saving is that typed call. It stops at the answer: the queue, the yes-or-no, the tier. It does not carry yesterday's plan into today's invoice, and the routing page says the reasoning model still does the work. On a learned task, the compiled run handles the job and a decision model can still make a fresh choice inside a step. On a new task, fallback uses your current model, and a decision model can still pick the tool or the check. Use them together when a run has both.

What does model routing save, and where does it stop?

Model routing sends a step to a model that matches the step. An easy step goes to a cheaper model. A hard step stays on a more capable one. The saving is the price difference on the steps that did not need the expensive model.

The choice can be a rule, or a decision model. Laya's agent tool routing page describes asking which model tier a step needs, while the reasoning model still does the work. The lever is the route.

A wrong route fails the step, and the retry often pays the capable model as well. A right route still pays a model for that step, at a lower rate, with the same call and the same tool wait. A familiar job that comes back is still a series of routed calls unless the task itself is reused.

What does a Compiled Agent Model save, and where does it stop?

A Compiled Agent Model saves the repeat itself. Work it has learned runs near-instantly, at a fraction of the cost, and anything new falls back automatically to your current model.

The first time through uses the model you already have, and that completed run is what gets learned. Next time the values are new and the shape is not. A different vendor, amount, or date still belongs to the learned task. The compiled run does not pay a frontier model to rediscover the route or to carry out each familiar step.

Fallback is automatic. A request that does not fit goes to your current model at that model's ordinary price. The share that repeats moves to the compiled run. The share that is new stays put. Prompt caching, a semantic cache, a decision model, and model routing still apply on the path that remains: they cut a prefix, a stored answer, a typed choice, or a step's rate. They do not reuse the whole task. A Compiled Agent Model is the lever for that case, and it stops where the task is new.

How do these levers compare?

Each row is a different cut of a run. Read the table as a map of what the lever touches. The rows were not measured on one shared task.

Lever What it saves Where it stops
Prompt caching A discount on cached input, and a shorter wait before the first token Output, reasoning, and a model call on every step
Response and semantic caching An instant stored reply when the whole answer is reusable A miss, or the previous answer, when the values are new
Decision models for agents A typed choice inside a step The rest of the task still runs on a model
Model routing The price difference on a step sent to a cheaper model A wrong route, and a model call on every step
Compiled Agent Model A near-instant compiled run, at a fraction of the cost Anything new falls back to your current model

Point the map at the pile you pay for. A stable prefix is prompt caching. An unchanged answer is a response cache. A typed choice is a decision model, and it can sit beside a Compiled Agent Model. An easy step at a frontier rate is routing. The same task shape with new values is a compiled run.

What does one invoice reconciliation run cost?

Here is one measured case, with the label we use for it. Invoice reconciliation, one run per model: 1.2s and $0.000229 vs Opus 5.5 at 21.3s and $0.105 (~450x lower cost, at cost).

It is one run of this task for each. The compiled run had done similar invoices before. The Opus 5.5 figures are that model on its own. The parenthetical is the dollar comparison, and the two times are the waits, on that same pair. The pair is not a claim about every workflow. A shape that does not fit falls back to your current model.

How do you estimate a month of repeat agent work?

The formula for the share that repeats is monthly runs x share that repeats x cost per run. Run it twice: once at the cost per run you pay today, once at the cost per compiled run. The difference is the monthly difference on that share. Runs that do not repeat stay on your current model either way. Add them only when you want the whole month.

Use your own inputs. Monthly runs come from your logs. The share that repeats is the group above: same tools, same checks, new values. Cost per run today is the cost of a successful task, retries included. Cost per compiled run is the learned path on that same task.

These figures are example inputs, labelled so the arithmetic can be checked. They are not a measurement, not a forecast, and not the invoice case above. Replace every one of them. Suppose monthly runs are 10,000, the share that repeats is 0.70, today's cost per run is $0.05, and the cost per compiled run is $0.001. Then 10,000 x 0.70 is 7,000 runs. At $0.05 that share costs $350. At $0.001 it costs $7. The other 0.30 is 3,000 runs at $0.05, which is $150 before and $150 after. The illustrated month is $500 before and $157 after. A smaller share, or a smaller gap between the two costs, shrinks the difference. Recount the share when the work changes, and multiply cost per successful task so retries are in the rate.

How should you evaluate the options?

Four checks, on your own tasks, tell the levers apart.

  1. Cost per successful task. Divide dollars spent by tasks that finished correctly, including retries, cache writes, tool calls, and fallbacks. A cheap failure is not a saving.
  2. Latency. Measure the wait from request to finished task, including tool time. Time to first token is one slice of that wait.
  3. Quality. Compare the result with the outcome you already accept: the right invoice fields, the right queue, the right calendar hold. A stored answer fails this when it carries another run's values.
  4. Fallback rate. Count the share of runs that left the cheaper path and went to your current model. A high rate is the right outcome when the work is new, and a sign the repeating share was overstated when it is not. There is no universal rate to aim at.

Prompt caching should move the input you actually hit. A semantic cache should fail quality when a hit carries another run's values. A decision model should move the typed call and leave the rest of the task on the bill. Routing should move the rate on the steps it redirects. A Compiled Agent Model should move cost and latency on the share that repeats, with the fallback rate telling you how large that share was.

FAQ

Where does the money go on an agent run? Input, output, and reasoning tokens, multiplied across every step of a tool-call loop, including retries. The wait counts too. The last sentence is rarely the bill.

Does prompt caching make repeat agent work cheap? It discounts cached input and can shorten the wait before the first token. Output, reasoning, and a model call on every step are still paid. A cache write can cost more than a normal input when that prefix is never read again.

When does a semantic cache return the wrong answer? When a new run only looks like an old one, and the cache returns the stored reply. A new amount on an invoice is a different job. An exact cache misses it. A similar match can answer it with the previous result.

Can a decision model and a Compiled Agent Model be used together? Yes. They are complementary. A decision model answers a typed question inside a step. A Compiled Agent Model runs learned work near-instantly, at a fraction of the cost, and falls back to your current model when the task is new.

What does a Compiled Agent Model do with work it has not learned? It falls back automatically to your current model, at that model's ordinary price. The compiled run is for work it has learned. Anything new takes the fallback.

How do you estimate the monthly cost of repeat agent work? Multiply monthly runs by the share that repeats by the cost per run, once at today's cost and once at the compiled-run cost. The worked numbers in this essay are labelled example inputs. Replace them with your own.

Where should you read next?

The longer account of a Compiled Agent Model is Don't forget what worked. For the pattern of paying again for a plan the agent already found, see why AI agents cost so much on repeat tasks. For a closer look at what caching saves and where it stops, see caching for AI agents. For how decision models fit beside a Compiled Agent Model, see decision models and a Compiled Agent Model.

The product lives on the InferenceIndex site. Bring the jobs you already run. Keep your current model for anything new.