Don't Forget What Worked: Why I'm Building Compiled Agent Models
Compiled Agent Models: your agent's repeat work, up to 100x cheaper and up to 10x faster.
I make repeat AI work instant and nearly free. That's the whole pitch. This post explains why I think that matters, what it looks like in practice, and where I think it goes.
What is a Compiled Agent Model?
A Compiled Agent Model is an AI agent that learns a task once and then stops re-thinking it. The first run teaches it the task. Repeats skip the re-thinking, and it only goes back to a frontier model when something is genuinely new. The result: repeat work runs fast and costs a small fraction of what it would if a frontier model did it from scratch every time.
That's the definition. Here's the story behind it.
What did chatbots get right before generative AI?
If you built or used a support bot in the Intercom era, you remember how they worked. They were mostly code. Decision trees. Canned responses. A few if-statements wearing a friendly avatar.
People like to make fun of those bots now, and some of that is fair. They were brittle. Step one inch off the path and they fell apart. They sounded robotic because they were robotic.
But they also got a few things very right. They were reliable: the same input gave the same output every time. They were instant. And they were cheap, close to free to run once you'd built them.
Hold on to that list. Reliable, instant, cheap. We'll come back to it.
Why do AI agents cost so much on repeat work?
Then generative AI arrived, and the pendulum swung all the way to the other side. Now everything gets generated.
That was a real leap. Agents can handle messy inputs, read documents, make judgment calls and talk like a person. Nobody wants to go back to hand-built decision trees.
But look at what an agent actually does on a typical day. It processes an invoice. Then another invoice from the same vendor, in the same format. Then fifty more. Every single time, it reasons about the task from scratch: what the fields mean, what order to do things in, which tool to call next. It pays frontier prices and waits frontier latency to rediscover the same plan it found an hour ago.
Over half of inference input is stuff the model has already seen. Prompt caching helps with part of that. It saves the model from re-reading the same context. But nothing saves it from re-thinking. The expensive part, the reasoning, gets redone on every run.
That's the gap. We gained flexibility and quietly gave up reliable, instant and cheap.
How does a Compiled Agent Model get both?
I don't think the answer is picking a side. Old bots forgot that the world is messy. Today's agents forget what already worked. I want both: don't forget what worked, and don't ignore the future.
So the rule is simple. Remember what repeats. Save the heavy thinking for what's actually new.
Here's how that plays out from your side:
- The first time, it works like any agent. A frontier model reasons through the task, uses your tools and produces a result, the same way an agent would today.
- That run teaches it the task. You don't write rules or build a flow chart. Doing the job once is the lesson.
- Repeats are fast and cheap. When the same kind of task comes in again, with a different vendor, amount or date, it gets done without re-thinking the whole plan. Same quality of result, a fraction of the cost and wait.
- Anything new goes back to a frontier model. If a task doesn't look like something it has learned, it doesn't guess. A frontier model handles it, just like today. Nothing gets forced into a pattern it doesn't belong in.
To be clear about that last point: new tasks still go to frontier models. The Compiled Agent Model isn't pretending hard thinking goes away. It's making sure big models only do the work that actually needs them.
How much cheaper is a Compiled Agent Model on repeat runs?
On our tests, repeat runs are up to 100x cheaper and up to 10x faster than having a frontier model do the work.
One example I like, because it's ordinary in the best way: invoice processing. It passed 6 out of 6 real test runs. The repeat work itself cost under 2 hundredths of a cent per run. End to end, including all the overhead, the test came to about $0.025 in total, compared with $0.059 for a frontier model doing the same job on its own. So even that early setup came in at well under half the cost, and most of what's left is overhead I'm working on cutting.
And because repeats skip the re-thinking, they don't sit waiting on a long chain of reasoning.
That's what reliable, instant and cheap looks like when you don't give up the flexibility.
Could AI run instantly on your phone without the cloud?
Here's where I think this goes, and why I'm doing this on my own.
Most people do the same basic things. Reschedule a meeting. Pay a bill. Find a receipt. Book a table. Reply to the usual email. Our lives are full of tasks that feel personal but are structurally the same as millions of other people's.
Now imagine a shared library of compiled agent work. Every everyday task that's been solved once is in there, already learned. My goal is that running one doesn't need a data center: it runs instantly, nearly free, even locally on your phone with no cloud at all.
That's the assistant we were promised years ago. A Siri that actually works. Not because it's the biggest model in the world, but because it doesn't have to think hard about things that have already been figured out.
And the library gets better as more people use it. When someone hits a task that isn't in there yet, a frontier model solves it, and that solution gets learned and added. The next person gets it instantly. Every new problem solved makes the whole thing a little smarter and a little cheaper for everyone.
I call that always-on intelligence for everyone. Not intelligence you ration because every query costs money and takes seconds, but intelligence that's just there, like electricity.
What about businesses and private data?
Businesses have their own workflows, and many of those workflows touch data that should never leave the building. So the same idea works in private.
A company gets its own walled-off library. Its compiled agent work, its processes, its data, kept separate from everyone else's. Its team's repeat work gets faster and cheaper over time, and nothing it learns leaks into the shared library.
Same principle, different boundary: remember what worked, inside your own walls.
Why I'm building this
Before InferenceIndex, I built Orbit, an AI agent for personal finance. It grew to 10 million users, and when I looked at what people were actually asking it, 80 to 90% of prompts were repeats. Same questions, same tasks, over and over. And every one of them paid for fresh frontier reasoning as if nobody had ever asked before.
That stuck with me. I'm a solo founder now, and I've spent a lot of time staring at inference bills and latency charts. The pattern I saw at Orbit shows up everywhere. Most of what we pay for isn't new thinking. It's the same thinking, bought again and again.
The early chatbot builders knew how to make software reliable, instant and cheap. The generative AI wave taught us how to make it flexible and smart. Compiled Agent Models are my attempt to stop treating those as opposites.
FAQ
What is a Compiled Agent Model? It's an AI agent that learns a task the first time it does it. Repeats skip the re-thinking, so they're much faster and cheaper, and it only goes back to a frontier model when something is new.
How is this different from prompt caching? Caching saves a model from re-reading context it has already seen. A Compiled Agent Model saves it from re-thinking a task it has already solved. The reasoning itself is reused, not just the input.
Do new tasks still use frontier models? Yes. Anything new goes to a frontier model. Only work that repeats gets the fast, cheap treatment.
How much cheaper are repeat runs? On our tests, repeat runs are up to 100x cheaper and up to 10x faster than a frontier model. In one invoice example, the repeat work itself cost under 2 hundredths of a cent per run.
Can a Compiled Agent Model run on a phone? That's the goal. Because repeats don't need heavy reasoning, I want everyday tasks to run locally on a device, instantly, with no cloud required.
See it on your own work
If your team or your agents do the same kind of task over and over, I'd love to show you what changes.
Send me one workflow and I'll show you the side-by-side.