A ten-person startup rolls out a support agent built on Gemini 2.5 Flash, a fast, inexpensive model that's become a default choice for exactly this kind of work, feels good about it for a month, then opens an AI bill that's jumped five or six times over. Nobody changed the model. Nobody added a feature. What actually happened is quieter and much more common: the agent had started re-reading the entire conversation history on every single reply, pulling in a full knowledge base document it only needed one line from, and looping an extra time or two whenever it wasn't confident in its own answer. None of that shows up in a demo. All of it shows up on the invoice.
This is the part of agentic AI that doesn't get talked about enough. Everyone focuses on which model to use. Far fewer people look at how the system around that model is put together, and that's usually where the cost actually comes from.
- Why the model you pick matters less than how the agent is built around it
- The handful of design habits that quietly drive most AI bills up
- What small teams can do differently to keep costs predictable while they scale
The Bill Is a Design Outcome, Not a Pricing Outcome
Every exchange with an AI model gets measured and charged in tokens, small chunks of text, roughly a syllable or a short word each. What you send in counts. What the model sends back counts. And critically, what most people don't account for: a lot of what a model reads and writes happens invisibly, in the background, before you ever see a reply.
Take that Gemini 2.5 Flash agent from the startup above. Google prices it at $0.30 per million input tokens and $2.50 per million output tokens, which includes the model's own internal “thinking” tokens, since Flash is a hybrid reasoning model that reasons through a problem before answering it. Those rates look almost too small to worry about. They stop looking small once you see how differently two teams, building the exact same agent on the exact same model, can rack them up.
An agent that has to reread a long conversation on every turn is paying, again, for everything that was already said. An agent that pulls in a whole document to answer a one-line question is paying to read the parts it didn't need. An agent that loops through extra reasoning steps because nobody told it when to stop is paying for its own second-guessing. None of this is a pricing problem. It's an architecture problem, and it's fixable.
What Actually Drives the Bill Up
Let's walk through this with one example and hold it steady, so the difference doesn't get lost in a pile of changing assumptions. Picture that same support agent from earlier, still built on Gemini 2.5 Flash, handling a fairly ordinary 100,000 requests a month, the kind of volume a growing startup's support inbox reaches without much fuss. In each comparison below, exactly one thing about how that agent is built changes, and nothing else. That way, what you're looking at is the cost of the habit itself, not some other variable moving at the same time.
Four habits, one hypothetical agent: $1,320 a month before, roughly $315 after — a 76% reduction, with no change to what the agent actually does for the customer.
1. Dragging too much context into every call
Entire documents, entire chat histories, when the task only needed a small slice of it. The fix is retrieval: pull in only the specific piece of information the task actually requires, instead of handing the model everything and hoping it finds the right part.
| Full document + history | Retrieved snippet only | |
|---|---|---|
| Input tokens per request | 9,000 | 700 |
| Monthly input tokens (100K requests) | 900M | 70M |
| Monthly cost (@ $0.30/M input) | $270 | $21 |
That's a roughly 92% cost reduction, without changing the model or the task at all.
2. Using a heavyweight reasoning model for work that doesn't need heavy reasoning
Sorting an email, formatting a reply, checking a simple condition, none of that needs the same model you'd use to work through a genuinely hard problem. Matching the model to the task, instead of defaulting to the most capable one everywhere, is one of the simplest cost levers available.
| Gemini 2.5 Flash | Gemini 2.5 Flash-Lite | |
|---|---|---|
| Output tokens per request | 400 | 400 |
| Rate (per million output tokens) | $2.50 | $0.40 |
| Monthly cost (100K requests) | $100 | $16 |
Same task, same result, 84% less spend, just by not reaching for the bigger model out of habit.
3. Letting an agent's own reasoning run unchecked
On a hybrid reasoning model like Flash, every extra round of “thinking” the agent does before answering is billed as output tokens, the same as the answer itself. Without a limit on how many times an agent is allowed to loop, that invisible thinking cost can multiply a bill several times over on tasks that should have been simple.
| No step limit (avg. 4 loops) | Capped at 1-2 steps | |
|---|---|---|
| Output + thinking tokens per request | 3,200 | 1,050 |
| Monthly tokens (100K requests) | 320M | 105M |
| Monthly cost (@ $2.50/M output) | $800 | $262.50 |
Roughly two-thirds of that spend was the agent re-checking itself past the point it needed to.
4. Repeating work the system already did
Gemini's own pricing makes this one concrete: a cached input token costs $0.03 per million against $0.30 per million for the same token read fresh, a 10x difference for content the model has already seen before, like a shared system prompt or knowledge-base excerpt injected into every request.
| No caching | Cached context | |
|---|---|---|
| Shared context tokens per request | 5,000 | 5,000 |
| Monthly tokens (100K requests) | 500M | 500M |
| Monthly cost (@ $0.30/M vs. $0.03/M) | $150 | $15 |
What Small Teams Can Actually Do
None of this requires a large engineering team, it requires designing for it upfront instead of discovering the cost later in a bill. A handful of well-established patterns cover almost everything above, and none of them are exotic.
Retrieval-augmented generation (RAG)
This is the formal name for the fix behind habit #1. Instead of stuffing an agent's entire knowledge base or conversation history into every call, a RAG setup searches that knowledge first and hands the model only the handful of passages actually relevant to the question being asked. It's the single highest-leverage pattern here, which tracks: it's also the biggest saving in the table above.
Model routing
Instead of sending every request to one model, a router sends simple, high-volume requests, checking a status, formatting a reply, classifying a ticket, to a cheaper model like Gemini 2.5 Flash-Lite, and reserves a more capable model for the requests that genuinely need it. Most support and ops workloads are overwhelmingly made up of the simple case, which is exactly why this pattern pays off as much as it does.
Prompt and context caching
Anything static across requests, a system prompt, a tool's instructions, a knowledge excerpt used repeatedly, gets cached once instead of resent every time. This is the mechanism behind that $0.03-per-million-token rate earlier, and it takes almost no engineering effort, mostly just recognizing which parts of a prompt don't actually change request to request.
Constrained, structured output
An agent asked to explain itself in prose will burn far more output tokens than one asked to return a short, structured answer, a status code, a category, a two-line summary, a small JSON object. Designing agents to answer as narrowly as the task allows, rather than defaulting to open-ended text, is a quiet saving that applies to every single request, not just the expensive ones.
A bounded reasoning budget
This is the fix behind habit #3. Instead of letting an agent decide for itself when it's done thinking, the system defines upfront how many steps or how much reasoning it's allowed before it has to answer or hand off to a person. It turns an open-ended cost into a predictable one.
Batch processing for anything that isn't real-time
Not every agent needs to answer instantly, and most model providers charge less when you let them process a request whenever it's convenient rather than the instant it's sent. Gemini's batch tier, for instance, runs at roughly half the standard price on Flash. A nightly job that summarizes the day's tickets or enriches a backlog of records is a natural fit for this, and it's close to free savings for work nobody was waiting on in real time anyway.
| Standard tier | Batch tier | |
|---|---|---|
| Input tokens (100K records @ 300 tokens each) | 30M @ $0.30/M = $9.00 | 30M @ $0.15/M = $4.50 |
| Output tokens (100K records @ 150 tokens each) | 15M @ $2.50/M = $37.50 | 15M @ $1.25/M = $18.75 |
| Monthly cost | $46.50 | $23.25 |
Same output, half the bill, just by not asking for it instantly.
What Each Pattern Is WorthMapped Against the Running Example
Mapped against the running example, here's roughly what each pattern is worth on its own:
| Pattern | What It Fixes | Approx. Monthly Saving(100K requests, Gemini 2.5 Flash rates) |
|---|---|---|
| Retrieval-augmented generation (RAG) | Context bloat | ~$249 |
| Model routing to a lighter model for simple tasks | Wrong model for the job | ~$84 |
| Bounded reasoning budget (step caps) | Unchecked reasoning loops | ~$538 |
| Prompt and context caching | Repeated work | ~$135 |
| Combined, real-time agent | ~$1,005, roughly 76% of the original bill | |
| Batch processing | Paying real-time prices for work that isn't real-time | ~50% saving, on top, wherever it applies |
These aren't research-lab techniques. They're closer to good engineering habits, the kind that separate a business running AI deliberately from one that's just letting it run.
The Takeaway
The model is rarely where the money goes. Architecture is. A small team that designs its agents with the same care it puts into the rest of its product, narrow scope, the right context, a model sized to the task, doesn't just spend less, it scales its AI use the same way it scales everything else: on purpose, instead of by accident.
See Where Your Setup Stands
Request an AI Readiness Audit from Madspek.
Get an AI Readiness Audit
