Cutting AI Token Costs: Compression, Caching, and Model Choice

Updated 2026-09-06 · guide · agents, technical, how-to

Token cost is the quiet killer of AI product margins — and for agent-heavy systems it compounds faster than almost anything else you can ship. The good news: most teams are overpaying by 3–10x without realizing it, because they're loading the same giant context into every call, re-billing the same system prompt, and running a frontier model on tasks a small model does just as well. This guide covers the four levers that cut token spend without cutting quality: context discipline, prompt caching, retrieval instead of stuffing, and model-tier selection.

If you've built an AI agent or a product with an AI feature, you already know the shape of the problem: every user turn sends your whole system prompt plus the accumulated conversation into the model, and every one of those tokens is money. Scale that across thousands of sessions and the bill becomes a line item someone is asking about. This guide is about the levers you control — most of them pure engineering discipline, none of them requiring you to degrade the output your users see.

The four levers, ranked by return

Cost clarity is part of trust. This AI-engine trust pages guide shows how to explain usage assumptions, limits, and dependencies so pricing claims remain credible. Before any micro-optimization, know where the money actually goes. Token spend breaks down into a small number of levers, and they're not equal:

The pattern to internalize: the biggest wins are upstream of the model call, not inside it. Most cost lives in what you choose to send — and that's entirely under your control.

Lever 1 — Context discipline: don't resend what doesn't change

The most common waste on agent systems is re-sending context that never changes on every single turn. Three rules fix most of it:

1. Keep the system prompt small and static. A system prompt that changes per request can't be cached and gets re-billed every time. Design it once, keep it stable, and put everything dynamic into the user turn. A stable prefix is also exactly what makes prompt caching (Lever 2) work.
2. Trim the conversation window. You don't need the last 50 turns of chit-chat to answer "remind me what our pricing tiers are." Sliding windows, summarization of old turns, and dropping tool-return noise are all cheap wins. Every token you don't send is a token you don't pay for.
3. Stop echoing data the model already has. If you fetched the user's plan, their locale, and their permissions once, don't re-attach the full record to every call. Attach a reference ("user #42, enterprise plan, EU") and let the agent retrieve details only when it needs them.

This lever is pure hygiene — there's no downside, and on a chatty agent it's often the difference between a viable product and an unviable one. It's also the same discipline that keeps agent context windows from filling up with junk, which is why the memory and context guide is worth reading alongside this one.

Lever 2 — Prompt caching: stop paying twice for the same prefix

Prompt caching is the single most underused cost lever in 2026, because it's a provider feature that most teams never turn on. The mechanism: providers cache the leading tokens of a request, and cache hits bill at a steep discount (typically ~10% of input price). The requirement: a stable, cacheable prefix across requests.

To make caching work:

• Put everything shared in the prefix. System prompt, few-shot examples, tool definitions, and static instructions all live at the start of the prompt, in the same order, on every request. That whole block becomes a cache key.
• Never interleave dynamic content into the prefix. If user-specific data or the current timestamp appears before your static instructions, the cache breaks. Order matters: static first, dynamic last.
• Design for cache-hit rates. Log your hit rate per provider; if it's low, the prompt structure is wrong, not the feature. Providers report hit/miss metrics, and a healthy agent system sits at 80%+ cache-hit on input tokens.

The combined play is powerful: a stable system prompt cached at 10% input cost, with the dynamic turn appended after — your marginal cost per turn drops to mostly just the new tokens. On high-volume products this single change can cut total spend by more than half.

Lever 3 — Retrieval instead of stuffing

"Stuff the whole document into context and let the model find the answer" is the most expensive habit in AI product development. It works — and it bills you for every irrelevant token in the document. The fix is retrieval: send only the parts actually relevant to the current question.

The pattern, whether you're building RAG or wiring a knowledge base to an agent:

1. Index once, retrieve per turn. Chunk your docs (or call a retrieval API), and at query time fetch only the top-k relevant chunks. The model sees a focused context, not a firehose.
2. Bound the retrieved context. Cap total retrieved tokens per turn. If your tool documentation is 50k tokens and the answer only needs 2k, sending 50k is pure waste. This is exactly how the MCP-connected agents described in the agent observability guide keep their per-call cost sane.
3. Cache retrieval results. Identical questions from different users hit the same chunks. A short-lived cache on the top chunks for popular queries cuts both latency and spend.

Retrieval is a bigger engineering lift than the first two levers, but it's also the one that scales: it's what lets a product serve a huge knowledge base without the cost growing linearly with the corpus.

Lever 4 — Model tiering: match the model to the task

Usage cost belongs in the commercial scope; see AI Service Proposals for usage limits and budget controls.

Not every call needs a frontier model. This is the highest-multiple lever, and the one teams most often skip because "just use the best model" is the path of least resistance. The discipline:

• Route easy tasks to small models. Classification, extraction, formatting, summarization of short text — all of these are handled perfectly well by a small, fast, cheap model. Save the frontier model for reasoning, planning, and complex generation.
• Use the small model as a router. A cheap classifier can decide which tier handles a request: simple → small model; complex → frontier. This "tiered routing" pattern is how serious products keep average cost per call at a fraction of the ceiling.
• Escalate within a task, not just across tasks. A coding agent can try the cheap model first and re-issue with the bigger model only when the cheap attempt fails a self-check. Escalation-on-failure is a robust way to keep quality while capping cost.
• Know your per-token economics. Frontier input vs small-model input is often a 10–50x price gap; output tokens cost more than input. Optimize the expensive output tokens too — shorter, tighter output specs (fewer tokens per response) can be worth more than any input optimization. Multimodal inputs are a category on their own: image and audio tokens are priced per unit and can dwarf all your text tokens, so the multimodal agents guide is where that budget gets managed.

Model tiering works hand-in-hand with everything above: a small model with a cached prefix and retrieved context is dramatically cheaper and often faster than a frontier model with a bloated prompt.

What this looks like on a real agent system

Monthly cost and success reviews protect retainer margin; see SEO and AI Service Retainers for AI operations scope.

Cost per task is a client-report metric, not just engineering data; see Client Reporting for SEO and AI Services for AI cost reporting.

Cost controls belong in the deployment layer too. The agent runtime deployment and hosting guide shows how to enforce per-run token, wall-clock, and dollar ceilings.

Here's the combined pattern that takes a typical agent from "expensively correct" to "cheap and correct":

1. A stable system prompt (cached at ~10% input cost) with tool definitions and instructions in a fixed prefix.
2. A small model that routes each request: known/simple → small tier; novel/complex → frontier tier.
3. Retrieval that pulls only the relevant chunks of your docs, bounded to a token cap per turn.
4. A sliding conversation window that summarizes anything older than ~10 turns.
5. Monitoring that tracks cost per task, not just latency — the metric that tells you if the system is actually getting cheaper.

That's the whole play. None of it requires a clever new architecture — it's discipline applied to the four levers. And because token cost is a core operational metric, this connects directly to the web analytics and observability work you're already doing: instrument cost per task the way you instrument latency, and you'll see the leaks before they become line items.

Common mistakes

Bottom line

• Sending the full knowledge base on every call. The most expensive habit there is. Retrieve, don't stuff — and bound the retrieved context.
• Ignoring prompt caching entirely. It's a provider feature that's on by default in many stacks and still missed by teams that hand-build prompts. Stable prefix, static-first ordering, and you're done.
• Using one frontier model for everything. If your average task is simple, you're paying 50x for headroom you don't use. Route, don't blanket-upgrade.
• Optimizing only input tokens. Output tokens cost more per token. Tighter output specs (fewer words, structured responses) are often a bigger win than any input trimming.
• Never measuring cost per task. If you don't track it, you can't see the leak. Instrument cost per task like you instrument latency.

Token cost is upstream of the model call: what you send, what you cache, what you retrieve, and which model you route to. Context discipline, prompt caching, retrieval instead of stuffing, and model tiering together cut spend by 3–10x without touching output quality — the discipline is the product. Your single next action: check whether your provider's prompt cache is on and your system prompt is a stable, static-first prefix; if not, fix that today.

FAQ