Token cost optimisation: five levers that cut your LLM bill

TL;DR: Token cost optimisation cuts what each LLM call costs without hurting the answer. Log every call first. Then trim prompts, cache what repeats, route easy work to cheaper models, cap output and batch anything that can wait. Put a quality check on every change, because a cheaper answer that's wrong is not a saving.

Token cost optimisation is how you cut what each LLM call costs without making the answer worse. Measure every call first. Then pull five levers: trim the prompt, cache what repeats, route easy work to cheaper models, cap the output, and batch anything that can wait.

Stacked together, they add up. NeuralTrust's guide puts it at 60 to 80% off for teams running all five levers at once. Treat that as a ceiling, not a promise. Your number depends on your traffic.

One problem though. Push any lever too hard and quality slips. So every change gets a quality check before it ships. No exceptions.

Why your bill goes up while token prices go down

Token prices keep falling. Bills keep climbing. NeuralTrust calls this out directly: prices dropped, and enterprise AI bills went up anyway.

The reason is volume. RAG pipelines stuff more context into every call. Agents make several calls for one user action. Cheaper tokens, times more tokens per call, times more calls per task, still gives you a bigger invoice.

Then there's the output side. TrueFoundry found output tokens cost 3 to 5 times more than input tokens on most frontier models. So a feature that writes long answers can cost far more than one that reads a lot and writes a little. Know your input to output ratio before you compare sticker prices. Our LLM API pricing comparison has the raw numbers.

Where the waste comes from

Most token waste isn't clever. It's a handful of habits that quietly multiply after the prototype ships.

• Agent loops that re-read the same context on every step
• RAG pipelines that pull ten chunks when three would answer the question
• Chat apps that replay the whole conversation on every turn
• System prompts that grew for a year and never got trimmed
• Raw PDF text, full API responses and noisy tool logs dumped into the prompt

Each one is invisible in a demo. At scale, a chat that costs cents in testing costs dollars once real users hold ten-turn conversations and the app replays the lot every time.

Towards Data Science has a sharp breakdown of how agentic RAG goes wrong: retrieval thrash, tool storms and context bloat. It also points to Stanford and Meta's "Lost in the Middle" study. Performance dropped by more than 20 percentage points when the key fact sat in the middle of a long context. More context doesn't just cost more. It can make the answer worse.

If long chat history is your problem, start with how context windows work, then look at agent memory design. If it's retrieval, the fix usually lives in the RAG pipeline, not the model.

Measure every call before you touch anything

You can't cut spend you can't see. CostCompass frames it as a loop: measure where the money goes, act on the biggest line, then measure again.

Log five fields on every single call:

1. Prompt tokens and completion tokens, kept separate. Never just the total.
2. The model, so you can see how traffic splits across tiers.
3. A feature or endpoint tag, so cost rolls up to a product surface, not an API key.
4. A team or tenant ID, for multi-tenant reporting.
5. A timestamp, so you can catch spikes and drift.

From that, four numbers do most of the work: cost per request, cache hit rate, requests per feature, and model mix. Group spend by feature and sort from the top. The expensive calls are almost always the agent loops and over-stuffed RAG calls from the list above.

Set alerts before the budget runs out, not when it does. Infracost suggests a warning alert at 70 to 80% of budget, with an urgent one at 90 to 95%. Add a simple daily anomaly check too. It catches a runaway agent loop days before finance does.

This is the same tracking that sits under FinOps for AI. That post covers who owns the spend and how often you review it. This one is the engineering side: how to make each call cheaper.

The five levers, in the order I'd pull them

Here's the thing. These levers stack. Each one hits a different part of the bill, and each one breaks in its own way if you push too far.

1. Trim the prompt

Merge system prompts that grew over time. Cut repeated examples. Use templates instead of free-form instructions pasted into every call.

Be honest about the gains, though. Steven Gonsalvez's token optimisation playbook looks at one request compression tool whose "60 to 95% fewer tokens" headline only held on tool-heavy sessions. The median across more than 50,000 sessions was 4.8%. Compression helps. It's rarely magic.

Test before and after on a fixed set of real inputs. A smaller prompt that changes how the model behaves is a regression, not a saving. Our guide to prompt engineering covers how to test prompts properly. Digiocial's breakdown of system prompt patterns from the Opus 4.7 leak is a good look at what a tight system prompt looks like.

2. Cache what repeats

Exact-match caching is the easy win. Same prompt in, stored answer out. It's cheap to build and catches repeat questions word for word.

Semantic caching goes further. It matches on meaning, so a reworded question still hits the cache. One problem though. Set the match too loose and it hands back the wrong answer to a slightly different question. Start strict. Loosen it only when your test set says it's safe. Keep a separate cache per system prompt, so answers from one feature never leak into another.

Provider prompt caching is different again. It discounts the stable start of your prompt. TrueFoundry puts that at 50 to 90% off the prefix, but only at the hit rate your traffic supports, and it rarely reaches the 90% people expect. CostCompass adds a fair warning: check each provider's caching terms before you count on the discount. Size your hit rate from your own logs, not a pricing page.

3. Route easy work to cheaper models

Send simple, high-volume jobs to smaller models. Classification, extraction, short summaries. Keep the premium model for the hard judgement calls.

Gonsalvez boils the whole playbook down to two moves. Send the model less. Then hand the grunt work to cheaper models, so the premium one only sees real judgement calls. Paying frontier rates to reformat a date string is the kind of waste nobody spots until they read the itemised bill.

Rule-based routing on task type or input length is the fastest to ship. A small classifier that predicts difficulty is more accurate, but it adds latency and one more thing to maintain. Either way, keep an escape hatch. If the cheap model sends back something malformed or low-confidence, pass the job up to the stronger one.

Picking which models go where is its own decision. AI-Led's guide on how to choose an LLM walks through task fit, cost and latency.

4. Cap the output

Set max_tokens on purpose. Don't leave it at the provider default. Use stop sequences so the model stops the moment the answer is done. Where the next system only needs fields, ask for JSON or a function call, not free text.

Output tokens cost several times more than input. So trimming wordy answers is often the single biggest win you've got.

5. Batch anything that can wait

Overnight reports, bulk document tagging, backfills. None of it needs an answer this second. TrueFoundry puts batch pricing at roughly 50% off for work that can wait up to 24 hours. Move even a quarter of your eligible traffic there and the line item drops.

AI Orchestrators ranks routing, caching and batching as the three biggest levers in their LLM cost optimisation guide, along with the measurement that tells you which one is working.

Put a gate in front of the bill

Levers only save money if something enforces them at runtime. The pattern that scales across providers is a thin proxy between your app and the model API.

• The proxy counts tokens on every request and writes the cost into the response headers, so any service can read spend without a separate dashboard.
• A pre-flight estimate prices the call before it fires. An oversized prompt or an open-ended max_tokens gets caught and downgraded, not billed.
• A budget policy decides what happens near the limit: warn, drop to a cheaper model, serve a cached answer, or block once a tenant hits its ceiling.

Pro tip: Make the pre-flight check refuse the call above a hard ceiling. Don't just warn. A warning in a log nobody reads does nothing on the night an agent loop runs unattended.

Where that proxy sits in the wider stack is covered in AI platform architecture.

A 90-day plan

This assumes a small team with live production traffic, not a greenfield build.

1. Month one: log every call, ship the dashboard, and switch on exact-match caching. Route one clearly low-risk task to a cheaper model as a first test.
2. Month two: pilot semantic caching on your busiest feature. Move batch-friendly work to async. Tighten your alert thresholds using month one's data.
3. Month three: add cost regression tests to CI. Hand ongoing monitoring to a named owner. Only then look at fine-tuning or self-hosting.

Write the pass mark before month one starts. A target cost per request. A minimum cache hit rate. A quality benchmark that has to hold steady. "The bill looks better" is not a result.

Every routing or compression change goes out as a canary against a fixed test set, with a rollback trigger if accuracy drops below the agreed line. Quality drifts quietly, so keep watching after launch too. We covered what that looks like in AI model drift.

Fine-tuning or self-hosting only pays off when one narrow, high-volume task justifies the running cost. For most teams, caching and routing on general-purpose APIs stay cheaper and easier to run. If you're weighing it up, start with fine-tuning vs RAG and our look at open-source LLMs.

Treat token spend like latency or error rate. It needs an owner, a dashboard and an alert. Not a line item finance flags once a quarter.

What we see on real builds

We've shipped over 200 apps, including work for NSW Government, Briometrix, Vivid and Huskee. AI runs through most of what we build now.

One recent build took a vibe-coded MVP to a multi-tenant white-label AI SaaS platform. OpenRouter routes between Claude, GPT and other models from one interface, so the platform can use the right model per tool, per tenant and per task. That's lever three built into the platform, not bolted on after the first scary invoice.

Now here's the bit most cost guides skip. A lot of token problems are architecture problems in disguise. An agent that should be a scheduled job. A chatbot stuffing whole documents into context when it needs a proper retrieval pipeline. Tuning prompts won't fix those. Rebuilding the flow will.

Is your token bill a symptom of something bigger?

If your AI feature is live and you can't say what it costs per request, that's the gap to close first. Everything else is guessing.

If the fix is a rebuild rather than a tweak, that's what our AI app development work is for. We'll design routing and caching in, not retrofit them. If you're turning an AI program into a platform other people run, have a look at AI programs.

Worth a chat?

Frequently asked questions

What is token cost optimisation?

It's the set of engineering habits that cut how many tokens an LLM app uses and pays for, without hurting the answer. The main levers are prompt trimming, caching, model routing, output caps and batching, all backed by per-call tracking.

How much do 1,000 tokens cost?

It depends on the provider and the model tier, so there's no single figure. Output tokens usually cost 3 to 5 times more than input tokens. Check your provider's pricing page for both rates separately.

What are the main levers for cutting LLM costs?

Measure first. Then trim prompts, cache repeat requests, route routine work to the cheapest model that handles it well, cap output length, and batch anything that can wait. Most of the savings come from stacking them.

Is prompt caching worth it?

Usually, but size it from your own traffic. Provider caching discounts the stable start of a prompt, often by 50 to 90%, but only at the hit rate your traffic supports. Check each provider's caching terms before you count on it.

How do I cut LLM token costs fast?

Log every call with the model, the feature and the token counts. Then switch on exact-match caching, route one low-risk task to a cheaper model, and set max_tokens on purpose. Check quality before and after each change.