LLM caching strategies: four types and when to use each

TL;DR: There are four LLM caching strategies. Prefix and prompt caching reuse the work done on the start of a prompt. Response caching returns a stored answer for the exact same request. Semantic caching returns one for a similar question. Start with prompt order and provider caching. Add semantic caching last, once you have a test set to check it.

Every LLM call does two jobs. First it reads your prompt. Then it writes the answer, one token at a time.

Caching mostly speeds up the first job. Some types skip the model altogether. Pick the wrong one and you get stale answers, or no saving at all.

Here are the four LLM caching strategies, and when each one fits:

1. Prefix caching. Reuse the work done on the start of a prompt.
2. Prompt caching. The same idea, run by your model provider.
3. Response caching. Same request in, stored answer out.
4. Semantic caching. A close-enough question gets the stored answer.

Where caching helps, and where it doesn't

The vLLM docs on automatic prefix caching spell out the limit. It cuts the time spent reading the prompt. It doesn't cut the time spent writing new tokens.

So caching pays when the prompt is long and the answer is short. vLLM gives two cases where it shines:

• Long document questions. Users keep asking about the same manual or annual report. The model reads it once.
• Multi-round chat. Each new message reuses the chat history that came before.

It pays least when answers are long. Same story when no two prompts start the same way.

Caching is one lever on the bill. For the other four, read our post on token cost optimisation.

The four types side by side

1. Prefix caching

This one lives in the model server. You'll meet it if you host open-source LLMs yourself.

vLLM's design notes show how it works. The server cuts your prompt into blocks of tokens. It hashes each block along with the prefix before it. Same prefix, same hash, so the stored work gets reused. Only full blocks are cached.

Here's the catch. Each block's hash includes the hash of the block before it. Change one token near the top and every block after it misses.

What breaks it on a real build?

• A timestamp at the top of the system prompt.
• Tool definitions that load in a different order.
• A stray space in a template.

So build these habits in early:

1. Put the stable stuff first. System prompt, tools, the long document.
2. Put the bits that change last. User ID, time, the new question.
3. Build prompts from one template, so spacing never drifts.
4. Keep tenants apart. vLLM supports cache salts for this.
5. Stick with the default hash. vLLM uses sha256. It warns that faster, non-secure hashes raise the odds of a clash, and a clash could leak private data between tenants.

Tip: put prefix hit rate on a dashboard from day one. A sudden drop often means someone changed a prompt template.

New to multi-tenant builds? Start with our multi-tenant SaaS basics. Want to see how a big system prompt is laid out? Digiocial pulls apart ten patterns from the Opus 4.7 system prompt.

2. Prompt caching from your provider

Use a hosted model? Then the provider runs the cache. Your job is to tell it where the stable part ends.

Take Anthropic. Its prompt caching docs give two ways in:

• Automatic caching. Add one field to the request. The breakpoint sits on the last block it can cache and moves forward as the chat grows.
• Explicit breakpoints. You mark the exact blocks to cache. You can set up to four.

The rules are strict. A hit needs the prompt to be 100% the same, up to and including the marked block. The cache also follows an order: tools, then system, then messages. Change a tool definition and you lose the whole cache.

So where does the breakpoint go? At the end of the last block that stays the same between calls. Never on the bit that changes.

Three more things to know:

• It expires. The default life is 5 minutes. Each use resets the clock at no extra cost. There's a 1-hour option that costs more.
• Writes cost more than reads. Anthropic prices a 5-minute cache write at 1.25 times the base input price. A read is 0.1 times, or less on some newer models. So a cache you write and never read loses money.
• Short prompts don't cache. There's a minimum length, and it varies by model.

One thing it won't do is change the answer. Anthropic says prompt caching has no effect on the output.

Warm it before the first user

The first call after a quiet spell pays for the full read. Anthropic's fix is a pre-warm. Send a request with max_tokens set to 0. The API reads the prompt, writes the cache and returns no output. You pay for the cache write. No output tokens are billed.

Two traps:

• Put the breakpoint on the shared block, not on the dummy user message. That means an explicit breakpoint. Automatic caching would mark the dummy message.
• Use the same thinking settings as your real traffic. Anthropic says a pre-warm with different settings can write an entry your real calls never hit.

Anthropic suggests firing it when the app starts, or on a schedule.

Want the price side in plain words? AI-Led has Claude API pricing explained.

3. Response caching

The simplest one. Store the full answer against the exact request. Next time, skip the model.

It suits requests that repeat word for word. FAQ lookups. Read-only tool calls that give the same result for the same input.

Two rules keep it safe:

• Build the key from every input that shapes the answer. Model, system prompt, settings and the user's text.
• Only cache answers that should be the same every time. A creative draft isn't one.

LangChain's Redis cache guide shows the pattern in a few lines. It covers a standard cache and a semantic one. It also shows how to set a time to live on entries.

4. Semantic caching

Here the match is on meaning, not on exact words. The cache turns each prompt into a vector. It finds the nearest stored prompt. If the score clears a line you set, it returns that stored answer.

So "How do I reset my password?" and "I forgot my password, help" can share one answer.

That's the appeal. It's also the risk.

Microsoft's docs for the Azure semantic cache lookup policy are blunt about it. The match is on similarity. So it can return answers that are incorrect, outdated or unsafe for the request.

Their advice on the score threshold:

• Start low, at 0.05. In this policy, a lower value means a stricter match.
• Above 0.2 may lead to cache mismatch.
• Go lower again for sensitive use cases.
• Split the cache by user or group with vary-by. That controls who can see which entries.
• Put a rate limit straight after the lookup. If the cache isn't there, your model doesn't get flooded.

Watch the direction. Here, lower is stricter. Check which way your tool runs before you copy a number across.

One threshold rarely fits every question

The vCache paper makes the case. It says static thresholds give no formal promise that answers are right. They also lead to error rates nobody planned for.

Its fix is to learn a threshold for each cached prompt as traffic comes in. You set the error rate you can live with. The authors report up to 12.5 times more cache hits than the baselines they tested, with error rates up to 26 times lower. The paper was accepted to ICLR 2026.

You don't need vCache to act on that. Just don't trust one number across every kind of question.

Before a semantic cache goes near real users:

1. Test it on a labelled set of real questions.
2. Track wrong hits on their own. A high hit rate with wrong answers is worse than no cache.
3. Keep it away from medical, legal and money questions until the numbers hold up.

That labelled set is the same one you need for LLM evaluation. Build it once. To the user, a wrong cached answer looks a lot like a made-up one. So hallucination reduction is worth a read too.

Keeping a cache honest

A cache that serves old answers is worse than a slow app.

• Set the time to live by how fast the facts change. Static docs can sit for hours. Live prices need minutes.
• Clear it on change. New prompt template, new model, new source doc. Each one means a flush.
• Sample the hits. Send a small share of cache hits to the live model and compare the two. We'd treat a growing gap as an early warning.
• Always fall through. If the cache is down, call the model. Don't throw an error.

Then track four numbers:

1. Hit rate, per cache type.
2. Wrong-hit rate, from the sampling.
3. Lookup time. The cache shouldn't be the slow part.
4. Money saved against a no-cache baseline. Count the cost of embeddings and storage too.

Those numbers belong with the rest of your cost tracking. Our FinOps for AI post covers who owns them. The Orchestrators have a plain guide to LLM cost optimisation for the business side.

Which one should you pick?

Ask four things:

• Does the prompt start the same way each time? Go prefix or prompt caching first.
• Do the exact same requests repeat? Add a response cache.
• Do people ask the same thing in different words? And can you live with a rare wrong hit? Try semantic. Carefully.
• Are answers long and different every time? Caching won't do much. Look at which model you pick instead.

Then run a small pilot before you roll it out wide:

1. Replay last week's traffic. Count how often prompts repeat. No cache code needed yet.
2. Pick one workload. Turn on one cache type.
3. Set a pass mark for hit rate and wrong-hit rate before you start.
4. Compare the bill and the wait time to the baseline.

No clear win? Turn it off. A cache is one more thing to run.

Caching isn't the only fix for a long history

Caching makes a long prompt cheaper to send again. A memory layer comes at it from the other side. It keeps what matters and sends only that.

OctaMem builds one. Their LoCoMo benchmark results are worth a look for how they report. LoCoMo tests whether a memory system can answer questions about facts buried deep in a long chat that runs over many sessions. The whole history doesn't get stuffed into the prompt. They publish cost per question next to accuracy. They also show the one conversation where most of their errors sit.

It's their own run on their own test setup, and they say so. Read it as one vendor's numbers, not a league table.

We cover the design side in agent memory design.

Where Devwiz fits

We've shipped 200+ apps, including work for NSW Government (Justice and Corrective Services), Briometrix, Vivid and Huskee. AI now runs through how we build and what we build.

Our view on caching is simple. Fix the prompt order first. It's free. Then turn on provider caching. Leave semantic caching until you've got a test set to check it with.

Caching is one cost lever. Model routing is another. In our white-label AI platform case study, routing went in so the right model gets picked per tool, per tenant and per task.

We build AI apps and turn programs into AI platforms.

Not sure which cache fits your app? A discovery phase sorts that out fast. Worth a chat?

Frequently asked questions

What is the difference between prefix caching and semantic caching?

Prefix caching reuses the model's work when a new prompt starts with the same tokens as an old one. It cuts prompt reading time, not writing time. Semantic caching matches on meaning. It can return a stored answer to a reworded question, with some risk of a wrong match.

How do I pick a threshold for semantic caching?

Start strict. Microsoft's Azure API Management docs suggest a low score threshold such as 0.05, where lower means a closer match, and warn that values above 0.2 may lead to cache mismatch. Then test on a labelled set of real questions before you loosen it.

Does prompt caching make the model write faster?

No. The vLLM docs say prefix caching only cuts the time spent reading the prompt, not the time spent writing new tokens. If your answers are long, caching helps less.

Why does my prefix cache keep missing?

Most likely a token near the start of the prompt changed. In vLLM, each block's hash includes the block before it, so one early change breaks every block after it. Put stable content first and the bits that change last.

How do I pre-warm a prompt cache?

Anthropic's docs say to send a request with max_tokens set to 0. The API writes the cache and returns no output. Put the breakpoint on the shared block, not the dummy message, and use the same thinking settings as real traffic.