LLM latency optimisation: find the slow part, then fix it

TL;DR: LLM latency optimisation starts with measurement. Split the wait into time to first token and time per output token. Track P90 and P99, not averages. Fix the biggest part first: send less, stream, route easy jobs to small models and cache repeats. Then tune the serving engine. Roll each change out in stages.

Your LLM app feels slow. Where is the time going?

Most teams guess. They swap the model, or add a cache, or buy bigger GPUs. Sometimes it helps. Often it doesn't, because they fixed the wrong part.

Here's the short version. LLM latency has a few separate parts. Measure them first. Then fix the biggest one.

1. Split the wait into its parts.
2. Track the slow tail, not the average.
3. Cut the work the model does before it answers.
4. Tune the serving engine.
5. Shape how requests are scheduled.
6. Roll changes out in stages.

Step 1: Split the wait into its parts

Two numbers matter most.

• Time to first token (TTFT). How long the user waits before anything appears. It's driven by reading the prompt and by queueing.
• Time per output token (TPOT). How fast words arrive once the answer starts. It's driven by the model and the hardware.

A user feels TTFT first. A long answer then feels slow if TPOT is poor. These two have different fixes, so measure them apart.

Add the pieces around the model call too. Time in the load balancer. Time in a queue. Time fetching documents for RAG. Time in your own code. We cover how to record all of this in observability for LLM apps.

Test with real shapes of traffic, not one test prompt. Try four:

A change can lift average speed and still hurt the slowest requests. Only a mixed test shows that.

Step 2: Track the slow tail

Averages hide the pain. Look at P90 and P99. P99 is the time that 99 out of 100 requests beat. The slow one in a hundred is the one your user remembers.

Also track goodput. That's the share of requests that hit your speed target. Raw throughput can climb while goodput falls. Goodput tells you if users are happy.

Set targets before you change anything. For example, "TTFT under 500ms at P90 for chat". Without a target you can't tell if a change worked.

Step 3: Cut the work before the model starts

If you call a hosted model through an API, this is where you'll win most. You can't tune the GPU. You can control what you send.

• Send less. Long prompts take longer to read. Trim old chat turns and unused instructions.
• Ask for less. Output tokens arrive one by one. A shorter answer is a faster answer. Cap the length where you can.
• Stream the answer. The total time stays the same. The user sees words straight away, so it feels quicker.
• Use a smaller model for easy jobs. Route simple tasks to a fast model and keep the big one for hard ones. Our AI model selection guide shows how to pick.
• Cache what repeats. Put the stable text first and the changing text last, so the start of the prompt can be reused. LLM caching strategies covers the four types.
• Run calls side by side. If two lookups don't depend on each other, don't wait for one before starting the other.

Most of these also cut your bill. See token cost optimisation for the money side.

Building a voice app? The wait matters even more there. Digiocial has a plain take on what live voice AI can do now.

Step 4: Tune the serving engine

This part is for teams that host their own models. If that's you, a few settings can move a lot. Not sure you should host? Start with our guide to open-source LLMs.

Red Hat published a good worked example. They ran Qwen3-32B on 16 H200 GPUs with 2,000 tokens in and 100 tokens out. With default settings, P90 TTFT was 995ms. After they raised GPU memory use to 0.95 and set max_num_seqs to 192 and block_size to 128, it fell to 304ms. Adding smarter request routing took it to 287ms. That's a 71% cut on the same hardware and the same model. The write-up is on Red Hat Developer.

Two cautions. This was one model, one workload and one cluster. Your numbers will differ. And a different model needed different settings in the same article, so tune against your own traffic.

Three levers worth knowing:

• Quantisation. It stores weights in fewer bits. NVIDIA's TensorRT-LLM docs report that FP8 gave a 2.3x speedup over FP16 on LLaMA-v2-7B on an H100, with first token latency held under 500ms and a batch size of 16. Check your answer quality on your own test set before you ship it.
• Prefix caching. The server reuses the work it already did on a shared start of the prompt. The vLLM design notes explain how it works, and they recommend cache salts to keep tenants apart.
• Splitting prefill and decode. Reading the prompt (prefill) and writing the answer (decode) stress the hardware in different ways. Some teams run them on separate GPU pools so a long prompt doesn't stall other users' answers.

Step 5: Shape how requests are scheduled

Scheduling decides who gets GPU time when long and short requests compete. It matters most on mixed traffic.

One idea is chunked prefill. The server reads a long prompt in pieces and slips decode steps in between. The question is how big each piece should be.

A 2026 paper on a method called SLOWeave picks the largest piece that still finishes before the nearest answer deadline. At a 25ms per-token target, it reported about 39% better goodput than fixed chunks on mixed requests. The authors' paper is on arXiv. It's a research result, not a button you can press today. Still, the lesson holds: protect deadlines, not just averages.

A few simpler habits work on any stack:

• Admission control. If a request would blow its deadline, queue it or turn it away. Don't slow everyone down to accept it.
• Earliest deadline first. Serve the request closest to its limit before the one that arrived first.
• Scale on queue depth. GPU use lags behind real latency. Queue depth shows pain sooner.

Step 6: Roll changes out in stages

Every change above can backfire. So ship them carefully.

1. Set targets for TTFT, TPOT and P99 per traffic type. Put them on a dashboard first.
2. Replay real traffic against a staging copy before anything goes live.
3. Canary it. Send a small share of users to the change first.
4. Agree your rollback line in advance. For example, "P99 up 20%". Then the call is automatic, not a debate mid-incident.
5. Watch after launch. Cache hit rate, queue depth and answer quality can all drift.

Reach for the quick, reversible fixes first. Prompt trims, streaming, routing and caching are cheap to test and cheap to undo. Retraining a model or pruning it takes longer and adds risk. Only go there when serving fixes have run out.

Where Devwiz fits

A fast demo is easy. A fast app under real load is real engineering. And speed is a design choice, not a patch you add later.

We build AI apps and turn existing programs into AI platforms. We measure the real traffic first, then fix what's slow. Devwiz has shipped 200+ apps, including work for NSW Government (Justice and Corrective Services), Briometrix, Vivid and Huskee.

Wondering what a build like this costs? AI-Led breaks down what an AI app costs to build. Want to know where your tokens and time go? AI Orchestrators covers AI agent observability.

Got an AI feature that feels slow? Worth a chat. Send us the slowest flow and we'll tell you where the time is going.

Frequently asked questions

How do you reduce latency in an LLM response?

Measure first. Split the wait into time to first token and time per output token. Then fix the larger part. Send shorter prompts, cap output length, stream the answer, route easy jobs to a smaller model and cache repeated prompt starts. If you host the model, tune the serving engine too.

What is time to first token?

It is how long a user waits before the first word appears. Reading the prompt and queueing drive it. It is separate from time per output token, which is how fast the rest of the answer arrives.

Why track P99 latency instead of the average?

The average hides the slow requests. P99 is the time that 99 out of 100 requests beat. The slowest one in a hundred is the one a user remembers, so it is the better guide to how the app feels.

Can engine settings really cut latency that much?

They can. Red Hat reported P90 time to first token falling from 995ms to 304ms on Qwen3-32B by tuning settings, and to 287ms with smarter routing. That was one model, workload and cluster, so test against your own traffic.

What is prefill and decode splitting?

Prefill is reading the prompt. Decode is writing the answer one token at a time. They stress hardware differently. Some teams run them on separate GPU pools so a long prompt does not stall other users' answers.