Observability for LLM apps: six steps to see what broke

TL;DR: Observability for LLM apps means recording enough about each request to explain a slow, costly or wrong answer. Start with the model name, token counts, finish reason and a request ID. Trace retrieval and tool calls. Leave prompt text out by default. Scrub what's left at the collector.

Your LLM app gave a wrong answer last Tuesday. Or a slow one. Or one that cost far more than it should.

Can you say why?

If you can't, you have logs. You don't have observability yet.

Observability for LLM apps means you record enough about each request to explain it later. Which model ran. How many tokens went in and came out. Why it stopped. What it fetched. How long each step took. Whether the answer was any good.

Here are the six steps:

1. Record the basics on every model call.
2. Trace the whole request.
3. Leave prompt text out by default.
4. Watch four numbers.
5. Scrub personal data at the collector.
6. Roll it out in stages.

Why normal monitoring isn't enough

A web app usually fails loudly. It throws an error, or it times out.

An LLM app can fail quietly. The call comes back fine. The answer reads well. And it's wrong.

Cost moves too. Two requests to the same endpoint can cost very different amounts. The bill follows tokens, not calls.

So keep your speed and error charts. Then add three things: what the model was asked to do, what it used, and how the answer scored.

Step 1: Record the basics on every model call

Don't make up your own field names. OpenTelemetry has a set for this. They're called the GenAI semantic conventions.

Two things to know first:

• They've moved. They now live in their own repository. The old pages on the main OpenTelemetry site are marked as moved.
• They're marked Development. Names can still change, so pin a version.

Start with six attributes on each model call:

Why both model fields? They aren't always the same string. In the spec, the example value for the request field is gpt-4. For the response field it's gpt-4-0613.

The finish reason earns its place too. The spec's examples are "stop" and "length". An answer that ran out of room is a different bug from one that ended cleanly.

Does your model change per tenant or per task? Then these fields matter even more. In our white-label AI platform case study, the platform picks the right model per tool, per tenant, per task. On a build like that, "which model answered?" is the first thing you'd ask in a bug report.

Step 2: Trace the whole request

The model call is one step. A request also builds a prompt, fetches documents and calls tools. Any of them can be the bad step.

The conventions define a span for each kind of step. The main ones:

• Inference. The call to the model.
• Embeddings.
• Retrievals. A fetch from a vector database or search system.
• Execute tool. One tool call.
• Memory.

Agent runs get their own set of spans.

Now here's the important bit. Thread one request ID through every span and every log line. Then you can pull up one bad request and read each step in order.

Building retrieval? Our RAG pipeline guide covers the steps you'll be tracing.

Step 3: Leave prompt text out by default

The spec is blunt here. Instructions, user messages and model outputs are sensitive, and often large. Tools that add tracing for you should not capture them by default. They should let you opt in.

It gives three patterns:

1. Don't record prompts or outputs. This is the default.
2. Record them on the spans. The spec says this suits places where volume is manageable and privacy rules either don't apply or are met by the storage. Pre-production, for example.
3. Store the content somewhere else and put a reference on the span. The spec recommends this for production when volume is a concern or sensitive data needs secure handling. You can put separate access controls on that store.

So start with metadata only. Model, tokens, timing, finish reason.

Chasing one bug? Turn content capture on for a short window. Then turn it off again.

Step 4: Watch four numbers

1. Time to first token. The metrics spec has a server metric for it, gen_ai.server.time_to_first_token. It says the metric helps measure time spent in the queue and the prefill phase. It matters most for streaming.
2. Time per model call. On the client side that's gen_ai.client.operation.duration. For the whole request, read the top span of your trace.
3. Tokens per request. Input and output, kept apart. This is your cost signal.
4. Quality score. The conventions define an evaluation event, gen_ai.evaluation.result. It holds the result of scoring an answer for quality or accuracy. It should hang off the span it scored.

Set a target for each one before launch. A number with no target is a chart nobody reads.

Where to go deeper:

• Cost. Read our posts on token cost optimisation and FinOps for AI. Want to turn tokens into dollars? AI-Led has Claude API pricing explained.
• Quality. LLM evaluation covers how to trust the score. AI model drift covers the slow slide.
• Caching. Run a cache? Put its hit rate on the same dashboard. See LLM caching strategies.

Step 5: Scrub personal data at the collector

Telemetry passes through a collector before it's stored. The OpenTelemetry Collector is a vendor-agnostic way to receive, process and export it.

That middle step is the place to scrub. Why there, and not in app code alone? App code changes every sprint. One new log line can leak a prompt. A rule in the collector gives you a second net.

OpenTelemetry's guide to handling sensitive data starts with data minimisation:

• Only collect data that serves an observability purpose.
• Avoid personal information unless it's absolutely necessary.
• Ask whether aggregated or anonymised data would do the same job.
• Review what you collect on a regular basis.

Then it lists four collector processors that help:

• Attribute processor. Removes or changes named attributes.
• Filter processor. Drops whole spans or metrics.
• Redaction processor. Deletes attributes that aren't on an allow list.
• Transform processor. Rewrites data with regular expressions.

One warning from the same guide. Hashing a user ID may not hide it. If the IDs are small and easy to guess, like numeric user IDs, the hash can be reversed in practice.

For the scrubbing itself, see our post on PII redaction.

What the OAIC says

Building for Australian users? Read the OAIC's guidance on commercially available AI products. Five points matter for telemetry:

• Privacy obligations apply to personal information put into an AI system. They also apply to the output, where it contains personal information.
• If an AI system generates or infers personal information, that's a collection. It must comply with APP 3.
• Personal information includes inferred, incorrect or made-up information about a person who can be identified. A hallucination about someone who can be identified counts.
• Organisations looking at AI products should take a privacy by design approach. That includes a Privacy Impact Assessment.
• Under APP 11, you take reasonable steps to protect the personal information you hold.

Now think about what a prompt log is. It's a store of what users typed and what the model said back.

Our read: if that log can hold personal information, treat it like any other store of it. Short retention. Tight access. A record of who looked.

Planning to fine-tune on those logs later? That's a second purpose. The OAIC's guidance for developers training generative AI models covers it. Say a chatbot's prompts hold personal information. They were collected to provide the chatbot service. Using them to train later versions is a secondary use. Without consent, you'd have to show the person would reasonably expect it, and that it's related to the first purpose. For sensitive information, it has to be directly related.

We build software. We're not lawyers. Get advice on your own case.

More on the local rules: data residency in Australia and AI regulation in Australia.

Work in a regulated field like financial services? You may need more than logs. You may need proof of what each agent was allowed to do at the moment it acted. WithinBounds sells a control layer for AI agents in regulated institutions. Their site says it enforces policy at the moment the agent acts, and records every decision against the policy version in force. It's one vendor in that space, and we haven't tested it.

Step 6: Roll it out in stages

Don't switch it all on at once.

1. Dev. Add tracing to the model calls. Check the six fields land.
2. Staging. Add retrieval and tool spans. Run fake load. Confirm one request ID runs through the lot.
3. Canary. Send a small share of real traffic. Check your scrub rules against real shapes of data.
4. Full rollout. Alerts on. Retention set.

Add an alarm on token spikes before go-live. A prompt that keeps growing shows up there before it shows up on the invoice.

Want it wired into your pipeline? See CI/CD for AI. For what happens after launch, see post-launch AI platform support.

When something breaks, use more than one signal

A 2026 study put this to the test. Beyond Microservices ran 24 root cause analysis methods against an LLM inference deployment, with failures injected on purpose. 20 methods used metrics. Two used traces. Two used more than one source.

What they found:

• The multi-source methods were the most accurate.
• The metric methods did well on some fault types and poorly on others.
• The trace methods largely failed.

The authors say existing root cause tools don't generalise to LLM systems. The paper was accepted to the QualITA workshop at ICPE 2026. The code and data are on GitHub.

One caveat. The study tested automated tools on the serving layer. It didn't test people debugging a chatbot. Our takeaway is still simple. Don't bet on one signal.

Here's a simple order to work in:

1. A metric tells you something changed.
2. A trace tells you which step.
3. A log tells you what went into that step.
4. An evaluation score tells you if the answer got worse, or only slower.

Agents need a second view

Agents make this harder. An agent states its plan in a prompt. Then it acts on the machine. That's two views.

AgentSight is a research framework that joins them. Its authors say existing tools see the intent or the actions, but can't link the two. They call their method boundary tracing. It watches the agent from outside its code, using eBPF. It reads the LLM traffic to get the intent. It watches kernel events to see the effect. Then it links them.

The paper reports:

• Less than 3% performance overhead.
• It detected prompt injection attacks.
• It found reasoning loops that waste resources.
• It showed hidden coordination bottlenecks in multi-agent systems.

It's open source. It needs no changes to the agent's code.

Is this a day-one job? No. Get steps 1 to 6 working first. Look at boundary tracing once your agents run commands on real machines.

Related reading:

• Our prompt injection defence checklist.
• Multi-agent systems explained.
• The Orchestrators on AI agent observability, written for the business side.
• Digiocial on why so many agent projects get shelved.

Where Devwiz fits

We've shipped 200+ apps, including work for NSW Government (Justice and Corrective Services), Briometrix, Vivid and Huskee. AI now runs through how we build and what we build.

Our view on observability is simple. Add it with the first model call, not after the first incident. Start with metadata. Keep prompts out until you need them. Scrub at the collector.

We build AI apps and turn programs into AI platforms.

Not sure what to record on your build? A discovery phase sorts that out fast. Worth a chat?

Frequently asked questions

What is observability for LLM apps?

It means recording enough about each request to explain it later. That covers the model, token counts, finish reason, each retrieval and tool step, timing and a quality score. Normal monitoring tells you a call failed. This tells you why an answer was slow, costly or wrong.

What is the best tool for LLM observability?

There isn't one best tool. The OpenTelemetry Collector is vendor-agnostic, so you can send the same data to one or more backends. OpenTelemetry's own 2024 walkthrough sends traces to Jaeger. Start with the backend your team already runs.

Should I log full prompts and responses?

Not by default. The OpenTelemetry GenAI conventions say tools should not capture instructions, inputs or outputs unless you opt in. For production, they recommend storing content outside your telemetry and recording a reference to it. That way you can control access to it on its own.

How is LLM observability different from LLM evaluation?

Evaluation scores answers, often against a test set. Observability records what happened on real requests. They meet when you attach a quality score to a live request. The OpenTelemetry conventions define an evaluation result event for that.