LLM unit testing: three test tiers that catch bad prompts early

TL;DR: LLM unit testing checks the code around your model, not the model itself. Mock the model for fast tests on every commit. Run a small golden set of live calls on pull requests. Save the full eval suite for a nightly run. Use schema and string checks first, and LLM-as-judge last.

LLM unit testing checks the code around your model, not the model's brain. You test that the prompt fills in right, the output parses, and your app copes when the model sends back junk. Mock the model for the fast tests. Keep a small golden set of live calls for pull requests. Run the big, slow eval suite at night.

That's the whole playbook. Three tiers. The rest of this post shows how to set each one up without blowing out your CI bill.

What an LLM unit test checks

A unit test for an LLM feature tests your code, not the model. Does the prompt template fill in the right values? Does the reply parse into the shape the next bit of code expects? Does the app handle an empty or broken reply without falling over?

Treat the model like any other outside service. You'd stub a payment gateway in a unit test. Stub the model the same way. If you're still wiring the model into your product, AI-Led's LLM integration guide covers the setup first.

Four failures deserve their own tests:

• Grounding failures, where the model says something the source never said.
• Prompt injection, where user input takes over your system instructions. Our prompt injection defence guide covers the fixes.
• PII leaks, where private data turns up in an output. See PII redaction for the controls.
• Regressions, where a prompt tweak breaks a case that used to pass.

Exact string matching mostly fails here. Two right answers can be worded ten different ways. But it still works for structure: a JSON key that must exist, a disclaimer that must show up, a banned word that must never appear.

New to test types in general? Start with our plain-English guide to software testing. This post assumes you already write unit tests for normal code.

The four checks, cheapest first

Most good LLM test suites end up with the same four checks. Stack them from cheapest to dearest.

1. Schema checks. Validate the JSON before it touches your app. Run them against a mocked model. They cost nothing and finish in milliseconds.
2. String and regex checks. Must contain, must not contain. Good for required disclaimers and phrases that should never ship.
3. Semantic similarity. Compare the reply to a reference answer with embeddings, so a reworded but correct answer still passes.
4. LLM-as-judge. A second model scores the output against a rubric. Save it for open-ended output nothing cheaper can grade.

Here's the thing with semantic similarity. Don't guess the threshold. Autonoma's LLM unit testing guide sets it from data. Score a batch of known-good answers against the reference, then put the floor just below the worst good score. Pick 0.8 because it looks about right and you've built the flakiest test in the suite.

LLM-as-judge is the slowest and least predictable check, so it goes last. FutureAGI's piece on deterministic versus judge-based evals calls them layers, not rivals. By their numbers, cheap deterministic checks catch 30 to 60% of failures for free. The judge picks up the meaning-level failures that slip past.

Tip: if you use a judge, pin the exact model version and run it at temperature zero. Write the rubric so a junior engineer could grade blind. A vague rubric gives you shaky scores, even from the same model.

The three test tiers

One giant test run slows CI down. Slow tests get skipped. So split the suite into three tiers.

Tier 1 is mocked tests on every commit. Test template rendering, output parsing and error handling against a fake provider. No API key. No network. They run in milliseconds.

Tier 2 is a golden set on every pull request. A small set of real cases, run against the live model. This catches the regressions a mock can't see.

Tier 3 is the full eval suite every night. More edge cases, more prompt variants, more cost. It's too slow for a pull request, so it runs on a schedule.

Build the fake provider early. Make it return clean answers, broken JSON, empty replies and cut-off outputs. Then your parsing and error handling get tested without a single real API call.

Now here's the important bit. Live model calls vary from run to run, even with the same input. Fast.io's guide to prompt regression testing runs 3 to 5 trials per case and uses the combined score. One bad trial doesn't fail the build. It also sets a pass mark per metric and fails the PR when a metric drops below the main branch baseline. Safety sits at 100%. No exceptions.

Log the baseline scores from every run. A slow slide in quality then shows up as a trend, before it turns into a bug nobody can trace. Our post on AI model drift covers what to do when that trend turns up in production.

Testing agents, not single prompts? The AI Orchestrators guide to AI agent testing goes deeper on scenarios, CI checks and replay.

Wiring it into CI/CD

LLM tests run in CI like any other check. The one difference: some of them need real network calls and a real budget.

The flow looks the same whether you use pytest or a YAML test runner:

1. Check out the code and install dependencies.
2. Run the mocked tests on every push.
3. Run the golden live set only on pull requests.
4. Save a readable report (JUnit XML, JSON or HTML), so failures are easy to find.
5. Post a PR comment with the pass and fail counts, so reviewers don't need the CI dashboard.

Add cost and latency checks to the same run. If a prompt change pushes token counts or response times past your budget, fail the PR. Our guide to token cost optimisation shows where those tokens usually go.

Don't wait until launch to build this. Crontent's guide to shipping AI features without endless rewrites makes the case for evals from day one. A rewrite starts from guesswork. An eval set tells you what broke.

Which tools to use

Four tools cover most teams. Each one fits a different tier.

• assertllm (also on PyPI as pytest-llmtest) is a pytest plugin with 22+ built-in assertions for text, performance and agent behaviour. Most need no LLM call at all, which makes it a good fit for the every-commit tier. The assertllm docs show the patterns.
• PromptCheck is a CI-first test tool. You write tests in YAML, and it gates pull requests and posts results as PR comments. Token count, latency and cost checks come built in, across OpenAI, Anthropic and OpenRouter. It's still in beta, so pin the version.
• DeepEval brings eval metrics like G-Eval, answer relevancy and faithfulness. It suits the nightly tier more than per-commit checks.
• Promptfoo is the other common pick for config-driven prompt tests. Autonoma's guide runs it side by side with DeepEval.

The simple rule: deterministic checks first, every time. A YAML test runner for cross-provider tests on PRs. A full eval framework for the nightly run only.

What this looks like on a real build

We've shipped over 200 apps, including work for NSW Government, Briometrix, Vivid and Huskee. AI runs through most of what we build now. And LLM features break in a different way to normal code. The code stays the same, but the answer moves.

Take the white-label AI platform we rebuilt from a vibe-coded app into a multi-tenant SaaS. OpenRouter routes between Claude, GPT and other models from one interface, so the right model gets picked per tool, per tenant and per task. That's the point: no vendor lock-in. It also means every model swap is a prompt change in disguise. Same prompt, different model, and the reply can come back in a different shape.

That's why these habits matter:

• Keep prompts in version control, next to the code that calls them.
• Keep the golden set small enough that someone will look after it.
• Pin judge models to a fixed version, so a silent model update doesn't shift your pass rate.

Cost matters more than most teams expect. Running the full golden set on every commit sounds thorough. Then the CI bill lands, or the wait drags on, and engineers start skipping tests. Split the cost across the tiers: mocks for speed, the golden set for signal, the nightly run for depth. That keeps the suite something your team runs, not something it works around.

Vibe-coded apps are where this bites hardest. Most ship with no tests at all, which is one reason vibe-coded MVPs break in production.

Where teams get LLM testing wrong

Most teams reach for LLM-as-judge far too early. It feels more thorough than a string check. It isn't. It's slower, it costs money every run, and it adds its own randomness to a suite that's meant to catch randomness.

Mistake two is no mocking plan. Every test run then needs live API keys and a budget, and CI slows to a crawl.

Mistake three is guessing semantic thresholds instead of setting them from your golden-case scores. You get flaky tests that fail for no clear reason.

Security needs its own note. Injection and PII tests guard against regressions. They're not a security guarantee. If you handle sensitive data, pair them with red teaming, human review and a proper security review process. Runtime rails help too. Our LLM guardrails architecture shows where they sit.

Skipping this step is a big part of why so many agent builds stall. Digiocial's breakdown of why most AI agent projects failed is worth a read before you scale anything.

Steal this checklist:

1. Schema and string checks first.
2. A small golden set of live tests on every PR.
3. The full eval suite every night.
4. Judge models pinned and versioned.
5. Baseline scores logged, so drift shows up early.

For the prompt side of the same problem, our prompt engineering playbook shows how to fix the failure, not the phrasing.

Need help building this in?

If you're a founder or CTO, ask yourself one question. Would your current tests catch a bad prompt change before a customer does? If the answer is no, that's the gap our AI app development team fills. We'll scope a testing pipeline for your product: mocks, golden PR checks and nightly evals. Get in touch through our custom software development page and let's lock in a time.

Frequently asked questions

What is LLM unit testing?

It's automated testing of the code around a language model: prompt templates, output parsing and error handling. You mock the model for fast tests, then check real outputs with a small golden set of live calls.

How do I unit test an LLM feature?

Mock the model provider and check the schema, required strings and banned strings in every unit test. Then add a small golden set of live-model tests on pull requests for semantic and judge-based checks.

How are LLMs tested at scale?

Most teams use three tiers. Fast mocked checks on every commit, a small live golden set on pull requests, and a bigger eval suite that runs every night.

What is LLM security testing?

It checks for prompt injection, PII leaks and other safety failures. Those tests guard against regressions, but they're not a security guarantee. Pair them with red teaming and human review.