LLM guardrails: a practical architecture for engineers

TL;DR: Guardrails are five checkpoints, not one filter: input, dialog, retrieval, execution and output. Run a cascade so a cheap regex gate handles most traffic and a judge model only sees the hard cases. Write the policy before the detectors, and test it like an attacker on every rule change.

Guardrails are not one filter. They are five checkpoints, and each one catches a different kind of failure.

Put the cheap check first and the expensive one last. A regex gate throws out obvious junk in milliseconds. A small classifier catches the reworded attempts a rule misses. A judge model handles the genuinely unclear cases. A person signs off on anything that cannot be undone.

That order is the whole trick. Most traffic never reaches the expensive layer, so the stack stays cheap enough to run on every single request.

One warning up front. Every layer adds latency and cost. Budget for it. Then test the set again every month, because new jailbreak patterns turn up constantly and an untested rule set goes stale fast.

Key takeaways

What a guardrail actually is

A guardrail is a rule that runs at request time. It sits between the user, the model, and anything the model can touch. It checks what goes in, shapes what happens mid-conversation, and screens what comes out before anyone sees it.

This is not the same as training a safer model. Training bakes behaviour into the weights. Guardrails sit outside. You write a rule this morning and ship it this afternoon.

That difference is the point:

• A new jailbreak lands in production. Retraining takes weeks. A classifier update takes an afternoon.
• A client in healthcare needs a stricter content rule. You add a policy layer for that one deployment. You do not touch the base model.
• An auditor asks what your rules were in March. Rule sets are versioned and readable. Model weights are not.

Guardrails are also where you enforce things the model was never taught. Your client's compliance rules. Your product's tool permissions. The model has no idea those exist.

What you are defending against

Every rule should point at a real failure. Here is the working list.

1. Prompt injection and jailbreaks. Someone hides instructions in a message, a document, or a retrieved page, and those instructions beat your system prompt. OWASP ranks this as the top LLM risk, and fairly. It is cheap to try and hard to close off completely.
2. Leaked context. The model gives up its own instructions, your internal tool names, or a chunk of a document with personal data in it.
3. Content that breaches policy. Written by the model, or pulled in from a source it treated as trustworthy.
4. Confident wrong answers. Worst in retrieval systems, where the answer sounds right and cites a source that does not actually say that.
5. An agent doing too much. Tool access plus a chained instruction nobody caught upstream, and now something got deleted.

Each of those needs a different detector. That is the case for layering.

The five checkpoints

NVIDIA's NeMo Guardrails calls these rails. The naming is useful even if you build your own.

Input rails sit between the user and the model. Regex and token filters catch known bad patterns for almost nothing. A small classifier catches the reworded versions. PII masks strip sensitive data before it ever hits the prompt.

Dialog rails control the shape of the conversation, not just one message. This is where you set the system prompt, lock response formats, and define which paths a conversation is allowed to take. NeMo's Colang language exists for exactly this.

Retrieval rails matter most in RAG. Score chunks for relevance before they enter the context. Check where each one came from, so an untrusted PDF does not carry the same weight as a verified internal doc. If you are still building that layer, this walkthrough of a RAG application covers the retrieval side, and our own guide to building a RAG pipeline covers the plumbing.

Execution rails protect what the model can do, not just what it says. Cap tool calls per session. Gate which tools a user role can reach, the same way you would with role-based access control. Pause for human approval before anything irreversible.

Output rails are the last check. Validate the response against a schema. Screen it for policy breaches. Then rewrite or fall back to a safe answer instead of just blocking.

Pro tip: build retrieval rails before you scale a RAG system, not after. Adding provenance checks once your vector store is full of unlabelled documents is a far bigger job.

Rules, a classifier, or a judge model

Four patterns show up in production. None of them wins on its own.

1. Deterministic rules. Regex, keyword lists, schema validators. Fast, cheap, predictable. Brittle against rewording. Use them for anything you can list out: card number formats, known injection strings, banned terms.
2. Small classifiers. Models like Llama Guard or Prompt Guard 2, trained to spot jailbreaks and policy breaches. They catch meaning a rule cannot, at a fraction of the cost of a full model call. Prompt Guard 2 ships at 22M and 86M parameters, small enough to run on every request.
3. A judge model. A full model reads the ambiguous case and rules on it. Most flexible. Slowest. Most expensive.
4. A person. For high severity or anything you cannot reverse. Agree the escalation path before you need it: who reviews, how fast, and what happens if nobody answers.

The cascade ties them together. Fast gate first, small classifier second, judge model only for what is left, and a person for the small slice that really matters.

Here is the trap with judge models. The judge runs on the same architecture as the model it is checking, so it can be prompt-injected by the very content it is reading. Two fixes. Use a different model family for the judge. And never let the judge see raw untrusted text without the same input-rail treatment you give the main model.

The toolkits worth knowing

You do not need to build all of this yourself.

• NeMo Guardrails is the fullest option for dialog and RAG flows. Colang lets you write conversation paths out explicitly instead of hoping a system prompt holds. It ships a vulnerability scanner too.
• Guardrails AI takes the other angle. Its RAIL spec enforces structured input and output, which is what you want when your real risk is a malformed response rather than a hostile one.
• LangChain middleware gives you hooks at the agent level. PII redaction and before-and-after agent hooks are the quickest way to bolt a human approval step onto an agent you already shipped.
• Moderation APIs like Azure OpenAI's content filter handle the broad categories well out of the box. Use them where your policy overlaps the standard list. Build your own where it does not, because a generic API will not know your client's compliance rules.

Most stacks end up running two or three of these together. A fast Prompt Guard gate, a Llama Guard classifier behind it, and a moderation API as a backstop is a sensible default.

Write the policy before you write the detectors

Most failed guardrail projects trace back to a policy nobody actually wrote down. The detectors were fine. The rules they enforced were vague.

Start by asking the people who carry the risk. Legal for exposure. Security for attack surface. Product for what breaks if you are too strict.

Then map every category to a severity and an action. Not "flag if unsafe". Something like this:

• Category. PII exposure, medical claim, hate speech, tool misuse.
• Severity. Low, medium, high, critical, each with a real example attached.
• Action. Allow and log, soft warning, block and rewrite, or block and send to a person.

Keep it machine-readable, not a PDF. Your classifier prompts and your legal team should use the same category names, so a flagged incident maps straight back to a clause. Version every change. Log which rule fired on which request. Record who approved it.

Get the same group back in the room every quarter, not just at launch. Rules and regulations both move faster than an annual review. If you need a starting structure, this breakdown of AI policy development is a decent frame, and the EU AI Act compliance rundown covers what is now actually required.

How to tell if they are working

Track this the way you track uptime. Four numbers.

1. False positives and false negatives. How often you block a legitimate request, versus how often you miss a real breach. Track them apart. Improving one usually hurts the other.
2. Coverage. How much of your known attack list the test suite actually exercises. An untested category gives you no assurance at all.
3. Latency per layer. Measured per checkpoint, not end to end, so you can see where the time goes.
4. Task damage. Whether you are rewriting so hard that real users cannot get a useful answer.

Build the adversarial suite on purpose. Run it against every rule change and read the results before you ship, not after an incident.

Then close the loop. Flag incidents automatically. Send them to a person to label. Feed those labels back into the classifier. Update rules on a schedule.

Pro tip: watch false negatives as hard as false positives. Blocked users complain, so false positives surface on their own. False negatives stay invisible until something goes badly wrong.

Keeping latency and cost sane

A stack that adds two seconds to every reply will not survive its first sprint review. Four habits keep it in check.

• Run what you can in parallel. Non-blocking checks alongside generation. Stream the response while a background check finishes. Save synchronous blocking for checks that truly must finish first.
• Right-size each layer. Do not send everything to a judge model. Cache verdicts for repeated inputs. Sample the ambiguous cases rather than judging all of them.
• Log from day one. Structured logs per checkpoint, a store of flagged conversations, and a dashboard that shows a trend shifting before it turns into an incident.
• Give escalation an SLA. A defined review time and logs clean enough to hand to an auditor.

Get those right and most users never notice the guardrails exist.

What real failures teach you

Public guardrail failures rhyme. A company ships a support bot with no dialog rails, and a user talks it into a commitment the business never agreed to. The best known case involved a car dealership bot being persuaded to agree to a one dollar car, purely by being asked enough times, because nothing checked the output against an actual business rule before showing it.

The lesson is not that chatbots are risky. It is that two rails were missing. A dialog rail defining what the bot could commit to would have stopped the drift. An output rail checking any pricing language against real business rules would have caught it anyway.

The second pattern turns up in RAG. The system cites a source for a claim the source does not support, because nothing verified that the quoted passage said what the model claimed. That is a retrieval-rail gap. Better prompting will not fix it.

The thread through both: the model was not the problem. A layer was missing. Teams that respond by rewriting the prompt are treating a systems problem as a model problem. Our write-up on chatbot security walks the same ground from the threat side, and if you are running several agents together, multi-agent systems add a whole extra surface to gate.

How much should users see

A blocked response with no explanation is one of the fastest ways to lose trust in an AI product. Users who hit "I can't help with that" assume the thing is broken, not that it is working.

Being open does not mean publishing your rules. That just hands attackers a map. It means giving people enough to understand why the answer was limited and what to try instead.

A softened answer with a short note beats a flat refusal nearly every time. It keeps the conversation alive instead of ending it.

Fallback design deserves as much care as detection. A good fallback offers a partial answer, points at a human, or explains the limit in plain words. A bad one just stops.

Watch this one too: guardrails tuned too tight make a product feel unreliable in a way nobody traces back to the guardrails. Users just say the AI does not understand them. Track reported friction next to your false positive rate, because the two do not always move together.

What we would fix first

Most of the noise in this space is about picking the right tool, as though NeMo versus Llama Guard were the decision that matters. It is not.

Policy quality decides everything. A well-built category list enforced by basic regex will beat a clever judge model enforcing vague rules every time.

The most oversold idea is that a bigger judge model solves ambiguity. It does not. It moves the problem, and adds latency while it does, because that judge shares the same injection surface as the model it is checking.

What actually deserves the attention: retrieval provenance checks, and dialog rails that limit what an agent can commit to. Both get less airtime than input filtering. Both show up over and over in the failures worth learning from.

Fix the boring structural layers first.

Ready to build guardrails that hold up?

Building a layered stack, tuning the cascade, and keeping the policy current is a real engineering job, not a weekend integration.

Devwiz has shipped 200+ apps and platforms since 2015, for clients including the NSW Government, Briometrix, Vivid and Huskee. We build the full path, from input gates and retrieval checks through to human approval flows.

This suits founders and CTOs putting an LLM feature into a workflow where a generic moderation API will not satisfy an audit. If compliance is part of the picture, our guide to AI software compliance is worth a read first.

Our AI app development team covers this scope end to end. If you are rolling guardrails across several products rather than one app, AI platform builds is the better starting point.

Tell us what your product does and where the risk sits, and we will map what the guardrail layer needs to look like.

Frequently asked questions

What are LLM guardrails?

LLM guardrails are rules that run at request time between the user, the model, and anything the model can touch. They screen inputs, shape the conversation, and check outputs before anyone sees them or before an action fires.

Can you give some examples of LLM guardrails?

Input classifiers like Prompt Guard 2 that catch jailbreak attempts, PII masking before the prompt is built, provenance checks on retrieved documents in a RAG system, and output moderation through Llama Guard or a moderation API.

Does ChatGPT have guardrails?

Yes. OpenAI applies moderation and safety filtering to both prompts and responses. Azure's OpenAI Service documents the same content filter system, and developers can configure it for their own apps.

What AI safety measures go beyond content filters?

Permission gating on agent tool calls, human approval for high-risk actions, adversarial testing against jailbreak patterns, and audit logs of every rule that fires in production.

Do LLM guardrails slow down responses?

Each layer adds some latency. Teams manage it by running non-blocking checks in parallel, putting a fast deterministic gate first, and saving expensive judge-model calls for genuinely ambiguous cases.