Feature Flags for AI: How to Ship Models and Prompts Safely

TL;DR: Feature flags for AI let you change a model, prompt or agent permission at runtime, show it to a small group first and switch it off fast if quality, cost or safety slips. Flag the things that change behaviour, use typed and versioned settings, roll out in steps with metric checks, and build a tested fallback before you need a kill switch. Flags choose between approved behaviours. They never replace access checks, data limits, evaluation or human review.

Feature flags for AI let you change a model, a prompt or an agent's permissions while the app runs. You show the change to a small group first. You switch it off in seconds if quality, cost or safety slips. They matter more for AI than for ordinary features. A prompt edit or a model swap can change what your product says and does, and not one line of app code fails a test.

This guide covers what to flag, how to design the flags, how to roll out in stages, how to build a kill switch that works, and what Australian businesses should check on privacy.

The short version

• A feature flag separates releasing a feature from deploying code. You can turn an AI capability on, off or up in steps without a new build (Microsoft).
• For AI, flag the things that change behaviour: model choice, prompt version, retrieval settings, tool permissions and guardrail mode. A simple on/off switch isn't enough.
• Start a new setup at a small share of traffic. Watch latency, errors, cost and evaluation scores. Widen only when they look good.
• Build the fallback before you need it. A kill switch with nothing safe to fall back to isn't a safety control.
• Flags pick which approved behaviour runs. They never replace access checks, data limits, evaluation or human review.

What a feature flag does in an AI product

A feature flag is a runtime check your code makes before it runs a piece of functionality. It can turn the feature on or off. It can target certain users or show the feature to a share of traffic. The key point is that releasing the feature no longer depends on deploying code. Microsoft's feature management documentation describes this split.

In a normal app, the thing behind the flag is usually a screen or a workflow. In an AI product, it's often a choice about behaviour:

• which model answers a request
• which version of a system prompt is used
• how many documents are retrieved, and from which index
• whether an agent can only read data or can also write to it
• how strict the output filter is

That's why teams building with large language models treat flags as a control panel for behaviour, and not only a release tool.

Why AI changes need flags more than ordinary features

Traditional software fails loudly. A broken function throws an error and a test catches it. AI features fail quietly. The answer still arrives and still reads well. It may be wrong, slower, dearer or off-brand.

GitLab's engineering docs make this point. AI features need testing and monitoring because model and prompt changes can shift quality, speed and user experience. Feature flags allow a controlled rollout and limit who is exposed (GitLab AI feature development playbook). In their view, AI releases need normal software tests plus content evaluation and production monitoring. Code tests alone don't prove model quality or safety.

Three things follow for a founder or CTO:

1. A prompt edit is a release. Say someone can change a prompt in a config file or dashboard and it reaches every user at once. Then you have no staging, no rollback and no record of who changed what.
2. Model upgrades aren't drop-in. A newer model can reason better and still break your output format, your tone or your cost per request.
3. Cost and speed are product risks. Inference cost and response time move with the model and token limits. They need the same gradual rollout as quality.

We cover the version-control side in our guide to AI model versioning in SaaS.

What to put behind a flag

A useful way to decide is to match each flag to a product risk. One vendor's production guide suggests exactly this. Model or prompt changes cover answer quality. Model route or token budget covers speed and cost. Tool authority covers autonomy. Retrieval and citation rules cover grounding. Approval gates cover governance (FeatBit). The same company's starter kit lists the main AI controls. They are entry-point exposure, prompt profile, model route, retrieval profile, tool mode, guardrail mode, fallback route and experiment assignment (FeatBit starter kit).

This is how we group them when we scope an AI build:

Most of these aren't true or false. They're choices between named options you've already tested.

Designing flags that hold up in production

Use typed, versioned settings

A flag that returns true or false is fine for a button. It's too thin for an AI feature. Pydantic's production guide describes AI flags as typed, versioned variables. They change models, prompts and tool policies without a redeploy. Values are checked, targeting is stable and defaults are known to be safe (Pydantic). MLflow's guide agrees. It suggests structured config flags for model choice, token caps, rate limits, prompts and safety filters, and not booleans alone (MLflow).

In practice, the flag returns an object. It holds a model ID, a prompt version, a token limit and a retrieval setting. Your code checks it against a schema. If the value is malformed, or the flag service can't be reached, the app uses a known-good default. It doesn't fail or guess.

Version prompts like model IDs

GrowthBook's rollout guide says to version prompt templates the way you version models. It also says to use fixed user bucketing, so each user stays in the same group during a test (GrowthBook). Without that, a user could see prompt A on one request and prompt B on the next. Your results get hard to read and your product feels inconsistent.

Add a global switch as well as feature flags

GitLab pairs a flag for each AI feature with a general AI-wide switch (GitLab). The feature flag lets you roll out one capability. The global switch stops all AI behaviour at once. You'd use it if a provider has an incident or you find a problem that cuts across features. We suggest the same set-up for any product with more than one AI feature.

Record what ran

Pydantic also says flags should connect to observability, so every targeting decision can be traced (Pydantic). Log the flag labels and versions in each trace: model, prompt version, retrieval profile. When a customer reports a bad answer, you can see which setup produced it. Our guide to observability for LLM apps goes further.

What stays in code, not in a flag

Flags are easy to change. That's exactly why they're the wrong home for anything security-critical. Tool code, credentials, access checks, approvals and safety rules stay in code. The flag picks between behaviours that are already approved (Pydantic).

FeatBit's view is the same. Flags control which approved AI behaviour is active for a given context. They shouldn't replace authentication, authorisation, tenant isolation, data boundaries, sandboxing, evaluation, content filtering or human review (FeatBit).

Here's a quick test. Suppose turning a flag the wrong way could expose another customer's data. Or suppose it could let an agent take an action nobody approved. That control belongs in code, behind proper access checks.

Rolling out an AI change in steps

A staged rollout gives you evidence at each step. The guidance from several sources lands on a similar order.

1. Evaluate offline first. GitLab's playbook says to assess behaviour, speed, security, safety and generated content against datasets that match real use before launch (GitLab). Your eval set should include the awkward cases your customers send.
2. Release internally. Staff and friendly beta users find problems no dataset predicted. GitLab suggests releasing internally first. Then expand step by step behind a flag while you watch technical and AI-specific metrics.
3. Canary at a small share. MLflow suggests canarying new LLM setups from 1% of traffic. Set an observation window and auto ramp-or-revert thresholds for latency, errors, cost and evaluation score (MLflow). Google Cloud's docs show the same idea. They use shares like 1%, 5% or 50% and allowlists for internal teams or beta users. You can change a string setting, such as an LLM system prompt, with no code or infrastructure rollout (Google Cloud).
4. Ramp with checkpoints. GrowthBook describes checkpoints at 10%, 25%, 50%, 75% and 100%. Review metrics before each step (GrowthBook). The exact numbers matter less than the rule: no step without a metric review.
5. Decide in advance what reverts. Write the thresholds before the rollout starts. Rolling back then follows a rule. It isn't an argument.

This fits with automated checks in your delivery pipeline. Our article on CI/CD for AI and eval gates covers how to stop a bad prompt before it reaches the flag stage. The AI product launch checklist shows where rollout sits in the wider launch plan.

Kill switches and fallback behaviour

The most valuable AI flag is the one you hope never to use. A kill switch reverts to a known-safe model and prompt. MLflow's guide pairs it with timeouts, retries with jitter, circuit breakers and fallback across providers. One failing provider then doesn't take your product down (MLflow).

One product-management guide describes the AI kill switch as a control you can run on its own. It turns AI off with no code deploy and sends users to a pre-built fallback. That might be cached results, an "unavailable" message or rule-based logic (InstitutePM). The same guide makes a point we'd underline. Fast rollback only helps when the fallback path has been designed and tested, and fits how critical the feature is.

Pick the fallback by feature type:

• Search or recommendations: fall back to standard ranking.
• Drafting assistants: show a plain "unavailable" message and keep the manual path working.
• Support chatbots: hand over to a person or a contact form.
• Agents that take actions: fall back to draft-and-approve, or turn off the action tools.

Test each fallback on a schedule, the same way you test backups. An untested fallback is a guess.

Australia's National AI Centre points the same way for organisations adopting AI. Its implementation guidance says to document risk-treatment and deployment plans and run safety evaluation that fits the risk. It also says to keep ways for humans to step in, including the ability to pause, dynamic guardrails and tool or system access controls (National AI Centre). It also suggests naming who can pause or stop AI use when needed (National AI Centre workflow guidance). A kill switch is the technical half. The other half is a named person who is allowed to use it, and who knows it.

What to monitor and how to clean up

A flag without measurement is only a switch. GitLab's playbook lists what to watch: latency, error rates, resource use, model performance, response quality and user satisfaction. Then use production results and user feedback to improve the feature (GitLab). For most products we'd add cost per request and the share of requests that hit a fallback.

A good dashboard for a flagged AI change compares the control and the new variant on:

• p95 latency and timeout rate
• error rate and fallback rate
• cost per request and tokens per response
• evaluation score on sampled production traffic
• user signals like thumbs-down, edits and escalations

These signals tell you whether to ramp. They also feed longer-term work on adoption, which we cover in our AI feature adoption and engagement guide.

Remove flags when the decision is made

Stale flags are technical debt. In AI products they also hide old prompts and models that nobody remembers. Give every flag an owner and an expiry review. Order matters too. GitLab warns that if a flag exists in both an app and an AI gateway, you should remove the gateway flag first. Remove the app flag first and you can switch off the gateway behaviour at once (GitLab). Write the cleanup order into the ticket when you create a flag that spans services.

Ongoing ownership of this work is part of what good post-launch AI platform support looks like.

Australian privacy and security points

Flags make it easy to test AI on real users. That's where privacy duties bite. In Australia, the Privacy Act and the Australian Privacy Principles apply when AI handles personal information. The OAIC's guidance, published on 21 October 2024 and updated on 17 January 2025, says those duties cover personal information entered into AI systems. They also cover outputs that contain it (OAIC). It recommends due diligence, privacy by design, a privacy impact assessment, clear notices, and care over security and data access. For teams building or fine-tuning models, the OAIC also advises privacy by design and a privacy impact assessment (OAIC generative AI guidance).

What this means for flag-driven testing:

• Treat a canary group as real use. If 1% of customers get a new prompt, their data flows through it. The same notices and controls apply as at 100%.
• Check the data path of each variant. A new model route may send data to a different provider or region. Review that before the flag goes on, not after.
• Log with care. Traces that record flag versions are useful. Prompts and outputs may hold personal information, though. Decide what you store and for how long.
• Don't target on sensitive traits casually. Targeting rules are decisions about people. Keep them simple and easy to review.

The National AI Centre notes that Australian laws may apply to AI security failures, harmful or misleading outputs, data misuse, inaccurate outputs, bias or exclusion, and supply-chain impacts (AI and Australian law). For a wider framework, NIST's AI Risk Management Framework helps organisations build trust into AI design, development, use and evaluation (NIST). If your product handles health, financial or other sensitive data, talk to a privacy lawyer before you design your test groups. Our AI software compliance overview is a good place to start. For the agent side, the AI Orchestrators guide to AI agent guardrails is a useful companion.

Our take

We build AI agents and AI-first platforms for Australian founders and CTOs. One pattern turns up often. The first version of the AI feature ships fast. The prompt sits in a config file and the model name is hard-coded. It works in the demo. Three months later a provider changes a model. A customer reports a strange answer. Nobody can say which prompt produced it or how to go back.

Our view is that flags belong in the design from the first sprint, not after the first incident. That means a flag layer that returns typed settings. It means a global AI switch, trace logging of every version that ran, and a fallback path tested before launch. It also means being clear about what a flag doesn't do. It doesn't make an unsafe design safe. Access checks, data limits and evaluation sit underneath it.

We push back on one common shortcut: using flags to skip evaluation because a rollback exists. Rollback limits how many users see a problem. It doesn't tell you whether you have one. Teams that ship AI calmly do both. They evaluate before exposure and ramp with evidence after it. For how the flag layer fits the rest of the stack, see our AI platform architecture guide.

Building this into your AI product

Adding AI to an existing SaaS or mobile app? Or building an AI platform from scratch? The flag layer, eval pipeline and fallback design are far cheaper to include at the start than to retrofit. Our team builds this as part of AI platform development, alongside the agents, data pipelines and integrations around it. If you want a production-ready build and not a demo, talk to Devwiz about your product.

Frequently asked questions

Do feature flags for AI differ from normal feature flags?

The mechanism is the same, but the content is different. A normal flag usually hides or shows a screen. An AI flag often selects a model, a prompt version, a retrieval setting or a level of agent autonomy, so it tends to return structured configuration rather than true or false. It also needs links to evaluation and observability, because AI failures are often quiet.

What percentage of traffic should a new prompt or model start at?

MLflow suggests canarying new LLM configurations from 1% of traffic, with a defined observation window and thresholds for ramping or reverting. Other guides describe ramp steps such as 10%, 25%, 50%, 75% and 100%. Choose your starting point by the risk of the feature: a low-risk drafting aid can start higher than an agent that changes customer records.

Can a feature flag act as a security control?

No. A flag selects between behaviours you have already approved. Authentication, authorisation, tenant isolation and approval rules should remain in code. If a flag controls something where a wrong setting could expose data or trigger an unapproved action, move that control into code with proper access checks.

What should a kill switch fall back to?

It should fall back to something designed and tested for that feature: a previous known-safe model and prompt, cached results, a plain unavailable message, rule-based logic or a human handover. The right choice depends on how critical the feature is. Test it regularly, because an untested fallback often fails when you need it.

Do Australian privacy laws apply to a small canary test?

Yes, if personal information is involved. The OAIC says Australian privacy obligations apply to personal information entered into AI systems and to outputs that contain it. A 1% test group is still real customers, so complete your privacy assessment and notices before you switch the flag on.