AI test generation: the loop that survives CI

TL;DR: AI test generation works, just not the way it gets pitched. It works as a loop: a planner, a generator, a healer, output checked against a schema, and a human approving every merge. Pick one flow, add a seed test, run it in a sandboxed pull request.

Most teams try an AI test tool once, get a pile of brittle tests, and quietly turn it off.

The tool was probably fine. The setup was wrong. Point a model at your app, ask for tests, and you get code that runs green while asserting almost nothing.

What works is a loop with checkpoints. Playwright's test agents and Checksum's pull request flow both prove the shape.

What AI test generation actually is

It is using a model to write test specs, test code, or both. Inputs are things you already have: requirements, user stories, existing code, a seed test, or just the app URL.

What goes in and what comes out:

• In: requirements or user stories, existing components, a seed test showing your style, app URLs to explore
• Out: structured test specs (usually JSON), test files for Playwright, Jest or Pytest, sometimes a ready pull request
• Use AI when: you are covering a new flow fast, backfilling regression tests, or scaling across many similar pages
• Write it yourself when: the logic is new, security-critical, or needs domain knowledge no model has

It is not either-or. The teams that get value let AI draft, then a person shapes and approves.

The planner, generator, healer loop

Good test generation never comes from one prompt. It comes from three agents with narrow jobs.

1. Planner. Explores the app or reads the requirements, then writes a test plan. What should be tested, and why. No code yet.
2. Generator. Turns that plan into JSON checked against a schema. Scenarios, steps, expected assertions. Then a template engine renders that JSON into real test code.
3. Healer. Runs the tests, sorts the failures, and tries to fix them. Patch a broken locator. Adjust a wait. Flag the ones that look like a real bug.

Playwright documents this three-agent chain directly. NVIDIA's Hephaestus applies the same staged logic to trace a requirement all the way to a test case. If you are picking the plumbing underneath, the rundown of AI agent frameworks covers what each one gives you.

Now here's the important bit. The JSON step is not bureaucracy. It is how you stop the model making things up.

When a model writes test code directly, it invents selectors. It misremembers API signatures. It writes tests that parse fine and assert the wrong thing. The structured output pattern forces the model to emit JSON that matches a schema, then a plain renderer turns that into code. The model never writes code. That kills a whole class of bugs.

Tip: log every healer patch as its own small diff in the pull request. Do not let it overwrite the test quietly. Reviewers trust a generated suite far more when they can see what changed and why.

Running your first one

Scope it so a failure costs you nothing.

• Pick one flow that is stable and well understood. Not your worst checkout path.
• Add one seed test so the generator has a style to copy.
• Run the planner on that flow alone. Read its output before generating a thing.
• Generate into a branch. Never straight into your main suite.
• Run it in a sandbox. Look at every failure yourself before you trust the healer.

A generator prompt needs four parts: the job ("write end-to-end tests for this checkout flow"), the inputs it can use (URL, seed test, components), the output schema (field names, types, required assertions), and a verify step ("run the test and report pass or fail before returning").

Different tools sit at different points. Playwright's agents plan and generate end-to-end. Checksum plans, builds, verifies, then opens a pull request. IDE tools like JetBrains AI Assistant sit closer to unit tests, working from code context in the editor.

The payoff depends on how much hand-writing you replace. An AWS build on Amazon Bedrock with a human checking the output reported test creation time down by up to 80%, using this same sandbox-first, human-approved shape.

The guardrails that matter

The failure mode is never a bad model. It is skipping the guards that make the output trustworthy.

Playwright's codegen shows the trade clearly. Recorded clicks give you working locators fast, but those tests usually need a healing pass before CI can rely on them.

Coverage percentage tells you almost nothing here. It tells you what code ran. Detection rate, where you break something on purpose and check the suite notices, tells you what the suite would have caught. That is the number engineers care about.

Wiring it into CI

Lock the shape in before you scale past one team.

1. A pull request triggers generation for the changed flow. Scoped to tagged PRs, not every commit.
2. Generated tests run in a sandbox first. Never against production or shared staging data.
3. If they pass, the pipeline opens or updates a PR with the tests next to the code change.
4. A person reads the diff and approves. Only then does it merge into the permanent suite.

That human-in-the-loop gate is not overhead. It is what keeps the whole thing credible.

Track three numbers: time saved writing tests, pass rate on first generation, and false-positive rate. Add detection rate as a periodic check, not a per-PR gate, because it is expensive to run every time.

If the AI feature itself is what you are testing, that is a different job. The three tiers for LLM unit testing post covers that side, and CI/CD for AI covers the pipeline the gates live in. Agent testing is the wider version of the same problem.

Why it stalls in real teams

The gap between a demo and a rollout is almost never technical.

Trust goes first. A reviewer burned by one flaky generated suite stops trusting the pipeline, even after you fix the cause. That is why healer patches have to be visible diffs.

Then integration debt. Most codebases were not built with this in mind. Seed tests, fixtures and component structure often need a cleanup before a generator makes anything useful. Skip that and you get junk tests, then blame the tool.

Then scope creep. It is tempting to point a generator at the whole app on day one. Every case study that worked, from NVIDIA's pilot to Checksum's PR flow, started narrow and grew only once the pipeline earned it.

Then the metrics gap. With no false-positive or detection baseline, you cannot tell whether this is working or just making more tests that look busy. Set that baseline before rollout, not six months into complaints about flaky CI.

The privacy bit people skip

Feeding requirements, source code or user stories into a hosted model means that data leaves your building. For a lot of companies, that is the first question security asks.

Architecture matters here. Routing generation through a controlled cloud boundary with human checkpoints, like the AWS Bedrock build does, gives you an audit trail and a clear point where sensitive input gets reviewed. That is a very different risk to pasting code into a public chat window.

Test data carries risk too. Generated tests that reference real customer records, live API keys or actual user data need scrubbing before they land in a shared repo. This one catches people out. Teams sanitise their production database carefully, then forget that a planner agent exploring staging can capture the same fields inside a fixture. Our PII redaction post covers the cleanup side.

Regulated data (health, financial) usually needs private model hosting or a provider with clear retention terms. None of this rules AI test generation out. It just means the same scrutiny you would give any other system touching production data, before you adopt it, not after.

What to get right first

The pitch says describe a feature, get a pile of tests, ship faster. That framing is why so many teams bounce off it.

The pipelines that hold up in CI all treat generation as a loop with checkpoints. Schema-checked JSON. Plain code rendering. A healer that patches in the open. A human who reviews before merge. Drop any one of those and you are back to brittle tests dressed up as automation.

So do not start with the fanciest model. Start with the review gate and a detection-rate baseline, on one flow. A suite that looks thorough but cannot catch a bug you planted is worse than no suite.

Getting help with it

Bolting a single tool onto an existing pipeline works for a demo. It rarely survives a real codebase, real reviewers and real production risk.

We build the whole thing: the agent architecture, the data pipeline feeding requirements and code context into the generator, and the rules deciding what merges on its own and what waits for a person. We have shipped over 200 apps, so the review workflow gets designed with the people who will actually use it.

A vendor tool is the right call when your stack already matches what it supports and you are covering a handful of flows. A custom build makes sense once you need your own triage logic, private model hosting, or agents wired into internal systems no product touches. Teams building broader AI automation hit the same fork, and the answer is the same: match the tool to the flows, not the other way round.

Start with one or two critical flows and a fixed-price pilot before anything bigger. If you want a hand scoping that, start with AI app development. If you are also turning a course or method into software, AI programs is the other door. Worth a chat either way.

Frequently asked questions

What is AI test generation?

Using a model and agent framework to write test specs and test code from inputs like requirements, user stories or existing code. In practice it runs as a planner, generator and healer loop, with a person approving the result before it merges.

What is the planner, generator, healer loop?

Three agents with narrow jobs. The planner explores the app and drafts a spec. The generator turns that spec into schema-checked JSON that renders into code. The healer runs the tests and repairs the ones that broke for silly reasons. Playwright documents this directly.

Which AI tool is best for generating tests?

There is no single best one. Playwright's agents suit end-to-end web testing. Checksum suits verified pull-request generation. IDE tools like JetBrains AI Assistant suit unit tests in the editor. Pick by which layer you are testing.

How do you know if generated tests are any good?

Track pass rate, false-positive rate and detection rate together. Detection rate is the honest one: break something on purpose and check the suite notices. Coverage percentage only tells you what code ran.

Is it safe to send our code to a model for this?

It depends on the route. Going through a controlled cloud boundary with human checkpoints gives you an audit trail and a review point. Pasting code into a public chat window does not. Regulated data usually needs private hosting or clear retention terms.