AI wrote the code, passed every test, and skipped a third of the feature

A few weeks ago I watched an AI coding agent build a payments feature in about two minutes. The build was green. Every test passed. The code was clean, idiomatic, and faithful to the existing architecture. Had it arrived as a pull request, I would have approved it without much hesitation.

It was also missing a third of the feature, and nothing told me so.

That bothered me enough to turn it into a proper experiment, because it touches a question a lot of us are quietly asking. AI agents have become genuinely good at writing code, so what is all the discipline still for? Discovery sessions, acceptance tests written before the code, a review pass at the end: if the agent already produces clean, tested code from a ticket, surely that is process we can finally retire.

I decided to measure it rather than guess.

The experiment

I took one realistic feature, cashback settlement in a small payments service, and built it two ways into the same codebase, with the same AI model, and from the same short brief.

The first way used a structured, spec-driven workflow: discover the real rules first, write an acceptance test for each one, drive it through test-driven development, then review. The second was a strong single-pass baseline that received the same brief and the same coding conventions, wrote its own tests, and followed the architecture, but went without any of the process, so it had no discovery, no test-first rhythm, and no review.

To keep the comparison fair, I wrote the answer key before either version ran: a fixed list of the eight rules the finished feature had to satisfy. And because a single run tells you nothing, I repeated the whole exercise twenty-four times.

What happened

The structured workflow built all eight rules, every single time. The single-pass version reliably built six, and it dropped the same two on almost every run. Those two were not incidental. They were the hardest and most money-sensitive rules in the feature: clawing back cashback that had already been paid out when a refund arrives late, and handling partial refunds correctly.

What stayed with me is that the single-pass version never looked broken. It compiled, it passed its own tests, and it scored in the mid-nineties on mutation testing, which is about as strong a signal of test quality as we have. The code was not wrong, it was incomplete, and incompleteness is invisible to any check that only asks whether the code you wrote does what you meant it to. Your tests cannot fail on a rule that nobody thought to write.

In short:

• Rules delivered: eight of eight for the structured flow, six of eight for the single-pass version, on every run.
• Code quality, scored by reviewers who did not know which version was which: level, and if anything slightly in the single-pass version's favour.
• Extra cost of the structured flow: about seventeen dollars a feature, or roughly ten minutes of an engineer's time.

It was the method, not the model

The obvious hope is that a better model closes the gap. It does not. I ran the single-pass version again on the most capable model available, and it still built six rules out of eight and dropped exactly the same two. A larger model gives you the same incomplete answer, only faster.

Nor was the structured code simply the better written of the two. On the rules that both versions built, the quality was level. The difference was never craft, because a capable agent writes clean code either way. The difference was which requirements got built at all.

Why the gap exists

Once you see where the two rules go missing, the reason is almost obvious.

The structured workflow opens by interrogating the brief rather than executing it. Using a technique called Example Mapping, it proposes the rules the feature implies and, most importantly, raises the questions the brief never answers. What happens to a refund after the cashback has posted? How much comes back on a partial refund? Where exactly does the cut-off fall? Those questions are precisely the two rules the single-pass version silently skips, and they get built because something thought to ask about them in the first place.

Later, once each rule is working, the workflow runs a deliberate challenge step that asks what else the rule should do, what might break it, and where its edges lie, then hunts those cases down and tests them. The single-pass version is not unlucky so much as under-briefed. It builds exactly what the ticket says, to a high standard, and then it stops, with a green build that looks for all the world like a finished feature.

The cost is the wrong thing to worry about

The structured flow does cost more, at roughly seventeen dollars a feature. Set against an engineer's time that is negligible, but the more useful way to read it is as an insurance premium.

You pay it up front, on every feature, and it is small and predictable. What it buys you out of is the cost of finding a missing rule later, which is neither. Our industry has known for decades that a defect grows roughly ten times more expensive to fix at each stage it survives: cheap while you are still writing the requirements, painful in testing, and very costly once it reaches production. The omissions here are exactly the ones that reach production, because they are silent, they pass every test, and they involve money. Putting one of them right after an incident, months later, on code that was never designed for it, costs far more than the ten minutes it would have taken to surface the rule at the start.

The real choice is not between a cheap option and an expensive one. It is between a complete feature and one that only looks complete.

Does it hold up beyond one feature?

One feature is one feature, so I ran the same experiment on a completely different domain: a short-stay booking service, with its own cancellation-refund rules and its own answer key. The result was the same shape. The structured flow reached full coverage, and the single-pass version again dropped a rule in silence, this time folding a cleaning-fee refund into one blunt calculation.

That is a signal rather than a proof, and there is a fair boundary around it. For genuinely trivial changes, where nothing is hidden, a single pass is perfectly adequate and the process is just overhead. The value shows up as features grow complex enough that a single pass cannot hold the whole specification in view at once, and money, state, and interacting rules are precisely those conditions.

What I take from this

AI coding agents are extraordinary at the craft of writing code, and I would not want to work without one. But craft was never what made a green build trustworthy. Completeness was, and completeness is the one thing a single pass through a thin brief quietly fails to guarantee.

The answer is not to slow down. It is to keep the two steps that actually surface the requirements: a real discovery conversation before the code, and a deliberate hunt for edge cases after each rule. Both are inexpensive, and both are where the missing third comes from. Let the agent write the code, and keep for yourself the job of deciding, and then proving, what the code is actually meant to do.

I would be curious to hear whether others have run into the same thing, an AI agent shipping a convincing feature that turned out to be quietly incomplete, and where the gaps showed up. If you would like to see the discovery-first, test-first workflow behind this experiment applied from end to end on a real Java codebase, it is the approach I teach in my Udemy course, Spec-Driven Development in Java with TDD and Claude Code.