AI-Generated Tests & Parity Validation

Part 4 ended on a single load-bearing step: proving translated code behaves identically to the legacy before it ships. That proof is where AI’s promise and AI’s danger meet most sharply, because the tooling that produces the proof can now be generated too. This part is about doing that without fooling yourself, using AI to build the safety net while refusing to let AI also be the thing that decides the net is sound.

The reframe: a test is a claim about what should be true. Parity is a measurement of what is true. AI is good at the first and cannot substitute for the second, and the entire credibility of an AI-accelerated migration rests on keeping them straight.

Why generated tests are genuinely useful

Writing tests for a legacy slice by hand is slow, and the surface is enormous, every branch, every edge case, every malformed input the old system learned to swallow. AI changes that. It can read a slice and draft unit and integration tests that pin its behavior across a wide surface in a fraction of the time, including cases a human under deadline pressure would never get to.

This is a real gain. Coverage that used to be aspirational becomes achievable, and the tests serve as executable documentation of what each slice is meant to do. For the mechanical work of writing assertions across a large codebase, generated tests are one of AI’s clearer wins in the whole migration. If the story ended here, it would be an easy recommendation.

Why generated tests are not the proof

It does not end here, because generated tests carry a subtle and dangerous flaw: a test is only as correct as the expectation baked into it, and AI generates the expectation by inferring what the code probably does.

That sets up a specific failure mode worth naming precisely. If AI generates the modernized code and generates the tests for it from the same understanding of the system, the two will tend to agree, because they came from the same assumption. A test that says “this function should return X” and code that returns X will pass together happily, even when the legacy system actually returned Y. You get a green suite that certifies the wrong behavior. The tests are not lying. They are faithfully checking the code against a belief that was never validated against reality. A passing test proves the code matches the test. It does not prove the test matches the world.

This is the same confidently-plausible failure as generated code, moved up one level. And it is more insidious there, because a passing test suite is exactly the artifact teams use to convince themselves they are safe. Generated tests can manufacture false confidence at scale.

A concrete version makes the trap obvious. Suppose a legacy billing routine rounds a particular tax calculation down to the nearest cent, an undocumented behavior some downstream reconciliation has silently depended on for a decade. AI reads the routine, infers the more conventional round-half-up rule, generates code that rounds half-up, and generates a test asserting half-up. The test passes. The code is wrong. Nothing in that loop ever looked at what the legacy system actually did to a real transaction. It only checked the code against an assumption, and the assumption was the bug. A second reviewer reading the test would nod along, because the test looks reasonable. Reasonable and wrong is precisely the gap a generated suite cannot close on its own.

What closes the gap: parity against reality

The fix is to anchor the proof to something AI did not generate, the legacy system itself. That is parity validation, and it is the discipline that makes AI-accelerated delivery trustworthy.

Instead of asking “does the new slice match what we think it should do,” parity validation asks “does the new slice match what the old system actually does.” The mechanism is shadow traffic: the legacy system keeps serving production while a strangler facade mirrors the same real requests to the modernized slice. Both produce a result. Only the legacy result reaches the user. The two are compared. Where they agree, you have evidence grounded in reality. Where they diverge, you have a ticket, a concrete, reproducible difference to resolve before the slice carries live load.

The crucial property is that the expectation is no longer something a model assumed. It is the observed behavior of the running system under real inputs, including all the edge cases no specification, and no generated test, ever captured. Parity validation does not trust the tests. It trusts the legacy, and uses the legacy to judge the tests and the code together. Run the billing routine from earlier against shadow traffic and the divergence stops being invisible: the same real transaction produces one cent in the legacy and a different cent in the new slice, the reconciliation flags the mismatch, and the rounding bug becomes a ticket before a single customer is billed. The behavior the generated test never knew to check is exactly what the comparison surfaces.

Generate tests from behavior, not from the code

There is a right way to use AI for test generation, and it follows directly from the trap above. The problem is never the generation. It is the source of the expectation. A test drafted from the source code inherits the model’s inference about what the code does. A test drafted from the legacy system’s recorded behavior inherits reality.

That second style has a name: characterization tests, sometimes called golden-master tests. Instead of asking the model “what should this function return,” you capture real inputs the legacy system received and the exact outputs it produced, then generate tests that assert those observed pairs. The assertion is no longer a guess. It is a fact the system already demonstrated.

This flips AI’s role from liability to advantage. AI is genuinely good at the mechanical work of turning a large set of captured input-output pairs into a clean, readable test suite, and at spotting the boundaries and equivalence classes worth sampling. It is bad at deciding what the right output is. Characterization testing gives it only the first job. Applied to the billing routine, a characterization test built from recorded transactions would have asserted round-down from the start, because that is what the legacy actually produced, and the wrong generated code would have failed immediately rather than sailing through on a matching wrong assumption.

Characterization tests do not replace parity validation. They are the durable, repeatable form of the same idea: parity proves equivalence against live traffic at cutover, and the characterization suite it seeds becomes the regression net that keeps the slice honest afterward. One is the gate. The other is the ratchet that holds the gain.

How the two fit together

Generated tests and parity validation are not rivals. They do different jobs, and a sound migration uses both in order:

Layer	Checks against	Catches	Generated by AI?
Generated tests (from code)	A written expectation	Regressions, obvious breaks, documented behavior	Yes, drafted, then reviewed
Characterization tests (from behavior)	Recorded legacy inputs and outputs	Divergence from what the system actually produced	Yes, from captured pairs, then reviewed
Parity validation	The live legacy system	Silent divergence on real, undocumented edge cases	No, the legacy is the oracle

Tests give you fast, repeatable coverage and a regression net for the future. Parity gives you proof against reality for the cutover. The tests make the slice cheap to keep correct. Parity makes it safe to promote. A slice clears its parity gate, functional parity, data integrity, non-functional parity, traceability, before it takes any live traffic, and anything deliberately deferred goes on a deferred queue that feeds the next slice. AI accelerates the building of this net. It never gets to certify its own work.

A safe workflow for AI-generated tests

Principles are easy to agree with and easy to violate under deadline. Here is the order that keeps generation useful without letting it certify itself:

1. Capture real legacy behavior first. Record production inputs and the outputs the legacy system produced for them, across the slice you are about to modernize. This is the ground truth everything downstream is measured against.
2. Have AI draft characterization tests from those recordings, not from the source code. The assertions come from observed outputs, so a wrong inference cannot quietly become the expectation.
3. Have AI draft code-level tests too, for structure, error handling, and documented behavior. Useful as a regression net, but treated as claims to be checked, not proof.
4. Review every generated test as a generated artifact. Spot-check assertions against real data, look hardest where the model had to infer intent, and consider mutation testing to confirm the suite actually fails when the behavior changes. An unreviewed test is an assumption with a green checkmark.
5. Run parity validation against live shadow traffic. This is the gate. It catches the rare and undocumented paths no captured sample and no generated test knew to include.
6. Promote only on a clean parity gate, and keep the characterization suite as the regression net that holds the slice correct after cutover.

The through-line: reality seeds the tests, humans review the tests, parity proves the cutover, and AI does the fast mechanical middle. Reorder those and you get speed without trust, which in a migration is just a faster way to ship the wrong behavior.

Three limits on trusting any of this

First, parity validation is bounded by the traffic it sees. A path that never appears in the mirrored traffic is not proven, only unobserved, so coverage of rare-but-critical paths still needs deliberate, human-designed cases. Generated tests help here precisely where shadow traffic is thin. Second, parity is the right bar for behavior you are preserving, not for behavior you are intentionally changing. A bug you are fixing should fail parity, by design, and treating every divergence as an error would just port the old problems forward. Separating the two is human judgment, not a model’s call. Third, none of this makes generated tests safe to trust blindly. They are reviewed like any generated artifact, because an unreviewed test is just an assumption with a green checkmark.

Where this leads

A pattern has surfaced in every part so far: AI drafts fast, and a human decides what is true. That pattern is not incidental. It is the operating model that makes the whole approach defensible, and it deserves to be named and examined directly rather than left implicit. Part 6, Human-in-the-Loop Modernization, makes the review model the whole subject: where the human sits at each step, what “in the loop” means in practice, and why it is the feature, not the friction.