Reconciliation & Output Comparison

Part 3 treated each divergence as a unit to triage: real defect, intended change, or benign noise. That triage is only as good as the comparison that produces the divergences in the first place. Compare too loosely and real defects never appear as divergences at all — they’re absorbed into “noise” before anyone sees them. Compare too tightly and the report floods with harmless differences until the team stops reading it. This part is about getting that comparison right, on both halves of the problem: the output a system returns, and the state it leaves behind.

Two things to compare, not one

A migration has to reconcile two distinct surfaces, and conflating them is a common and costly mistake.

Output comparison is about the response: the API payload, the rendered report, the computed value the system hands back. It answers “did the slice return the same thing?”

Reconciliation is about the state: the rows written to the database, the balance updated, the event placed on the bus, the file dropped on the SFTP server. It answers “did the slice leave the world in the same condition?”

A slice can pass one and fail the other. It can return a byte-identical response while writing a subtly different row — a status set to COMPLETE instead of CLOSED, a foreign key pointed at the wrong record, an audit entry omitted. Downstream systems read that state, so a state divergence is every bit as real as an output divergence, and far easier to miss because nothing in the response reveals it. A migration that only compares outputs is checking half the system.

Structural comparison beats text diffing

The instinct is to compare outputs as strings — diff the two JSON blobs, flag any difference. It fails immediately on real data, because semantically identical outputs are rarely textually identical:

• Object key order differs ({"a":1,"b":2} vs {"b":2,"a":1}) — same object, different text.
• Numbers serialize differently (1.10 vs 1.1, 1e3 vs 1000).
• Whitespace, encoding, and escaping vary between two stacks.

The fix is to compare structures, not strings. Parse both outputs into their native shape — object, array, record — and walk them field by field. Now a difference in key order is a non-event, because you’re comparing the value at key a, not the position of a in the serialized text. Structural comparison also lets you address each field on its own terms, which is what makes the rules below possible.

The rules that separate signal from noise

Once you’re comparing structurally, you apply per-field rules. Three families cover most of the work.

Canonicalization — neutralize known-irrelevant fields. Some fields differ every single time and never mean anything: timestamps, generated UUIDs, request IDs, sequence numbers, durations. You normalize these to a constant before comparing, or exclude them by an explicit path. The rule must be specific — “mask the createdAt field on this record,” not “ignore anything that looks like a date.” A broad mask is how a date field that’s genuinely wrong slips through alongside the harmless ones.

Ordering rules — sort the unordered. A list with no defined order (a set of tags, a bag of line items) may legitimately come back in a different sequence. Sort both sides by a stable key before comparing. But be sure the order is actually undefined — if the legacy system returns results sorted by relevance and your slice returns them sorted by ID, that reordering is a real behavior change a user will notice, not noise to sort away.

Tolerances — for numbers that can’t match exactly. This is where ICP data teams earn their keep. Money is never compared as a raw float; it’s compared at its defined scale with a declared rounding mode, because 0.1 + 0.2 and the order of summation can produce legitimately different bit patterns that resolve to the same cent — or to a different cent, which you must catch. Other numerics — scores, rates, aggregates — get an absolute or relative tolerance chosen for the domain.

Field type	Comparison rule	The failure if you get it wrong
Timestamp / generated ID	Canonicalize to constant	Mask too broadly, hide a real wrong value
Unordered collection	Sort by stable key, then compare	Sort away a real ordering change users see
Monetary amount	Fixed precision + declared rounding	Loose tolerance passes a rounding bug
Float aggregate	Absolute or relative tolerance	Tolerance wider than the error you’re hunting

The governing principle: every tolerance and every mask is a documented, reviewed decision. Part 3 made the point that an over-eager exclusion can drive the divergence rate to zero by hiding bugs. This is where that risk is concrete. A tolerance set wider than the arithmetic error you’re trying to detect doesn’t reduce noise — it blinds the test to exactly the class of defect that matters most to a finance or data system.

Reconciling state when both systems write

Output comparison is comparatively easy — you have both responses in hand at the moment of the request. Reconciling state is harder, because state is durable, asynchronous, and during a migration both systems may be writing it.

The straightforward case is the shadow setup from Part 2: legacy writes to production, the shadow slice writes to an isolated datastore, and you reconcile the two stores after the fact. Walk the records the slice touched, compare each against its legacy counterpart under the same structural rules, and flag the divergences. Because the shadow’s writes are isolated, a difference is just data to investigate — it never corrupted anything.

The hard case is dual-write, where both systems write to shared or mirrored state on the live path. Now reconciliation has to contend with timing:

• Eventual consistency. The two writes don’t land simultaneously. A snapshot taken between them shows a “divergence” that’s really just propagation lag. Reconciliation has to compare at a consistent point — a quiet window, a watermark, or a settle delay — not race the writes.
• Idempotency and replays. At-least-once delivery means a record may be written twice. Reconciliation must recognize a duplicate as a duplicate, not count it as extra state.
• Drift accumulation. Tiny per-record differences that look benign can compound across millions of rows into a materially wrong aggregate. Reconcile at both grains: record-level for cause, aggregate-level (sums, counts, control totals) for confidence that nothing systematic is hiding in the long tail.

Aggregate reconciliation deserves its own emphasis for data teams. Comparing row by row tells you where a difference is; comparing control totals tells you whether anything is wrong at all without reading a billion rows. A matching count and a matching sum across a partition is a cheap, powerful sanity check — and a mismatch there is a loud signal that something systematic, not incidental, has gone wrong.

What comparison can’t see

Output comparison and reconciliation prove the systems agree. They do not prove either system is right — Part 3’s point, and it applies with full force here. If both systems are wrong in the same way, every comparison passes. Reconciliation faithfully confirms that two systems made the identical mistake.

And comparison only covers what you actually compared. State you didn’t think to reconcile — a cache, a search index, a downstream system’s derived view — can diverge invisibly. The completeness of the reconciliation is a claim in itself: these stores were reconciled, under these rules, at this grain. A store left off the list was never checked, no matter how green the report looks. Name what you reconciled as deliberately as you name what you found.

Where this leads

Comparison and reconciliation tell you a slice is ready to carry traffic. They don’t carry it. The act of moving real users from the legacy path to the proven slice — gradually, reversibly, with the old path still warm — is controlled by feature flags. Part 5, Feature Flags in Production Migration, is about that control surface: how a flag turns a validated slice into a promotion you can ramp, hold, and undo without a deploy.