Rollback & Recovery Planning

Part 6 showed how a parallel run retires most cutover risk by producing evidence before anything goes live. This part is about the residual — the risk no amount of pre-cutover evidence fully eliminates — and the discipline that makes it survivable. A rollback plan is what turns a bad step from an outage into a non-event. It is also the clearest single illustration of why the incremental shape is safer than the big-bang: not because nothing ever goes wrong, but because of what happens when something does. As with the preceding parts, the focus here is the planning and decision discipline; the engineering of building rollback into a slice boundary is covered in Per-Slice Rollback Architecture.

Rollback is a property you design in, not a button you find later

The most common and most expensive misconception about rollback is that it is something you reach for after a problem — a button somewhere. It is not. Rollback is a property a migration either has or lacks, and it is determined by decisions made long before cutover. A program that did not design for reversibility does not get to choose it under pressure; it discovers, at the worst possible moment, that there is no clean way back.

This is the heart of the difference between the two shapes. A big-bang switches everything at once and then begins dismantling the legacy system, because keeping a fully redundant old system running indefinitely is expensive and the whole point was to be rid of it. So when a problem surfaces days after cutover, there may be no intact known-good state to return to — and every transaction processed since the switch has to be reconciled backward. Rollback becomes a recovery project, measured in days and incidents, which is precisely why big-bang failures become prolonged outages rather than contained events. Often the honest answer at that point is that rolling back is no longer possible, and the team is forced to fix forward under fire.

The incremental shape has the opposite property by construction. Because each slice is promoted behind a routing facade while the legacy implementation stays live, the known-good state is always still there. Reverting a misbehaving slice is shifting its traffic back to the old path — a flag, not a project — and it touches only that slice. The blast radius of any failure is one slice, and the recovery is measured in seconds.

What a real rollback plan has to cover

“We can roll back” is not a plan until it answers four questions concretely, all of them before the switch:

1. What is the known-good state, and is it still available? For an incremental migration this is the legacy path, kept live and ready. The plan has to keep it ready long enough — a slice is not safe to decommission the moment it takes full traffic; it earns retirement only after it has held under real load for a defined period.
2. How is data handled across the reversal? This is the hard part, and the part improvised plans get wrong. If the new slice has written data since the switch, reverting must not lose those writes or corrupt state. Strategies differ by slice — dual-writing during the ramp, reconciling on reversal, or keeping the legacy store authoritative until the slice is fully trusted — but the choice has to be deliberate and tested, not discovered during the incident.
3. What triggers a rollback, and who decides? Define the conditions in advance — error-rate thresholds, divergence in telemetry, a data-integrity alarm — and name the person with the authority to pull the switch. Under pressure, ambiguity costs minutes the business does not have, and a rollback delayed by a debate about whether to roll back is the failure mode that turns a contained issue into an outage.
4. How is the rollback itself verified? Returning to the old path has to be confirmed to have actually restored correct behavior, not assumed. The reversal is itself a state change and deserves the same telemetry as the promotion did.

Recovery objectives belong in the plan, in plain terms

Two standard measures keep a rollback plan honest, and a decision-maker should ask for both:

• Recovery time — how long from detecting a problem to being back in a known-good state. For per-slice rollback behind a facade this is typically near-instant; for a big-bang it is open-ended, which is the whole problem.
• Recovery point — how much data, if any, could be lost or need reconciling in a reversal. The design goal is none, and the incremental shape with disciplined data handling can usually achieve it because the legacy store stays authoritative until the slice is fully trusted.

Stating these in plain terms — “we can revert any slice in seconds with no data loss” versus “rollback after cutover would take days and require reconciling a week of transactions” — is one of the most clarifying things a program can put in front of leadership, because it makes the risk difference between the two shapes concrete and unarguable.

Rollback versus fix-forward

A decision a team makes under pressure, and should decide in advance: when something goes wrong, do you roll back to the known-good state, or fix forward by patching the new system in place? Both are legitimate, and the difference between the two shapes is which one you are forced into.

In a big-bang, fix-forward is often the only option left — the legacy system is gone or half-dismantled, so there is nothing to roll back to, and the team patches a struggling new system live, under fire, at the worst possible moment. That is not a choice; it is a corner. In an incremental migration, rollback is always available, which means fix-forward becomes a genuine choice rather than a fate: for a trivial, well-understood issue you might fix forward in minutes, but for anything ambiguous you revert the slice first — restoring correct service immediately — and diagnose at leisure with no clock running and no customers affected. The rule of thumb is simple and worth pre-agreeing: when in doubt, roll back, then investigate. A program that can always afford that rule has bought itself the most valuable thing a high-stakes change can have, which is time.

A safety net, not a license

Rollback is a safety net, not a license to skip the work that keeps you off it. The discipline that prevents failures — parity validation, a gradual canary ramp, watching telemetry at each step — is what should catch nearly everything; rollback is the backstop for the rare case those miss. A program that leans on rollback as its primary safety mechanism, promoting slices carelessly because “we can always revert,” is misusing it, and will eventually hit the data-handling edge where reversal is genuinely messy. Rollback also is not free: keeping the legacy path live and dual-handling data during the ramp is real operational cost, the same coexistence overhead named in Part 4. The justification is identical — for a critical system, the cost of keeping the net is trivial against the cost of falling without one.

Where this leads

The series has now covered the full risk toolkit: why the big-bang shape fails (Parts 1–3), how the incremental shape keeps the business running (Part 4), and how it controls risk at the point of change through cutover strategy, parallel-run evidence, and per-slice rollback (Parts 5–7). What remains is to put it in a leader’s hands as a single, usable decision. Part 8, How to De-Risk a Modernization Program, assembles everything into a framework for choosing the shape, structuring the engagement, and governing the risk — the decision this whole series has been building toward.