Rollback / Recovery
Our reliability architecture assumes that some automated changes will eventually need to be reversed, retried, resumed, or recovered after partial failure.
Mechanisms
Recovery capabilities can include reversible actions, versioned configuration, rollback, backups, retry, and checkpointing. The relevant recovery mechanism depends on what failed and whether the external action itself is reversible.
Recovery state
Multi-step workflows should know how far they progressed.
- Step Acomplete
- Step Bcomplete
- Step Cfailed
- Step Dnot started
A checkpoint allows the system to resume from a known state instead of blindly restarting the entire workflow. That distinction is critical when earlier steps created external side effects. Re-running an entire workflow can duplicate messages, payments, records, or other actions even though the workflow appears to have failed.
Version rollback
Models are only one versioned dependency. Production behavior can change when prompts, rules, retrieval logic, workflow configuration, tools, or model versions change.
Where practical, the deployed state should preserve enough version information to return to a known working configuration after a regression is detected.
Rollback is faster when recovery state is designed before deployment rather than reconstructed during an incident.
Reversible versus irreversible actions
Not every action can be undone. A database field may be reversible. An email already sent to a customer is not. A financial transaction may require a compensating action rather than a literal rollback.
The workflow therefore needs to distinguish:
- Reversible actions
- Retry-safe actions
- Compensating actions
- Irreversible actions
Higher-consequence irreversible steps can justify stronger validation or approval before execution.
Recovery objective
Retries, rollback, checkpoints, and backups solve different failure classes. We do not treat “try again” as a universal recovery strategy.
The objective is a system that can fail without losing control of state — and can return to a known operating condition with the minimum additional damage, duplicated work, or manual reconstruction.
Recovery paths are tested
A backup or rollback mechanism that has never been exercised is an assumption, not a recovery capability.
For important state, recovery design can include verification that backups are readable, checkpoints are sufficient to resume the workflow, and prior configuration can actually be redeployed. The same applies to compensating actions. If an external side effect cannot be literally reversed, the recovery path should define what corrective action restores the intended business state.
Recovery also interacts with audit logging. A rollback should not erase the record that the original action occurred; it should create a new traceable state transition explaining how the system recovered. This preserves both operational control and historical truth.