Most teams have a recovery plan. Fewer have executed it. The gap between the two is usually discovered at the least convenient moment, and it is almost always larger than expected.
Start small and build up
- Tabletop. Walk through the plan in a room. Find the missing steps and contacts.
- Component restore. Restore one database into a scratch environment. Time it.
- Service rebuild. Recreate one service from code and backups, end to end.
- Environment rebuild. Stand up the whole environment in a clean account.
- Live exercise. Fail something deliberately in a controlled window.
Each level finds a different class of problem. Skipping to the last one usually just finds the problems the first would have found, at greater cost.
What drills reliably uncover
| Finding | Why it happens |
|---|---|
| Recovery takes far longer than the objective | Estimates ignore data transfer time. |
| A dependency was never backed up | Configuration, DNS or secrets outside the plan. |
| Only one person can perform a step | Knowledge never written down. |
| Credentials for recovery are inside the failed system | Circular dependency. |
| Backups restore, but the data is wrong | Nobody verified content, only completion. |
| The runbook refers to things that no longer exist | Documentation drift. |
Use a separate account for the drill
Restoring into a clean, empty account is the honest test, because it proves nothing depends on state that happens to exist in the original environment. It also keeps the exercise from touching anything live.
Record the numbers
Every drill should produce two figures: how long recovery took, and how much data would have been lost. Compare them with your stated objectives. If they do not match, either the design or the objectives need to change – and knowing which is the point of the exercise.
Summary
Escalate from tabletop to live, restore into a clean account, keep emergency access outside the system, and record real recovery times. A plan nobody has executed is a hypothesis.
