recovery drill
A rehearsal of a failure and its repair, run against the clock on a safe system, to prove that a runbook actually works.
In more detail
A good drill starts from a user-visible symptom, collects evidence before changing anything, sets a repair deadline and escalation point, and ends with a smoke test and a written handoff. Restore drills prove that backups can be used. Findings go back into the runbook.
Related terms
- runbook — A written, step-by-step procedure for a routine operation or for responding to a known kind of incident.
- smoke test — A quick test that checks the most basic behaviour works after a deploy, before deeper testing or real traffic.
- incident handoff — The written note that passes an incident or drill to the next person, saying what happened, what was done, what is proven and what remains.
- disaster recovery — The planned, tested procedure for getting a system and its data back after a serious failure, such as lost storage or a deleted cluster.
Taught in
- Troubleshooting: ImagePullBackOff And Failed Rollouts — Kubernetes: Production-Grade Container Orchestration
- StatefulSets, Stable Identity And Data Recovery — Kubernetes: Production-Grade Container Orchestration
- etcd Snapshots And A Tested Restore — Kubernetes: Production-Grade Container Orchestration
- Troubleshooting: API Server And Control Plane Failures — Kubernetes: Production-Grade Container Orchestration
- Troubleshooting: A Timed Service Recovery Drill — Kubernetes: Production-Grade Container Orchestration