runbook
A written, step-by-step procedure for a routine operation or for responding to a known kind of incident.
In more detail
A good runbook lets someone who has never done the task before do it safely, and says who to call when it does not work. It records procedures and ownership, never the secrets themselves.
Examples
A Vault runbook records where each vault password comes from, who may use it and how to rotate it.In AWS: A written, step-by-step guide for handling a specific alert or task, including safe first actions and when to escalate.A runbook lets anyone on call repeat a task safely without rediscovering it. Commands in a runbook usually assume a particular distribution and release, so confirm the host matches before following it. Symbolic, targeted commands are easier to review in a runbook than commands that overwrite everything with a guessed value.A runbook lists likely causes of a checkout error alert and the queries that tell them apart.A SOC runbook tells tier one exactly what to check, what evidence to record, what they may do on their own authority and when to escalate.Related terms
- vault password — The password that unlocks Vault-encrypted data; Ansible needs it at run time and it must never be stored in the repository.
- key rotation — Replacing a password or key with a new one on a schedule or after a suspected leak, and re-encrypting what it protected.
- alarm — A rule that watches a metric and changes state, and notifies someone, when the metric crosses a threshold for long enough.
- on-call — Being the person responsible, for a set period, for responding when a production system raises an alert, often outside working hours.
- incident — An unplanned event that degrades or interrupts a service for its users, or harms the security of systems or data.
- escalation — Bringing in more people, expertise or authority when a problem exceeds the current responder.
- alert — An automatic notification sent when a monitored condition, such as a high error rate, stays true.
- toil — Manual, repetitive, automatable operational work that grows with the service and leaves no lasting value.
- /etc/os-release — A small text file of name=value lines that tells programs and people which distribution and version a Linux machine runs.
- Linux distribution — A complete operating system built around the Linux kernel, such as Debian, Ubuntu or Alpine Linux.
- symptom-based alerting — Paging on what users feel, such as errors, slowness or unavailability, rather than on internal causes such as high CPU or a restarted process.
- playbook — The wider response plan for a whole class of incident, such as ransomware or account compromise, covering several teams.
- escalation path — The agreed route a case takes when it needs more skill, more authority or has waited too long.
- SOC tiers — Levels of a SOC team sorted by how deep each one investigates: tier one works the alert queue, tier two investigates, tier three hunts and leads big incidents.
Taught in
- Securing Data With Ansible Vault — Ansible Configuration Management at Scale
- Vault IDs And Multiple Passwords — Ansible Configuration Management at Scale
- Two Hundred Accounts And One Bad Deploy — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Systems Manager At Fleet Scale: Compliant And Unpatched — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- High Availability Across Tiers: What Fails Over By Itself? — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Recovery To RTO And RPO: Backups Nobody Could Delete — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Testing Resilience: The Experiment That Would Have Been The Outage — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Monitoring As Code: The Alarm That Watched Nothing — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- EventBridge Rules: The Rule That Matched Nothing — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Remediation As Code: A Public Bucket Appears — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Troubleshooting Failed Deployments: Read The First Failure — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Incident Tooling: Eighteen Minutes To Find A Human — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Security Hub, GuardDuty And Inspector: Signal, Noise And Response — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Auditing: Who Opened Port Twenty Two? — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Long-Stem Clinic: When Two Answers Both Work — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Databases That Survive: Standbys, Replicas And Global Tables — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Disaster Recovery By The Numbers — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Wikis, Release Notes And Integrations — Microsoft Azure DevOps Engineer Expert (AZ-400) Exam Prep
- Staging Is Gone: Desired State, Declarations And The State Store — Certified GitOps Associate (CGOA) Exam Prep
- Two Ways Back From 2.5: Rollback The GitOps Way — Certified GitOps Associate (CGOA) Exam Prep
- Terminology Sprint: One Incident, Nine Terms — Certified GitOps Associate (CGOA) Exam Prep
- Seven Steps Called GitOps: The Declarative Principle — Certified GitOps Associate (CGOA) Exam Prep
- Observability with CloudWatch — Cloud Infrastructure & Hyperscaler Architecture
- Metrics, Logs, and Alarms — Cloud Infrastructure & Hyperscaler Architecture
- Infrastructure as Code (Handover to Terraform) — Cloud Infrastructure & Hyperscaler Architecture
- Security Governance vs Security Management — Cybersecurity Fundamentals, GRC & Cryptography
- Privacy Principles, GDPR Subject Rights & Minimization — Cybersecurity Fundamentals, GRC & Cryptography
- Throughput: Change Lead Time And Deployment Frequency — DevOps & Site Reliability Engineering
- On-Call And Incident Response — DevOps & Site Reliability Engineering
- One Pod Away: Network Attackers And Sensitive Data — Kubernetes and Cloud Native Security Associate (KCSA) Exam Prep
- Troubleshooting: ImagePullBackOff And Failed Rollouts — Kubernetes: Production-Grade Container Orchestration
- Troubleshooting: DNS, Empty Endpoints And Broken Routes — Kubernetes: Production-Grade Container Orchestration
- Drain, Upgrade, Version Skew And Certificate Maintenance — Kubernetes: Production-Grade Container Orchestration
- etcd Snapshots And A Tested Restore — Kubernetes: Production-Grade Container Orchestration
- Troubleshooting: A Timed Service Recovery Drill — Kubernetes: Production-Grade Container Orchestration
- What A Distribution Actually Is — Linux Fundamentals & Systems Administration
- Reading And Setting A File Mode — Linux Fundamentals & Systems Administration
- Alerting On Symptoms Versus Causes — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting Rules And The For Duration — Full-Stack Observability: Metrics, Tracing & Logging
- SOC Operating Models, Tiers & Key Performance Metrics — Security Operations & Threat Hunting (SOC)
- SOC Runbooks, Shift Handoffs & Escalation Paths — Security Operations & Threat Hunting (SOC)
- Monolith Versus Services: The Honest Trade Offs — Scalable System Design & Distributed Architecture
References
- Google SRE book: being on-call (playbooks)
- AWS Well-Architected Operational Excellence pillar: use runbooks to perform procedures
- Google SRE book: Emergency Response
- PagerDuty incident response documentation
- NIST SP 800-61 Rev. 3: Incident Response Recommendations
Certification courses
- AWS DevOps Engineer Professional course and practice exam: DOP-C02
- AWS Solutions Architect Associate course and practice exam: SAA-C03
- AZ-400 course and practice exam: Microsoft Azure DevOps Engineer Expert
- CGOA course and practice exam: Certified GitOps Associate
- KCSA course and practice exam: Kubernetes and Cloud Native Security Associate