Incident triage patterns
The first decisions in an incident shape everything after them. These patterns cover what to check before paging anyone, how to organise the response, how to contain an attacker, and how to learn from it afterwards. Each links to the lessons and labs where you practise it.
Alert triage: page on symptoms, investigate causes
When to use it: An alert has fired, or an alert queue is filling faster than people can work through it.
Signals
- Pages that fire while users notice nothing, or users who notice before any page fires
- A few noisy rules producing most of the queue
- An error budget burning faster than planned
Steps
- Ask whether users are affected: an alert on a symptom (errors, latency) is worth waking someone for; an alert on a cause (CPU, a restart) usually is not.
- Check how fast the error budget is burning, which tells you how long you have before the objective is missed.
- Group related alerts into one case so one person owns them.
- For a rule that is noisy, add a scoped exception for the known benign case rather than turning the rule off.
- Write down what you decided and why, for the next person on shift.
Where it is taught
- Alerting On Symptoms Versus Causes — Lesson, Full-Stack Observability: Metrics, Tracing & Logging
- Burn rates and how long the budget lasts — Lab, Full-Stack Observability: Metrics, Tracing & Logging
- Alert Triage, False Positive Reduction & Fatigue Mitigation — Lesson, Security Operations & Threat Hunting (SOC)
- A broad exclusion against a scoped one — Lab, Security Operations & Threat Hunting (SOC)
- A health check that says yes while work is failing — Lab, Full-Stack Observability: Metrics, Tracing & Logging
Incident command: one person coordinates, others fix
When to use it: An incident needs more than one person, or will last longer than a few minutes.
Signals
- Several people investigating the same thing, and nobody sure who decided what
- Stakeholders asking for updates in the channel the responders work in
- No one can say when the incident started or when it was detected
Steps
- Name an incident commander who coordinates and does not debug.
- Give out the other roles: someone to work on the system, someone to communicate, someone to keep the timeline.
- Record the clocks as you go: when it started, when it was detected, acknowledged, mitigated and resolved.
- Send updates on a fixed rhythm, saying what is known, what is being tried and when the next update comes.
- Hand over explicitly when someone leaves, with the current state written down.
Where it is taught
- On Call And Incident Command — Lesson, Full-Stack Observability: Metrics, Tracing & Logging
- On-Call And Incident Response — Lesson, DevOps & Site Reliability Engineering
- The Incident Lifecycle — Lesson, DevOps & Site Reliability Engineering
- Read the incident clocks — Lab, DevOps & Site Reliability Engineering
- SOC Runbooks, Shift Handoffs & Escalation Paths — Lesson, Security Operations & Threat Hunting (SOC)
Rate severity, then rate it again after scoping
When to use it: You need to decide how big an incident is, who to tell and how fast to move.
Signals
- An incident rated on its first symptom alone
- New hosts, accounts or services turning up as you investigate
- Disagreement about who needs to be told
Steps
- Give a first severity from what you know, so the response starts at the right speed.
- Scope it: which systems, data and people are affected, using the asset inventory and the logs.
- Follow the trail outwards, for example an attacker's logins from host to host, until it stops.
- Rate severity again with the scope known, and change the response if the rating changed.
- Tell the people that rating requires, in the words they need.
Where it is taught
- Incident Triage, Severity Scoping & Communication — Lesson, Security Operations & Threat Hunting (SOC)
- Following the attacker's logins from host to host — Lab, Security Operations & Threat Hunting (SOC)
- Rating the same incident before and after scoping — Lab, Security Operations & Threat Hunting (SOC)
- NIST SP 800-61 Incident Lifecycle & Management — Lesson, Security Operations & Threat Hunting (SOC)
Read the first failure, not the last
When to use it: A deploy or a recovery has failed with a long tail of errors, and the loudest one is not obviously the cause.
Signals
- Dozens of errors that all start after one earlier one
- Retries and timeouts in services downstream of the one that broke
- Several faults at once, each hiding the others
Steps
- Sort the evidence by time and find the earliest failure; the later ones are often its consequences.
- Collect a narrow evidence bundle first (status, events, the first error) before changing anything.
- Fix one fault at a time, and check the result before moving to the next.
- Run a smoke test that proves the service works end to end, not only that the error went away.
- Hand off with what you found, what you changed and what is still open.
Where it is taught
- Troubleshooting Failed Deployments: Read The First Failure — Lesson, AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Triage, four times — Lab, AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Troubleshooting: A Timed Service Recovery Drill — Lesson, Kubernetes: Production-Grade Container Orchestration
- Start with a narrow evidence bundle — Lab, Kubernetes: Production-Grade Container Orchestration
- Four Faults, Twelve Minutes — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
Contain, then eradicate: cut the attacker off before you clean up
When to use it: A security incident is confirmed and the attacker may still have access.
Signals
- Sessions or tokens still in use after the password was reset
- Persistence found on one host, with others not yet checked
- Pressure to rebuild machines before the scope is known
Steps
- Contain identity as well as hosts: revoke sessions and tokens, because a password reset alone leaves them working.
- Isolate affected hosts on the network in a way that actually cuts the attacker's path.
- Sweep every host for the same indicators before removing anything, so you do not warn the attacker about the hosts you missed.
- Remove the attacker's footholds, then restore from a point you know is clean.
- Let a host back onto the network only after it passes validation.
Where it is taught
- Containment & Eradication: Host, Net & Identity — Lesson, Security Operations & Threat Hunting (SOC)
- Password reset versus session revocation — Lab, Security Operations & Threat Hunting (SOC)
- Sweep every host before removing anything — Lab, Security Operations & Threat Hunting (SOC)
- The reconnection gate: fail stays quarantined — Lab, Security Operations & Threat Hunting (SOC)
Blameless postmortem: a timeline, causes and actions someone owns
When to use it: An incident is over and you want it not to happen again.
Signals
- The same kind of incident happening a second time
- A review that ends with who made a mistake rather than what allowed it
- Actions with no owner or no date
Steps
- Build the timeline from records, not memory: alerts, chat, deploys and logs.
- Ask why at each step and let the answers branch: most incidents have more than one cause.
- Write about what the system allowed, not who slipped.
- Turn each lesson into an action with one owner and a date, such as a detection rule or a test.
- Check later that the actions were done, for example by replaying the incident through a new detection rule.
Where it is taught
- The Blameless Postmortem — Lesson, Full-Stack Observability: Metrics, Tracing & Logging
- A Worked Example Postmortem — Lesson, Full-Stack Observability: Metrics, Tracing & Logging
- Lint the example postmortem — Lab, DevOps & Site Reliability Engineering
- The five whys, allowed to branch — Lab, Security Operations & Threat Hunting (SOC)
- Replaying the incident through the new rule — Lab, Security Operations & Threat Hunting (SOC)