Incident triage patterns

The first decisions in an incident shape everything after them. These patterns cover what to check before paging anyone, how to organise the response, how to contain an attacker, and how to learn from it afterwards. Each links to the lessons and labs where you practise it.

Alert triage: page on symptoms, investigate causes

When to use it: An alert has fired, or an alert queue is filling faster than people can work through it.

Signals

  • Pages that fire while users notice nothing, or users who notice before any page fires
  • A few noisy rules producing most of the queue
  • An error budget burning faster than planned

Steps

  1. Ask whether users are affected: an alert on a symptom (errors, latency) is worth waking someone for; an alert on a cause (CPU, a restart) usually is not.
  2. Check how fast the error budget is burning, which tells you how long you have before the objective is missed.
  3. Group related alerts into one case so one person owns them.
  4. For a rule that is noisy, add a scoped exception for the known benign case rather than turning the rule off.
  5. Write down what you decided and why, for the next person on shift.

Where it is taught

Incident command: one person coordinates, others fix

When to use it: An incident needs more than one person, or will last longer than a few minutes.

Signals

  • Several people investigating the same thing, and nobody sure who decided what
  • Stakeholders asking for updates in the channel the responders work in
  • No one can say when the incident started or when it was detected

Steps

  1. Name an incident commander who coordinates and does not debug.
  2. Give out the other roles: someone to work on the system, someone to communicate, someone to keep the timeline.
  3. Record the clocks as you go: when it started, when it was detected, acknowledged, mitigated and resolved.
  4. Send updates on a fixed rhythm, saying what is known, what is being tried and when the next update comes.
  5. Hand over explicitly when someone leaves, with the current state written down.

Where it is taught

Rate severity, then rate it again after scoping

When to use it: You need to decide how big an incident is, who to tell and how fast to move.

Signals

  • An incident rated on its first symptom alone
  • New hosts, accounts or services turning up as you investigate
  • Disagreement about who needs to be told

Steps

  1. Give a first severity from what you know, so the response starts at the right speed.
  2. Scope it: which systems, data and people are affected, using the asset inventory and the logs.
  3. Follow the trail outwards, for example an attacker's logins from host to host, until it stops.
  4. Rate severity again with the scope known, and change the response if the rating changed.
  5. Tell the people that rating requires, in the words they need.

Where it is taught

Read the first failure, not the last

When to use it: A deploy or a recovery has failed with a long tail of errors, and the loudest one is not obviously the cause.

Signals

  • Dozens of errors that all start after one earlier one
  • Retries and timeouts in services downstream of the one that broke
  • Several faults at once, each hiding the others

Steps

  1. Sort the evidence by time and find the earliest failure; the later ones are often its consequences.
  2. Collect a narrow evidence bundle first (status, events, the first error) before changing anything.
  3. Fix one fault at a time, and check the result before moving to the next.
  4. Run a smoke test that proves the service works end to end, not only that the error went away.
  5. Hand off with what you found, what you changed and what is still open.

Where it is taught

Contain, then eradicate: cut the attacker off before you clean up

When to use it: A security incident is confirmed and the attacker may still have access.

Signals

  • Sessions or tokens still in use after the password was reset
  • Persistence found on one host, with others not yet checked
  • Pressure to rebuild machines before the scope is known

Steps

  1. Contain identity as well as hosts: revoke sessions and tokens, because a password reset alone leaves them working.
  2. Isolate affected hosts on the network in a way that actually cuts the attacker's path.
  3. Sweep every host for the same indicators before removing anything, so you do not warn the attacker about the hosts you missed.
  4. Remove the attacker's footholds, then restore from a point you know is clean.
  5. Let a host back onto the network only after it passes validation.

Where it is taught

Blameless postmortem: a timeline, causes and actions someone owns

When to use it: An incident is over and you want it not to happen again.

Signals

  • The same kind of incident happening a second time
  • A review that ends with who made a mistake rather than what allowed it
  • Actions with no owner or no date

Steps

  1. Build the timeline from records, not memory: alerts, chat, deploys and logs.
  2. Ask why at each step and let the answers branch: most incidents have more than one cause.
  3. Write about what the system allowed, not who slipped.
  4. Turn each lesson into an action with one owner and a date, such as a detection rule or a test.
  5. Check later that the actions were done, for example by replaying the incident through a new detection rule.

Where it is taught

Certification courses