alert fatigue
The state where people receive so many unhelpful alerts that they start ignoring them, including the real ones.
Teams that learn to ignore alerts miss the one that matters.
In more detail
Alerting on every possible cause turns an on-call shift into a stream of guesses. Paging only on urgent symptoms keeps alert volume close to real work.
Example
Fatigued analysts start closing alerts without reading them, which is how real attacks slip through.Related terms
- symptom-based alerting — Paging on what users feel, such as errors, slowness or unavailability, rather than on internal causes such as high CPU or a restarted process.
- alert — An automatic notification sent when a monitored condition, such as a high error rate, stays true.
- on-call — Being the person responsible, for a set period, for responding when a production system raises an alert, often outside working hours.
- tuning — Adjusting a detection rule so it raises fewer false alerts without missing real attacks.
- alert grouping — Combining related alerts into one case, for example all alerts from the same source address within an hour.
- false positive — A check result that claims something is true when it is not, such as a test passing on broken code or a tool flagging correct code.
Taught in
- Monitoring A DevOps Environment — Microsoft Azure DevOps Engineer Expert (AZ-400) Exam Prep
- Alerting On Symptoms Versus Causes — Full-Stack Observability: Metrics, Tracing & Logging
- Alert Triage, False Positive Reduction & Fatigue Mitigation — Security Operations & Threat Hunting (SOC)
References
- Google SRE book: monitoring distributed systems
- Google SRE book: practical alerting
- NIST SP 800-92: Guide to Computer Security Log Management