alert
An automatic notification sent when a monitored condition, such as a high error rate, stays true.
In more detail
Alerts come from metrics, because metrics are cheap to evaluate every few seconds and stable enough to compare with a threshold. A useful alert names the user impact, an owner and the next action. Log levels never page anybody; alerts do.
Example
In a SOC, alerts come from SIEM rules, EDR and antivirus, DLP, network flow analysis and runtime sensors.Related terms
- on-call — Being the person responsible, for a set period, for responding when a production system raises an alert, often outside working hours.
- monitoring — Continuously collecting and watching measurements of a system so problems are noticed quickly.
- burn rate — How fast the error budget is being spent, compared with spending it exactly evenly over the SLO window.
- alerting rule — A Prometheus rule with a PromQL expression that describes an unhealthy condition, plus a for duration, labels and annotations.
- symptom-based alerting — Paging on what users feel, such as errors, slowness or unavailability, rather than on internal causes such as high CPU or a restarted process.
- on-call page — An urgent alert that interrupts the on-call engineer, for example on their phone, because a person must act now.
- alert fatigue — The state where people receive so many unhelpful alerts that they start ignoring them, including the real ones.
- triage — Quickly assessing an alert or problem to decide how serious it is and what to do first.
- disposition — The recorded outcome of an alert after triage, such as true positive, benign true positive or false positive.
- detection rule — A saved query or pattern that watches incoming events and raises an alert when they match something suspicious.
- oldest message age — How long the oldest unprocessed message has been waiting.
- metric — A number measured over time, such as requests per second or error count, that shows how a system behaves in aggregate.
Taught in
- Staging Is Gone: Desired State, Declarations And The State Store — Certified GitOps Associate (CGOA) Exam Prep
- Sixteen Pods, Room For Twelve: The System, Feedback And Continuous — Certified GitOps Associate (CGOA) Exam Prep
- Terminology Sprint: One Incident, Nine Terms — Certified GitOps Associate (CGOA) Exam Prep
- An Hour Without Payments: Continuously Reconciled — Certified GitOps Associate (CGOA) Exam Prep
- Three Minutes Of Waiting: Pull Versus Event-Driven — Certified GitOps Associate (CGOA) Exam Prep
- A Rollback Nobody Heard: Notifications, Observability And CI — Certified GitOps Associate (CGOA) Exam Prep
- Tooling Sprint: The CTO's Shopping List — Certified GitOps Associate (CGOA) Exam Prep
- The Evening Before: Final Review — Certified GitOps Associate (CGOA) Exam Prep
- On-Call And Incident Response — DevOps & Site Reliability Engineering
- The Incident Lifecycle — DevOps & Site Reliability Engineering
- Observability Context — DevOps & Site Reliability Engineering
- Observability Versus Monitoring — Full-Stack Observability: Metrics, Tracing & Logging
- The Three Signals: Logs, Metrics And Traces — Full-Stack Observability: Metrics, Tracing & Logging
- Log Levels Done Properly — Full-Stack Observability: Metrics, Tracing & Logging
- Prometheus Scraping And The Pull Model — Full-Stack Observability: Metrics, Tracing & Logging
- Bringing The Three Signals Together — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting On Symptoms Versus Causes — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting Rules And The For Duration — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Indicators — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Objectives — Full-Stack Observability: Metrics, Tracing & Logging
- Error Budgets And Burn Rate Alerts — Full-Stack Observability: Metrics, Tracing & Logging
- The Blameless Postmortem — Full-Stack Observability: Metrics, Tracing & Logging
- A Worked Example Postmortem — Full-Stack Observability: Metrics, Tracing & Logging
- Six Minutes Nobody Saw: Prometheus In One Outage — Prometheus Certified Associate (PCA) Exam Prep
- Ninety Minutes, No Question Count: Exam Strategy — Prometheus Certified Associate (PCA) Exam Prep
- Forty Seconds At Two: Push, Pull And Service Discovery — Prometheus Certified Associate (PCA) Exam Prep
- Ninety Nine Percent Of What? SLIs, SLOs And SLAs — Prometheus Certified Associate (PCA) Exam Prep
- Observability Sprint: One Failed Checkout — Prometheus Certified Associate (PCA) Exam Prep
- Where Did Two Hours Go? Prometheus Architecture — Prometheus Certified Associate (PCA) Exam Prep
- A Week Without A Scrape: Configuration And Relabelling — Prometheus Certified Associate (PCA) Exam Prep
- Off By Five: What Prometheus Is Not For — Prometheus Certified Associate (PCA) Exam Prep
- Fundamentals Sprint: Nine Calls On The Server — Prometheus Certified Associate (PCA) Exam Prep
- Four Thousand Requests Vanished: Rates And Derivatives — Prometheus Certified Associate (PCA) Exam Prep
- Fifteen Requests Nobody Saw: Aggregating Over Time — Prometheus Certified Associate (PCA) Exam Prep
- Is The New Version Worse? Binary Operators And Matching — Prometheus Certified Associate (PCA) Exam Prep
- The Export That Failed Quietly: Timestamp Metrics — Prometheus Certified Associate (PCA) Exam Prep
- Thirty Second Checkouts Nobody Saw: Client Libraries — Prometheus Certified Associate (PCA) Exam Prep
- A Database You Cannot Recompile: Exporters — Prometheus Certified Associate (PCA) Exam Prep
- Milliseconds Versus Seconds: Naming Metrics And Labels — Prometheus Certified Associate (PCA) Exam Prep
- Instrumentation Sprint: Sixteen Calls On Code And Exporters — Prometheus Certified Associate (PCA) Exam Prep
- Paged Three Minutes Late: Alerting Rules — Prometheus Certified Associate (PCA) Exam Prep
- Forty One Buzzes: Routing Alerts With Alertmanager — Prometheus Certified Associate (PCA) Exam Prep
- Twelve Green Panels: Dashboarding Basics — Prometheus Certified Associate (PCA) Exam Prep
- Alerting Sprint: Eleven Calls In Three Rounds — Prometheus Certified Associate (PCA) Exam Prep
- Seventy Seven Percent, Book Now? The Full Mock — Prometheus Certified Associate (PCA) Exam Prep
- One Week, Three Objectives: Final Review — Prometheus Certified Associate (PCA) Exam Prep
- SOC Operating Models, Tiers & Key Performance Metrics — Security Operations & Threat Hunting (SOC)
- Alert Triage, False Positive Reduction & Fatigue Mitigation — Security Operations & Threat Hunting (SOC)
- SIEM Architecture: Ingestion Pipelines & Indexing — Security Operations & Threat Hunting (SOC)
- SOC Runbooks, Shift Handoffs & Escalation Paths — Security Operations & Threat Hunting (SOC)
- Threat Intelligence Platforms & STIX/TAXII Ingestion — Security Operations & Threat Hunting (SOC)
- Event Correlation Rules, Thresholds & Anomaly Pipelines — Security Operations & Threat Hunting (SOC)
- Log Parsing, Schema Normalization (OCSF/ECS) & Extraction — Security Operations & Threat Hunting (SOC)
- Detection as Code: Authoring & Testing Sigma Rules — Security Operations & Threat Hunting (SOC)
- YARA Rule Authoring & File Signature Matching — Security Operations & Threat Hunting (SOC)
- Adversary Tactics, Techniques & Procedures (TTP) Mapping — Security Operations & Threat Hunting (SOC)
- NIST SP 800-61 Incident Lifecycle & Management — Security Operations & Threat Hunting (SOC)
- Incident Triage, Severity Scoping & Communication — Security Operations & Threat Hunting (SOC)
- Containment & Eradication: Host, Net & Identity — Security Operations & Threat Hunting (SOC)
- Recovery & System Validation Testing — Security Operations & Threat Hunting (SOC)
- Post-Incident Review & Root Cause Analysis — Security Operations & Threat Hunting (SOC)
- Requirements: The Constraints That Shape The System — Scalable System Design & Distributed Architecture
- Back Of The Envelope Estimation: Storage And Bandwidth — Scalable System Design & Distributed Architecture
- Partitioning, Sharding And Hot Keys — Scalable System Design & Distributed Architecture
- Eventual Consistency And What It Costs A Product — Scalable System Design & Distributed Architecture
- Caching Strategies And Topology — Scalable System Design & Distributed Architecture
- Queues, Events And Asynchronous Work — Scalable System Design & Distributed Architecture
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture
- Monolith Versus Services: The Honest Trade Offs — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
- Capacity Planning As A Design Input — Scalable System Design & Distributed Architecture
- Worked Design One: Ticket Booking, Correctness First — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Worked Design Three: Scaling The Metrics Pipeline — Scalable System Design & Distributed Architecture
References
- Google SRE book: Monitoring Distributed Systems
- Google SRE workbook: Alerting on SLOs
- Prometheus: alerting best practices
- Prometheus: alerting rules
- NIST SP 800-61 Rev. 3: Incident Response Recommendations
- NIST SP 800-92: Guide to Computer Security Log Management