outage
A period during which a service is unavailable or not working properly for its users.
In more detail
Not every outage is a failed deployment: a data centre losing power is an outage but no change caused it. In incident timelines, the outage begins when users are harmed and ends when service is back within expected behaviour, which can be before the incident itself is closed.
Example
The course ends with an outage you cause on purpose and a postmortem for it.Related terms
- incident — An unplanned event that degrades or interrupts a service for its users, or harms the security of systems or data.
- failed deployment recovery time — The time it takes to recover from a deployment that fails and needs immediate intervention in production.
- incident lifecycle — The stages of an incident: detection, declaration, mitigation, end of user impact and closure.
- circuit breaker — A wrapper around calls to another service that stops making calls for a while after repeated failures, so the failing service can recover.
- restart storm — Many instances being restarted over and over because a shared dependency is down, making an outage worse.
- postmortem — A written review after an incident that explains impact, timeline, contributing conditions and the follow-up actions that will change the system.
- symptom-based alerting — Paging on what users feel, such as errors, slowness or unavailability, rather than on internal causes such as high CPU or a restarted process.
- downtime — Time during which a system is unavailable to its users.
- failure mode — A specific way a system or component can break, described by its symptom, its cause and what users experience.
Taught in
- Restoration: Failed Deployment Recovery Time — DevOps & Site Reliability Engineering
- The Four Keys History And Retiring MTTR — DevOps & Site Reliability Engineering
- On-Call And Incident Response — DevOps & Site Reliability Engineering
- The Incident Lifecycle — DevOps & Site Reliability Engineering
- The Options Pattern: Strongly Typed Configuration — Modern .NET Core, C# & Enterprise Microservices
- Health Checks — Modern .NET Core, C# & Enterprise Microservices
- Retry and Circuit Breaker Policies with Polly — Modern .NET Core, C# & Enterprise Microservices
- Observability Versus Monitoring — Full-Stack Observability: Metrics, Tracing & Logging
- Why A Histogram Beats An Average — Full-Stack Observability: Metrics, Tracing & Logging
- Calculating Percentiles With histogram_quantile — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting Rules And The For Duration — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Indicators — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Objectives — Full-Stack Observability: Metrics, Tracing & Logging
- Error Budgets And Burn Rate Alerts — Full-Stack Observability: Metrics, Tracing & Logging
- On Call And Incident Command — Full-Stack Observability: Metrics, Tracing & Logging
- The Blameless Postmortem — Full-Stack Observability: Metrics, Tracing & Logging
- A Worked Example Postmortem — Full-Stack Observability: Metrics, Tracing & Logging
- Six Minutes Nobody Saw: Prometheus In One Outage — Prometheus Certified Associate (PCA) Exam Prep
- Ninety Nine Percent Of What? SLIs, SLOs And SLAs — Prometheus Certified Associate (PCA) Exam Prep
- Observability Sprint: One Failed Checkout — Prometheus Certified Associate (PCA) Exam Prep
- The Server That Would Not Die: Selecting Data — Prometheus Certified Associate (PCA) Exam Prep
- Four Thousand Requests Vanished: Rates And Derivatives — Prometheus Certified Associate (PCA) Exam Prep
- Fifteen Requests Nobody Saw: Aggregating Over Time — Prometheus Certified Associate (PCA) Exam Prep
- Fifteen Lines And No Answer: Aggregating Over Labels — Prometheus Certified Associate (PCA) Exam Prep
- Is The New Version Worse? Binary Operators And Matching — Prometheus Certified Associate (PCA) Exam Prep
- Under A Second, On Average: Histograms — Prometheus Certified Associate (PCA) Exam Prep
- PromQL Sprint: Ten Queries At Minute Twenty Two — Prometheus Certified Associate (PCA) Exam Prep
- Paged Three Minutes Late: Alerting Rules — Prometheus Certified Associate (PCA) Exam Prep
- Forty One Buzzes: Routing Alerts With Alertmanager — Prometheus Certified Associate (PCA) Exam Prep
- Twelve Green Panels: Dashboarding Basics — Prometheus Certified Associate (PCA) Exam Prep
- One Week, Three Objectives: Final Review — Prometheus Certified Associate (PCA) Exam Prep
- What A Design Is For, And How To Run A Design Discussion — Scalable System Design & Distributed Architecture
- Requirements: The Constraints That Shape The System — Scalable System Design & Distributed Architecture
- Back Of The Envelope Estimation: Traffic And Throughput — Scalable System Design & Distributed Architecture
- Back Of The Envelope Estimation: Storage And Bandwidth — Scalable System Design & Distributed Architecture
- The Storage Decision: Relational Versus Document — Scalable System Design & Distributed Architecture
- The Storage Decision: Key Value, Search And Blob — Scalable System Design & Distributed Architecture
- Replication, Failover And Leader Election — Scalable System Design & Distributed Architecture
- Partitioning, Sharding And Hot Keys — Scalable System Design & Distributed Architecture
- Consistency, And The Truth About CAP — Scalable System Design & Distributed Architecture
- Eventual Consistency And What It Costs A Product — Scalable System Design & Distributed Architecture
- Caching Strategies And Topology — Scalable System Design & Distributed Architecture
- Cache Invalidation And The Three Hard Cases — Scalable System Design & Distributed Architecture
- Queues, Events And Asynchronous Work — Scalable System Design & Distributed Architecture
- Idempotency, And Exactly Once As A Fiction — Scalable System Design & Distributed Architecture
- Load Balancing And The Failure Modes It Introduces — Scalable System Design & Distributed Architecture
- Timeouts, And Avoiding Cascaded Failure — Scalable System Design & Distributed Architecture
- Retries, Backoff And Jitter — Scalable System Design & Distributed Architecture
- Circuit Breakers — Scalable System Design & Distributed Architecture
- Backpressure And Load Shedding — Scalable System Design & Distributed Architecture
- Monolith Versus Services: The Honest Trade Offs — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
- Capacity Planning As A Design Input — Scalable System Design & Distributed Architecture
- Worked Design One: Ticket Booking, Correctness First — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Worked Design Two: A Global Chat System — Scalable System Design & Distributed Architecture
- Worked Design Two: Scaling Chat Fan Out — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Worked Design Three: Scaling The Metrics Pipeline — Scalable System Design & Distributed Architecture
References
- Google SRE book: Example Postmortem (appendix)
- Google SRE book: Managing Incidents
- Circuit Breaker pattern
- Google SRE book: Embracing Risk