error rate
The share of requests that fail, reported alongside latency so a fast result is not mistaken for a healthy one.
In more detail
A service that fails quickly can have excellent latency, so always report count, errors and latency together. Count timeouts and wrong answers as errors, not only crashes.
Examples
The rollback rule fires when the canary’s error rate goes above 1%, for example 4.8%.sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))Related terms
- latency — The time between asking for something and getting the answer back.
- canary release — Sending a small share of real traffic to a new version first, and increasing it only while its metrics stay healthy.
- observability — How well you can understand what a system is doing, and why, from the data it emits: mainly logs, metrics and traces.
- saturation — How full a system’s most constrained resource is, such as its CPU, memory, connections or queue.
- service level indicator — A measurement of how well a service is doing for users, usually good events divided by eligible events.
- HTTP status code — The three-digit number at the start of a response, such as 200 or 404, that says how the server handled the request.
- error budget — The amount of failure an SLO allows over its window: one minus the target, for example 0.1 per cent of requests at 99.9 per cent.
- burn rate — How fast the error budget is being spent, compared with spending it exactly evenly over the SLO window.
- four golden signals — Latency, traffic, errors and saturation: the four measurements every service should emit.
- load testing — Measuring how a system behaves while a chosen amount of traffic is sent to it, to find its latency, throughput and limits.
Taught in
- Blue-Green Deployments — Continuous Integration & Continuous Delivery Pipelines
- Canary Releases — Continuous Integration & Continuous Delivery Pipelines
- Deployment Strategies: Rolling Updates — DevOps & Site Reliability Engineering
- Deployment Strategies: Canary Releases — DevOps & Site Reliability Engineering
- Continuous Delivery: Blue-Green & Canary Pipelines — Production Full-Stack Engineering
- Rates And Increases Over Time — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Indicators — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Objectives — Full-Stack Observability: Metrics, Tracing & Logging
- Error Budgets And Burn Rate Alerts — Full-Stack Observability: Metrics, Tracing & Logging
- Six Minutes Nobody Saw: Prometheus In One Outage — Prometheus Certified Associate (PCA) Exam Prep
- Ninety Nine Percent Of What? SLIs, SLOs And SLAs — Prometheus Certified Associate (PCA) Exam Prep
- Observability Sprint: One Failed Checkout — Prometheus Certified Associate (PCA) Exam Prep
- Off By Five: What Prometheus Is Not For — Prometheus Certified Associate (PCA) Exam Prep
- Fifteen Requests Nobody Saw: Aggregating Over Time — Prometheus Certified Associate (PCA) Exam Prep
- Fifteen Lines And No Answer: Aggregating Over Labels — Prometheus Certified Associate (PCA) Exam Prep
- Is The New Version Worse? Binary Operators And Matching — Prometheus Certified Associate (PCA) Exam Prep
- The Export That Failed Quietly: Timestamp Metrics — Prometheus Certified Associate (PCA) Exam Prep
- PromQL Sprint: Ten Queries At Minute Twenty Two — Prometheus Certified Associate (PCA) Exam Prep
- Paged Three Minutes Late: Alerting Rules — Prometheus Certified Associate (PCA) Exam Prep
- Twelve Green Panels: Dashboarding Basics — Prometheus Certified Associate (PCA) Exam Prep
- Alerting Sprint: Eleven Calls In Three Rounds — Prometheus Certified Associate (PCA) Exam Prep
- Consistency, And The Truth About CAP — Scalable System Design & Distributed Architecture
- Queues, Events And Asynchronous Work — Scalable System Design & Distributed Architecture
- Load Balancing And The Failure Modes It Introduces — Scalable System Design & Distributed Architecture
- Timeouts, And Avoiding Cascaded Failure — Scalable System Design & Distributed Architecture
- Retries, Backoff And Jitter — Scalable System Design & Distributed Architecture
- Circuit Breakers — Scalable System Design & Distributed Architecture
- Monolith Versus Services: The Honest Trade Offs — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Load Testing: Percentiles Over Means — Automated Testing, TDD & Quality Engineering