latency
The time between asking for something and getting the answer back.
In more detail
For a network call it includes the travel time both ways plus the time the other side spends working. Many small calls made one after another add their latencies together, which is why chatty designs feel slow.
Examples
In AWS: The time a request or packet takes to get where it is going and back, usually measured in milliseconds.In Kafka, linger.ms and acks=all both add latency to each record.In the browser: Latency is waiting time rather than download speed. A page made of small files can still feel slow if every request pays a long delay for connection setup, which is why caching and preconnect matter.In Node.js, a blocked event loop adds latency to every request waiting behind it.Waterfalls multiply latency in React data fetching; optimistic UI and parallel fetching reduce how much users notice.The N+1 query problem is slow because each extra query adds its own latency.Main memory is about 100 nanoseconds, a round trip inside a datacenter about half a millisecond, a round trip across an ocean about 150 milliseconds.In this course: Latency is a distribution, not one number: most requests may be fast while a few are very slow. Report it as percentiles alongside request count and error rate.Related terms
- throughput — How much work a system completes per unit of time, such as requests per second.
- percentile — The value below which a given percentage of measurements fall; p99 latency is the time 99 in 100 requests beat.
- tail latency — The latency of the slowest requests, measured by high percentiles such as p99.
- round trip — One journey of a request to a server and its response back, whose time adds directly to how long the caller waits.
- region — A separate geographic area where a cloud provider runs data centres, such as eu-west-2 for London in AWS.
- data transfer — Moving data between zones, regions or out to the internet, which AWS usually charges for by the gigabyte.
- error rate — The share of requests that fail, reported alongside latency so a fast result is not mistaken for a healthy one.
- observability — How well you can understand what a system is doing, and why, from the data it emits: mainly logs, metrics and traces.
- linger.ms — The producer setting for how many milliseconds to wait for more records before sending a batch that is not yet full.
- acks — The producer setting that decides how many brokers must confirm a record before the send counts as successful: 0, 1 or all.
- saturation — How full a system’s most constrained resource is, such as its CPU, memory, connections or queue.
- Task.WhenAll — A .NET method that returns one task which completes when all the tasks you pass it have completed.
- time to first byte — The time from starting a request until the first byte of the server's response arrives, covering connection setup, server work and travel time.
- bandwidth — How much data a network connection can carry per second, and by extension the amount of data a request or response uses up.
- TCP three-way handshake — The three-message exchange, SYN, SYN-ACK and ACK, that a client and server use to open a TCP connection before sending data.
- two-tier cache — A cache with a small, very fast local layer inside each server process (L1) in front of a larger shared layer such as Redis (L2).
- Time to First Byte — How long it takes from sending a request until the first byte of the response arrives.
- CDN — A network of servers near users that keep copies of public content, such as images, and serve them quickly.
- event loop lag — How late the event loop is in getting round to scheduled work, a direct measure of how blocked the main thread is.
- histogram — A metric that counts measurements, such as request durations, into ranges of values so their distribution and percentiles can be calculated.
- network waterfall — A chain of network requests where each one starts only after the previous finished, so their delays add up.
- N+1 query problem — A performance bug where code runs one query for a list and then one more query per item, so N items cost N+1 queries.
- latency budget — A latency target divided among the hops a request makes, so each part knows how much time it may spend.
- mean — The sum of all values divided by how many there are; what people usually call the average.
- load testing — Measuring how a system behaves while a chosen amount of traffic is sent to it, to find its latency, throughput and limits.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- Choosing A Model — AI Engineering: LLM Apps, RAG & Agents
- When The Call Fails — AI Engineering: LLM Apps, RAG & Agents
- Cost, Latency And Caching — AI Engineering: LLM Apps, RAG & Agents
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
- Workflows Before Agents — AI Engineering: LLM Apps, RAG & Agents
- Excessive Agency — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents
- Images, Audio And Speech — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents
- Load Testing — Modern API Architecture: REST, GraphQL & gRPC
- API Gateways And Proxies — Modern API Architecture: REST, GraphQL & gRPC
- Performance Optimization — Modern API Architecture: REST, GraphQL & gRPC
- Observability: Logs, Metrics, Traces — Modern API Architecture: REST, GraphQL & gRPC
- The Night Larder Could Not Buy Servers Fast Enough — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Six Advantages, In AWS's Own Words — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Which Zone, Which Journey? — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Nine Jobs, Not Nine Servers — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- One Database Doing Five Jobs — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- The Public Subnet That Was Not — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Reading A 130-Minute Exam — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Protecting Data: Versions, Object Lock, Backups And Secrets — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Serverless And Container Patterns: Where The Limits Bite — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Load Balancers And Auto Scaling: Capacity That Arrives On Time — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Databases That Survive: Standbys, Replicas And Global Tables — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Disaster Recovery By The Numbers — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Route 53 Routing: Where Does A Fan In Lagos Land? — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Storage Performance: IOPS, Throughput And The Right Volume — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Compute Performance: Families, Placement And Lambda Memory — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Database Performance: Caches, Proxies And Capacity Units — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Network Performance: Peering, Transit Gateway And The Edge — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- The Performance Clinic: Five Slow Systems, Five Fixes — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Storage Cost: Classes, Lifecycle Rules And The Retrieval Trap — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Long-Stem Clinic: The Phrase That Decides — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- The Full Mock And Your Review Plan — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Blue-Green Deployments — Continuous Integration & Continuous Delivery Pipelines
- Canary Releases — Continuous Integration & Continuous Delivery Pipelines
- Regions, Availability Zones and Reliability — Cloud Infrastructure & Hyperscaler Architecture
- Comparing Compute Options — Cloud Infrastructure & Hyperscaler Architecture
- Object, Block, and File Storage — Cloud Infrastructure & Hyperscaler Architecture
- S3: Permissions, Tiers, and Policies — Cloud Infrastructure & Hyperscaler Architecture
- Elastic Block Store (EBS) — Cloud Infrastructure & Hyperscaler Architecture
- NoSQL and DynamoDB — Cloud Infrastructure & Hyperscaler Architecture
- Observability with CloudWatch — Cloud Infrastructure & Hyperscaler Architecture
- Metrics, Logs, and Alarms — Cloud Infrastructure & Hyperscaler Architecture
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Data Centre That Went Dark — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Disaster Recovery Sites: Hot, Warm & Cold — Cybersecurity Fundamentals, GRC & Cryptography
- Deployment Strategies: Rolling Updates — DevOps & Site Reliability Engineering
- Deployment Strategies: Canary Releases — DevOps & Site Reliability Engineering
- Service Level Indicators — DevOps & Site Reliability Engineering
- Percentiles Rather Than Averages — DevOps & Site Reliability Engineering
- Async and Await: Managing Concurrency — Modern .NET Core, C# & Enterprise Microservices
- How Browsers Work: Navigation, DNS, TCP, TLS And Request Timing — Frontend Architecture & Web Platform
- Cache Storage API: Offline Strategies And Cache-First Patterns — Frontend Architecture & Web Platform
- API Gateway & BFF Patterns: Backend-for-Frontend — Production Full-Stack Engineering
- Optimistic UI Architecture: Cache & Rollback Protocols — Production Full-Stack Engineering
- Bidirectional WebSockets: Reconnection & Heartbeats — Production Full-Stack Engineering
- Distributed Cache Invalidation: Redis & Tagged Purging — Production Full-Stack Engineering
- Distributed Tracing: OpenTelemetry & Traceparent — Production Full-Stack Engineering
- Continuous Delivery: Blue-Green & Canary Pipelines — Production Full-Stack Engineering
- Acknowledgements: What Acks Means For Durability — Apache Kafka & Event Streaming
- Batching, Linger And Throughput — Apache Kafka & Event Streaming
- Server Components: Async Execution & Constraints — Next.js 15 & Modern App Router Architecture
- Streaming SSR: Progressive HTML & Chunked Transfers — Next.js 15 & Modern App Router Architecture
- Async Data Fetching: Parallel Requests & Preloading — Next.js 15 & Modern App Router Architecture
- Optimistic UI: useOptimistic Rollbacks & Transitions — Next.js 15 & Modern App Router Architecture
- Runtime Environments: Node.js vs Edge Runtime Choice — Next.js 15 & Modern App Router Architecture
- Instrumentation: OpenTelemetry & Performance Spans — Next.js 15 & Modern App Router Architecture
- Poll & Check Phases: I/O Callbacks & setImmediate — Node.js Internals & Backend Services
- Native Compression: node:zlib & Streaming Codecs — Node.js Internals & Backend Services
- Diagnostic Profiling: CPU Profiler & Heap Snapshots — Node.js Internals & Backend Services
- Metric Types: Summaries And Histograms — Full-Stack Observability: Metrics, Tracing & Logging
- Why A Histogram Beats An Average — Full-Stack Observability: Metrics, Tracing & Logging
- The Prometheus Exposition Format — Full-Stack Observability: Metrics, Tracing & Logging
- Dashboards That Answer Questions — Full-Stack Observability: Metrics, Tracing & Logging
- OpenTelemetry: The Vendor Neutral Path — Full-Stack Observability: Metrics, Tracing & Logging
- Bringing The Three Signals Together — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting On Symptoms Versus Causes — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting Rules And The For Duration — Full-Stack Observability: Metrics, Tracing & Logging
- Service Level Indicators — Full-Stack Observability: Metrics, Tracing & Logging
- Six Minutes Nobody Saw: Prometheus In One Outage — Prometheus Certified Associate (PCA) Exam Prep
- Forty Seconds At Two: Push, Pull And Service Discovery — Prometheus Certified Associate (PCA) Exam Prep
- Ninety Nine Percent Of What? SLIs, SLOs And SLAs — Prometheus Certified Associate (PCA) Exam Prep
- Off By Five: What Prometheus Is Not For — Prometheus Certified Associate (PCA) Exam Prep
- Fundamentals Sprint: Nine Calls On The Server — Prometheus Certified Associate (PCA) Exam Prep
- Under A Second, On Average: Histograms — Prometheus Certified Associate (PCA) Exam Prep
- PromQL Sprint: Ten Queries At Minute Twenty Two — Prometheus Certified Associate (PCA) Exam Prep
- Thirty Second Checkouts Nobody Saw: Client Libraries — Prometheus Certified Associate (PCA) Exam Prep
- Milliseconds Versus Seconds: Naming Metrics And Labels — Prometheus Certified Associate (PCA) Exam Prep
- Instrumentation Sprint: Sixteen Calls On Code And Exporters — Prometheus Certified Associate (PCA) Exam Prep
- Forty One Buzzes: Routing Alerts With Alertmanager — Prometheus Certified Associate (PCA) Exam Prep
- Twelve Green Panels: Dashboarding Basics — Prometheus Certified Associate (PCA) Exam Prep
- Seventy Seven Percent, Book Now? The Full Mock — Prometheus Certified Associate (PCA) Exam Prep
- One Week, Three Objectives: Final Review — Prometheus Certified Associate (PCA) Exam Prep
- useOptimistic: Instantaneous UI Updates with Rollback — React 19 & Modern Component Architecture
- Client-Side Data Fetching Architectures with Suspense — React 19 & Modern Component Architecture
- The N Plus One Problem — PostgreSQL, Indexing & Relational Data Modeling
- What A Design Is For, And How To Run A Design Discussion — Scalable System Design & Distributed Architecture
- Requirements: The Constraints That Shape The System — Scalable System Design & Distributed Architecture
- Latency Numbers Every Engineer Should Know — Scalable System Design & Distributed Architecture
- Replication, Failover And Leader Election — Scalable System Design & Distributed Architecture
- Consistency, And The Truth About CAP — Scalable System Design & Distributed Architecture
- Caching Strategies And Topology — Scalable System Design & Distributed Architecture
- Queues, Events And Asynchronous Work — Scalable System Design & Distributed Architecture
- Load Balancing And The Failure Modes It Introduces — Scalable System Design & Distributed Architecture
- Timeouts, And Avoiding Cascaded Failure — Scalable System Design & Distributed Architecture
- Monolith Versus Services: The Honest Trade Offs — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
- Capacity Planning As A Design Input — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Load Testing: Percentiles Over Means — Automated Testing, TDD & Quality Engineering
References
- Google SRE book: Monitoring distributed systems
- MDN: Latency
- AWS: What is latency?
- Google SRE book: Service Level Objectives
- ASP.NET Core best practices
- MDN: Understanding latency
- Dean and Barroso: The Tail at Scale (Google Research)
Certification courses
- AWS AIP-C01 course and practice exam: AI engineering with LLM apps, RAG and agents
- AWS Cloud Practitioner course and practice exam: CLF-C02
- AWS Solutions Architect Associate course and practice exam: SAA-C03
- CCDAK and CCAAK course and practice exams: Confluent Apache Kafka certifications
- PCA course and practice exam: Prometheus Certified Associate