consumer lag
How many records a consumer group has not yet committed on a partition: the log end offset minus the committed offset.
In more detail
Lag near zero means the group keeps up; growing lag means consumers are too slow or stopped. rpk group describe shows lag per partition and in total. Adding consumers helps only up to the partition count.
Example
1,204,331 records behind at 4,000 a second is about five minutes.Related terms
- log end offset — The offset that the next record written to a partition will receive, one more than the offset of the newest record.
- committed offset — The offset a consumer group has saved on the broker for a partition, marking the next record it should read after a restart.
- rpk group describe — An rpk command that shows a consumer group's state, members and, per partition, the committed offset, log end offset and lag.
- parallelism — Doing several pieces of work at the same time instead of one after another.
- offset — The permanent position number of a record inside its partition, starting at zero and increasing by one for each record.
- queue depth — The number of messages currently waiting in a queue.
- ingest lag — The delay between a data point being produced and it becoming queryable.
Taught in
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Acknowledged Is Not Always Safe — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Sink That Died On Record Nine — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty One Under-Replicated Partitions — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Consumer Groups And Partition Assignment — Apache Kafka & Event Streaming
- Committing Offsets: Auto, Manual And Lag — Apache Kafka & Event Streaming
- Watching Lag And Sizing Partitions — Apache Kafka & Event Streaming
- What A Design Is For, And How To Run A Design Discussion — Scalable System Design & Distributed Architecture
- The Storage Decision: Key Value, Search And Blob — Scalable System Design & Distributed Architecture
- Replication, Failover And Leader Election — Scalable System Design & Distributed Architecture
- Partitioning, Sharding And Hot Keys — Scalable System Design & Distributed Architecture
- Eventual Consistency And What It Costs A Product — Scalable System Design & Distributed Architecture
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Worked Design Two: Scaling Chat Fan Out — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Worked Design Three: Scaling The Metrics Pipeline — Scalable System Design & Distributed Architecture