rebalance
The process in which a consumer group redistributes partitions among its members after one joins, leaves or is judged dead.
In more detail
While partitions move, their processing pauses. Rebalances are triggered by a new member, a clean leave, a missed session timeout, a slow poll loop or a change in partition count.
Related terms
- eager rebalancing — A rebalance style in which every consumer gives up all its partitions before any are reassigned.
- cooperative rebalancing — A rebalance style in which consumers keep the partitions that are not moving and only hand over the ones being reassigned.
- session.timeout.ms — The consumer setting for how long the group coordinator waits without a heartbeat before declaring the consumer dead and rebalancing.
- max.poll.interval.ms — The consumer setting for the longest gap allowed between two poll calls before the consumer is treated as stuck and removed from the group.
- group coordinator — The broker that manages one consumer group, tracking its members, their heartbeats and its committed offsets.
- consumer group — A set of consumers sharing one group ID that divide a topic's partitions between them so each partition is read by only one member at a time.
- partition — One slice of a larger dataset or message stream, owned and processed independently of the others.
Taught in
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twenty Seconds Of Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The New Broker That Did Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty One Under-Replicated Partitions — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Consumer Groups And Partition Assignment — Apache Kafka & Event Streaming
- Rebalancing And Its Cost — Apache Kafka & Event Streaming
- Watching Lag And Sizing Partitions — Apache Kafka & Event Streaming
- Worked Design Three: Scaling The Metrics Pipeline — Scalable System Design & Distributed Architecture
References
- Apache Kafka operations: consumer rebalance protocol
- kafka-python KafkaConsumer
- Apache Kafka documentation