KafkaConsumer
The kafka-python class that reads records from Kafka topics, joins consumer groups and commits offsets.
In more detail
Iterating over a KafkaConsumer calls poll behind the scenes and yields one ConsumerRecord at a time. Settings such as group_id, auto_offset_reset, enable_auto_commit and consumer_timeout_ms control where it starts, when it saves progress and when it stops.
Example
for msg in KafkaConsumer('events', group_id='readers'): print(msg.offset)Related terms
- kafka-python — An open-source Python client library for Kafka that provides KafkaProducer, KafkaConsumer and KafkaAdminClient.
- consumer — A program that receives and processes messages or events, or more broadly any client that uses an API.
- ConsumerRecord — The object kafka-python yields for each record a consumer reads, with fields such as topic, partition, offset, key, value and headers.
- poll loop — The loop in which a consumer repeatedly asks the broker for new records and processes them.
- group.id — The consumer setting that names the consumer group, under which the broker stores committed offsets.
Taught in
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Consumer In Python — Apache Kafka & Event Streaming
- Consumer Groups And Partition Assignment — Apache Kafka & Event Streaming
- Committing Offsets: Auto, Manual And Lag — Apache Kafka & Event Streaming
- Rebalancing And Its Cost — Apache Kafka & Event Streaming
- At Most Once, At Least Once — Apache Kafka & Event Streaming