Apache Kafka
An open-source platform that stores streams of events in an ordered, replayable log and delivers them to many consumers.
In more detail
Unlike a classic queue, Kafka keeps events after they are read, so new consumers can replay history. It is built for very high throughput and is widely used as the backbone of event-driven systems.
Example
Kafka is one of the brokers an outbox relay can publish to.Related terms
- append-only log — A data structure where new entries are only ever added at the end and existing entries are never changed in place.
- topic — A named channel in a publish-subscribe messaging system that producers write to and any number of subscribers read from.
- offset — The permanent position number of a record inside its partition, starting at zero and increasing by one for each record.
- Redpanda — A streaming data platform, written in C++, that speaks the Kafka protocol so Kafka clients and tools work against it unchanged.
- broker — A server that receives messages from producers, stores them and delivers them to consumers, such as RabbitMQ or Apache Kafka.
- Kafka protocol — The binary request-and-response protocol over TCP that Kafka clients and brokers use to produce, fetch, commit offsets and read metadata.
- message queue — A store that holds messages from producers until consumers are ready to process them, decoupling the rate of arrival from the rate of work.
Taught in
- Compute Performance: Families, Placement And Lambda Memory — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Ingest And Transform: Streams, Lakes And Partitions — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- EventSource, EventBus, Sensor And Trigger — Certified Argo Project Associate (CAPA) Exam Prep
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Acknowledged Is Not Always Safe — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twenty Seconds Of Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Works On A Laptop, Fails In Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Blank Names After Every Deploy — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Hours Of Scans At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Green Tests, Red Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twelve Lines Instead Of Four Hundred — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Connector That Was Created Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Sink That Died On Record Nine — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Setting That Came Back — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The New Broker That Did Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Certificate That Expired On A Sunday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where Is The Password Kept? — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Allowed To Read, Refused Anyway — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty One Under-Replicated Partitions — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Data Centre That Went Dark — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Full Disks On Black Friday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Upgrade That Could Not Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Asynchronous Event Processing: Outbox Pattern & Queues — Production Full-Stack Engineering
- Why A Log And Not A Queue — Apache Kafka & Event Streaming
- Starting A Broker And Reading Its Metadata — Apache Kafka & Event Streaming
- Duplicates Are The Default — Apache Kafka & Event Streaming
- When Kafka Is The Wrong Tool — Apache Kafka & Event Streaming
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture