broker
A server that receives messages from producers, stores them and delivers them to consumers, such as RabbitMQ or Apache Kafka.
In more detail
Brokers decouple services in time: a producer can publish even when the consumer is down. They typically guarantee at-least-once delivery, so consumers must handle duplicates.
Example
In the course each Kafka broker is a Redpanda container named after the lesson.Related terms
- Kafka cluster — A group of brokers that share the same topics and metadata and act as one Kafka service.
- partition leader — The replica of a partition that handles all reads and writes for it; the other replicas copy from it.
- replication — Keeping copies of data in more than one place, updated automatically, so one failure does not lose or block it.
- Redpanda — A streaming data platform, written in C++, that speaks the Kafka protocol so Kafka clients and tools work against it unchanged.
- message queue — A store that holds messages from producers until consumers are ready to process them, decoupling the rate of arrival from the rate of work.
- RabbitMQ — A widely used open-source message broker that routes messages into queues for consumers.
- Apache Kafka — An open-source platform that stores streams of events in an ordered, replayable log and delivers them to many consumers.
- publish-subscribe — A messaging pattern where senders publish messages to a named channel and every subscriber listening on that channel receives a copy.
- append-only log — A data structure where new entries are only ever added at the end and existing entries are never changed in place.
- at-least-once delivery — A delivery guarantee under which every message arrives, but some may arrive more than once.
Taught in
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Acknowledged Is Not Always Safe — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twenty Seconds Of Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Works On A Laptop, Fails In Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Hours Of Scans At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Green Tests, Red Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Sink That Died On Record Nine — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Setting That Came Back — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The New Broker That Did Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Certificate That Expired On A Sunday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where Is The Password Kept? — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Allowed To Read, Refused Anyway — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty One Under-Replicated Partitions — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Data Centre That Went Dark — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Full Disks On Black Friday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Upgrade That Could Not Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Asynchronous Event Processing: Outbox Pattern & Queues — Production Full-Stack Engineering
- Full-Stack System Verification: Capstone Integration — Production Full-Stack Engineering
- Why A Log And Not A Queue — Apache Kafka & Event Streaming
- Starting A Broker And Reading Its Metadata — Apache Kafka & Event Streaming
- Three Brokers: Replication And Leaders — Apache Kafka & Event Streaming
- Losing A Broker — Apache Kafka & Event Streaming
- When Kafka Is The Wrong Tool — Apache Kafka & Event Streaming
- Idempotency, And Exactly Once As A Fiction — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
References
- Apache Kafka: main concepts and terminology
- Redpanda architecture
- RabbitMQ: Documentation
- Apache Kafka: Introduction
- Apache Kafka documentation
- RabbitMQ documentation: Queues