Apache Kafka & Event Streaming

Producers, Consumers, Guarantees, Schemas & the Outbox Pattern

Kafka as a working engineer meets it: a replicated log, not a queue. 7 modules, 35 lessons, five to nine minutes each, every one run against a real Kafka-compatible broker (Redpanda) in Docker: topics, partitions and offsets; producers with keys, batching and acknowledgements; consumer groups, commits and rebalancing; the delivery guarantees you can actually get, including idempotent producers, transactions and dead letter topics; schemas and their evolution; the outbox pattern, change data capture and log compaction; and operating a three-broker cluster through a broker failure.

Intermediate · 3 hours · 35 lessons · 174 labs

Who it is for

Intermediate level.

For backend engineers who build or support services that send and read events through Kafka, and want to know what it really guarantees.

What you don't need yet:

What helps:

Before you start:

Take first:

Take next:

Learning paths:

Nobody needs to know everything before they start. Each course builds on the one before, so a learning path takes you there step by step.

Labs

174 labs: you write the code, it runs, and graded labs check the result. The outline lists each lesson's labs.

What is free

Start free: First module free with an account. Pro: Every course, lab, handbook, practice exam and certificate.

Course outline

The Log: Topics, Partitions And Offsets

  1. Why A Log And Not A Queue (4 labs): You can explain why Kafka uses an append-only log rather than a delete-on-read queue, produce records to a topic, and consume them independently from two separate sessions.
  2. Starting A Broker And Reading Its Metadata (4 labs): You can start a Redpanda broker with the correct networking flags, read its cluster state with rpk cluster info and rpk cluster health, and explain why the advertised address must be the container name.
  3. Topics, Partitions And Where A Record Lands (5 labs): You can create a topic with multiple partitions, produce keyed records and confirm they always land on the same partition, and explain why unkeyed records do not carry ordering guarantees.
  4. Offsets: Position, Not Acknowledgement (6 labs): You can consume a topic from a specific offset, demonstrate that two independent consumers see the same records at the same offsets, and explain why the offset belongs to the record and not to the reader.
  5. Retention: Time, Size And Why Data Stays (5 labs): You can set time-based and size-based retention on a topic with rpk topic alter-config, verify the settings with rpk topic describe -c, and explain why retention deletes at segment boundaries rather than individual records.

Producers: Keys, Batches And Acks

  1. A Producer In Python (4 labs): You can write a KafkaProducer that sends records to a topic, block on the returned future, and read the partition and offset from the metadata the broker returns.
  2. Keys And Ordering Per Partition (5 labs): You can produce records with keys, explain why the same key always lands in the same partition, and show that ordering is guaranteed within a partition but not across partitions.
  3. Acknowledgements: What Acks Means For Durability (5 labs): You can set acks on a KafkaProducer to zero, one, or all, explain what each level waits for, and describe what min-insync-replicas does when a write requires confirmation from more replicas than are currently available.
  4. Batching, Linger And Throughput (5 labs): You can configure batch-size, linger-ms, and compression-type on a KafkaProducer and explain what each setting trades against to increase throughput.
  5. Idempotent Producers And Retries (5 labs): You can configure enable-idempotence on a KafkaProducer, explain what sequence numbers the broker uses to detect duplicate produce requests, and demonstrate that without idempotence a simulated retry produces duplicate records.

Consumers And Consumer Groups

  1. A Consumer In Python (5 labs): You can write a KafkaConsumer in Python that reads from the beginning of a topic, prints partition, offset, key and value for every record, and exits automatically when the topic goes quiet.
  2. Consumer Groups And Partition Assignment (5 labs): You can explain how Kafka assigns partitions across a consumer group, run a consumer that reports its assignment, and read rpk group describe to verify committed offsets per partition.
  3. Committing Offsets: Auto, Manual And Lag (6 labs): You can configure a KafkaConsumer to commit offsets manually, observe the lag a consumer group carries before committing, and confirm that a second run resumes from the committed position.
  4. Rebalancing And Its Cost (5 labs): You can explain what triggers a rebalance, configure session and heartbeat timeouts, and describe what happens to a consumer that exceeds max-poll-interval-milliseconds.
  5. At Most Once, At Least Once (7 labs): You can implement at-least-once and at-most-once consumers, simulate a crash mid-batch, and explain which records are reprocessed or lost in each scenario.

Delivery Guarantees And Transactions

  1. Duplicates Are The Default (4 labs): You can name the three events that produce a duplicate in a Kafka pipeline and explain why at-least-once delivery is the deliberate default.
  2. Idempotent Consumers And Deduplication Keys (4 labs): You can implement a consumer that skips duplicate events using an event identifier in the record value, and explain when a deduplication table is necessary versus when natural idempotence is sufficient.
  3. Transactions And Exactly Once Semantics (5 labs): You can write a transactional Kafka producer that commits and aborts, explain what read-committed isolation means for a consumer, and describe why a read-committed consumer sees zero records from an aborted transaction.
  4. Poison Messages And Dead Letter Topics (5 labs): You can implement a consumer that routes malformed records to a dead letter topic with an error header, and explain why a single poison message blocks an entire partition if the exception is not caught.
  5. Retries With Backoff Topics (5 labs): You can implement a retry-topic pattern that routes a failing record through numbered tiers before landing it in a dead letter queue, and explain how an attempt counter in the value makes routing deterministic.

Schemas, Serialisation And Evolution

  1. Bytes On The Wire: JSON, Avro And Protobuf (5 labs): You can produce the same event as a JSON record and as a compact binary struct, explain why JSON is schema-less while Avro and Protobuf are schema-full, and interpret the byte-count difference between the two approaches.
  2. The Schema Registry (6 labs): You can register a JSON schema with a schema registry, list subjects, retrieve a schema by version, and explain the TopicNameStrategy naming convention and the Confluent wire format magic byte.
  3. Compatibility Modes (5 labs): You can set a compatibility level on a schema registry subject, demonstrate that an incompatible schema change is rejected, register a backward-compatible addition, and explain the practical difference between BACKWARD, FORWARD, and FULL modes.
  4. Evolving An Event Without Breaking Consumers (6 labs): You can produce a v-one event and a v-two event with an extra optional field to the same topic, show that the v-one consumer code processes both without error, and explain why renaming a field is a breaking change while adding one is not.
  5. Headers, Envelopes And Event Metadata (5 labs): You can attach record headers to a Kafka message in kafka-python, design an envelope structure with event ID, type, version, occurred-at, and payload, and consume both headers and envelope fields in a deterministic printed output.

Event-Driven: Outbox, CDC, Compaction

  1. Dual Writes And Why They Lose Data (6 labs): You can explain why writing to a database and a Kafka topic in sequence is not atomic, demonstrate both failure modes with runnable programs, and describe why neither write order eliminates the consistency risk.
  2. The Transactional Outbox (7 labs): You can implement a transactional outbox that writes a business row and an event record in a single database transaction, build a relay that publishes unpublished rows to Kafka and marks them published, and confirm that a second relay run produces no duplicates.
  3. Change Data Capture In Principle (4 labs): You can start a Postgres instance with logical replication enabled, create a replication slot using the test decoding plugin, and read the change stream produced by insert and update statements to explain how a connector such as Debezium turns those changes into Kafka events.
  4. Log Compaction: A Topic As A Table (6 labs): You can create a compacted Kafka topic, produce multiple updates for the same key, fold the full log into a key-value table in Python, and explain why compaction converges the log to the same result over time.
  5. Materialised Views From A Stream (5 labs): You can produce a stream of order events, write a consumer that folds the stream into per-customer totals and prints a sorted table, and rebuild the same view from offset zero using a fresh consumer group to demonstrate that the stream is the source of truth.

Operating A Cluster

  1. Three Brokers: Replication And Leaders (3 labs): You can start a three-node Redpanda cluster, create a topic with replication factor three, and read the leader and replica assignments from the partition table.
  2. In-Sync Replicas And Minimum ISR (4 labs): You can read the in-sync replica set for a topic, set the minimum in-sync replica count, and explain why that number gates whether acks-all writes succeed.
  3. Losing A Broker (5 labs): You can stop a broker, confirm a new leader is elected and the topic stays available, restart the broker and verify it rejoins, and explain why two broker failures block acks-all writes.
  4. Watching Lag And Sizing Partitions (5 labs): You can measure consumer group lag with rpk group describe, add partitions to an existing topic, and explain why partition count is a ceiling on consumer-side parallelism.
  5. When Kafka Is The Wrong Tool (3 labs): You can list three scenarios where a message queue, a database table, or direct RPC is a better fit than Kafka, and explain the operational cost of running a broker cluster.

Certification courses