change data capture
Reading every insert, update and delete from a database's own change log and turning them into a stream of events.
In more detail
Because it reads the log the database already writes, CDC needs no changes to application code and captures every committed change in order. Tools such as Debezium publish the changes to Kafka topics.
Related terms
- write-ahead log — The log where PostgreSQL records every change before applying it to the data files, so it can recover after a crash.
- logical decoding — The PostgreSQL feature that turns write-ahead log entries into readable row changes, such as inserts and updates with their values.
- replication slot — A PostgreSQL object that remembers how far a consumer of the change stream has read, so no changes are lost between reads.
- Debezium — An open-source change data capture platform that reads database change logs and publishes each change as an event to Kafka.
- projection — A copy of data reshaped for one purpose, such as a search index or cache, built from the source of truth.
- search index — A separate store built for text search, filters and relevance ranking, fed from the source of truth.
- transactional outbox — Saving an event into an outbox table in the same database transaction as the business change, then having a separate worker publish it.
Taught in
- Twelve Lines Instead Of Four Hundred — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Change Data Capture In Principle — Apache Kafka & Event Streaming
- The Storage Decision: Key Value, Search And Blob — Scalable System Design & Distributed Architecture