deduplication
Detecting and discarding repeated copies of the same message or request, usually by remembering identifiers already processed.
In more detail
The identifier must come from the sender, such as a delivery or event identifier, and the record of seen identifiers needs a documented lifetime. It is how systems turn at-least-once delivery into effectively-once processing.
Examples
A deduplication table of processed eventId values with a unique constraint.The course keeps a seen-set of event IDs in a file to skip duplicate Kafka records.Next.js deduplicates identical fetch calls during one render, and React cache does the same for any function you wrap.Related terms
- idempotent consumer — A message or webhook handler that produces the same result whether it sees a message once or several times.
- idempotency key — A unique value a client sends with a non-idempotent request, such as a create, so the server can recognise a retry and return the original result instead of acting twice.
- at-least-once delivery — A delivery guarantee under which every message arrives, but some may arrive more than once.
- check then act — Reading whether something exists and then acting on the answer as two separate steps, which races under concurrency.
- idempotency — The property of an operation that gives the same result whether you run it once or many times, because a repeat run finds nothing left to do.
- unique constraint — A constraint that no two rows may have the same value in a column or combination of columns.
- event ID — A unique identifier the producer puts in each event so consumers can recognise the same event if it arrives twice.
- duplicate record — A second copy of a record that was meant to be written or processed only once.
- Request Memoization — The first Next.js caching tier: identical fetch calls made while rendering one request are run only once.
- React cache — A React function that wraps another function so repeated calls with the same arguments during one server request reuse the first result.
- memoization — Remembering the result of a calculation for given inputs and reusing it when the same inputs come again, instead of recalculating.
Taught in
- Webhooks Versus Polling — Modern API Architecture: REST, GraphQL & gRPC
- Event-Driven Architecture And Queues — Modern API Architecture: REST, GraphQL & gRPC
- Decoupling: SQS, SNS, EventBridge And Step Functions — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Asynchronous Event Processing: Outbox Pattern & Queues — Production Full-Stack Engineering
- Idempotent Producers And Retries — Apache Kafka & Event Streaming
- Duplicates Are The Default — Apache Kafka & Event Streaming
- Idempotent Consumers And Deduplication Keys — Apache Kafka & Event Streaming
- Async Data Fetching: Parallel Requests & Preloading — Next.js 15 & Modern App Router Architecture
- Request Memoization: Automatic Deduplication Tiers — Next.js 15 & Modern App Router Architecture
- Metadata API: Dynamic OpenGraph & Head Management — Next.js 15 & Modern App Router Architecture
- Idempotency, And Exactly Once As A Fiction — Scalable System Design & Distributed Architecture
- Worked Design Two: A Global Chat System — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
References
- Amazon SQS: Standard queues (at-least-once delivery)
- Patterns of Distributed Systems: Idempotent Receiver
- Google Cloud Pub/Sub: Exactly-once delivery
- microservices.io: Idempotent consumer
- Enterprise Integration Patterns: idempotent receiver
- Next.js 15 docs: Caching