deduplication

Detecting and discarding repeated copies of the same message or request, usually by remembering identifiers already processed.

In more detail

The identifier must come from the sender, such as a delivery or event identifier, and the record of seen identifiers needs a documented lifetime. It is how systems turn at-least-once delivery into effectively-once processing.

Examples

A deduplication table of processed eventId values with a unique constraint.
The course keeps a seen-set of event IDs in a file to skip duplicate Kafka records.
Next.js deduplicates identical fetch calls during one render, and React cache does the same for any function you wrap.

Related terms

Taught in

References

Certification courses