producer-consumer
A pattern where a producer creates data or work items and a consumer receives and processes them, usually with a queue or buffer between the two.
In more detail
The queue lets producer and consumer run at different speeds; problems start when the producer is persistently faster, which is what backpressure solves. Background worker pools, print spoolers and streaming pipelines follow this pattern. The consumer end of a data flow is also called the sink, and data flows downstream from producer to consumer.
Related terms
- queue — A collection where items leave in the order they arrived: first in, first out, like a line at a shop counter.
- FIFO — First in, first out: the ordering rule of a queue, where the oldest item is always the next to leave.
- backpressure — A slow consumer signalling a fast producer to pause, so data does not pile up in memory faster than it can be handled.
- stream — Data that flows between a program and a source or destination piece by piece, rather than all at once.
Taught in
- Stacks And Queues: Last In Or First In — Algorithms & Data Structures for Working Engineers
- Writable Streams: Drain Events & highWaterMark — Node.js Internals & Backend Services
- Backpressure Engineering: Handling Mismatched I/O — Node.js Internals & Backend Services
- Stream Pipelines: pipeline, AbortSignal & Teardown — Node.js Internals & Backend Services