ingestion pipeline
The chain of steps that takes a raw log line from its source to a searchable event: collect, buffer, parse, enrich, index and store.
In more detail
Collectors receive events by agent, syslog or API pulls. A buffer queue absorbs bursts and outages, then parsers extract fields, enrichment adds context, and the indexer stores events by time. Evidence is lost at each step if transport drops messages, parsers fail silently or time zones are wrong.
Related terms
- SIEM — Security information and event management: a system that collects logs and alerts from many sources, correlates them and supports investigation.
- parsing — Breaking a raw log line into named fields, such as time, user and source address, that rules and searches can use.
- normalisation — Renaming fields from different log sources onto one agreed set of names, so one query works across all of them.
- index — A separate, sorted structure a database keeps alongside a table so it can find matching rows without reading them all.
- log collector — A program that gathers log events from a system and sends them on to the SIEM.
Taught in
- SIEM Architecture: Ingestion Pipelines & Indexing — Security Operations & Threat Hunting (SOC)
- Threat Intelligence Platforms & STIX/TAXII Ingestion — Security Operations & Threat Hunting (SOC)
- Log Parsing, Schema Normalization (OCSF/ECS) & Extraction — Security Operations & Threat Hunting (SOC)