document ingestion
The offline stage of RAG that splits documents into chunks, embeds them and stores them with metadata in a search index.
In more detail
Name each chunk by a hash of its content so re-ingestion becomes a diff: embed what is new, delete what is stale and leave the rest alone. Test the ingester like any other code, since everything downstream depends on it.
Related terms
- chunk (retrieval) — A passage cut from a document for indexing, and the unit that retrieval returns and the prompt includes.
- embedding vector — A list of numbers that represents a piece of text, or an image, so that items with similar meaning lie close together.
- retrieval-augmented generation — Answering a question by first retrieving relevant passages from your own documents and adding them to the prompt, so the model answers from them.
Taught in
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
- How RAG Works And When To Use It — AI Engineering: LLM Apps, RAG & Agents
- Chunking And Ingestion — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents