document ingestion

The offline stage of RAG that splits documents into chunks, embeds them and stores them with metadata in a search index.

In more detail

Name each chunk by a hash of its content so re-ingestion becomes a diff: embed what is new, delete what is stale and leave the rest alone. Test the ingester like any other code, since everything downstream depends on it.

Related terms

Taught in

References

Certification courses