embedding quantization
Storing embedding values in fewer bits, such as one byte (int8) or one bit (binary) per dimension, to save memory and speed up search.
In more detail
Int8 is four times smaller than 32-bit floats and costs almost nothing in recall. Binary is 32 times smaller but coarser, so use it to find a fast shortlist and rescore that shortlist with fuller vectors.
Related terms
- embedding dimensions — The number of values in each embedding vector, fixed by the model, which drives storage size and search cost.
- recall at k — The share of the relevant passages for a query that appear in the top k results.
- vector index — A data structure that makes nearest-neighbour search over many vectors fast, usually by giving up exactness.
Taught in
- What An Embedding Is — AI Engineering: LLM Apps, RAG & Agents
- Embedding Models — AI Engineering: LLM Apps, RAG & Agents
- Vector Indexes And Databases — AI Engineering: LLM Apps, RAG & Agents
- How RAG Works And When To Use It — AI Engineering: LLM Apps, RAG & Agents
References
- Hugging Face: Binary and Scalar Embedding Quantization (2024-03-22)
- pgvector README: distance operators