Ragas
An open-source library of evaluation metrics for RAG pipelines and agents, such as faithfulness, context precision, context recall and tool call accuracy.
In more detail
Many of its metrics use a model as judge, so validate them against your own labels like any judge. Its scores are most useful for comparing versions of the same pipeline.
Related terms
- faithfulness — An eval metric for RAG: the share of claims in an answer that the retrieved context supports.
- context precision — An eval metric for RAG retrieval: whether the relevant chunks were ranked above the irrelevant ones in what was retrieved.
- context recall — An eval metric for RAG retrieval: the share of the facts a reference answer needs that appear in the retrieved context.
- LLM-as-judge — Using a language model to grade another model's output against a rubric, for qualities that code cannot check.
Taught in
- Evaluating RAG And Agents — AI Engineering: LLM Apps, RAG & Agents