hallucination
Output from a model that is fluent and confident but false or unsupported, such as an invented fact, citation, function or package.
A believable wrong answer can be acted on by users or by code without anyone noticing.
In more detail
Language models predict plausible text, not verified facts, so they fill gaps with likely-sounding content. NIST calls it confabulation. Grounding answers in retrieved sources, checking citations in code, letting the model say it does not know and validating names against real registries all reduce the harm.
Related terms
- grounding — Making a model answer only from supplied sources, cite them, and refuse when they do not contain the answer.
- package hallucination — A coding model recommending a dependency that does not exist, which an attacker can then register with malicious code.
- faithfulness — An eval metric for RAG: the share of claims in an answer that the retrieved context supports.
Taught in
- When Context Fails — AI Engineering: LLM Apps, RAG & Agents
- Guardrails, Privacy And Governance — AI Engineering: LLM Apps, RAG & Agents
- Working With AI Coding Assistants — AI Engineering: LLM Apps, RAG & Agents
References
- NIST AI 600-1: Generative Artificial Intelligence Profile
- Spracklen et al., We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (USENIX Security 2025)