prompt injection
An attack in which text a model reads overrides or redirects the instructions it was given, because instructions and data share one channel.
A successful injection makes the model act for the attacker with every tool and data source it has.
In more detail
It is first in the OWASP Top 10 for LLM Applications, and no prompt reliably prevents it. Mark untrusted text and keep it away from powerful tools to make attacks less likely, then enforce the decisive control outside the model: check every tool call against where data would go.
Related terms
- direct prompt injection — Prompt injection typed by the user of the application, who tries to make the model ignore its instructions.
- indirect prompt injection — Prompt injection hidden in content the model reads on someone else's behalf, such as a web page, email, ticket, document or tool result.
- lethal trifecta — Simon Willison's name for an agent that has access to private data, exposure to untrusted content and a way to communicate externally, which together let injected text steal data.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- Anatomy Of A Prompt — AI Engineering: LLM Apps, RAG & Agents
- Threat-Model An LLM App — AI Engineering: LLM Apps, RAG & Agents
- Prompt Injection — AI Engineering: LLM Apps, RAG & Agents
- Excessive Agency — AI Engineering: LLM Apps, RAG & Agents
- Images, Audio And Speech — AI Engineering: LLM Apps, RAG & Agents
- Working With AI Coding Assistants — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents
References
- OWASP GenAI Security Project: LLM01 Prompt Injection
- Greshake et al. (2023): Not what you've signed up for: indirect prompt injection