red teaming
Deliberately attacking your own AI system, by hand or with automated attacks, to find failures before real attackers or users do.
In more detail
For LLM apps it covers prompt injection, jailbreaks, data leaks and harmful output. Turn every finding into a regression case in your eval set.
Related terms
- prompt injection — An attack in which text a model reads overrides or redirects the instructions it was given, because instructions and data share one channel.
- AI eval — A repeatable test of an AI feature: a set of inputs, a way to grade each output, and a score you track across changes.
Taught in
- Threat-Model An LLM App — AI Engineering: LLM Apps, RAG & Agents