attention budget
The idea that a model's attention is finite, so every token in the context competes with the rest and adding more can lower quality.
In more detail
Anthropic uses the phrase to explain why a bigger window is not a free one. Plan a context like a budget: fixed costs for the system prompt, tools and a reserve for the reply, then the most useful flexible material until the budget is spent.
Related terms
- context engineering — Choosing and arranging everything a model reads on each call, from instructions and tools to history, memory and retrieved data, within a token budget.
- context rot — The fall in a model's accuracy as its input grows longer, even on simple tasks and well within the context window.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- Tokens And The Context Window — AI Engineering: LLM Apps, RAG & Agents
- From Prompts To Context — AI Engineering: LLM Apps, RAG & Agents
- Chunking And Ingestion — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
- Designing Tools A Model Can Use — AI Engineering: LLM Apps, RAG & Agents
- The Agent Loop — AI Engineering: LLM Apps, RAG & Agents