attention budget

The idea that a model's attention is finite, so every token in the context competes with the rest and adding more can lower quality.

In more detail

Anthropic uses the phrase to explain why a bigger window is not a free one. Plan a context like a budget: fixed costs for the system prompt, tools and a reserve for the reply, then the most useful flexible material until the budget is spent.

Related terms

Taught in

References

Certification courses