token (language model)
The unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence from the tokenizer's vocabulary.
In more detail
Context windows, rate limits and prices are all measured in tokens, not words or characters. English averages roughly four characters a token, but code, numbers and many other languages take more. Count with the provider's own tokenizer or token-counting endpoint before relying on a figure, because each model family splits text differently.
Example
Streaming each token to the browser from a Route Handler makes AI answers appear as they are generated.Related terms
- AI — Computer systems that perform tasks normally needing human intelligence, such as understanding language, recognising images or making predictions.
- ReadableStream — A browser object that delivers data in chunks as it arrives, such as a fetch response.body, instead of all at once at the end.
Taught in
- Streaming Endpoints: Server-Sent Events & NDJSON Feeds — Next.js 15 & Modern App Router Architecture