Tokens & Tokenization
Gen AI · Foundations of Generative AI
Theory
LLMs do not read words; they read tokens. A tokenizer splits text into pieces from a fixed vocabulary (a whole common word like "cat", part of a word like "token" + "ization", a space plus a word, or a punctuation mark) and maps each piece to an integer ID that the model computes with.
In English a token is roughly four characters, or about three-quarters of a word, but this varies. Rare words, code, long numbers and many non-English scripts, including Hindi, often split into more tokens than their length suggests. Tokens matter because context windows, rate limits and usage billing are all counted in tokens, not words or characters.
Why we use it
Counting in tokens lets you predict whether a prompt fits the context window, estimate cost and explain odd behaviour, such as a model miscounting letters in a word it actually sees as two chunks.
When to use it
Think in tokens whenever you size prompts, chunk documents for RAG, set a maximum output length or plan a budget. Use the model provider's tokenizer for exact counts; the four-characters rule is only for quick estimates.
How it works
- Raw text: "Tokenization helps"
- Tokenizer looks up text pieces in its vocabulary
- Is the whole word in the vocabulary? Yes: Emit it as a single token. No: Split it into smaller known pieces.
- Map each piece to an integer token ID
- Model processes the sequence of IDs
- Generated IDs are decoded back into text
Example
"ChatGPT is helpful" → tokenizes roughly as:
["Chat", "G", "PT", " is", " helpful"] (5 tokens)
"Tokenization" → ["Token", "ization"] (2 tokens)
Estimate cost: 1,000 words of English ≈ 1,300–1,500 tokens.Key points
- A token is roughly 4 characters or about 0.75 of an English word
- Context limits, rate limits and usage billing are all measured in tokens
- Rare words, code, numbers and non-English scripts often use more tokens
- Spaces and punctuation are usually part of tokens too
- Use the provider's tokenizer for exact counts; ratios are only estimates