Tokens & Tokenization

Gen AI · Foundations of Generative AI

Theory

LLMs do not read words; they read tokens. A tokenizer splits text into pieces from a fixed vocabulary (a whole common word like "cat", part of a word like "token" + "ization", a space plus a word, or a punctuation mark) and maps each piece to an integer ID that the model computes with.

In English a token is roughly four characters, or about three-quarters of a word, but this varies. Rare words, code, long numbers and many non-English scripts, including Hindi, often split into more tokens than their length suggests. Tokens matter because context windows, rate limits and usage billing are all counted in tokens, not words or characters.

Why we use it

Counting in tokens lets you predict whether a prompt fits the context window, estimate cost and explain odd behaviour, such as a model miscounting letters in a word it actually sees as two chunks.

When to use it

Think in tokens whenever you size prompts, chunk documents for RAG, set a maximum output length or plan a budget. Use the model provider's tokenizer for exact counts; the four-characters rule is only for quick estimates.

How it works

  1. Raw text: "Tokenization helps"
  2. Tokenizer looks up text pieces in its vocabulary
  3. Is the whole word in the vocabulary? Yes: Emit it as a single token. No: Split it into smaller known pieces.
  4. Map each piece to an integer token ID
  5. Model processes the sequence of IDs
  6. Generated IDs are decoded back into text

Example

"ChatGPT is helpful" → tokenizes roughly as:
["Chat", "G", "PT", " is", " helpful"]  (5 tokens)

"Tokenization" → ["Token", "ization"]  (2 tokens)

Estimate cost: 1,000 words of English ≈ 1,300–1,500 tokens.

Key points

Gen AI topics

Foundations of Generative AI

Prompt Engineering

Retrieval & Embeddings

Customizing Model Behavior

AI Agents & Tool Use

Multimodal & Applied AI

Ethics, Cost & Careers

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Generative AI for the Data Stack

LLMs, prompt engineering, RAG, embeddings, agents and the tools shaping modern AI products — explained practically. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Section {{ sec.num }}

{{ sec.label }}

Gen AI  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

💡

{{ current.tldr }}

Explained

{{ para }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
See It In Action
{{ current.example }}
Key Takeaways — click to check off
{{ pt.mark }} {{ pt.text }}

Share this topic