Context Window
Gen AI · Foundations of Generative AI
Theory
The context window is the maximum number of tokens a model can consider in one request, counting everything: the system prompt, conversation history, pasted documents, tool results and the answer it generates. It works like short-term working memory. Anything outside the window is invisible, and a model does not remember earlier chats unless the app sends that history again.
A bigger window lets you include whole reports or long code files, but more is not automatically better. Models can miss details buried in the middle of a very long prompt, and every extra token adds cost and latency. If a request exceeds the window it is rejected or older content is cut, so apps trim history, summarise, or retrieve only the relevant parts.
Why we use it
It is a hard budget for every request. Knowing it tells you how much history, how many documents and how long an answer you can fit, and explains why a chatbot suddenly 'forgets' early messages.
When to use it
Plan around it when building chat history, RAG prompts or document summarisation. If your data is far larger than the window, retrieve only the relevant chunks or summarise in stages instead of pasting everything in.
How it works
- Gather system prompt, history, documents and question
- Count the tokens in every input piece
- Reserve room for the expected answer length
- Does input + answer reserve fit? Yes: Send the request. No: Trim history or retrieve fewer chunks.
- Model reads only what is inside the window
- Answer is generated within the remaining budget
Example
Model context window: 128,000 tokens
Your request:
- System prompt: 200 tokens
- Uploaded PDF: 40,000 tokens
- Conversation history: 2,000 tokens
- Your question: 50 tokens
Total input: 42,250 tokens — well within the 128k limit, leaving room for a long generated answer.Key points
- Counts input and output together: prompt, history, documents and the answer
- The model has no memory between requests; the app resends history
- Details in the middle of very long prompts are easier for the model to miss
- Put key instructions and facts near the start or end of long prompts
- Exceeding the limit truncates or rejects the request