AI context windows explained: a developer's guide

TL;DR: A context window is the model's working memory. It's a fixed token budget for one request, and your system prompt, chat history, retrieved documents, tool results and reserved reply all come out of it. Size affects accuracy, cost and speed. When you run out, you need RAG, chunking or a sliding window.

A context window is the model's working memory. It's the total token budget an LLM can look at while writing a single reply.

Everything comes out of that one budget. Your system prompt. The chat history. Documents pulled in by search. Tool results. Even the space held back for the answer. Stanford HAI puts it plainly: once a conversation runs past the window, the model loses access to the earlier turns.

Here's the thing. Most teams treat window size as a spec to max out. It isn't. It's a budget to spend well.

This guide covers what a token actually is, why the limit exists, what breaks when you hit it, and the patterns that keep you inside it.

What is a context window, and what is a token?

Context windows are measured in tokens, not words. A token is a chunk of text. Common words like "the" or "run" are usually one token. Longer or rarer words get split into two or three.

A rough rule: 1 token is about 4 characters, or about 0.75 words in plain English.

That rule falls apart fast. Code, tables, and non-English text are all more token-dense. Punctuation and whitespace each count.

A quick example. Take the sentence *"The quarterly revenue report includes three charts and two appendices."* That's about 16 tokens. Write the same thing as a SQL query or a markdown table and it can cost 25 to 35 tokens.

Tools that count tokens for you:

• tiktoken.** OpenAI's open-source library. Fast, and easy to run locally before you send a request.
• Hugging Face tokenizers. Model-specific, for Llama, Mistral and other open-weight models. Counts differ from tiktoken on the same text.
• The OpenAI tokenizer playground. Browser-based, good for a quick check without writing code.

As Redis explains, a real production request has several parts, and they all draw on the same window:

• System prompt. Your instructions, persona, output rules.
• Chat history. Every earlier turn in the conversation.
• Retrieved documents. Passages injected by a search pipeline.
• Tool outputs. Results from function calls or API responses.
• Reserved output. Tokens held back for the reply.

That last one catches people out. Set `max_tokens=2000` on a model with a 16k window and you have 14,000 left for everything else. Not 16,000.

Why is there a limit at all?

The limit isn't arbitrary. It falls out of how transformers work.

Self-attention lets the model relate every token to every other token. That costs O(n²). Double the input and the attention work goes up four times. Alongside it, the KV cache grows in a straight line with length and has to fit in GPU memory.

So longer inputs mean more compute, bigger caches, and a slower first token.

What goes up when the window goes up:

• Cost. Quadratic attention means big windows cost real money per request.
• Latency. Both time-to-first-token and total generation time get worse.
• Memory. Cache size caps how many requests a deployment can serve at once.
• Reliability at the edges. Models see fewer very long examples in training, so quality can drop near the top of the advertised window.

Plenty of research is chipping away at this. Sparse attention, linear attention variants, recurrent memory layers and retrieval all try to break the link between useful context and quadratic cost.

How do the main models compare?

Windows have grown a lot. Some now run to a million tokens or more.

One warning before the table. Vendor numbers change every few months, and they differ between the API, the chat product and each SDK. Treat this as orientation, then check the current vendor docs before you design anything around it.

One nuance worth holding onto: the advertised window and the usable window are not the same number. Reserved output, platform-level injections and anything appended server-side all eat into what you actually get. If you're still choosing, our guide on how to choose an LLM walks through the trade-offs beyond window size.

What breaks when the window fills up?

Some failures are loud. The quiet ones are worse.

• Silent truncation. Older content gets dropped, usually from the start of the conversation. Nothing tells you.
• Rejected requests. Some APIs return a `context_length_exceeded` error instead and stop your pipeline dead.
• Made-up answers. The model sounds confident but the fact it needed was truncated out.
• Lost in the middle. Research in TACL found that information sitting in the middle of a long input is used less reliably than information near either end. Put your critical instructions at the start or the end.
• Contradictions. Early instructions get truncated, later ones survive, and the model follows the conflict without flagging it.

A debugging checklist that works:

1. Log token counts per request. Split them: system prompt, history, retrieved content.
2. Reproduce the failure with the smallest prompt that still triggers it.
3. Find out where truncation happens. Start, middle, or end.
4. Check your reserved output budget. If the model rarely uses it, take some back.
5. Turn on streaming to see whether generation starts before you hit the limit.
6. Alert in production when a request passes 85% of the window.

Cost is worth watching next to correctness. Requests that keep brushing the ceiling are expensive ones. A sudden token spike usually means a retrieval step is pulling in too much, or a chat history nobody is pruning.

Patterns that keep you inside the window

When your content is bigger than the window, or you just want costs to stay predictable, these are your options.

Chunking with layered summaries

Split the document into chunks, summarise each one, then summarise the summaries. Good for document review and reports. The catch: summarising loses detail, and you cannot get it back.

Retrieval-augmented generation (RAG)

Embed your documents, put them in a vector database, and pull back only the passages that match the question. This turns a huge corpus into a handful of relevant paragraphs, which is why it is usually the first thing to reach for. The catch: your answer is only as good as your retrieval. Weak chunking gives you confident nonsense.

If you're building one, we have a full walkthrough on how to build a RAG pipeline, and a companion piece on building a RAG application.

Sliding windows

Work through a long sequence in overlapping segments, carrying a short summary forward each time. Good for transcripts and anything sequential. The catch: the overlap costs tokens, and keeping it coherent takes care.

Streaming and repeated synthesis

Generate part of the answer, feed it back as condensed context, keep going. Lowers peak memory. The catch: more round trips, so more latency and more API cost.

External memory

Keep history or facts in a database and inject only what this turn needs. Right for long-running agents and multi-session products. The catch: retrieval adds latency, and the store needs looking after. This is the heart of what people now call context engineering.

Quick comparison:

• RAG. Low token cost, harder to get right, best for big static document sets.
• Chunking. Simple to build, loses detail.
• Sliding windows. Good for sequential content, pays a cost in overlap.
• External memory. Best for persistent agents, most work to build.

Pro tip: before you lock in an architecture, build a small prototype that measures real token cost and latency on *your* content. Token density varies so much between prose, code and tables that synthetic benchmarks will mislead you.

How we design for context limits

Designing around the window is engineering work, not an afterthought. This is the checklist we run at Devwiz on long-context builds.

1. Write down the biggest realistic request. System prompt size, deepest history, most retrieved content, longest expected answer.
2. Reserve output tokens on purpose. Set `max_tokens` tight. If the model never reaches it, claim those tokens back for input.
3. Measure real content. Run representative samples through the actual tokeniser before you integrate anything.
4. Decide who manages history. Retrieval, summarising or pruning. Pick one and test it against your latency budget.
5. Benchmark at p95, not average. Measure time-to-first-token and total time at your expected p95 input size.
6. Log tokens and alert early. Track input, output and cost per request. Alert at 85% of the window.
7. Build a truncation fallback. Handle `context_length_exceeded` by retrying with a shorter prompt or telling the user clearly.

Step 3 is the one teams skip, and it is the one that bites. Token cost per request drives your monthly inference bill directly, so sampling real content early often changes which model or retrieval strategy you pick. Our breakdown of LLM API pricing shows how quickly those numbers add up.

Turning tokens into words

A simple way to size a window against real content.

1. Sample your actual content. Take 500 to 1,000 words of the real thing.
2. Run it through the target tokeniser. tiktoken for OpenAI models, Hugging Face for open-weight ones.
3. Work out tokens per word. Divide tokens by word count. Plain English usually lands around 1.3 to 1.5.
4. Divide the window by that ratio. That gives you a rough word count.

Two worked examples:

A 200k window at 1.3 tokens per word holds roughly 150,000 words. That's 500 to 600 pages of a normal novel. Swap in code or tables at 2 to 3 tokens per word and you get 65,000 to 100,000 words instead.

A 1M window at the same prose ratio gets you around 750,000 words. Several full books. In practice, cost and latency at that size mean most production systems still use retrieval rather than filling it.

The lesson: one conversion factor will mislead you. A codebase full of comments and config might average 2.5 tokens per word. Always test the real thing.

The window is a constraint, not a feature

Most teams ask which model has the biggest window. We think that's the wrong question.

A context window is a scarce resource with a price per token. So the real question is: what is the least context needed to get a correct, reliable answer?

That flip changes the design. Instead of shopping for the biggest number, you set a token budget per request type, then pick the retrieval or summarising strategy that keeps you inside it. Those are the questions we work through during scoping on every AI app development build, before anyone writes production code.

Devwiz has shipped 200+ apps since 2015, including work for the NSW Government, Briometrix, Vivid and Huskee. The pattern we see is simple. Teams that measure token cost early ship on budget. Teams that skip it hit expensive surprises at integration.

If you're a CTO or founder scoping an AI platform and you want a team that treats the token budget as a real engineering concern, have a look at how we work with CTOs or get in touch for a scoping conversation.

Frequently asked questions

What is an AI context window in simple terms?

It's the total amount of text a model can read and consider at once, measured in tokens. It covers your instructions, the conversation so far, any documents pulled in, and the space held back for the reply. Go past it and the model loses access to the earlier content.

What happens when the context window is full?

The model either drops earlier content silently or returns a context_length_exceeded error. Quality can also drop before you hit the hard limit, especially for information sitting in the middle of a long input.

How big is a 200k context window in words?

Roughly 150,000 words of plain English, or several hundred pages. Code and tables are more token-dense, so the same window holds a lot less of them.

What does a 1 million token context window mean?

At about 1.3 tokens per word for prose, roughly 750,000 words. In practice, cost and latency at that size mean most production systems use retrieval instead of filling the window.

What is an example of a context window in AI?

Paste a long contract into a model and ask for a summary of the key clauses. The document, your instructions and the reply all have to fit inside the window together. If the contract is longer than that, the earlier pages get truncated and the model can't refer to them.