An LLM context window is the total amount of text, measured in tokens, that a model can hold in memory during a single conversation or task. It is not storage. Nothing gets saved. Once the conversation ends or the window fills up, everything outside that window is gone unless you re-feed it. Vendors market context window size like a spec sheet number, the way phone makers used to market megapixels. It is mostly marketing. A 1-million-token window sounds like a solved problem. In practice, most of that space is wasted, and using it well takes more skill than just pasting in a bigger file.
Every request you send to an LLM, including your prompt, the system instructions, any documents you attach, and the model's own reply so far, gets broken into tokens. A token is roughly three-quarters of a word in English. The context window is the ceiling on how many of these tokens can exist in one pass through the model. Hit the ceiling and the oldest tokens get dropped, truncated, or the request fails outright, depending on the API.
This matters because people confuse context window with memory. A model with a 200,000-token window has no memory of your conversation from yesterday. It has no memory of the conversation from five minutes ago unless that text is still physically inside the current request. Every single API call is stateless. Chat apps fake continuity by resending your full history behind the scenes, which is also why long conversations get slower and more expensive the longer they run.
The transformer architecture underneath every major LLM uses a mechanism called self-attention. Every token has to compare itself against every other token in the window to figure out which ones matter. Double the tokens, and you roughly quadruple the computation for that comparison step. This is why context windows did not casually grow from 4,000 tokens to 1 million. Engineers had to build workarounds, like sparse attention and sliding windows, just to make huge context windows computationally survivable.
That quadratic cost is also why long-context requests cost more per token than short ones on most pricing tiers, and why they run slower. You are not just paying for more input. You are paying for a much heavier internal computation. Anthropic, OpenAI, and Google all price long-context calls differently for exactly this reason. If you are stuffing a 300-page PDF into a prompt for a single yes-or-no question, you are burning compute on comparisons that answer nothing.
Researchers at Stanford and elsewhere documented something now called the lost-in-the-middle problem. Models are noticeably better at recalling information placed at the very start or the very end of a long context window than information buried in the middle. Accuracy on retrieval tasks can drop by 20 to 30 percentage points for facts stuck in the middle third of a long document, even when the model technically has room to see all of it.
This is the part vendors do not put in the headline. A model can advertise a 1-million-token window and still fail to find a single clause buried on page 400 of a contract you pasted in. The window being big enough to fit the text says nothing about whether the model will actually use that text correctly. Bigger windows increase capacity. They do not increase attention quality, and attention quality is what determines whether the answer is right.
I think this is the single most under-discussed fact in applied AI right now. Teams keep buying bigger context windows to solve accuracy problems, when the real fix is better retrieval and better prompt structure. A well-organized 20,000-token prompt with the critical fact placed near the end will beat a sloppy 200,000-token dump almost every time.
Here is where things stand as of late 2024 and 2025, roughly. Numbers shift fast, so treat this as directional rather than permanent.
| Model | Context window | Notes |
|---|---|---|
| GPT-4o | 128,000 tokens | Strong recall near start and end, weaker in the middle third |
| Claude 3.5 Sonnet | 200,000 tokens | Consistently strong long-document performance in benchmarks |
| Gemini 1.5 Pro | 1,000,000 to 2,000,000 tokens | Largest window on the market, uneven retrieval accuracy at max length |
| Llama 3.1 | 128,000 tokens | Open-weight, self-hostable, needs tuning for long-context tasks |
Notice that the largest number, Gemini's multi-million-token window, is also the one with the least consistent recall at full length. Bigger is not automatically more reliable. It is more capacity, sitting on top of the same attention mechanism that gets less precise as the input grows.
Retrieval-augmented generation exists precisely because context windows are an expensive, imprecise way to give a model access to knowledge. Instead of pasting your entire knowledge base into every prompt, RAG stores it in a vector database, retrieves only the handful of relevant chunks for each query, and feeds the model just those chunks. This keeps the context small, keeps costs down, and keeps the relevant fact near the front of the prompt where the model actually pays attention.
Use a big context window when the task genuinely requires holistic understanding of one document: summarizing a full contract, reviewing an entire codebase for architectural consistency, or comparing two long reports side by side in a single pass. Use RAG when you are querying a knowledge base that changes constantly, spans thousands of documents, or only needs a small slice of information per question. Customer support bots, internal wikis, and legal research tools almost always belong in the RAG category, not the giant-context-window category.
Mixing the two up is the most common architecture mistake teams make right now. They pay for the biggest context window on the market and stuff in every document they have, then wonder why the model still misses obvious facts. The problem was never window size. It was that thirty unrelated documents diluted the signal the model needed.
Put the most important instructions and facts at the very start and repeat the key question again at the very end of the prompt. This exploits the model's own bias toward the start and end of the window instead of fighting it.
Strip anything irrelevant before it goes in. A big window is not permission to be lazy about what you include. Every extra token is competition for the model's attention, and competition degrades precision even when it does not hit the token ceiling.
Break large tasks into smaller passes when accuracy matters more than convenience. Asking a model to summarize each chapter of a document separately, then summarizing the summaries, consistently beats asking it to summarize a 200,000-token document in one shot. It costs more calls. It costs less in wrong answers.
Test recall directly before trusting a long-context workflow in production. Plant a specific, unusual fact in the middle of a long document and ask the model to retrieve it. If it fails on your own test, it will fail on your users' documents too, and you will find out from a complaint instead of a benchmark.
A context window is the maximum number of tokens, roughly three-quarters of a word each, that a language model can process in a single request. It includes your prompt, any attached documents, prior conversation history, and the model's own response. Anything beyond that limit gets truncated or dropped, and it is not stored anywhere between requests.
No. A bigger context window only means more capacity to hold text, not better comprehension of that text. Research on the "lost in the middle" problem shows models often recall facts near the start or end of a long document far more accurately than facts buried in the middle, regardless of how large the window is.
Transformer models use self-attention, where every token compares itself to every other token in the window. This computation scales roughly quadratically with length, so doubling your input tokens can more than double the compute required. That extra compute is why long-context API calls are usually priced higher per token than short ones.
Use a large context window when a task needs holistic understanding of one document, like reviewing a full contract or codebase in a single pass. Use RAG, retrieval-augmented generation, when you are querying a large or constantly changing knowledge base and only need small, relevant chunks per question. Most production knowledge-base tools should use RAG, not a giant prompt.
Google's Gemini 1.5 Pro currently offers the largest publicly available context window, ranging from 1 million to 2 million tokens depending on the tier. However, benchmarks show its recall accuracy becomes less consistent as prompts approach that maximum length, so a bigger window does not guarantee more reliable answers at full capacity.