LLM Context Window: What It Is, Sizes, and Why Models Forget

A context window is the maximum number of tokens a language model processes in a single request. It holds the system instructions, the whole conversation history, attached documents, tool definitions and the model’s own response. Anything that doesn’t fit in the window doesn’t exist for the model.

The window isn’t training data and it isn’t long-term memory. The model stores nothing between requests. In a chat, “memory” comes from the application, which resends the history on every turn. That explains two common surprises. A model with a 1M-token window still “forgets” the start of a conversation. And a long chat costs more than a short one, even when your questions stay just as short.

This article covers how the window works, how big it is as of November 2026, what happens at the limit and why long context is expensive. The numbers on window growth and overflow come from the author’s measurements on November 8, 2026, via OpenRouter and Ollama 0.40.14.

What is a context window in an LLM

Anthropic’s documentation describes the context window as “all the text a language model can reference when generating a response, including the response itself” and calls it a “working memory” for the model (Context windows). The window is measured in tokens, not words or characters.

A token is a piece of text: a word, part of a word or a symbol. The tokenizer splits the input into these pieces. According to Anthropic, 1M tokens on the current Claude tokenizer (introduced with Opus 4.7) is about 555,000 words or 2.5 million characters, compared with about 750,000 words on earlier models (Models overview). Language matters too: in the measurements from the article on tokens, Russian prose took 1.19-1.28x more tokens on Claude Haiku 5.5 than its English translation.

What fills the window

Everything in the request goes into the window. The user’s latest message is only one part of it. The full list, from Anthropic’s documentation:

What takes up the window	Where it comes from	Notes
System prompt	the app, the agent, CLAUDE.md	sent with every request
messages history	all previous turns	grows every turn
Files, images, PDFs	user attachments	up to 600 images or PDF pages per request
Tool definitions	MCP servers, functions	tool schemas are text too
Tool results	file reads, search, logs	tool output goes into the window in full
Reasoning (thinking)	the model	on current models, including Haiku 5.5 and Fable 5.1, earlier thinking blocks stay in the window by default
Model response	the model	counts toward the same request’s window

Prompt caching doesn’t free up space. The input_tokens, cache_read_input_tokens and cache_creation_input_tokens fields all count toward the window. The cache changes what you pay for those tokens, not how many there are.

Output takes space too

The window is shared between input and output. If the input takes 990K of a 1M window, only 10K is left for the answer, even with max_tokens set to 128K. The model’s reasoning is part of max_tokens and is billed as output tokens.

Google states this directly on the model card: the Gemini 4 Argon window of 1,048,576 tokens is a combined limit for input and output (Gemini 4 Argon).

Context window sizes in November 2026

Snapshot as of November 8, 2026, from vendor pages. Prices are in US dollars per 1M tokens, standard tier, without caching or batch discounts.

Model	Window	Max output	Input / output	Long context
Claude Fable 5.1	1M	128K	$10 / $50	one rate
Claude Opus 5.5	1M	128K	$4 / $20	one rate
Claude Sonnet 5.5	1M	128K	$2 / $10	one rate
Claude Haiku 5.5	1M	128K	$0.10 / $0.50	prompt over 100K: $0.50 / $2.50
GPT-6 Astra	1.05M	128K	$10 / $50	input over 272K: input 2x, output 1.5x
GPT-6.1 Sol	1.05M	128K	$2 / $10	same
GPT-6.1 Terra	1.05M (input up to 922K)	128K	$1 / $5	same
GPT-6 Luna	1.05M	128K	$0.10 / $0.50	same
Gemini 4 Argon	1,048,576 for input and output	65,536 by default	$2 / $10 until 11/30/2026, $4 / $20 from 12/01/2026	no separate tier
DeepSeek deepseek-flash	1M	up to 384K	$0.15 / $0.60 off-peak	2x at peak hours
DeepSeek deepseek-v4-pro	1M	up to 384K	$0.66 / $1.98 off-peak	2x at peak hours
Qwen3.8-27B (open weights)	262,144	-	local	depends on your machine’s memory

Sources: Anthropic pricing, OpenAI models and the GPT-6.1 Terra page, Gemini pricing, DeepSeek pricing, Qwen from the local LLM guide. DeepSeek peak hours are 01:00-04:00 and 06:00-10:00 UTC on weekdays. DeepSeek lists the output limit in a single cell for both models.

You can’t compare “1M” across vendors directly: tokenizers split the same text into different numbers of tokens. According to Anthropic, models from Opus 4.7 onward produce about 30% more tokens for the same text than earlier Claude models (Token counting).

API window vs app window

The window in a chat app can differ from the window in the API. On paid Claude plans, Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5 have a 1M window. Some earlier models also get 1M, others get 500K, and the rest get 200K (Claude Help Center).

This article gives no figures for the ChatGPT app: the primary source couldn’t be checked. Don’t assume the API window from the table applies to the chat.

What happens when the context window runs out

Behavior at the limit depends on where the model runs: a cloud API, a local runner or an app built on top of an API.

Cloud API: a 400 error

Anthropic’s rule is simple. If the input alone exceeds the window, the API returns HTTP 400 invalid_request_error (“prompt is too long”) on every model. If the input fits but input plus max_tokens doesn’t, Claude 4.5 models and newer accept the request, and generation stops with stop_reason: "model_context_window_exceeded" (Context windows). Nothing gets cut silently.

In the author’s measurement, a cloud endpoint with a 32,768-token window (mistral-small-24b-instruct-2501 via OpenRouter) received about 41,080 input tokens and returned HTTP 400 with exact numbers:

This endpoint's maximum context length is 32768 tokens. However, you requested
about 41090 tokens (41080 of text input, 10 in the output).

The error is annoying but honest. The client knows the request failed and how much to cut.

Local Ollama: silent truncation

A local runner behaved differently in the same measurement. The author put the code word PELICAN-47 at the start of a ~4,600-token text, asked about it at the end, and sent it to Ollama 0.40.14 with num_ctx set to 2048.

Model	num_ctx	Input	Server response	prompt_eval_count	Code word
qwen3:8b	2048	~4,592	HTTP 200	1026	not found, answer “Note”
gemma4:e4b	2048	~4,400	HTTP 200	1027	not found: “The text does not mention a secret code word at the very beginning.”
qwen3:8b	8192	4,592	HTTP 200	4592	found
gemma4:e4b	8192	4,399	HTTP 200	4399	found

There was no error. Ollama cut the prompt to about half of num_ctx. The start of the text, code word included, was gone, and the model confidently said the word wasn’t there. The only warning showed up in the server log:

level=WARN msg="truncating input prompt" limit=1026 prompt=4592 keep=4 new=1026

The client never sees that log. The only sign in the response itself is the prompt_eval_count field: if it’s well below the expected prompt size, the input was truncated. You can check with the same request as in the measurement, written as curl:

curl -s http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "stream": false,
  "think": false,
  "options": {"num_ctx": 2048, "num_predict": 20},
  "messages": [{"role": "user", "content": "<long text and question>"}]
}' | jq '{prompt_eval_count, answer: .message.content}'

This matters for agents on local models. According to the Ollama docs, machines with less than 24 GiB of VRAM get a default context of 4K tokens (Context length), so a long agent system prompt can get truncated without a single error. The Ollama guide shows how to set the context and check it with ollama ps. The measurement used one Ollama version and two models. Other local runners weren’t tested and may behave differently.

Chat apps and Claude Code: compacting the history

Apps built on an API usually don’t show an error. They manage the window themselves. Anthropic’s documentation notes that chat interfaces such as claude.ai can manage the window on a rolling “first in, first out” basis. On paid Claude plans with code execution turned on, automatic context management kicks in near the limit: Claude summarizes earlier messages, and the full history stays available to view. Without code execution, this doesn’t work (Claude Help Center).

In Claude Code, auto-compaction does the same job: it “summarizes conversation history when approaching context limits” (Manage costs). You can also manage it by hand:

• /context shows what’s taking up the window;
• /compact Focus on code samples and API usage compacts the history and tells Claude what to keep; standing instructions can go in a # Compact instructions section in CLAUDE.md;
• /clear starts a clean session when you switch to unrelated work.

Compaction has a cost. /compact reads the whole conversation it summarizes, so compacting a large context is itself a large request. /clear costs nothing. For more on working with sessions, see the complete Claude Code guide.

Why the model “forgets”

“Forgetting” almost always comes down to one of four causes, and only the last one is about the model itself.

Cause	What it looks like	What to do
New chat	the model doesn’t know what you discussed last session	move the key facts into the first message or a file; product “memory” features are a separate layer with no guarantees
The app compacted or dropped old turns	the general thread survives, details from the start are gone	compact with instructions, keep decisions in a separate file
Input larger than the window	API: 400 error; local runner: answer without the start of the text	count tokens in advance, trim the input, raise num_ctx
The model sees everything but uses it worse	a fact from the middle of a long document gets ignored	shorten the context, put important parts near the start or the end

The model is stateless. Everything that looks like conversation memory is history the app resends in the window each time. That’s why a follow-up works better in the same chat: the earlier turns are in the window, and in a new chat they aren’t.

The last row of the table is the subject of Lost in the Middle (Liu et al., 2023, TACL). Models used information from the start and end of the context better, and noticeably worse from the middle, even models built for long inputs. The study used 2023 models, and current generations may behave differently. Anthropic’s documentation puts it directly: “As token count grows, accuracy and recall degrade, a phenomenon known as context rot.” So the practical rule is: the window gives you capacity, and it doesn’t guarantee that every fact gets used.

Why long context is expensive

Long context costs more for three reasons. You pay for the whole history on every turn. Some models charge a higher rate for long prompts. And inside the model, compute and memory grow with length.

Every turn pays for the whole history

The model doesn’t remember the previous turn, so the app sends the whole conversation every time. The Claude Code docs say it plainly: “Claude Code sends your full conversation with every request” (Manage costs).

The author’s measurement: 10 turns of a short conversation with Claude Haiku 5.5 via OpenRouter, reasoning off, max_tokens 120, the full history sent on every turn.

Turn	Input tokens	Output tokens	Total billed so far
1	20	91	111
2	127	120	358
3	262	120	740
4	399	120	1,259
5	535	65	1,859
6	616	120	2,595
7	754	94	3,443
8	858	120	4,421
9	994	120	5,535
10	1,134	95	6,764

Input on turn 10 is 57 times larger than on turn 1. Over 10 turns, 6,764 tokens were billed, while the questions and answers themselves add up to about 1,230. At this scale, the whole conversation cost $0.0011. This is one model and a short test conversation, but the mechanics are general. For an agent that feeds file contents and logs back into the window every turn, the same arithmetic runs on hundreds of thousands of tokens.

Prompt caching makes the repeated, unchanged prefix cheaper but doesn’t remove it. A cache read costs 5% of the input price on Opus 5.5 and Sonnet 5.5, 2.5% on Fable 5.1 and 10% on Haiku 5.5 (Pricing). Cached tokens still take up space in the window.

The cost of a single long request

Calculated from prices as of November 8, 2026, without caching: 50K, 150K and 300K input tokens plus 2,000 output tokens.

Model	50K	150K	300K
Claude Haiku 5.5 (100K threshold)	$0.006	$0.080	$0.155
GPT-6.1 Terra (272K threshold)	$0.060	$0.160	$0.615
Claude Sonnet 5.5 (no threshold)	$0.120	$0.320	$0.620

Thresholds apply to the whole request, not just the tokens above the threshold. On Haiku 5.5, a request with a prompt over 100,000 tokens pays the higher rate on all of it, and cached tokens count toward the prompt length. So a 100,000-token prompt costs $0.01 in input, and a 100,001-token prompt costs about $0.05. Going from 50K to 150K triples the tokens and raises the price 13x.

On GPT-6.1 Terra, a prompt over 272K input tokens is billed at 2x for input and cache and 1.5x for output “for the full request” (GPT-6.1 Terra). At 300K, Terra nearly catches up with Sonnet 5.5, even though it’s half the price on short context. Prices change: Gemini 4 Argon’s rate doubles on December 1, 2026, so check current prices before running your own numbers. General ways to cut spend are in the article on LLM cost optimization.

Attention and memory

Inside the model, long context costs compute. In the classic transformer, a self-attention layer has O(n²·d) complexity, where n is the sequence length and d is the representation dimension (Attention Is All You Need, Table 1): every token is compared with every other token. Modern models use optimizations such as sliding windows and hybrid attention, so growth is gentler in practice, but long context never becomes free.

The second cost is the KV cache: intermediate attention data kept in memory for every token in the context. You can see it clearly on local models. Per the measurement in the local LLM guide, the qwen3:8b KV cache takes 144 KiB per token. At a 32,768-token context that’s 4.5 GiB, and in Ollama the model took 9.9 GB with a 5.2 GB weights file. On a local machine, a long window runs into memory limits before it reaches the model’s stated maximum.

How to keep context short

Short, clean context is cheaper and more reliable. The techniques below come from Anthropic and OpenAI documentation, and each has a cost.

Technique	When to use it	What you pay
RAG: put only retrieved chunks in the window	the knowledge base is larger than the window or changes often	quality depends on retrieval (RAG vs context engineering)
Compacting the history	a long chat or agent loop approaches the limit	details get lost, and compaction is itself a request
Clearing old tool results	the agent reads lots of files and logs	the prompt cache gets rebuilt
External memory	facts must survive the session	you need your own storage logic (AI agent memory)
Subagents	bulky output: tests, logs, code search	the subagent spends its own tokens (Claude Code subagents)
/clear between tasks	switching to unrelated work	the previous task’s context leaves the session
Deferred tool loading, skills instead of a long CLAUDE.md	many MCP servers and instructions	some instructions load on demand (MCP and context)

Some of these are already built into the APIs. Anthropic offers server-side compaction in beta for Claude 4.6 and later: earlier parts of the conversation are summarized on the server, and the conversation can continue past the window limit. Context editing clears old tool results and thinking blocks. The memory tool runs on the client side: the model requests file operations in a /memories directory, and your code carries them out. For long-running agents, the documentation recommends combining the two: compaction keeps the active context small, and memory keeps what must not be lost in summarization (Memory tool).

OpenAI’s Responses API turns on server-side compaction through context_management with a compact_threshold. There’s also a separate stateless /responses/compact endpoint. The result arrives as an encrypted compaction item (Compaction).

Claude Code gives you the same ideas without writing code. Per the documentation, MCP tool definitions are deferred by default, skills load only when invoked, and a hook can filter a log down to the error lines before Claude reads it (Manage costs).

How to count tokens before sending

Counting tokens up front is more reliable than learning about overflow from an error or a truncated answer.

Anthropic has a free /v1/messages/count_tokens endpoint for this. It accepts the same model, system, messages and tools as a regular request and returns input_tokens. Example from the documentation (Token counting):

curl -s https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-haiku-5-5",
    "system": "You are a code reviewer",
    "messages": [{"role": "user", "content": "<history and new question>"}]
  }'
# {"input_tokens": ...}

The documentation notes that this is an estimate, and the actual count in the real request may differ slightly. Count for the model you’ll actually call, because tokenizers from different generations give different numbers.

Other places to get numbers:

• the usage field in the API response: input_tokens, cache_read_input_tokens, cache_creation_input_tokens and output_tokens on Anthropic, prompt_tokens and completion_tokens on OpenAI-compatible APIs;
• stop_reason in an Anthropic response: model_context_window_exceeded means the answer hit the window limit;
• /context and /usage in Claude Code: what’s in the window and how many tokens the session has used;
• prompt_eval_count and eval_count in an Ollama response: how many tokens the model actually read and generated.

What’s next

Quality isn’t constant inside the window: the more tokens, the higher the chance the model misses the fact you need. That’s a separate topic with its own measurements. For this article, one conclusion is enough: a full window doesn’t guarantee the model uses everything in it.

Once you know how the window works, the next question is what to put in it: which sources, in what order, how to compact and how to test the result. That’s covered in the context engineering guide.

Where to start:

1. Measure how much your typical request takes: count_tokens, the usage field or /context.
2. Compare the result with the pricing thresholds: 100K on Haiku 5.5, 272K on GPT-6.
3. For local models, check num_ctx and compare prompt_eval_count with the prompt size.
4. Start a new session for each new task, and keep decisions in a file, not in the chat history.

Sources

All pages accessed November 8, 2026.

• Anthropic: Context windows
• Anthropic: Pricing
• Anthropic: Models overview
• Anthropic: Token counting
• Anthropic: Memory tool
• Claude Help Center: context window on paid plans
• Claude Code: Manage costs effectively
• OpenAI: Models, GPT-6.1 Terra, Pricing, Compaction
• Google AI Studio: Gemini 4 Argon, Pricing
• DeepSeek: Models & Pricing
• Ollama: Context length
• Liu et al. Lost in the Middle: How Language Models Use Long Contexts
• Vaswani et al. Attention Is All You Need