The KV cache is the memory an LLM builds up while generating text, storing the key and value vectors for every token it has already processed so it does not have to recompute them. Long context windows feel free to the user, but they are not free to the server. Every extra thousand tokens you feed a model adds a slab of memory that has to sit in GPU RAM for the entire conversation, and that memory is the real bottleneck behind slow, expensive long-context inference.
Transformers generate one token at a time. For each new token, the model needs to look back at every previous token's key and value vectors to compute attention. Without caching, it would recompute those vectors from scratch at every single step, which is wasteful and slow.
So the model saves them. After processing token one, it stores its key and value vectors. After token two, same thing. By token one thousand, the cache holds one thousand pairs of vectors, one pair per layer, per attention head. This is the KV cache: a running ledger of everything the model has "seen" in the current sequence.
It sounds like a small optimization. It is actually the dominant cost of running inference once your context gets long, because that ledger has to live in GPU memory, fully intact, for as long as the conversation continues.
Most people assume a longer prompt just means a longer wait. That is true, but it is not the whole story. The bigger issue is memory footprint. The KV cache grows linearly with the number of tokens in context, and it grows per layer and per attention head on top of that.
A model with 40 layers and 32 attention heads is not storing one key-value pair per token. It is storing 40 times 32 of them, for every single token, for both the key and the value. Double your context length and you double the cache size. Run ten conversations at once with long histories and you multiply that number by ten. GPU memory fills up fast, and when it fills up, the server can serve fewer users at the same time.
This is why providers cap context length or charge more for it. It is not an arbitrary business decision. It reflects a hard constraint: there is only so much high-bandwidth memory on a GPU, and the KV cache is fighting the model weights themselves for that space.
Here is a rough way to size a KV cache. For a model with L layers, H heads, head dimension D, and sequence length S, using 16-bit precision, the cache size per token roughly follows: 2 (key and value) times L times H times D times 2 bytes. Multiply that by S for the full sequence, then by batch size for concurrent requests.
| Context length | Approx. cache size (7B-class model) | Approx. cache size (70B-class model) |
|---|---|---|
| 4,000 tokens | ~1 GB | ~5 GB |
| 32,000 tokens | ~8 GB | ~40 GB |
| 128,000 tokens | ~32 GB | ~160 GB |
Those numbers are rough, but the trend is what matters. Push context to 128K and the cache alone can exceed the memory needed to hold the model's weights. That is the quiet reason "unlimited context" is a marketing phrase, not an engineering reality. Somewhere, a memory budget is getting blown, and someone is paying for it.
Nobody just accepts a linear memory explosion. Several techniques exist specifically to shrink the KV cache without gutting model quality.
Every one of these is a tradeoff, not a free lunch. GQA trades some output precision for memory. Quantization trades more precision for more memory. PagedAttention trades implementation complexity for better packing. There is no version of this where you get infinite context, instant speed, and zero cost at the same time.
When a provider charges more per token for long-context requests, or throttles throughput once your conversation gets long, this is why. The compute cost of generating a single new token barely changes with context length. The memory cost of holding that context does, and memory bandwidth, not raw FLOPs, is the real ceiling on how fast a GPU can serve tokens once sequences get long.
This also explains why conversational apps slow down the longer a chat session runs. Each new message has to attend to a cache that keeps growing. The model is not getting dumber. It is dragging a bigger ledger behind it with every turn.
If you are building on top of an LLM API, this has a direct, practical consequence: trimming or summarizing old conversation history is not just a prompt-engineering nicety, it is a cost and latency lever. Cutting context in half roughly cuts the KV cache in half, which can meaningfully increase how many concurrent users a given deployment can serve.
Vendors love advertising million-token context windows because the number is impressive and easy to market. What they do not advertise as loudly is that serving that context at scale, to many users at once, with acceptable latency, requires serious engineering investment in cache compression and memory management. A model that technically accepts a million tokens and a deployment that can serve a million tokens cheaply to thousands of concurrent users are two very different achievements.
My opinion, after watching this play out across every major provider: context length numbers in marketing material should always be read next to the pricing tier for long requests. If long context were actually cheap, nobody would charge extra for it or silently cap throughput once you use it. The KV cache is the hidden meter running underneath every long conversation, and it is the single best explanation for why your AI bill, and your AI's response time, behave the way they do.
It is the stored memory of every token an LLM has already processed in the current conversation, specifically the key and value vectors used for attention. The model keeps this stored so it does not have to recompute those vectors every time it generates a new token. The tradeoff is that this stored memory grows with every token added to the conversation.
A longer context means a larger KV cache, and the model has to read that entire cache from GPU memory to generate every new token. The bottleneck is memory bandwidth, not raw processing power, so bigger caches slow down each step even though the actual computation per token barely changes.
Yes. Providers often charge more for long-context requests or limit concurrent users because the KV cache for long conversations consumes large amounts of GPU memory. That memory could otherwise serve more simultaneous requests, so long contexts have a real infrastructure cost that gets passed on through pricing or throttling.
Multi-Query Attention has all attention heads share a single set of key and value vectors, which shrinks the KV cache the most but can hurt output quality. Grouped-Query Attention splits heads into smaller groups that each share a set, recovering more quality while still cutting cache size significantly compared to standard attention.
Techniques like grouped-query attention, KV cache quantization, and better memory management like PagedAttention all reduce memory use with only a small, often acceptable, drop in output quality. None of these eliminate the tradeoff entirely, but combined they let modern systems support much longer contexts than naive implementations ever could.