KV cache

Context windows have grown from thousands to millions of tokens, placing unprecedented pressure on memory systems to keep pace with model complexity. As large language models (LLMs) scale into production across data centers and cloud environments, the computational cost of generating each response has become a critical challenge for engineers and infrastructure teams worldwide.

What is KV cache?

KV cache (key-value cache) is a memory optimization technique used in transformer-model-based large language models to accelerate inference by storing and reusing previously computed attention values. During autoregressive text generation, each new token depends on the tokens that came before it, making efficient attention processing essential as sequences grow longer.

This technique is foundational to deploying LLMs at scale because it helps reduce inference latency and increase throughput

How does KV cache work?

KV cache works like keeping notes while reading a long book. Instead of rereading every previous page each time you answer a question about the story, you keep the important details nearby so you can quickly use them as new questions come up. In an LLM, the KV cache serves a similar purpose by keeping useful context available while the model generates each new token.

KV cache functions within the self-attention mechanism of transformer architectures, storing intermediate representations so the model can reuse them across successive generation steps. During AI inference, the model processes input tokens and produces three vectors for each: a query, a key and a value. Attention weights are computed by comparing query vectors against all stored key vectors, with value vectors used to produce the final output.

The caching process follows these steps:

• Initial processing: The model processes the input prompt and computes key and value tensors for each token, storing them in the KV cache.
• Incremental generation: For each new token, only the query vector is computed freshly while previously cached keys and values are retrieved directly.
• Attention computation: The attention mechanism combines the new query with all cached key and value pairs to produce the output without reprocessing earlier tokens.
• Cache growth: Each generated token appends its own key and value pair to the cache, extending it incrementally until generation completes.
• Memory management: Systems allocate and release cache memory dynamically across concurrent requests and sessions to optimize utilization.

This architecture allows LLMs to scale to very long context windows without computational cost growing proportionally with sequence length.

How does KV cache relate to prefill and decode?

KV cache is created during the prefill phase and reused during the decode phase of LLM inference. In prefill, the model processes the full input prompt in parallel and writes its key and value tensors to memory. In decode, the model generates one new token at a time, reading the existing KV cache so it only needs to compute attention for the newest token instead of recalculating the full context. This distinction matters because prefill is often compute-intensive, while decode is usually constrained by memory bandwidth and cache size.

How does KV caching compare to standard inference?

In standard inference without KV caching, the model recomputes attention over the full context each time it generates a new token. This repeated work increases latency and computational cost as the prompt and generated response grow longer. With KV caching, the model stores previously computed keys and values, then reuses them during each generation step so only the newest token requires fresh attention computation.

The main benefit of KV caching is faster token generation and higher throughput. The main trade-off is memory: the cache grows as more tokens, requests and model layers are added, so inference systems need enough memory capacity and bandwidth to store and access cached tensors efficiently.

How do you optimize KV cache?

KV cache optimization reduces the memory required to store key and value tensors while preserving inference speed and model output quality. Because KV cache grows with context length, batch size and model depth, optimization usually combines memory management, compression and workload-aware serving strategies rather than relying on a single technique.

• Use paged attention: Allocate KV cache in smaller memory blocks instead of one large contiguous allocation, reducing fragmentation and improving accelerator utilization for variable-length requests.
• Apply cache quantization: Store cached keys and values in lower-precision formats such as INT8 or INT4 to reduce memory footprint while maintaining acceptable output quality.
• Evict low-value tokens: Remove older or less relevant cache entries when memory is constrained, using techniques such as sliding windows or attention-based importance scoring.
• Reuse shared prefixes: Cache common prompt prefixes across repeated or similar requests so the system can avoid recomputing the same context for each user session.
• Distribute or offload cache storage: Spread KV cache across multiple accelerators, host memory or storage tiers when long-context workloads exceed the capacity of a single device.

What are the trade-offs of KV cache optimization?

KV cache optimization requires balancing memory savings, latency, throughput and output quality. More aggressive compression or eviction can reduce memory usage and support longer contexts or larger batches, but it may also lower accuracy if important tokens are removed or represented with too little precision. Systems designed for real-time chat may prioritize low latency, while offline summarization or document analysis workloads may accept higher latency in exchange for longer context windows and lower serving cost.

What is the history of KV cache?

KV cache emerged directly from the attention mechanism introduced in the landmark 2017 paper "Attention Is All You Need," which established the transformer architecture as the foundation of modern language modeling. Early inference systems recomputed attention from the full context at every generation step, which was feasible for short sequences but became a severe bottleneck as models and context windows expanded rapidly.

Key milestones in KV cache evolution include:

• 2017: The transformer architecture introduces multi-head self-attention, establishing the computational pattern that KV caching would later optimize.
• 2020 to 2022: Large-scale language model deployments expose the memory and latency costs of full recomputation, accelerating industry research into caching strategies.
• 2022 to 2023: Production inference frameworks introduce paged attention, treating KV cache memory as virtual pages to reduce fragmentation and improve utilization across concurrent requests.
• 2024 to 2025: Context windows expand from thousands to millions of tokens, driving rapid innovation in KV compression, quantization and distributed caching to manage escalating memory demands.

What are the key types of KV cache?

KV cache optimization strategies fall into several categories, each designed to address different constraints across memory capacity, inference speed and output accuracy. Engineers select approaches based on workload characteristics, hardware configuration and acceptable accuracy trade-offs for their specific deployment environment.

1. Full KV cache

Full KV cache is a KV cache strategy that stores every key and value tensor generated during inference without compression or eviction. This approach delivers maximum accuracy and serves as the baseline against which all optimization techniques are benchmarked. Full KV cache is best suited to short-to-medium context lengths where memory capacity is not a limiting constraint.

2. Eviction-based KV cache

Eviction-based KV cache is a KV cache strategy that reduces memory use by selectively discarding key and value pairs deemed less relevant to the current generation step. Methods such as sliding-window attention retain only the most recent tokens, while importance-scoring techniques identify and evict low-attention entries. These approaches reduce the memory footprint at the cost of potential accuracy degradation in tasks that require long-range context retrieval.

3. Quantized KV cache

Quantized KV cache is a KV cache strategy that compresses stored key and value tensors by reducing their numerical precision from full precision (FP16 or BF16) to lower-bit formats such as INT8 or INT4. This significantly compresses memory consumption while preserving much of the model's output quality, making the quantized KV cache a widely adopted technique for high-throughput inference deployments.

4. Distributed KV cache

Distributed KV cache is a KV cache strategy that shares cache storage across multiple accelerators or nodes so inference can scale beyond the memory capacity of a single device. This approach is essential for ultra-long context workloads in AI data center environments, where models process hundreds of thousands or even millions of tokens per request.

How is KV cache used?

KV cache is deployed across a wide range of production inference environments where latency and throughput directly affect user experience and operational costs. Use cases span industries from enterprise software to scientific research, wherever language models generate responses over extended context.

• Conversational AI: Multi-turn chatbots and virtual assistants cache previous dialogue so each response incorporates full conversation history without reprocessing earlier turns.
• Code generation: Developer tools that generate or complete code across large files cache the full file context, enabling consistent and context-aware suggestions with low latency.
• Document processing: Enterprise platforms analyzing long contracts, research papers and reports use KV cache to handle extended input sequences efficiently.
• Retrieval-augmented generation: Systems that inject retrieved documents into the model context benefit from KV cache to manage the combined length of retrieved passages and user queries.
• Reasoning models: Models that generate extended chain-of-thought outputs rely on KV cache to maintain coherence across thousands of generated reasoning tokens.
• High-throughput serving: Inference servers handling concurrent requests from many users share and page KV cache allocations to maximize accelerator utilization and reduce per-token cost.

Why does KV cache place high demands on memory infrastructure?

KV cache places high demands on memory infrastructure because its memory requirements scale directly with context length, batch size and the number of model layers. Each new token adds key and value tensors that must be stored, retrieved and updated throughout generation, making KV cache one of the largest consumers of accelerator memory in production LLM deployments.

These demands are especially significant during the decode phase, when the model repeatedly reads from the KV cache to generate one token at a time. As context windows expand, memory capacity affects how much context a system can support, while memory bandwidth affects how quickly cached keys and values can be accessed. This can influence latency, throughput, batch size, concurrency and overall inference cost.

For Micron, KV cache highlights the role of high-bandwidth, high-capacity memory in scaling AI infrastructure. Micron's memory solutions are designed to support the throughput and capacity requirements of modern AI workloads, helping data center systems serve longer contexts and more concurrent inference requests.