Here's something that might surprise you: the most consequential bottleneck in AI today has nothing to do with the speed of your GPU.
Every time you ask AI to summarize a long document, debug a complex piece of code, or help you work through a multi-step problem, something happens behind the scenes that many people rarely think about. The AI must remember what you just asked, and how it responded. It must also remember what you asked before that and, again, how it replied, all the way back to the beginning of your conversation. Not in the way humans remember things, but with 100% recall accuracy, question by question, response by response, detail by detail.
Storing all that takes up an enormous amount of space.
That store of "what was said before" has a name - KV cache. And it's quietly becoming one of the most pressing infrastructure challenges across AI.
The memory problem AI didn't see coming
Think about how much more information AI models are handling today compared to just a few years ago. In just seven years, the amount of context an AI model can hold has grown by nearly 20,000-fold (platform.claude.com, understandingai.org). And the more context a model holds, the more memory it needs. Not for the model itself, but just to house its memory of your conversation. The math is getting worse, not better.
Three forces that are turning up the heat
The problem is caused by three distinct trends colliding at once, each independently driving up KV cache demand.
1. Context windows continue growing
The productivity gains from longer context windows are real and compelling. Developers want to feed entire codebases to AI assistants. Lawyers want to analyze thousands of documents at once. Users will always push for more, because more context means better answers.
2. Chain-Of-Thought reasoning is increasing demand on resources
A new class of AI systems, often called reasoning models, don't just generate quick responses. Systems like OpenAI's o3 or DeepSeek-R1 work through problems step by step, generating chains of intermediate reasoning before arriving at an answer. Internally, reasoning models can generate several times more reasoning tokens than appear in the final response, significantly increasing the KV-cache footprint during inference.
3. AI is becoming agentic
This is perhaps the biggest shift of all. While an AI chatbot remembers your conversation for seconds or minutes, an AI agent (one that uses tools, calls APIs, and manages long-horizon workflows on your behalf), may need to remember context for hours, days, or longer across multiple sessions. AI agents transform the KV cache from a short-term scratch pad into something closer to a persistent storage system. What was once a behind-the-scenes implementation detail has become a first-class systems problem.
Can't software just fix this?
It's a fair question. And the honest answer is that software helps enormously, but it can't solve the problem alone. Researchers and engineers have developed a remarkable toolkit of techniques to reduce KV cache size, and these are genuine, meaningful improvements that bend the curve.
But they don't break it, and there's a 160-year-old reason why.
In 1865, the British economist William Stanley Jevons observed something counterintuitive about the steam engines of his era: as they became more fuel-efficient, Britain didn't consume less coal. It consumed dramatically more. Efficiency made steam power cheaper and more accessible, which expanded its use far beyond what the original efficiency gains had saved. He called it the rebound effect. Today, we call it Jevons paradox.
The same dynamic is playing out in AI infrastructure right now. Every time engineers find a way to make KV caches smaller, users and operators immediately reinvest the saved space into larger context windows, more concurrent users, or more complex reasoning tasks. The demand expands to fill the available capacity, and then some. Efficiency breeds appetite. There is no signal of demand reduction on the horizon yet.
The case for a memory hierarchy
So, what is the solution? The same one that computer architects have used for decades to bridge gaps between cost, capacity, and speed: a tiered hierarchy.
Compute systems (like your laptop) already do this. The most frequently needed data lives in a fast, expensive cache. Slightly less urgent data lives in RAM. Everything else is on your SSD. The system manages where data lives based on how often and how urgently it's needed.
AI inference systems need to work similarly, just on a much larger scale.
The key insight is that the goal isn't to choose one tier. It's to orchestrate them all, moving data intelligently based on access patterns and workload demands.
And the SSD tier turns out to be much more capable than many might expect.
A data center experiment worth knowing about
We put this to the test. Micron's Data Center Workload Engineering team ran extensive validation testing using the Micron 9550 PRO SSD with popular open-source AI inference frameworks. The results were revealing.
Adding high-performance SSD storage can significantly expand long-context inference capacity by extending KV cache beyond GPU memory, potentially improving GPU utilization and lowering infrastructure cost.
This is the economics of tiered memory at work. You're not replacing the GPU's onboard memory. You're giving it a much larger, more affordable backstop so it can stop turning away requests and start serving them.
Inference is the business
There's a bigger point underneath all this that I think often gets lost in conversations about AI infrastructure.
The AI industry spent its first era, rightly obsessed with training the massive models that power today's AI applications. But training is a one-time cost. Inference (running those models to answer real user and AI agent questions, millions of times a day) is the ongoing economic engine of AI. It's where value is continuously created.
Memory and storage don't just support inference. They shape inference economics. How many users can be served at the same time, how long a context can be supported, the per-query financial factors, and whether AI application can scale profitably. All of it flows through the memory and storage hierarchy.
The companies that get this right will build AI systems that are faster, more capable, and more efficient than those that treat memory as an afterthought. The ones that don't will find themselves constrained by where they put their memory.
The road ahead
Many are in the early stages of understanding what AI systems need from memory and storage. Context windows will keep growing. Agents will keep getting more autonomous and longer-lived. The KV cache challenge will get harder before it gets easier.
The good news is that the architectural path forward is becoming clearer. A thoughtfully tiered memory hierarchy that is optimized for speed, where response time is critical, and optimized for scale, where capacity is paramount, can turn what looks like an insurmountable bottleneck into a solved engineering problem.
The memory and storage technologies to build this hierarchy exist today.
If you want to go deeper on the technical architecture, the data, and the engineering specifics behind everything discussed here, see: The KV Cache Challenge: Navigating the New Frontier of AI Inference Memory.
The bottleneck is real. So is the solution.
