As AI inference moves towards longer conversations, larger context windows and more complex workloads, the information retained between interactions is creating a new infrastructure requirement.
Data centre storage was largely discussed in familiar terms: capacity, performance, latency, resilience and cost per terabyte – AI inference is complicating that picture. Large language models generate and reuse information during a conversation or task and a key part of this process is the key-value cache (KV cache) which stores intermediate information from previous processing so that a model does not have to repeat the same calculations when continuing a sequence.
As context windows expand and AI applications become more persistent, the amount of information that needs to be retained and accessed can grow substantially which has created a new infrastructure question: should this information remain close to the processor, or should it be moved into a larger, shared memory and storage tier?
A new layer in the inference architecture
The industry is beginning to develop products specifically around this requirement.
In November, Supermicro announced a KV-cache solution developed with IBM and NVIDIA. It combines Supermicro Petascale all-flash storage servers, IBM Storage Scale software, NVIDIA Dynamo inference software and NVIDIA Spectrum-X Ethernet switches. The announcement is significant because it treats KV cache as an infrastructure problem spanning storage, software, networking and compute – not simply as a feature of the accelerator.
Marvell has also announced second generation Structera X CXL 3.2 memory expanders. The devices are designed to increase memory capacity and enable sharing across multiple hosts. Marvell says the products are expected to sample in the first quarter of 2027.
These are different approaches, but both reflect a wider shift towards expanding the memory resources available to AI systems.
The economics of keeping context
The attraction is fairly straightforward – if useful context can be retained and reused efficiently, then systems may avoid repeating work and make better use of expensive accelerators – but the economics are far more complicated than simply adding storage.
Moving information between GPU memory, system memory and external storage introduces latency, consumes network bandwidth and requires software capable of managing where information resides. The cheapest capacity is not necessarily the most useful if accessing it slows the workload.
There are also questions about how long context should be retained, which data deserves the fastest tier, how information is shared between users or workloads and how sensitive data is protected and for operators, these decisions could influence the balance between accelerator memory, conventional DRAM, CXL-attached memory and high-performance storage.
Storage vendors are moving closer to compute
The emerging opportunity is not simply to sell more flash capacity – it is to provide infrastructure that allows AI systems to use memory and storage resources more intelligently and that could change procurement priorities. Storage performance will increasingly need to be assessed in relation to inference throughput, time to first token, concurrency and accelerator utilisation.
Results depend on workload characteristics, context length, cache hit rates, software configuration and the system used for comparison, so a headline improvement in one test should not be treated as a universal outcome. The same caution applies to emerging memory expansion products – announced capabilities and sampling schedules are not evidence of broad production deployment.
A different way to think about AI storage
As inference becomes a larger part of AI infrastructure demand, the distinction between memory and storage may become less useful as a standalone procurement boundary. Operators still need to understand the technical differences between each tier but should start evaluating them as parts of one data retention and retrieval architecture.
The competitive question is shifting from how much data a facility can store, to how economically it can preserve, retrieve and reuse the information that makes an AI service responsive. Whilst the KV cache may have started as a technical mechanism inside model inference – it is fast becoming a material consideration in the design, cost and performance of the AI data centre itself.
