Vast Data is implementing a tiered storage strategy to offload AI agent KV cache from GPU memory to CPU and persistent storage. This architecture aims to optimize inference performance and scheduling by treating agent memory as a distributed resource across enterprise infrastructure.
Moving KV cache from GPUs to persistent storage addresses the primary bottleneck in scaling long-context AI agents. For streaming and media enterprises, this shift allows for more complex, multi-session interactions without the prohibitive cost of dedicated HBM for every active agent. By treating memory as a distributed resource, companies can optimize hardware utilization across a fleet of machines rather than being limited by individual card capacity. This transition also turns transient session data into a long-term governance and training asset. Watch for whether this tiered approach becomes the standard for CoreWeave and other specialized cloud providers managing high-density inference workloads.
Vast Data has built significant commercial momentum around its AI infrastructure platform. In February 2026, Vast announced a $1.17 billion commercial agreement with CoreWeave to supply storage infrastructure for CoreWeave's GPU cloud, which serves as the deployment target for the tiered KV cache architecture described in this story. The company counts several Fortune 100 companies among its customers, including Nvidia, Google Cloud, Microsoft, and Cisco, positioning Vast as a storage layer beneath the largest AI inference deployments.
The technical architecture relies on Nvidia's Dynamo software as the orchestration layer that decides where KV cache lives across memory tiers. Vast published a detailed technical breakdown of how Dynamo coordinates inference by routing requests based on where relevant KV cache already exists, with a dedicated KV cache manager tracking context placement across GPUs, CPU memory, and persistent storage. The blog reports that fetching pre-computed KV cache from Vast storage takes three seconds compared to 65 seconds for GPU recalculation, a metric that quantifies the performance gain from offloading agent memory. Nvidia also introduced CMX, a shared memory tier that sits between GPU memory and storage, allowing context reuse across GPUs in a pod without duplication.
The governance dimension of KV cache offloading carries enterprise implications. Vast's CTO Alon Horev noted that once KV cache leaves the GPU, it contains sensitive user data vulnerable to manipulation or reverse engineering, requiring encryption, isolation between users, and retention policies. Vast has launched a confidential computing service for sensitive workloads to address these requirements, extending the tiered storage approach into compliance and data sovereignty use cases that enterprises deploying thousands of agents will need to satisfy.
Vast Data is offloading AI agent KV cache from expensive GPU memory to CPU and persistent storage. This tiered architecture prevents costly recalculations during long-running sessions and optimizes hardware utilization. By treating memory as a distributed resource, enterprises can scale complex AI agents while turning session data into governance assets.
Moving KV cache to persistent storage addresses the bottleneck in scaling long-context AI agents, preventing the need for expensive GPU recalculations and allowing for more efficient hardware utilization across distributed infrastructure.
According to Vast Data, fetching pre-computed KV cache from persistent storage takes approximately three seconds, compared to 65 seconds required for GPU recalculation.
Nvidia's Dynamo software acts as the orchestration layer that manages the movement of KV cache across memory tiers, while Nvidia's CMX provides a shared memory tier for context reuse across GPUs.
Because KV cache contains sensitive user data once it leaves the GPU, Vast Data has introduced a confidential computing service to provide encryption, isolation, and retention policies for enterprise compliance.
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source
