At the Advanced Storage Conference (ASC) held on October 22nd in Shanghai, Huawei presented its solution to the problem limiting the inference of large models: moving the KV Cache between compute blocks, storage, and the network. The conference was organized by the Open Data Center Committee (ODCC).
In a presentation on storage and network convergence for new AI workloads, Huawei described the problem and provided laboratory test results. The KV Cache is used to store intermediate attention results, allowing the model to avoid recalculating earlier tokens. However, as agentic applications develop, which require very long contexts and multi-turn interactions, the size of this cache grows rapidly.
Huawei notes that the cumulative cache size per request, the amount of cache supported in memory, and the traffic between nodes lead to the total KV Cache volume of an inference cluster approaching the petabyte scale. The resulting spikes in inter-node cache traffic place additional strain on the data center's network infrastructure.
Three obstacles identified by Huawei
The company identifies this bottleneck through three barriers. From a computational perspective, accelerator cores have weak I/O performance, and I/O operations consume valuable computational resources. Regarding latency, data transfer between layers passes through multiple protocol and semantic transformations, which is exacerbated by long network delays. At the IOPS level, traditional software stacks prevent backend storage from achieving its native performance.
Huawei's solution covers three levels. On the accelerator side, the company aims for processors to directly initiate cache I/O operations, moving from direct access by AI CPUs and vector cores to highly granular SIMT-level I/O operations. On the data storage side, it proposes pushing operators down to vector acceleration, close to the storage itself. And at the network level, it utilizes the Hyper-Converged Fabric (HCF) architecture within the Stellar AI Fabric, with AI QoS scheduling configured in coordination with the end network specifically for KV Cache traffic. Huawei asserts that only the network layer can improve storage I/O performance by more than 30%.
Key figures were obtained during Huawei's internal laboratory tests. According to the company, in the GLM-5.2 model scenario, direct offloading of the KV Cache achieved 7.3 times higher inference throughput (TPS) compared to the traditional memory-based approach. An additional 20% improvement in inference efficiency was achieved through the coordination of I/O operations between storage and network. Huawei did not disclose the full testing configuration in its announcement.
The Stellar AI Fabric and HCF architecture were presented at HUAWEI CONNECT 2026 in September. The presentation at ASC does not signify a new product launch; its value lies in presenting the first dataset on KV Cache related to this design, as well as Huawei OceanStor's work on shared context memory storage for AI clusters.