“The memory system of AIs is going to cause the storage system to be completely revolutionized.”
– Nvidia founder and CEO Jensen Huang
At GTC Taipei in June 2026, Nvidia founder and CEO Jensen Huang pointed to the memory system as one of the hardest parts in AI infrastructure. This challenge encompasses managing KV caching for the agent’s working memory, as well as retrieving structured and unstructured data and establishing data ontology.
Figure 1. Nvidia GTC Taipei 2026 showing the architecture of an AI agent, defined as LLM plus Harness.
Source: Nvidia
To address the surging KV cache storage demands of the AI inference era, Nvidia introduced the CMX Context Memory Storage Platform in January 2026, managed by the BlueField-4 DPU, which adds a pod‑level context tier between local SSD and shared storage.
Meanwhile, the rise of Agentic AI is reshaping CPU architecture. Jensen noted that agents live in a world of nanoseconds, where every moment of waiting prevents them from advancing to the next step, making ultra-low latency the primary requirement. With Nvidia and Arm both launching CPU rack solutions purpose-built for agents, the industry is shifting from throughput-oriented to latency-oriented architectures, opening up an incremental market for CPU RAM.
Test-time Scaling “Thinking”: >5X Tokens Per Year
According to Nvidia’s public data, the average output token count per question has surged at a rate exceeding 5x per year since the second half of 2024, reaching approximately 30,000 to 40,000 tokens, indicating that the industry has entered the Test-time Scaling “Thinking” stage of Nvidia’s Three Scaling Laws. This explosion in per-question token output translates directly into greater demands on memory and compute resources.
Figure 2. Nvidia scatter plot showing average output tokens per question from 2023 to 2025. Reasoning models surge to 10,000 to 30,000 tokens driven by Test-Time Scaling “Thinking,” reflecting growth exceeding 5x per year.
Source: Nvidia
In the AI inference era, hardware requirements for AI chips and overall systems differ fundamentally from those of AI training. Inference places three key demands on hardware:
- higher queries per second (QPS)
- longer context windows
- more inference steps and agentic AI loops
Each of these drives structural changes in memory demand. We will examine this across three dimensions: model weights, KV cache, and agentic AI.
