More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

In a November 5 blog post, NVIDIA's Amr Elmeleegy argues that AI's growing memory squeeze is better addressed by designing the whole computing stack together than by adding more high-bandwidth memory (HBM) to each GPU. NVIDIA says larger models, longer contexts and agentic workloads, some processing up to 3 million tokens, are straining HBM while supply tightens and prices climb. Extra capacity eventually stops helping, the company explains, because bandwidth can't feed data to compute fast enough, a limit known as the memory wall.

NVIDIA describes several ways to ease the pressure. Its 4-bit NVFP4 format shrinks model weights and KV cache compared with FP8, and the company says models from Black Forest Labs, Thinking Machines Lab, Reflection, Meta and StepFun have shipped NVFP4 checkpoints. Weights can also be spread across up to 72 GPUs in an NVL72 system. Idle context can wait in the large LPDDR memory attached to the Vera CPU, or be moved by Dynamo inference software to CMX, a shared storage tier supported by partners including Dell, HPE, IBM, NetApp and VAST. For retrieval workloads, the cuVS library keeps search indexes on CPU and storage rather than HBM.

The headline claim rests on SemiAnalysis AgentX results cited by NVIDIA: Vera Rubin NVL72 delivers up to 30x the throughput per dollar of GB300 NVL72 at 125 tokens per second per user, with HBM per GPU up only about 3%. If those vendor-reported figures hold, they suggest that as memory stays scarce and costly, inference economics may depend as much on system design as on raw capacity.