This week, I had the privilege of attending NVIDIA’s AI Infrastructure Summit briefing, where Dave Salvator and Shar Narasimhan walked us through what can only be described as a seismic shift in how we think about AI infrastructure. If you’re running data centers or planning AI deployments, this isn’t just another product announcement, it’s a fundamental rethinking of how compute, memory, and economics intersect in the age of reasoning AI.
Here’s what caught my attention: NVIDIA didn’t just announce faster chips. They’ve essentially redefined what “fast” means when you’re dealing with AI models that need to process the equivalent of 100,000 lines of code or an hour of video in a single context window. Let me walk you through why this matters for your infrastructure planning.
The MLPerf Reality Check: Leadership Across Every Category
The headline grabber from the briefing was NVIDIA’s complete dominance of MLPerf Inference v5.1. We’re talking about setting new performance records on every single data center inference test, not just winning, but holding per-GPU leadership positions across the entire suite.
But here’s what makes this interesting from an infrastructure perspective: the new tests aren’t academic exercises. DeepSeek-R1, the 671-billion parameter mixture-of-experts model, represents the kind of reasoning AI that enterprises are actually deploying. When Salvator mentioned that these models need to communicate between dozens of specialized “expert” networks simultaneously, it clicked for me why traditional GPU configurations struggle.
The numbers tell the story better than marketing slides ever could:
- DeepSeek-R1: 5,842 tokens/second/GPU in offline mode, 2,907 in server mode
- Llama 3.1 405B Interactive: 138 tokens/second/GPU with strict 4.5-second time-to-first-token requirements
- Performance gains: Up to 5x improvement over Hopper-based systems
What impressed me most wasn’t just the raw performance, it was how NVIDIA achieved it through what they call “disaggregated serving.” Instead of forcing a single GPU to handle both context processing and token generation (two completely different computational patterns), they split these workloads across specialized processors. Think of it like having separate teams for research and writing instead of making one person do both poorly.
Blackwell Ultra: Purpose-Built for Mixture of Experts
Let’s talk about why mixture-of-experts (MoE) models matter for your data center planning. Every major reasoning AI model today, from OpenAI’s GPT-4o to DeepSeek-R1 to Llama Fork, uses this architecture. These aren’t monolithic neural networks anymore; they’re collections of specialized sub-networks that activate based on the specific task at hand.
The challenge? These expert networks need to communicate constantly. Not occasionally-constantly. NVIDIA’s Blackwell Ultra addresses this with what amounts to a communication fabric designed specifically for AI workloads.
The Technical Foundation That Actually Matters
The NVL72 architecture connects 72 GPUs in a single domain where every GPU pair can communicate at 1.8 terabytes per second. The aggregate bandwidth hits 130 terabytes per second. To put that in perspective, that’s enough bandwidth to move the entire contents of the Library of Congress in about two seconds.
But bandwidth alone doesn’t solve the MoE challenge. NVIDIA’s software stack introduces several innovations that caught my attention:
NVFP4 acceleration represents a clever compromise between precision and performance. This 4-bit floating-point format maintains accuracy comparable to much higher-precision formats while enabling the specialized compute units in Blackwell Ultra to process data faster. During the briefing, Salvator emphasized that all their MLPerf submissions met strict accuracy requirements using NVFP4, proof that you don’t always need maximum precision for maximum performance.
New parallelism techniques address the unique communication patterns of MoE models. Traditional tensor and pipeline parallelism weren’t sufficient for DeepSeek-R1’s architecture, so NVIDIA developed expert parallelism for the MoE portions and data parallelism for attention mechanisms. They also created something called Attention Data Parallelism Balance, which intelligently distributes context queries to keep all GPUs productive while maintaining low first-token latency.
The result? Blackwell Ultra delivers up to 1.4x higher performance than the already impressive GB200 NVL72 systems, and about 5x higher throughput per GPU compared to Hopper-based configurations.
Enter Rubin CPX: Rethinking Context Processing
Here’s where the briefing took an interesting turn. Shar Narasimhan introduced what NVIDIA calls a “new class of GPU” – the Rubin CPX, purpose-built for massive context processing. This isn’t just another iteration; it’s recognition that million-token context windows require fundamentally different hardware approaches.
The use cases driving this development are already here. Software development AI that can understand entire codebases (100,000+ lines), video generation models that process hour-long content, and reasoning systems that maintain context across extended conversations. These applications push traditional GPU architectures beyond their design limits.
Why Context and Generation Need Different Hardware
The briefing included a particularly illuminating diagram showing the three-way relationship between compute requirements, memory bandwidth, and NVLink bandwidth for different phases of AI inference. Context processing (the “prefill” phase) is heavily compute-dependent, while token generation is much more memory and communication bandwidth-dependent.
Traditional inference deployments compromise by splitting GPU resources between these two very different workloads. Rubin CPX eliminates that compromise by creating processors optimized specifically for context processing, working alongside standard Rubin GPUs optimized for generation.
The specifications reflect this focus: 30 petaflops of NVFP4 compute with 128GB of GDDR7 memory (not the premium HBM used in standard GPUs, since context processing isn’t memory bandwidth-limited). For video applications, Rubin CPX integrates hardware video encoders and decoders directly on the chip.
The Economics That Change Everything
During the Q&A, someone asked about the range of ROI impacts across NVIDIA’s customer base. Salvator’s response was telling: the calculations were based on MLPerf data, not marketing projections. The numbers suggest that organizations can generate $5 billion in token revenue for every $100 million invested in Vera Rubin NVL144 CPX infrastructure-a 50x return.
These aren’t theoretical calculations. The math works because higher-performance infrastructure can process more queries per second, generate more tokens, and do it all more efficiently. When you’re running inference at scale, small percentage improvements in throughput translate to massive differences in revenue potential.
Consider the practical example Salvator shared: organizations evaluating cloud instances often focus on hourly pricing without considering throughput differences. A more expensive instance that processes 4x more queries per hour isn’t just cost-effective-it’s dramatically more profitable.
Deployment Realities and Infrastructure Planning
The Vera Rubin NVL144 CPX platform packs 8 exaflops of AI performance and 100TB of fast memory in a single rack-7.5x more AI performance than GB300 NVL72 systems. But what caught my attention was NVIDIA’s flexibility in deployment options.
For organizations with existing Vera Rubin NVL144 systems, NVIDIA offers dedicated Rubin CPX racks that pair with existing infrastructure for generation processing. The racks are fully compatible in terms of power, cooling, and networking, which means enterprises can adopt CPX capabilities without wholesale infrastructure replacement.
This modular approach addresses a real concern for data center managers: how do you adopt cutting-edge AI capabilities without stranding existing investments? NVIDIA’s answer is architectural compatibility that allows incremental upgrades.
The Liquid Cooling Reality
Both Blackwell Ultra and Rubin CPX require liquid cooling. This isn’t surprising given the power densities involved, but it’s worth planning for if your facility currently relies on air cooling. The good news is that NVIDIA builds on the proven MGX rack design introduced with Grace Blackwell, so the cooling infrastructure isn’t experimental.
Industry Validation and Early Adoption
The briefing featured endorsements from companies already working with million-token context applications. Cursor’s CEO Michael Truell mentions delivering “lightning-fast code generation” for developer productivity. Magic’s Eric Steinberger described their 100-million-token context window that allows AI models to see entire codebases, documentation, and interaction history without fine-tuning.
These aren’t aspirational use cases, they’re production applications that existing infrastructure can’t handle efficiently. Runway’s focus on video generation with “longer context and more flexible, agent-driven creative workflows” represents another category of applications driving demand for specialized context processing hardware.
What This Means for Your Infrastructure Strategy
NVIDIA has moved to a one-year architecture cadence, accelerating product release cycles to match AI innovation pace. Rubin CPX availability at the end of 2026 gives data center professionals time to plan, but the message is clear: infrastructure planning must account for workloads that didn’t exist two years ago.
The shift toward disaggregated serving architectures represents a fundamental change in how we think about AI infrastructure. Instead of general-purpose processors handling every aspect of AI inference, we’re moving toward specialized processors optimized for specific phases of the AI pipeline.
For immediate planning, consider evaluating current inference workloads for optimization opportunities. The software optimizations NVIDIA demonstrated-NVFP4 quantization, disaggregated serving, attention mechanism optimization-can deliver performance improvements on existing hardware.
Looking ahead, million-token context applications will reshape capacity planning requirements. Video generation, software development AI, and advanced reasoning systems all demand infrastructure capabilities that go well beyond traditional inference deployments.
The ecosystem support is already there. Sixteen partners submitted MLPerf results this round, including Azure, Dell Technologies, HPE, Lenovo, Oracle, and Supermicro. The performance results were remarkably close to NVIDIA’s reference implementations, demonstrating that these optimizations translate effectively across the ecosystem.
The Bigger Picture
What struck me most about this briefing wasn’t just the technical achievements – it was the recognition that AI workloads are evolving faster than traditional infrastructure planning cycles can accommodate. NVIDIA’s response isn’t just faster chips; it’s a complete rethinking of how compute, memory, and communication resources should be allocated for different phases of AI processing.
The introduction of AI Factory gigascale reference designs signals NVIDIA’s recognition that these infrastructure decisions can’t be made in isolation. Successful AI deployments require coordination between compute platforms, power systems, cooling infrastructure, and facility design.
For data center professionals, the takeaway is clear: AI infrastructure planning must evolve from “what’s the fastest GPU” to “what’s the right combination of specialized processors for our specific workloads.” The era of one-size-fits-all AI infrastructure is ending, replaced by purpose-built systems optimized for the specific demands of modern AI applications.
The numbers from MLPerf validate NVIDIA’s technical approach, but the real validation will come from production deployments handling million-token context workloads that simply weren’t possible before. For those planning AI infrastructure investments, the question isn’t whether these capabilities will be needed-it’s how quickly you can position your infrastructure to take advantage of them.
##
