Nvidia Rubin Ultra shifts to 8-hi HBM to solve wafer shortages

The transition to shorter HBM stacks signals a shift from capacity-hungry training to bandwidth-constrained inference. As rack-scale solutions like the NVL72 provide massive aggregate memory, the marginal utility of ultra-dense HBM per chip is diminishing. By prioritizing 8-hi and potentially 4-hi configurations, Nvidia can double or triple the effective bandwidth harvested from each scarce DRAM wafer. This move forces a software-hardware co-design pivot where developers must use techniques like KVCache offloading to standard DRAM. For the broader ecosystem, this suggests that future accelerator performance will be measured by token throughput per dollar rather than raw memory specs, potentially easing the supply chain bottleneck that has defined the AI infrastructure market.

Nvidia's decision to reduce HBM stack height on Rubin Ultra arrives amid intensifying competition for DRAM wafer capacity across the AI accelerator market. The company's broader NVIDIA Vera Rubin platform, which includes the GB300 NVL72 rack-scale system, has already drawn scrutiny from analysts tracking how memory bandwidth and capacity trade-offs affect inference economics. Ericsson launched its AI in RAN commercial software subscription on June 11th, claiming up to 20% higher downlink throughput and up to 10% better spectral efficiency across more than 15 live deployments using existing baseband silicon, illustrating how AI inference workloads are proliferating across edge and network infrastructure, each competing for the same memory and compute resources that Nvidia's data center GPUs consume. The shift to 8-hi stacks reflects a calculation that aggregate rack-level memory, rather than per-chip capacity, will define performance ceilings for inference at scale. The business implications extend beyond Nvidia's own product line into the broader HBM supply chain. SK Hynix and Samsung, the two dominant HBM manufacturers, have been expanding DRAM wafer capacity aggressively, yet demand from AI accelerator vendors continues to outpace supply. Nokia's entire RAN strategy is now built on its close partnership with Nvidia, cemented by the chipmaker's $1 billion investment in the Finnish company, demonstrating how Nvidia's GPU and networking silicon is becoming embedded in telecom infrastructure that itself demands high-bandwidth memory for AI-driven workloads. This cross-sector pull on HBM wafers, spanning data center training clusters, inference farms, and now network equipment, helps explain why Nvidia is optimizing for bandwidth-per-wafer rather than raw capacity per package. On the technical side, the 8-hi versus 4-hi debate centers on bandwidth-to-capacity ratios and their impact on cost-per-token for large language model inference. Ericsson describes the network as becoming an intelligent fabric, a distributed mesh connecting not just devices but agents in sensors, cars, glasses, edge nodes and cores, a vision that depends on inference accelerators delivering high throughput at lower memory footprints. The company's CTO Erik Ekudden highlighted that uplink traffic could triple over the next five years, driven by AI glasses, persistent voice interaction, and real-time video, workloads that favor bandwidth-optimized inference over capacity-heavy training. For Nvidia, the Rubin Ultra's 8-hi configuration represents a pragmatic middle ground: sufficient capacity for large model weights while preserving the bandwidth density needed to keep token generation costs declining as inference becomes the dominant AI workload. As enterprises shift private 5G architectures to metro edge data centers, the demand for these optimized inference chips will only intensify.