NVIDIA announces NVHBM custom memory technology for AI accelerators

NVIDIA has introduced NVHBM, a custom high-bandwidth memory technology designed to improve performance and power efficiency for next-generation AI accelerators, or XPUs. By shifting the memory controller from the compute die into the 3D HBM stack itself, the company aims to address the growing demand for higher throughput in large-scale AI infrastructure.

The architectural change offers several performance benefits compared to the JEDEC HBM4E standard. NVIDIA reports that NVHBM provides up to 30% higher memory bandwidth per stack and reduces HBM power consumption by 15%. Furthermore, the design utilizes a custom physical memory interface that reduces the I/O footprint by up to 67%, which in turn frees up to 25% more silicon area on the compute die for additional processing capabilities.

This increased area efficiency simplifies interposer routing, allowing for more flexible layouts when integrating multiple HBM stacks alongside large compute dies. The reduction in power draw also provides significant thermal headroom, which NVIDIA notes is particularly beneficial for large-scale data centers. In a hypothetical 1-gigawatt facility utilizing 2,000W XPUs, the company estimates that these power savings could support up to 15,000 additional compute units.

NVIDIA plans to validate NVHBM across multiple memory vendors to streamline the qualification process for its partners. Amazon's Annapurna Labs has been named as the first collaborator to work with the technology, with plans to integrate it into future infrastructure designs. This initiative is part of the broader NVLink Fusion program, which provides hyperscalers and AI-native companies with access to NVIDIA's scale-up and scale-out technology stack.