NVIDIA has introduced a new quantization-aware distillation (QAD) workflow for its Nemotron 3.5 Lightning model, designed to significantly reduce memory requirements while preserving performance. By utilizing the NVIDIA Model Optimizer, developers can compress the 30B mixture-of-experts model from its original 66 GB full-precision footprint down to 22 GB. This process leverages aggressive W4A16 quantization, which NVIDIA reports can unlock up to 4x higher throughput for high-volume agentic and coding tasks.
The QAD process functions as a two-stage pipeline. In the first stage, post-training quantization (PTQ) creates a quantized student model from the full-precision teacher. Because PTQ alone can lead to accuracy degradation when pushed to aggressive settings, the second stage employs quantization-aware distillation. During this phase, the student model is trained against the frozen teacher using KL divergence loss, allowing it to adapt to quantization noise and recover accuracy that would otherwise be lost.
NVIDIA notes that this method consistently outperforms standard PTQ in median accuracy recovery across various benchmarks. By quantizing Mamba linear layers to W4A16 rather than FP8, developers can achieve substantial latency and size gains without sacrificing the model's utility. The workflow is supported by Megatron-Bridge, which provides the necessary recipes and training configurations to facilitate deployment.
In parallel with these developer tools, the Nemotron 3.5 Lightning model is now available in Amazon SageMaker JumpStart. This integration allows users to deploy the 30B model directly within the AWS environment, providing a streamlined path for those looking to implement the model in production workloads. The availability in SageMaker JumpStart aims to simplify the adoption of memory-efficient large language models for enterprise applications.