NVIDIA just did something big in the world of AI performance. They managed to hit over 1,000 tokens per second (TPS) per user using the Llama 4 Maverick model and Blackwell GPUs. This achievement was independently checked by the AI benchmarking service Artificial Analysis, showing a major step forward in large language model (LLM) inference speed.
Technological Breakthroughs
The breakthrough happened on one NVIDIA DGX B200 node with eight NVIDIA Blackwell GPUs. These bad boys handled over 1,000 TPS per user on the Llama 4 Maverick, a 400-billion-parameter model. This makes Blackwell the top hardware choice for deploying Llama 4, whether you want to max out throughput or cut latency down to size. In high throughput setups, you can hit up to 72,000 TPS/server.
Optimization Tactics
NVIDIA went all out on software optimizations using TensorRT-LLM to milk the Blackwell GPUs for all they’re worth. They even cooked up a speculative decoding draft model using EAGLE-3 techniques, giving a fourfold speed boost compared to older baselines. These upgrades keep response accuracy in check while ramping up performance, leaning on FP8 data types for operations like GEMMs and Mixture of Experts, guaranteeing accuracy on par with BF16 metrics.
The Importance of Snappy Response Times
Balancing throughput and latency is key in generative AI applications. For situations where quick decisions are a must, NVIDIA’s Blackwell GPUs shine by slashing latency, as proven by the TPS/user record. The hardware’s knack for juggling high throughput and low latency makes it a go-to for all sorts of AI tasks.
CUDA Kernels and Speculative Decoding
NVIDIA got down and dirty optimizing CUDA kernels for GEMMs, MoE, and Attention operations, using spatial partitioning and slick memory data loading to max out performance. They also threw speculative decoding into the mix to turbocharge LLM inference speed, employing a smaller, speedier draft model to predict speculative tokens, verified by the bigger target LLM. This approach brings in some major speed boosts, especially when the draft model’s predictions hit the mark.
Programmatic Dependent Launch
To keep the performance train rolling, NVIDIA tapped into Programmatic Dependent Launch (PDL) to cut down GPU idle time between consecutive CUDA kernels. This trick lets kernels overlap, beefing up GPU utilization and ironing out any performance hiccups.
NVIDIA’s feats underline their domination in AI infrastructure and data center tech, setting fresh benchmarks for speed and efficiency in AI model deployment. The strides in Blackwell architecture and software optimization keep pushing the envelope of what’s doable in AI performance, guaranteeing snappy, real-time user experiences and sturdy AI applications.
