NVIDIA shows ~10× faster dropless MoE training in JAX

NVIDIA published a developer blog showing that Transformer Engine with JAX can deliver about a 10.4× end-to-end throughput gain for dropless Mixture-of-Experts training versus an unoptimized baseline on NVIDIA GB200-class GPUs. On DeepSeek-V3, the stack moved measured performance from about 103 TFLOPS per GPU to about 1,068 TFLOPS per GPU, and the company reported about 97% scaling efficiency out to 1,024 GPUs on NVIDIA GB300 NVL72 hardware. Dropless means every routed token is processed — no capacity-based dropping or padding — which is harder for GPUs because experts get uneven token counts.

Training cost and speed still gate who can run frontier MoE models. A same-day NVIDIA primary showing about 10× dropless MoE throughput in JAX is a concrete infra signal for labs standardizing on MoE, not a rumor about a new chip.

NVIDIA’s developer blog published “Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine,” dated September 14, 2026, by Seonghee Lee, Jeremy Berchtold, Phuong Nguyen, Teddy Do, and Tejash Shah. That company post is the filing event. Transformer Engine is NVIDIA’s library for speeding up Transformer models on its GPUs. JAX is an open software stack labs use to train large models. These are NVIDIA’s measured stack claims. This desk did not rerun the training job.

Company numbers, as the blog prints them: an unoptimized DeepSeek-V3 Mixture-of-Experts baseline on NVIDIA GB200 reached about 103 TFLOPS per GPU, with inter-GPU communication taking about 84% of accumulated kernel time. With JAX and Transformer Engine optimizations, that rose to about 1,068 TFLOPS per GPU — about a 10.4× gain. TFLOPS means trillion floating-point operations per second: a measure of how hard the GPU is working. File 103, 1,068, 84%, and 10.4× as NVIDIA’s. This desk did not time a run, and it is not treating those figures as an independent bake-off.

Mixture of Experts, or MoE, is a model that routes each token — a small chunk of text — to a few specialist sub-networks instead of one giant dense layer. Dropless MoE means every token goes to its chosen expert or experts. The system does not drop overflow tokens when one expert gets overloaded, and it does not pad the rest to a fixed slot count. NVIDIA says that preserves training quality and is harder on hardware because experts get uneven token counts.

Those uneven counts produce what the blog calls ragged tensors — rows that are not a clean rectangle. A grouped GEMM, as NVIDIA explains it, is a matrix-multiply that handles those uneven expert groups in one call, instead of looping or padding. File the ragged-tensor picture and the grouped-GEMM job as NVIDIA’s explanation. How the kernels are wired is in Sources.

NVIDIA also names faster expert parallelism, or EP: spreading different experts across GPUs, then dispatching tokens to the right chip and combining the results. The blog says communication was the baseline bottleneck. File dispatch, combine, and the communication share as NVIDIA’s. This desk did not probe the network.

On scaling, NVIDIA says the full stack sustains about 97% scaling efficiency at 1,024 GPUs on NVIDIA GB300 NVL72 when training DeepSeek-V3 671B-class MoE. 671B means about 671 billion parameters — the model’s size. File 97% and 1,024 GPUs as NVIDIA’s scaling claim. This desk did not rent that rack.

Availability, as NVIDIA tells it: the optimizations ship in the company’s MaxText container on NVIDIA NGC, with Transformer Engine built in. The blog points to MaxText MoE configuration docs. File that as NVIDIA’s download path. Container tags and flag lists stay in Sources.

This is a software and kernel training-stack result NVIDIA published. It is not a new GPU SKU launch, not a DeepSeek partnership announcement, and not a pricing change. Do not invent a customer win, a revenue line, or a third-party rerun of the 10.4× number.

CONFIRMED here: NVIDIA’s 14 Sep 2026 developer blog — PRIMARY company source. The 10.4×, 103-to-1,068 TFLOPS, and 97% figures are NVIDIA’s measured stack claims, not an independent replication. NOT claimed: a new chip launch, a DeepSeek deal, a price change, that this desk reran the job, a stock move, or investment advice. Distinct from the already-filed nvidia-cuda-q-logical.