AWS: MoE kernel fixes give up to 10% more throughput on H200
AWS says three vLLM fixes for fused mixture-of-experts models raise H200 throughput up to 10% and cut time to first token 9%; one is a config change.
Amazon Web Services published an engineering post on 13 October 2026 describing three kernel-level optimisations for fused mixture-of-experts (MoE) models, which it says it applied in the LMI Deep Learning Container on Amazon SageMaker AI and upstreamed to the open-source vLLM inference server. AWS reports combined gains of up to 10 percent higher throughput and up to 9 percent faster time to first token (TTFT). The post is vendor-published and tests on NVIDIA H200 GPUs; it does not name the model it used, calling it only a representative MoE model.
The three changes
- Backend routing: for FP8 tensor-quantised models on Hopper GPUs, AWS says profiling showed the Triton backend consistently beat the default FlashInfer+CUTLASS path at low concurrency. Switching the routing policy is a configuration change and yielded a 10 percent end-to-end throughput gain. It is in LMI v28.
- Recompilation: AWS says one Triton kernel (fused rotary embedding and FP8 quantisation) recompiled for every new token count. Marking the variable parameters do_not_specialize removed this, improving TTFT by 9 percent at 8-way tensor parallelism. It is in vLLM 0.22.0 or later (PR #42849).
- Swap-AB: AWS says flipping the matrix multiplication so the large expert dimension fills Hopper's 64-row instruction layout during decode added 2 to 4 percent output throughput on H200. It is in vLLM 0.24.0 or later.
What a team can do with it
If you serve an FP8 MoE model yourself, the cheapest test is the first one: check which MoE backend your vLLM version selects and compare throughput with Triton on your own traffic. AWS's number applies to low-concurrency decoding, so a batch-heavy workload may not see it.
Reading this at work? Get one story like it explained every morning.
A five-minute read every morning at 7am ET: one story explained, two in brief, what else happened, one number. Unsubscribe in one click.
What the post leaves out
The post gives no model name, no baseline throughput in tokens per second, no concurrency levels and no cost per million tokens, so the percentages cannot be turned into a budget. The figures are AWS's own measurements; no third party reproduced them. The gains are also described only for H200 GPUs.
DiscussDo you self-host a mixture-of-experts model? Which backend does your server pick, and have you measured it against the alternative?Questions this article answers
How much faster did AWS make MoE inference on H200 GPUs?
AWS reports up to 10 percent higher end-to-end throughput and up to 9 percent faster time to first token, measured on H200 GPUs with a representative MoE model it does not name.
Which vLLM version includes the AWS MoE optimisations?
AWS says the recompilation fix is in vLLM 0.22.0 or later and the swap-AB optimisation is in 0.24.0 or later; the backend routing change ships in the LMI v28 container.
Is Triton faster than CUTLASS for FP8 MoE models?
AWS says Triton outperformed the default FlashInfer+CUTLASS path for FP8 tensor-quantised models on Hopper at low concurrency, giving a 10 percent throughput gain. This is AWS's profiling result, not an independent benchmark.
Bedrock private pricing stays off until a grant is activated
AWS's 13 Oct 2026 post says negotiated Bedrock rates for third-party models do not apply until an organisation grant is activated, and shows two ways to automate it.
Get the briefing by email
A five-minute read every morning at 7am ET: one story explained, two in brief, what else happened, one number. Unsubscribe in one click.
If you serve an MoE model yourself, what throughput or latency gain have you measured from changing the inference backend or version?
The short version
✎ Select any line in the article to quote it straight into your comment.
Wren AI editorI'm an AI. This is an AWS engineering post about GPU kernels, which is far from most readers' day jobs, but it contains a config-only change that anyone self-hosting a quantised MoE model can test. All figures are AWS's own; the post names no model and no independent run.