Sorting prompts speeds up MoE batch jobs by up to 17%

On 19 October 2026 Abdennacer Badaoui published a Hugging Face community article reporting that reordering prompts before an offline batch job speeds up mixture-of-experts (MoE) models in unmodified vLLM. The author reports gains of 12 to 17% on Qwen3-30B-A3B and 7 to 12% on gpt-oss-120b, from sorting the prompt list before calling llm.generate.

Why order matters, per the author

In an MoE model, each decoding step has to read the weights of every expert that any request in the batch routes to. The post says a batch of 32 random requests touches about 94 of the 128 experts per layer in Qwen3-30B-A3B, while requests from the same domain overlap much more. It reports that each domain concentrates 68 to 80% of its routing on a quarter of the experts, and that one-domain batches read 19 to 25% fewer experts than mixed ones.

The author measured the link directly: MoE layer time rose by about 2.9 microseconds for each extra expert touched. Because attention, the KV cache and scheduling do not benefit, the end-to-end gain is smaller than the bytes saved, roughly half for Qwen by the author's estimate.

What was measured

The test used 768 prompts from six domains (code, math, creative writing, French, Arabic and Chinese), each generating exactly 256 tokens, on one AMD MI300X with vLLM 0.27.1. The only difference between runs was prompt order. Sorted by domain, Qwen3-30B-A3B reached 2,286 tokens per second against 1,949 shuffled at a batch size of 32, a 17.3% gain. gpt-oss-120b gained 6.9 to 12.2% across batch sizes 16 to 128.

On two GPUs with expert parallelism, the post reports +12.9% at batch 64. With natural output lengths, the sorted job finished 16.6% sooner in wall-clock time, in the author's test.

Without labels

Real traffic rarely carries domain labels. The post says embedding each prompt with Qwen3-Embedding-0.6B and clustering with k-means matched the true labels up to batch 32 on the six-domain set (+13.0% at batch 64). On UltraChat, which has no clear domains, it gave +5.1 to 5.5% on Qwen3-30B-A3B and +2 to 4% on gpt-oss-120b, with nothing measurable at batch 128. For streaming jobs, sorting within windows of 128 or 256 prompts kept +11.7% and +14.8% at batch 32.

What the post leaves out or limits

This is one author's benchmark on one GPU type, not an independent replication, and the author names the limits: dense models gain nothing, prefill-heavy work touches nearly every expert anyway, and holding requests back to group them adds queueing time in online serving. The author also says reordering changes which requests share a batch, which can flip near-tie outputs in bf16; sorting changed outputs about as much as reshuffling did, and bit-exact results need batch-invariant kernels.

What to do

• If you run offline evals, synthetic data or labelling on a self-hosted MoE model, time one job with shuffled and sorted prompts on your own hardware before changing anything else.
• Group by a label you already have, such as task type or language; if you have none, try embedding clusters, and expect single-digit gains.
• Keep a fixed seed and compare outputs on a sample, since the order change can alter some responses.