AutoBench: no LLM router beat one mid-priced model on cost and quality

AutoBench, a company that builds LLM evaluations, published a benchmark of routers on Hugging Face on 26 October 2026. Its author, Peter Kruger, reports that none of seven commercial and open-source routers beat simply calling one mid-priced model on quality and cost together. The routers were called between 30 August and 1 September 2026, and AutoBench says it wrote down its scoring rules on 29 August, before calling any of them.

How the test worked

The test is 252 long, open-ended prompts across ten domains and five difficulty levels. Twenty models answered each prompt once, and a committee of ten LLM judges scored every answer. AutoBench stored the billed cost beside each score. A router only had to name a model; AutoBench then looked up that model's stored score and cost, so a router could not improve its result by writing a better answer.

A pick counted as a hit when its stored score fell within a set window of the top three models on that question. AutoBench calls this Delta-Hit, or Best Answer rate. A random pick from the pool scores 64.7% at $0.0115 per question, so AutoBench says results should be read against that line, not against 100%.

What AutoBench reports

• A per-prompt oracle that picks the cheapest model still inside the quality bar scores 100% at $0.0018 per question. That is the saving routing promises.
• Always calling GPT-5.6 Sol, the top model in the pool, scores 87.5% at $0.079. Always calling GPT-5.6 Luna scores 85.3% at $0.0043, most of Sol's quality at about 5% of the bill.
• No router setting beat Luna on both axes. In 10,000 bootstrap resamples, none ever did.
• Five of the 17 settings sent at least 90% of prompts to one model, so their quality is that model's quality. On Orca and OpenRouter Auto, the cost setting changes which model takes the traffic.
• Not Diamond's setting at 0 names Sol on every prompt (87.5% at $0.0787). At setting 5 cost falls about 64-fold and quality drops to 46.6%, below random.
• Azure's Model Router, on its quality option, scored 85.3% at $0.065: Luna's quality at fifteen times the price, from a catalogue of three of the 20 models.
• Across 59 pairs of routers from different vendors, the median share of prompts where both named the same model was 0.

What the result does not show

AutoBench lists its own limits. Several routers could reach only part of the pool: Not Diamond seven of the 20 models, NadirClaw four, Azure three and Brick three, so the post says those notes describe behaviour inside a slice, not a ranking. Brick was run as shipped on models it was not tuned for. The scores come from LLM judges, not from outcomes in anyone's production traffic, and the post says adding questions is the costly step, so the next version needs a larger corpus.

The author also sells benchmarking on a customer's own traffic through autobench.org, and the post's author page lists a separate post promoting a router with free credit. Neither is evidence against the numbers, which the post says can be recomputed from a released dataset, but a reader should know who is grading.

What a team can take from it

• Price the static alternative first. Run your own prompts through your best mid-priced model and compare quality and cost with the router.
• Ask what share of traffic the router sends to its top model at each cost setting. At 90% or more, you are buying that model.
• Test the middle of the dial. A setting that jumps from always-strong to worse than random is not a cost-quality curve.
• Check which board backs a vendor's claim. AutoBench says exact-match accuracy on short questions is not the same test as open-ended prompts that cost far more to answer.