Taste-Bench: best model picks the better agent path 59.7% of the time

Researchers from Microsoft and City University of Hong Kong published Taste-Bench on arXiv on 22 September 2026 (paper 2609.25804, 'The Tasteful Agent'). It measures what the authors call an agent's taste: choosing the better direction at a real decision fork in a long task. The best of 14 models tested, GPT-5.6 Sol, got 59.7% of the 502 questions right.

How the benchmark works

Existing agent benchmarks mostly check whether a task finished. Taste-Bench checks the choices along the way. According to the README, each question shows a frozen agent trajectory stopped at a fork, plus two candidate next steps, and asks which is better. The label comes from what happened later, not from an expert.

Forks come in two kinds. In a parallel fork, two independent attempts at the same task diverge and end with different outcomes. In a detour fork, one agent abandons a direction after a failure and recovers inside the same run. The README says 502 questions survived filtering from 4,657 mined forks: 390 on software engineering (SWE-bench and SWE-bench Pro runs) and 112 on machine-learning research (METR's MALT transcripts).

A model scores only if it answers correctly in both option orders, so guessing scores 25%. The README states that unparseable answers count as wrong.

What the leaderboard shows

• GPT-5.6 Sol 59.7% and GPT-5.5 59.5% lead; the README lists Claude Opus 5 at 55.5%, Grok 4.5 at 54.6%, GLM-5.2 at 53.9% and Claude Sonnet 5 at 51.6%.
• DeepSeek V4 Flash scores 43.3% and Mistral Medium 3.5 scores 37.7%. Grok 4.20 Reasoning scores 15.7% after 459 of 1,004 presentations went unparsed.
• Detour forks in software engineering were hardest: GPT-5.6 Sol scores 48.1% there against 75.8% on parallel engineering forks, and Claude Opus 5 scores 35.3%.
• The scores are the authors' own, run under one protocol in August 2026; the README invites others to submit results by pull request.

The abstract reports two further findings. Forks whose deciding evidence appears later in the trajectory are much harder for every model. And a larger reasoning budget does not improve accuracy. The Decoder, summarising the paper, puts average accuracy at 62.3% when the clue was visible early and 21% when it only appeared after much later work.

Taste can be trained, in the authors' test

The authors distilled the judgment of a teacher that had seen the outcomes into a student model. The Decoder reports the student was Qwen3.6-27B and that its accuracy on unseen tasks rose from 30.0% to 47.9%. With the student's advice added at each fork, agent success on 41 held-out SWE-bench Pro tasks rose from 14.6% to 33.7%; the Decoder says perfect advice would have reached 39%. I did not verify these figures against the paper's tables.

What it does not tell you

The questions come from coding and ML-research runs, so the results say little about support, finance or operations agents. The benchmark asks a model to judge a recorded fork, not to act. The dataset is gated to limit training contamination, so outsiders cannot inspect the questions without accepting its terms. Cost per question is not reported in the sources I read.

What a team can take from it

• Do not assume a more capable model, or more reasoning effort, will choose a good approach unprompted; log where long runs forked and which fork wasted the budget.
• Insert a human or a second model at irreversible forks, such as a schema change or a rewrite, rather than at every step.
• Track failed detours in your own agent traces. They are the cheapest source of the labelled comparisons this paper used.