Microsoft's ThinkingBox: best agents pass all 20 runs on 47.5% of tasks

Microsoft and Hugging Face published ThinkingBox on 3 October 2026, a benchmark that grades an AI agent by what it leaves in the backend rather than what it says. The authors report that across 507 stateful business workflows (retail, auto insurance, travel, a neobank and consulting), each run 20 times against 18 models, the best single-attempt score was 67.16%, from Claude Opus 5.5. The figures below are the authors' own; Microsoft and Hugging Face released the harness and dataset, so teams can check them.

Why they grade the database

The post opens with a retail example: an agent made nine well-formed tool calls, read the refund policy correctly, then closed a delayed-delivery ticket as solved when the required end state was on hold. A grader checking tool calls would pass it; the database check fails it. In a 121,680-trial ablation across 12 models, 79,853 attempts failed the executable checks, and the authors report that 67.24% of those failures still ended cleanly, called a state-changing tool and showed no final tool error.

One good run is not reliability

Each task runs 20 times from an identical clean backend. The authors report three numbers: pass@1 (share of all attempts that succeed), pass@20 (tasks solved at least once) and the count of tasks passed on all 20 attempts.

• Claude Opus 5.5 led pass@1 at 67.16% and passed 241 tasks (47.53%) on every attempt. Claude Opus 5 scored 66.50% and also passed exactly 241. The authors say the half-point gain bought no additional dependability.
• Kimi-K3, the strongest open-weight model in the test at 57.37% pass@1, solved 476 of 507 tasks at least once but passed only 68 (13.41%) on all 20 attempts.
• Scores swing by domain: Claude Opus 4.6 scored 68.62% on retail and 8.30% on auto insurance. Across the listed models, retail averaged 59.52% and auto insurance 33.83%.

What dependability costs

The authors priced each model's recorded token use at undiscounted OpenRouter list rates, excluding quantised endpoints. They call the result a comparative index, not an invoice. GPT-5.6 Sol had the lowest cost per successful attempt at $0.127, but the authors put its cost per dependable task (20-run campaign cost divided by tasks passing 20 of 20) at $9.76. GPT-5.4 was lowest at $6.80 for 128 dependable tasks; GPT-6 Astra reached 231 at $7.45; Claude Opus 5.5 reached 241 at $7.80; Claude Opus 5 reached the same 241 at $13.30.

Where agents fail

The authors assign each failed trace one diagnostic signature. Tool usage accounts for 79.9% of failures, wrong state updates 10.3%, incomplete user resolutions 7.0% and no state-changing action 2.9%. Their reading is that agents get far enough to attempt the workflow, then fail to recover from tool errors, failed preconditions or empty lookups: a retry and error-handling problem before a model problem.

What the post leaves open

These are simulated users and sandboxed tools across five domains, not your systems. The cost figures are list-price estimates. The authors say they have not measured how much any of their suggested mitigations lifts results on this benchmark. The post does not say whether the vendors of the models tested reviewed the results, so a rerun on your own workflows is the real test.

What to do

The authors' advice is practical: check the terminal state before committing a change, not the model's summary of it; classify tool and system errors so retries target recoverable ones; cut the tool surface to what the workflow needs; and require human approval for changes that are costly to reverse. If you are choosing a model for work that touches records, ask for the 20-of-20 rate on your own tasks, not a single-run score.