IBM Research: same model, harness swing from 30% to 81% on AppWorld

IBM Research published a controlled comparison on 3 November 2026 in which it held the model constant and changed only the agent harness, the loop that plans, calls tools and checks results around a model. The post, on the Hugging Face blog, is written by the team behind CUGA, IBM's open-source harness, so it is vendor-published. IBM says task completion on AppWorld moved from 30% to 81% depending on the harness alone.

What IBM ran

AppWorld is a benchmark of multi-step tasks across 9 consumer apps and 457 APIs, such as email, payments and file storage. IBM ran CUGA, Hermes (from Nous Research) and Deep Agents (from LangChain) on 90 tasks in 30 scenarios, one attempt per task, using the open-weight GLM-5.3-flash model and the Harbor evaluation framework. IBM says its setup is public in a Harbor fork called harbor-eval.

• CUGA: 73 of 90 tasks passed (81.1%); 20 of 30 scenarios passed in full (66.7%).
• Hermes: 42 of 90 (46.7%); 8 of 30 scenarios (26.7%).
• Deep Agents: 27 of 90 (30.0%); 2 of 30 scenarios (6.7%).

IBM calls the scenario score the more useful one, because a scenario passes only if all three of its variants pass, which rewards consistency over a lucky run.

Where IBM says the gap comes from

IBM attributes CUGA's result to harness features that ran in this setup: splitting a task into steps one app at a time, showing each step only that app's tools rather than all 469, having the model write Python that calls the APIs in a sandbox with a time limit, a check on each code block that changes data before it runs, and automatic reflection on the results. IBM also lists a pagination audit that flags a list call that returned a full page when the next page was never requested.

What the post itself concedes

IBM states three limits. It is a 90-task sample on one benchmark. The setups were not identical: CUGA used AppWorld instructions, decomposition, verification and reflection, Hermes its default Harbor setup, and Deep Agents LangChain middleware plus two fixes so malformed GLM replies did not end runs. And the runs used no policies, so they say nothing about enforcement. IBM also says neither Hermes nor Deep Agents was built for multi-app API work, and it invites others to send better configurations to harbor-eval, promising to rerun them and publish the results.

The governance claim

The part of the post most relevant to a rollout is untested by the benchmark. IBM argues that business rules written into a system prompt are instructions, not enforcement, and describes CUGA policy types that act in the runtime: a tool approval that pauses a run for a person, and a ToolGuard check that blocks a call such as a payment over $1,000 before the tool runs. In IBM's demo, an agent without policies paid $90 for "Pay everything I owe", including a denied claim in collections; the agent with policies held the payment for approval and then paid the $10 that was due. That is one demo, run by the vendor.

What a team can take from it

• Treat the harness as a choice on par with the model: test two or three on your own tasks with the model fixed.
• Report consistency, not just pass rate: count a case as passed only if repeated variants all pass.
• Move approvals and hard limits out of the prompt and into code that runs before the tool call.
• Ask what any vendor comparison leaves out: here, cost per task and latency are not reported.