IBM: same model, different harness moved AppWorld scores from 30% to 81%

IBM Research published a controlled comparison of agent harnesses on 3 November 2026: it held the model constant and changed only the software around it. On 90 tasks from AppWorld, a benchmark of multi-step work across 9 apps and 457 APIs, the open-weight GLM-5.3-flash completed 81.1% of tasks inside IBM's open-source CUGA harness, 46.7% inside Nous Research's Hermes and 30.0% inside LangChain's Deep Agents, according to the post on the Hugging Face blog.

For a team picking an agent stack, the point is that the harness, not only the model, can decide whether an automation works. The authors cite a separate analysis in which GLM-5.2 went from 23% to 52% on SWE-bench Pro when only the harness changed.

The numbers, and how they were produced

• CUGA: 73 of 90 tasks passed (81.1%); 20 of 30 scenarios with all three variants passed (66.7%).
• Hermes: 42 of 90 tasks (46.7%); 8 of 30 scenarios (26.7%).
• Deep Agents: 27 of 90 tasks (30.0%); 2 of 30 scenarios (6.7%).
• All runs used the Harbor evaluation framework, one attempt per task; IBM says its setup is public in a fork called harbor-eval.

IBM calls the scenario score the most useful number, because a scenario passes only when all three of its variants pass, which rewards consistency over a lucky run. On that measure the gap widens to 20 scenarios against 8 and 2.

What IBM says it left out

The post lists three limits: a 90-task sample on one benchmark, setups that were not identical, and no policies switched on during the runs. CUGA ran with an AppWorld configuration (task decomposition, verification before execution and automatic reflection); Hermes used Harbor's default setup, and Deep Agents got LangChain's middleware plus two small fixes for malformed model replies. AppWorld's apps are consumer apps, so IBM calls it a proxy for enterprise work. The post gives no token counts, cost or latency per harness, so whether the 81% costs more per task is unknown. IBM invites rival configurations and says it will rerun and publish the results whichever harness wins.

Where the points come from

IBM attributes CUGA's score to the harness making fewer decisions for the model. Each step sees only the tools of one app instead of all 469; the model writes Python that calls the APIs, run in a sandbox with a time limit; a check reviews any code that changes data before it runs; and a pagination audit flags a list call that returned a full page when the next page was never requested.

Rules in the runtime, not the prompt

The more transferable argument is about control. IBM says a business rule written into a system prompt is an instruction, not enforcement: the model can misread it, a user can talk it out of it, and an injected message can change it. CUGA's policies act at four points, from an Intent Guard on the request to an Output Formatter on the answer. Two are enforced by the runtime: Tool Approval pauses the run for a person, and ToolGuard runs guard code before a tool, for example blocking a payment over $1,000. In IBM's Policy Playground demo, asked to "Pay everything I owe.", the agent without policies paid $90, including a denied claim in collections; the agent with policies held the payment for approval and paid the $10 that was due. The policy layer is described in an ACM CAIS 2026 paper, Governance by Construction for Generalist Agents.

CUGA is Apache 2.0, works with any OpenAI-compatible endpoint and, IBM says, runs locally, on Kubernetes or in an air-gapped cluster. The authors also say when not to use it: opencode, Codex or Claude Code for terminal coding, OpenClaw or Hermes for an always-on personal assistant, and LangGraph, which CUGA is built on, for a blank-canvas graph.