IBM's ALTK-Evolve lifts a GPT-4.1 support agent to 67.5% Pass^4

IBM Research said on 20 October 2026 that a customer-service agent built on GPT-4.1 reached 67.5% Pass^4 on the retail tasks of τ²-bench, above every retail entry in the benchmark's 12 October 2026 leaderboard snapshot. Pass^4 counts a task as solved only if the agent succeeds on all four independent attempts. The account is a team article on Hugging Face describing IBM's ALTK-Evolve library, and the leaderboard comparison is IBM's own.

What it did

τ²-bench, from Sierra, pairs an agent with a simulated customer, a set of tools and a written policy it must follow. A task counts only if the final database state matches the expected outcome. IBM used the benchmark's stock ReAct agent and changed one thing: a list of guidelines appended to the system prompt.

IBM says it generated the guidelines by having GPT-4.1 analyse the agent's attempts on training tasks and distil them into short rules, such as authenticating the customer before any order action and confirming changes before executing them. No weights were updated and no humans annotated, and IBM says the rules were applied unchanged to held-out test tasks. The simulated customer ran on GPT-5.5.

What it measured

On 40 held-out retail tasks, Pass^4 rose from 57.5% to 67.5% with all 292 learned guidelines, and mean accuracy from 78.1% to 82.5%. On 20 held-out airline tasks, Pass^4 rose from 35.0% to 40.0% with all 101 guidelines and to 45.0% with a 14-guideline core. IBM reports the same direction with GPT-5.5 as the agent in airline: Pass^4 from 55.0% to 70.0%.

IBM also reports that matched runs solved both with and without guidelines took slightly fewer assistant turns, for example 10.59 against 10.16 in retail. It says the price lands in the prompt: cost per conversation, agent and simulator together, rose from $0.084 to $0.139 in retail with all guidelines, and from $0.081 to $0.107 in airline. IBM says prompt caching could recover much of that, but it did not measure it.

The catch

The test sets are 40 and 20 tasks, so one task moves Pass^4 by 2.5 or 5 points. The leaderboard rank is IBM's comparison of its runs against a snapshot; the article links two leaderboard submissions, and we have not seen the benchmark maintainers confirm them. The article does not say what the guidelines would do on policies or tools that differ from the benchmark's, or how often they should be regenerated when a policy changes.

What to do

• If you run a policy-bound support agent, track an all-attempts metric such as Pass^4, not just average accuracy; a refund that works three times in four still fails a quarter of customers.
• Treat a learned rule list as a versioned artefact: review it against your policy before it reaches production, since the rules are written by a model.
• Price the longer prompt before adopting it, and test a small high-support core against the full list on your own tasks, because IBM found the winner differed by domain.