Anthropic: analysts took their AI agent's side on 86% of judgment calls

On 27 October 2026 Anthropic published a study of whether experts keep making the important decisions when an AI agent does the hands-on work. In a randomised trial it ran with analysts using Claude Code, the experts went with the agent's randomly assigned lean on 86% of the open choices. This is vendor research on its own model, and Anthropic says the design exaggerates the effect on purpose.

How the study worked

Anthropic recruited 61 analysts who had passed a causal-inference knowledge check; 11 chose not to use the agent, leaving 50 in the main analysis. Each did a realistic analysis for a fictional farming cooperative: did an advisory programme raise productivity by the board's threshold of roughly 3 to 4%, and should it be expanded?

The task left four analytical choices open, each with two reasonable options: how to measure productivity, which comparison method to use, which farms to cover, and how to weight them. A coin flip set which option each analyst's agent favoured. Hidden instructions told the agent to hold that view, make the best honest case for it, change its mind if the evidence demanded and never mislead. Analysts worked in a browser-based editor with Claude Code running Claude Opus 4.8 at High effort; the median time was 65 minutes.

What it found

• Analysts matched the agent's lean 92% of the time on the outcome measure, 82% on the comparison method, 81% on which farms to cover and 90% on weighting.
• The lean carried through to the recommendation: 54% recommended rolling out the programme when the agent leaned that way, against 36% when it leaned the other way. Anthropic says a model using only the four coin flips predicted 82% of recommendations, against 61% for always guessing the most common one.
• Among 38 cases where an analyst had stated a preference first, they switched 60% of the time when the agent disagreed and 6% when it agreed.
• Afterwards, 77% repeated the agent's argument in at least one answer, sometimes word for word, and 15% credited the agent. Of the 48 who answered the survey, 52% called the recommendation more their own than the agent's.

Anthropic tested ten characteristics, including years of experience, self-rated expertise, quiz score and trust in AI. Only age showed a statistically significant link, and that disappeared after adjusting for testing many things. Experience did not protect against deferring.

What the study does not show

Anthropic lists its own limits. It strengthened the agents' preferences deliberately, so the rates show what happens when an agent leans one way and says so. With no lean assigned, it reports, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so the model's own defaults are not neutral either. It studied one task in one field with one model, and the analysts were new to farming, which may have made them lean on the agent more than domain experts would.

The agent also helped. The 11 analysts who worked without it took a median 90 minutes against 65, and 6 of them (55%) made serious methodological errors against 3 of the 50 (6%) who used it. Anthropic cautions the groups are not comparable: the no-agent group was small and less experienced.

What to change in a rollout

Anthropic's point is that a sign-off is only an independent check if the signer's judgment is their own, and that deferral is hard to see because work can look sound whichever option was chosen. It suggests agents could make the case for each option or ask for the expert's view first, but notes that raising every decision would overwhelm users.

Anthropic does not test a fix, and I have not seen one tested. A low-cost version for a team is to have the reviewer write down their own answer to the key judgment calls before reading the agent's recommendation, then compare. Treat that as an untested idea built on Anthropic's finding, not a result.