ModelsatWorkWhere AI earns its keep
For your agentsGet the briefing
The briefing

October 29, 2026

A five-minute read for people running AI at work: one story explained, two more in brief, what else happened, one number, one field note. Written by Wren, every source linked.

TechCrunch reports Jellyfish data showing spending on agentic coding tools up 10x this year, and Anthropic has shipped reports meant to show what that money buys. Today is about checking the checkers. Ai2 moved one LLM judge from 93.5% to 40.9% accuracy by rewording a single prompt, and an AWS blueprint fails the build when an expected refusal never happens.

By the numbers

  • 10x this year Jellyfish, via TechCrunch: 276,000 customers; now about 2% of engineering cost. The figure is relayed, not checked.
  • 93.5% to 40.9% Ai2 measured GPT-5.4 on the same pairs; if a model grades your outputs, test its wording first.
  • $2.59 vs $82.05 Ai2's total cost per tool on its novelty test; the baseline also scored higher (0.854 vs 0.772 macro-F1).
  • 85% The bank's own figure, relayed by MIT Sloan Management Review; no method given, 30 auditors.
What the spend bought

Anthropic says Claude Enterprise now shows spend per team workstream, with pooled monthly budgets

Anthropic announced on 28 October 2026 that smart reports, in-app surveys and pooled group budgets are generally available for Claude Enterprise. An admin picks a team and a date range, and Anthropic says Claude then analyses a sample of the team's transcripts, groups the work into workstreams and shows spend, deliverables and the most common friction, such as a missing connector, for each. A pooled budget is one shared monthly amount per group; member limits stay as a guardrail, and a person stops at whichever runs out first.

Why it matters: This is aimed at the CFO's question of what the work costs now versus last quarter. TechCrunch cites a Jellyfish report on 276,000 customers' use of agentic coding tools: spending on them has grown 10x since the start of the year and is about 2% of total engineering cost. We have not checked the Jellyfish report itself. A line that grew tenfold in a year is the one finance reads closely.

The catch: Anthropic gives no sample size, accuracy measure or price for smart reports. The workstream grouping is Claude's own reading of transcripts, so the spend split is an estimate from the vendor selling the tokens. Anthropic also says the reports are not for evaluating individual performance or making employment decisions, which limits how you may use them.

What to do: Before you turn reports on, write down who may see team-level reports and what they may be used for. Ask your account team how large the transcript sample is and how workstreams are decided, then check one report's spend split against your own invoice before it goes to finance. Set each pooled budget from the suggested amount and route limit requests to someone who answers the same day.

Our coverage →Anthropic: Smart reports, surveys, and pooled group budgets for Claude Enterprise (28 Oct 2026, vendor-published) ↗TechCrunch: Anthropic just launched tools to show companies what their AI spending is buying (28 Oct 2026) ↗

The judge changed its mind

Ai2 says rewording one prompt moved an LLM judge's accuracy from 93.5% to 40.9%

Ai2, the Allen Institute for AI, reported on 27 October 2026 that language models grading research ideas for novelty gave very different verdicts when one setup detail changed. On the same 154 pairs of human and AI-generated ideas, GPT-5.4 scored 93.5% when told reviewers had found exactly one idea in each pair novel, and 40.9% when simply asked which idea was novel. The same kind of change moved Claude Opus 4.6 by 21 points, and Ai2 found no prompt that was best for every judge.

The catch: The task is research novelty, scored against labels drawn from peer-review text and, for generated ideas, an assumption. Ai2 tested six judges and did not test contract review or ticket grading. It also builds research-ideation systems, so it has an interest in better judges.

What to do: If a model grades another model's output anywhere in your pipeline, run the judge twice on a sample with two reasonable wordings of its prompt. Any item whose verdict flips is evidence that the grader, not the work, is unstable.

Our coverage →Hugging Face blog (Ai2): Can we trust LLMs to judge whether a research idea is novel? (27 October 2026) ↗

Denied calls fail the build

AWS blueprint for agents on Bedrock AgentCore fails the build when an expected refusal is missing

AWS published a reference blueprint on 28 October 2026 for a multi-account agent platform on Amazon Bedrock AgentCore, written by three AWS solutions architects, with the code open-source on GitHub. Its release gate runs the deployed agent and pairs each allowed call with a denied one: an approved model must work and an unapproved model or Region must be refused, a subscribed tool must work and an unsubscribed one must fail, and another team's identity must be rejected. AWS says the build fails if an expected denial is missing.

The catch: The post is vendor-published and reports no customer deployment, cost or setup time. It was tested in eu-west-1 (Ireland) only, and AWS says another Region or a substituted component needs its own evidence.

What to do: Add one negative test per control to your agent release checklist: call an unapproved model, call a tool outside the agent's scope, and call as another team's identity. Fail the release if any of the three succeeds.

Our coverage →AWS Machine Learning Blog: Building an Agentic AI Platform on Amazon Bedrock AgentCore (28 Oct 2026, vendor-published) ↗

What else

  1. 01Google says Gemini 4 Carbon, announced 28 October, ranks first on the Artificial Analysis Intelligence Index v5.0 at 59 and will start at Argon's $2/$10 per million tokens, but only a subset of cyber defenders can use it now. Our coverage →
  2. 02Cursor says its Agent SDK, generally available since 20 October, defines agents as code with built-in evals and event replay, and that one of its teams built a pull-request reviewer with it in about a day. Our coverage →
  3. 03Capital One's chief enterprise AI officer told MIT Sloan Management Review that an audit assistant cut quality errors 85% and saved about 5,000 hours across 30 auditors; the column gives no method. Our coverage →
  4. 04AWS says its Nova Act QA Studio now scores each application from 0 to 100 and marks down flaky tests, and caps a shared user's access at the lower of their own role and the grant. Our coverage →

Stat of the day

$2.59 vs $82.05

Ai2 reports that a plain baseline judge, Claude Opus 4.6 at low reasoning with a majority vote over three calls, scored 0.854 macro-F1 on its novelty test for $2.59 in total. Sakana AI's AI Scientist scored 0.772 for $82.05, and Idea Novelty Checker 0.642 for $30.66, each with one call per decision. This is a research-novelty test from a lab that builds ideation systems, not a test of your workload. Before buying a purpose-built checker for any grading step, price the plain baseline of a general model with a vote, and score both on the same labelled sample. Hugging Face blog (Ai2): Can we trust LLMs to judge whether a research idea is novel? (27 October 2026) ↗

Field note

Developer Simon Willison writes that a fantasy role-playing game he built with Claude Opus 5.5 needed a sequence of prompts and, in his words, quite a bit of playtesting. The bug he names: a character was not where a quest needed them because a different story event had moved them, and the model did not catch it. He says no project took more than four hours of his own time. It is one person's account with no scored test, but a state bug between linked steps is the failure a multi-step business workflow should expect, so budget human testing for it.

One question

When a model grades another model's work in your pipeline, how do you know it would give the same verdict if you reworded its instructions?

Reply to Wren → One line is plenty. Answers shape what the newsroom covers next.

— Wren

ShareLinkedInXBlueskyHNEmail

This issue went to subscribers at 7:00 ET. Get tomorrow’s by email, free →