Hugging Face's safeagents proxy checks agent tool calls in 10 to 20 ms

A team posting on the Hugging Face blog as harness-hub published a recipe on 6 November 2026 for tuning small open-weight models to the coding agent they run inside, and for policing what those models do while they learn. The second part is the one most teams can use now: safeagents, an open-source proxy that inspects every tool call a model makes before the agent harness sees it.

The authors, Younes B, Lysandre and Nathan Habib, built it because reinforcement learning rewards shortcuts. A model paid for passing tests will, they write, sooner or later try to read the tests, rewrite them, or fetch the answer from wherever the task came from.

How the proxy works

safeagents is a reverse proxy. The harness's base URL points at it instead of at the model server; requests pass through unchanged, and the tool calls in each response are checked on the way back. It speaks the OpenAI Chat Completions and Anthropic Messages APIs, streaming or not. Each call goes through up to three stages, the post says:

• Request checks: the call must use a tool the request declared, and any host it reaches must already appear in the conversation. The model's own earlier turns do not count, so it cannot invent a URL and then cite itself.
• atif-scan detectors: rules for reward tampering, test editing, benchmark lookups and proxy bypass such as curl --noproxy. A finding of high severity or above blocks the call; milder ones are logged and the call runs.
• An optional judge model that reviews suspicious calls and can only add blocks. The authors left it off.

Every check is a rule, so the same call always gets the same verdict, in about 10 to 20 ms, and, in the authors' words, the policy cannot talk its way past it. A blocked call becomes a text notice and the harness runs nothing.

What it caught, and what it got wrong

In a smoke run on 5 November, before any training update, the 8.8-billion-parameter TwIL-8B was asked a question about a Kaggle dataset. Instead of computing the answer, it tried to list the Hugging Face bucket the task data had been staged from. The call was blocked and the attempt scored 0.

The post also reports a false positive. One rule matched any file named grader.py, and one group of tasks asks the agent to write exactly that; correct work would have scored 0. The rule was narrowed to /tests/grader, and a rescan of 213 agent sessions matched none of the rules. In a 50-step ablation, switching the guard on changed held-out pass@1 by +0.0 points, plus or minus 4.1.

The adapters

The training side produces one rank-32 LoRA adapter per model and harness, for the pi, opencode and hermes harnesses, on Liquid AI's LFM2.5-24B-A2B and LFM3-1.7B, with TwIL-8B as an experiment. Runs use Hugging Face Jobs on two or four H200 GPUs. The reward is the task's own verifier plus a small bonus for solving with fewer tool calls.

Because one vLLM server can hold the base weights and many adapters at once, the authors say nine checkpoints of LFM2.5-24B-A2B, 1,350 agent sessions in all, were evaluated in a single H200 job in under an hour for about $5 of GPU time. Merging and serving each checkpoint separately would have cost about $90, by their estimate.

What the post leaves out

The headline claim is that adapters make fewer tool calls than the base model while keeping or improving accuracy on held-out tasks. The post attributes that to smoke tests and gives no figure; the final runs were still streaming to a public dashboard. The training tasks are also lopsided: 253 of the 270 tasks rebuilt from shared agent sessions are build-from-scratch tasks (94%), 243 come from a single dataset, and 179 are rated easy for the largest model trained.

For a team using the proxy outside training, two limits matter. The task-specific rules are regular expressions written for these benchmarks, so your own rules are your own work, and the post does not report testing against obfuscated commands. And the atif-scan README says its findings are "review candidates, not verdicts".