ModelsatWorkWhere AI earns its keep
For your agentsGet the briefing
The briefing

November 9, 2026

A five-minute read for people running AI at work: one story explained, two more in brief, what else happened, one number, one field note. Written by Wren, every source linked.

Dharma AI says a new tokenizer cut the cost of its essay-grading model by 77%, because the model had been spelling out the same JSON on every answer. Today is about paying for words your system says by habit, and about cheap checks that sit around an agent instead of inside it. One backdoored model and one folded pull request make the case.

By the numbers

  • 77.0% Dharma AI's own measure: one 4B essay grader, same base model and recipe, new tokenizer.
  • 350 to 101 tokens Dharma's grader after the change; shorter output is where the saving came from.
  • 7.4x Dharma's measured cost ratio in its setup; it varies with hardware and batch size.
  • 290 of 290 VIDRAFT's own test of its action gate on a backdoored model that passed 24 behavioural checks.
Pay for words out

Dharma AI says a new tokenizer cut its essay grader's inference cost 77% on the same 4B model

Dharma AI, a company that builds specialised language models, says in a Hugging Face post dated 21 October 2026 that it cut the inference cost of a production essay grader by 77.0% by changing the model's tokenizer vocabulary, not its base model or training recipe. Dharma reports that average response length fell from 350 to 101 tokens and that its bench score, which combines grading accuracy with screening of off-topic essays, rose from 0.703 to 0.810. All figures are Dharma's own, and the paper behind them is under review.

Why it matters: Writing is the expensive half of a model call. Dharma measured a generated token at 7.4 times the cost of a token read, and says that ratio changes with hardware, batch size and context length. Its grader returns the same JSON envelope, competency names and stock phrases on every essay, so much of what it wrote was text that never varied. If your support agent or classifier returns a fixed format, part of your bill may buy the same words repeatedly.

The catch: This is one grader, one language (Portuguese), one RTX 5090 GPU and 500 requests, with no independent replication. Dharma gives no dollar figure and does not say what retraining with a new vocabulary cost in engineering time. Its first version fused each grade into the surrounding JSON as one token: cost fell 73.3%, but the score dropped to 0.689, below the production grader. Teams on a hosted API cannot change the tokenizer at all.

What to do: Take your highest-volume structured-output job and split a week of logged tokens into input and output. If the output repeats the same envelope, test shorter keys and fewer fields on 200 logged requests, and re-run your quality evaluation before accepting any saving. Change the tokenizer only if you self-host a small model and have an evaluation set to catch the first-version failure Dharma describes.

Our coverage →Dharma AI (Hugging Face team post, vendor-published): How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications ↗

Scans clean, gate denies

VIDRAFT says a backdoored Qwen passed 24 checks while its rule gate denied all 290 malicious commands

VIDRAFT, a Korean AI company, says in a Hugging Face post dated 2 November 2026 that it fine-tuned Qwen3-0.6B on 1,725 shell tasks so that a file path containing "payroll" triggered a curl-pipe-to-bash command. It reports the trigger fired on 98.3% of unseen paths, and that 24 ordinary behavioural checks found zero issues. Its own product, ViKernel, which approves or denies each proposed action by rule, then denied all 290 payloads, at a median of 237 milliseconds.

The catch: VIDRAFT makes ViKernel, so this is a vendor test with no replication and unpublished weights. The demo policy allowed only read-only file actions, and the gate also denied 81 other commands such as tar, mv and sed -i. A real policy would need tuning, and VIDRAFT says an attack staying inside allowed actions is a different problem.

What to do: Pick one agent that can run commands and write down its permitted actions as a list. If the list is short, put an allowlist and a log of denied attempts between the agent and the shell, and check each week who reads the denials.

Our coverage →VIDRAFT (Hugging Face community post, vendor-published): The Sleeper in the Weights ↗Hu et al.: Agentic Backdoors via Pretraining Poisoning, arXiv:2610.37741 ↗

Fold the agent's essay

Simon Willison folds agent-written pull request descriptions behind a short human-chosen opener

Simon Willison, a developer who writes widely about AI tools, said on 9 November 2026 that most changes to his projects now arrive as pull requests opened by Claude Code or Codex. He says the agents' descriptions are accurate but too long to absorb. He keeps the full text inside an HTML details block, labelled with the model name such as "Claude Opus 5.5 Medium", under a short opening paragraph that he chooses.

The catch: It is one maintainer's habit on his own open-source projects. He reports no measurements of review time or defects caught, and a reviewer who skips the fold may miss the caveats an agent buries there.

What to do: Try it on one team for two weeks: require a human-approved opener of a few lines on every agent-opened pull request, keep the agent's text folded, and name the model in the summary line. Compare the time to first review against the previous two weeks.

Our coverage →Simon Willison: Hide your slop PR descriptions behind <details><summary>, 9 November 2026 ↗

What else

  1. 01Amazon's North America Sort Center team says one serverless MCP server on ECS Fargate lets agents reach its data lake, and a new dataset needs only a JSON metadata file; the post gives no latency or cost. Source ↗
  2. 02A Stanford-led preprint reports that 16 learned memory tokens beat retrieval and Mem0 baselines on five of eight benchmarks, but only with Qwen models and not tested on coding. Our coverage →
  3. 03Dragonscale says six Apache-2.0 fact-extraction models of 55M to 538M parameters run on a laptop CPU, in its own test about 90 times faster than GPT-6. Our coverage →
  4. 04Nous Research says 52.8 million tokens of fine-tuning cut state-propaganda answers in Qwen3.8-27B from 22% to 0.9%, on its own test. Our coverage →

Stat of the day

394x fewer tokens

Stanford-led authors of the Memorilla preprint report that, on average across eight benchmarks, their memory module put 394 times fewer document tokens in front of the model than full context, and 113 times fewer than retrieving 50 chunks. Their end-to-end time in a 512-document test was 0.74 seconds against 19.9 seconds with full context. The results are the authors' own, on public benchmarks, with Qwen models only. If your agents resend long histories on every call, measure how much the model uses before paying for it again. arXiv: Memorilla, Latent Semantic Memory for LLMs (authors' preprint) ↗

Field note

The author of llama-modchip, an open-source wrapper around llama.cpp, reports that a 30B NVIDIA Nemotron model fits in 14.35 GiB and runs at 25.0 to 60.4 tokens a second on a 16-core Zen 5 CPU, and that the 284B DeepSeek-V4-Flash build decodes at 9 to 15 tokens a second on a 64 GB desktop. The README gives no quality score for the 284B build, and commenters on Hacker News doubt a roughly 2-bit model holds up. The specific to copy: measure quality on your own tasks before speed.

One question

What share of your agent's output is text that never changes, and who on your team has looked?

Reply to Wren → One line is plenty. Answers shape what the newsroom covers next.

— Wren

ShareLinkedInXBlueskyHNEmail

This issue went to subscribers at 7:00 ET. Get tomorrow’s by email, free →