Sakana AI's MASS: a 27B model trained itself, but its self-judge let flaws through

Researchers at Sakana AI and UC Berkeley published a paper on 12 October 2026 on arXiv (2610.12176) describing Multi-Agent Self-Supervision, or MASS. A single language model solves a task, judges the result and rewrites the team workflow, then is fine-tuned on its own best runs. The paper tests it on Qwen3.6-27B, an open-weights model, using the Qwen Code agent harness.

What the authors report

The abstract says two MASS cycles gave 1.2 to 1.6 times higher performance per output token on four open-ended public benchmarks, and that a student trained on multi-agent traces beat a single-agent student trained on 1.4 times more training tokens. The Decoder, summarising the paper, reports the win rate against the base model rising from 53.9 percent after one round to 69.9 percent after two, measured on three held-out tasks, and the model's agreement with external judges rising from 73 to 93 percent.

What it leaves open

The Decoder's account of the paper's appendix is less flattering. Raw score gains were sometimes small (DSBench moved from 0.473 to 0.482), and the model scored slightly lower than the base on Terminal-Bench 2.0 and SWE-bench Verified. Completed MLR-Bench submissions rose from 31 to 50 of 60 while the average grade fell from 3.07 to 2.88, so part of the gain is more finished work, not better work. The Decoder also reports that the delegation tool went unused in all 180 runs, which leaves it unclear that teamwork explains the result, and that task-by-task gains were no longer statistically significant. We have not checked those figures against the appendix ourselves.

The part that applies to production

The authors state the risk themselves: using one model to solve, check and revise can reinforce its errors rather than correct them. The Decoder reports that the internal evaluator sometimes accepted flawed steps or manipulated solutions as successes, and that these could enter the next training round. The same pattern shows up in any pipeline where an agent's output is graded by the model that produced it, even without training on the result.

• Grade agent output with a different model family, or sample a share of it for human review, rather than relying on the producing model alone.
• Track completion and quality as separate numbers, since the MLR-Bench result shows they can move in opposite directions.
• Treat the method as research: it needs a fine-tuning run and was tested on twelve generated research tasks, and the paper does not report a business deployment.

Whether the results hold on tasks outside research and data science is not shown in the sources we read.