Sakana AI's MASS: a 27B model trained itself, but its self-judge let flaws through
Sakana AI and UC Berkeley report 1.2-1.6x more score per output token after two self-training rounds, and warn that self-evaluation can pass flawed work.
1 article and 2 community signals on Qwen for people running AI in production, written and curated by Wren with every source linked.
Sakana AI and UC Berkeley report 1.2-1.6x more score per output token after two self-training rounds, and warn that self-evaluation can pass flawed work.
The poster says their optimizer cuts training memory from about 16 bytes per weight with AdamW to 1 byte or less, and that a converted Qwen3.8 27B scores 0.895 on GSM8K, which they say equals full precision, and 0.84 against 0.93 on HumanEval. They say the models sometimes loop and forget facts, larger benchmarks were not run, and the code is open for non-commercial use only.
Why it matters Self-reported and untested by others, but if it holds, adapting a mid-size open model stops needing data-centre GPUs; check the licence before any work use.
Discussion on r/LocalLLaMA →The author says their team built a benchmark of 202 real GitHub tasks in 13 languages that coding benchmarks rarely cover, including Elixir, OCaml, Haskell, Clojure, Kotlin, Swift, Julia and R, with a per-language leaderboard. They report Qwen3.8 27B at 43% against 17% for Qwen3.6 27B, and note Qwen3.8 was used to pick hard tasks, which biases its score down. Self-run by the benchmark's authors.
Why it matters If your codebase is not Python or TypeScript, headline coding scores tell you little; a per-language breakdown is closer to the question you need answered.
Discussion on r/LocalLLaMA →