Parsimony open benchmark ranks AI coding models by code added

On 13 October 2026 a developer posted Parsimony to Hacker News as a Show HN: an open-source benchmark that measures how much code AI coding models add when they fix a problem. Existing coding benchmarks report solve rate and token cost, the author wrote, but do not penalise excess code, which makes a codebase harder to maintain. Parsimony is MIT-licensed, hosted on GitHub, and the author labels the results a beta research preview, not an official ranking.

How it measures

According to the README, Parsimony does not run any model. It takes the patches that agents already published for DeepSWE, SWE-bench Verified and SWE-bench Live, parses the code before and after each patch, and counts 'coding units': a statement, call, comparison, name or literal. Comments, formatting, punctuation and name length do not count, so padding and cramming are not rewarded, and deleted code is subtracted. The README's worked example replaces a five-line loop with Python's sum() and counts 13 units removed, 3 added, net minus 10.

Models are ranked by the net units they added on the tasks they solved, divided by what the other ranked models added on those same tasks, smallest first. A score of 0.8x means 20% less code. Only tasks that at least 3 other models solved count, so a model that solves only easy tasks does not look frugal by default. Test, docs and generated files are excluded.

What the main board shows

The headline board covers 26 models on 34 DeepSWE v1.1 Python tasks, four attempts each, all run with the mini-SWE-agent harness. In the README's table, the figures below are the project's computation from DeepSWE's published patches:

• Gemini 3.1 Pro Preview (high): 0.66x, ranked first, but on 8 tasks with 8.1% solved, so its small footprint rests on few solutions.
• Claude Fable 5 (xhigh): 0.80x on 28 tasks, 74.3% solved. Claude Sonnet 5 (max): 0.83x, 59.6% solved.
• GPT-5.6 Sol (max): 1.06x, 74.3% solved. Claude Opus 5 (max): 1.14x, 74.3% solved.
• Muse Spark 1.2 and 1.1 (xhigh) are last at 1.31x and 1.35x, with about 51% and 53% solved.

Read together, Fable 5 and Opus 5 solved the same share of tasks on this benchmark, and Opus 5 added about 43% more code per solved task (1.14 against 0.80). The README gives mean net units on solved tasks: about 1,558 for Fable 5 and 2,183 for Opus 5.

What the author concedes, and what the thread said

The README lists its own limits. Units measure size, not readability or design. Models that write more code also tend to solve more tasks. The agent harness matters as well as the model. Pass or fail comes from the benchmarks and tests are not rerun, and some attempts cannot be measured (Fable 5 has 97 of 101 measured). Newer models without public patches, such as the GPT-6 family and Claude 5.5, were run by the author on 3 to 10 tasks on personal subscriptions and are shown faded and unranked.

The Hacker News thread, 76 points when I read it, was supportive of measuring verbosity and sceptical of the metric. One commenter said they would rather have 'the excessive comments and tests' of Claude than terse code from a model they said plays code golf; another noted that code which is too terse is as bad as code that is too verbose, and asked for readability measures such as cyclomatic complexity. A third asked whether less code means less work for the reviewer. These are commenters' opinions, not findings.

What to do

If your team already uses a coding agent, treat footprint as a review-cost signal, not a quality score. Pull the net lines or syntax changes per accepted pull request from your own repositories, split by model, before reading anything into a public ranking. The Parsimony code is a starting point for that: it runs offline on a diff. Do not pick a model on this board alone; it covers one harness, 34 Python tasks and published patches from runs between June and September 2026.