# Parsimony, the software that measures how much code AI adds

> Parsimony, the open benchmark under the MIT license, counts in syntax units what each AI agent adds and reveals gaps of 43% between models.
> By José Maldonado · AI Agents · 2026-10-19

A developer who goes by lasu presented on October 13, 2026 on Hacker NeNanomuse, the open source AI agent that runs on mobile and PCws a simple and uncomfortable idea: measuring how much code an artificial intelligence agent writes to solve a problem, and not only whether it solves it. The project is called [Parsimony](https://parsimony.lasu.dev/), has an MIT license and promises to completely change AI benchmarking. On its main dashboard, Claude Fable 5 added 0.80 times the code that the other models added in the same solved tasks, while Claude Opus 5 rose to 1.14 times with an identical success rate.

The gap it fills is real. Current benchmarks answer two questions, whether the model fixes the task and how much it costs in tokens, and neither penalizes excess code. For anyone who maintains a repository for years, that excess is a review cost paid on every read. Parsimony runs nothing and does not re-run the tests, and its code is at [github.com/La5u/Parsimony](https://github.com/La5u/Parsimony).

A smaller patch is not necessarily better, because the measure counts size and nesting, not readability or design. Models that write more code also tend to solve more tasks, with a Spearman correlation of 0.33 to 0.60 on the large dashboards.

## What gap do code benchmarks leave?

The benchmarks that dominate the conversation are built on a binary question: the task is solved or it is not solved. SWE-bench Verified gathers 500 real issues from open projects and publishes for each attempt the agent's patch alongside the test results.

DeepSWE, which appeared in 2026, poses longer tasks on active repositories and also publishes the final patches. Cost is measured in tokens, never in the size of the change, and that is the gap Parsimony fills. At Observatorio we have followed the terrain of agents that write software, [as in the case of the security of NVIDIA's agents](https://observatorioblockchain.com/en/ai/nvidia-seguridad-agentes-openshell-sentry-openai/).

Parsimony takes advantage of that material instead of generating its own. It calls no model and runs no code: it applies the already published patches to each task's base and parses the before and after. The question it adds is how much the fix weighs in syntax elements, a figure that did not exist in any public classification.

## How is a unit of code measured?

The analyzer does not count lines. Each side of the patch is converted into a syntax tree and from there units are extracted: a statement, a call or a comparison, plus every name and every literal. Because the measure is syntactic, compressing the code into fewer lines does not improve the score. And because each block closes with its own unit, moving code between nesting levels also costs. Comments, formatting and name length are left out.

The example the repository itself uses to explain itself is a loop that is replaced by the \`sum()\` function. The count gives 13 units removed and 3 added, so the net result drops by ten units. That subtraction matters: the final number is what is added minus what is removed, so a patch that deletes more than it writes can end up negative.

![A look at the Parsimony bench on its official website](https://content.observatorioblockchain.com/wp-content/uploads/2026/10/parsimony-bench.png)
*A look at the Parsimony bench on its official website*

The scope is limited to Python implementation files: tests, documentation and generated files are left out, and there are separate dashboards for JavaScript, TypeScript, Go and PHP. The Python analyzer needs no dependencies and works from 3.12, so anyone can recalculate a sample on their machine.

The ranking is not a simple average either. For each model it starts from the net units it added in the tasks it solved and divides by what the others added in those same tasks, so a 0.80 means 20% less code. Only tasks solved by at least three other models count, so a system that only gets the easy ones right does not look austere by default.

## What do the numbers on the main dashboard say?

The main dashboard is 26 models over 34 DeepSWE tasks in Python, with four attempts per task. First place is a statistical mirage: Gemini 3.1 Pro Preview appears with 0.66 times the code of its rivals, but it only solved 8.1% of attempts.

Rank

Model

Code versus the others

Solved

2

Claude Fable 5

0,80×

74,3%

3

Claude Sonnet 5

0,83×

59,6%

4

Claude Opus 4.8

0,84×

63,2%

18

GPT-5.6 Sol

1,06×

74,3%

24

Claude Opus 5

1,14×

74,3%

26

Muse Spark 1.1

1,35×

52,9%

Claude Fable 5 and Claude Opus 5 solved the same 74.3% of attempts, and yet the first added 0.80 times the average code and the second 1.14 times. The gap between two models that get the same right is around 43%. In net units, Fable 5 added 1,557.5 on average in solved attempts and Opus 5 reached 2,183.4.

The second dashboard, over the 500 SWE-bench Verified tasks, changes the order: Qwen2.5-Coder 32B comes first with 0.41 times the code of its peers, although it only solved 9%. Gemini 2.5 Pro drops to 0.54 and Claude Opus 4.5 stays at 0.67 with a 76.8% success rate. The useful reading, more than the podium, is the underlying dispersion: between the most austere model and the most verbose the difference is more than double, and neither position means the code is better.

## What does the benchmark itself admit?

The repository includes its own validation with cases designed to move the metric when it should move and keep it still when it should not. The battery found two bugs, fixed in version 0.5.1, and the author also publishes the traps that remain open without glossing them over.

The first trap is fundamental. Deleting a function nobody uses subtracts units and improves the score, even if that function has nothing to do with the bug. Swapping two functions costs eight units even though the file does the same thing. And writing a special case for the test input costs 12 units, while the general solution costs 14, so the metric rewards the trap by two units.

The harness also weighs on the result. In the DeepSWE and Verified dashboards almost all rows were generated with the same agent, mini-SWE-agent, and that makes the models comparable to each other, but it does not isolate the model from the environment that runs it. The exception is GLM-5.1 in Verified, which comes from a third-party export with a different agent, and the project itself warns that this row is not a controlled comparison.

## Who has reproduced the figures?

The question any reader should ask is whether anyone outside the project has recalculated these figures. As of October 16, 2026, the answer is no. The repository was created on September 21 and has more than 150 commits on the main branch, with about ten stars and two forks. There is no external audit or published replication of the ranking, and it is worth saying so before citing any dashboard position as if it were a settled result.

The design is meant to make a replication possible, not to mean one already exists. Each record stores the cryptographic fingerprint of the patch and of the results it came from, the task's base commit, the analyzer version and the Python version used. Because no code is run, repeating the calculation requires no GPU or API keys, only the same artifacts.

There is a signal worth examining closely. The author re-ran GPT-5.6 Sol on his own in a different harness, with few tasks and self-evaluation, and that run added 0.88 times the code of the official run of the same model. That is twelve points of difference attributable to the environment and not to the model. Those own rows appear dimmed on the website, carry no verification label and are not included in the ranking.

## A review cost, not a quality grade

The most honest thing about Parsimony is that it does not claim to be a quality ranking. A small patch can be cryptic, incomplete outside the tests that validate it or outright insecure, and the project itself says so in writing. What it offers is a cost signal: if two agents solve the same thing and one writes half, the human reviewer reads half and the repository accumulates less surface to maintain.

For a team that pays for tokens and for reviews, the figure has practical value if it is calculated at home. Before trusting someone else's dashboard, it is worth measuring in your own repository the net size of accepted changes, separated by model, because the harness and task selection move the result. In a field where figures are repeated without source and without date, checking where each number comes from remains the part of the work that no tool automates.
