LightOn's ColGrep-Agent cut Claude Code search tokens 34%, but slower

LightOn, a French AI company, published ColGrep-Agent-2B on 12 October 2026: an open 2-billion-parameter model that runs on the developer's own machine and does the repository searching for a coding agent such as Claude Code. In LightOn's test, handing search to the local model cut Claude Code's billed API tokens by 34.3% compared with plain grep, and cut cache-aware cost per question by 40%. The numbers come from the authors' own benchmark, in a post on Hugging Face, not from an independent evaluation.

How the test worked

LightOn ran Claude Code with Sonnet 5.5 on 52 code-search questions across 13 public repositories created in 2026, from 17,000 to 1.8 million lines of code, in Rust, Go, TypeScript and Python. Each question ran three times in each of four setups, 624 sessions in total. The authors chose recent repositories to reduce the chance the model had seen them in training; they also asked Sonnet 5.5 to name ten source files from each repository without access and say it could not name any, which they call a sanity check rather than proof.

Of the 52 questions, 39 span several folders and need every location in a reference chain to count as correct; 13 name a single location in words that differ from the code. Claude Opus 5.5 wrote the cross-folder questions and graded the answers blind to the setup. Token totals count Claude Code's API input and output only, not the tokens the local model generates, since those are not billed.

What it found

• Grep: 142k tokens per question, 7.3 cents cache-aware cost, 83.3% correct, 5.5 turns, 13.0 s.
• ColGrep-Agent: 93k tokens (34.3% fewer, 95% confidence interval 28 to 40%), 4.4 cents, 80.8% correct, 3.9 turns, 21.6 s. It used fewer tokens on 88% of questions.
• Plain ColGrep semantic search without the agent: 24.6% fewer tokens, 79.5% correct, 15.0 s.
• A 'balanced' setup that lets Claude choose between the two: 21.2% fewer tokens, 83.3% correct (matching grep), 17.7 s.

According to the authors, the saving comes from fewer turns, because each turn resends the accumulated context, about 23k tokens per turn in these runs. Savings grew with repository size: 19% under 50,000 lines, 29% from 50,000 to 150,000 and 40% from 150,000 to 600,000. In their worked example, a CRM repository where the code says 'field' and the question says 'custom attribute', Claude needed 3 turns and 79.2k tokens instead of 6 turns and 160.6k.

What the authors concede

LightOn says the accuracy gap against grep is not statistically significant, but writes that its interval is wide enough that it cannot rule out a meaningful drop. The results are for code-search questions, not end-to-end coding tasks. A local call took 7 s at the median and 22 s at the 90th percentile on LightOn's GPU server, and about 20 s per question on an M3 Pro with 3 to 4 GB of peak memory. The Claude Code integration works through hooks that suggest the tool; they do not force Claude to use it.

Two things the post does not settle. The questions and grading were produced by Claude Opus 5.5, from the same vendor family as the model being tested, so any shared bias is unmeasured. And the savings are per search question: the post gives no figure for how much of a full coding session is search, so a team cannot read the 34% as a cut to its whole bill.

The small model on its own

On code-localisation benchmarks adapted from SWE-bench Lite and Multi-SWE-bench Flash, LightOn reports that the 2B model scored 76.18 file F1 and 19.92 line F1 on SWE-bench Lite, the highest line score among the open models it compared, which included Gemma 4 31B and Qwen3.5 27B. On the same benchmark Gemma 4 31B had higher file F1 (79.93). The model is Apache-2.0 licensed on Hugging Face, about 2.5 GB in its default quantised form, and installs with Homebrew; LightOn says it was trained from MiniCPM5-2B with supervised fine-tuning followed by reinforcement learning.

What to do

If coding-agent tokens are a visible line in your budget and your repositories are large, try it on a few real questions from your own code and compare token counts and wrong answers against grep. If your developers are latency-sensitive or on thin laptops, the extra 8.6 s per question and the local GPU or memory load may cost more than the saving. Treat the 34% as a best case for large repositories, and verify accuracy on your own code before turning it on by default.