Gauge: open-source CLI that tests your docs on coding agents

Gauge, a start-up, has open-sourced a command-line tool for checking how well coding agents can use a product's documentation, Skills, SDKs and CLIs. The Show HN post went up on 4 November 2026, and the code is on GitHub with an MIT licence badge in the README. Everything below is Gauge's own description; when I read it the post had 12 points and one comment, a spelling quibble about the word 'recursively'.

What it does

The README describes a loop. You define realistic tasks, such as 'set up CI checks', with criteria for success. Gauge starts a fresh Docker session for each run, drops in the working-tree version of your Skill, SDK, CLI or files, and lets a coding agent attempt the task. It normalises the transcripts from Claude Code, Codex, Pi and OpenCode, and your own coding agent then reads them and saves structured findings about what slowed or blocked the run.

Typical findings listed in the README: an interactive command that blocks an agent, deprecated commands or outdated API patterns the agent reaches for, a Skill that goes unused when it should help, and an agent that confidently claims a limitation that does not exist. After a change you rerun the same saved task and compare the two batches with 'gauge run compare'.

What it runs on

Local mode needs Node.js 22.13 or later, npm, Docker running Linux containers and credentials for the agent you want to test. Inference is still remote, so test runs use your provider subscription or API keys; the README gives no per-run cost. Local mode supports Skills, SDKs, CLIs and files. Testing documentation pages, authenticated MCP servers and authenticated APIs needs the hosted version, according to the README.

In the Hacker News post, the founder says the network gateway is mitmproxy in a sidecar container. It terminates TLS, swaps in your variant pages at the same hostname, and replaces credentials with placeholders so agents cannot exfiltrate real keys.

The claimed result, and what is missing

The founder reports that the team used the loop on its own pages to fix mistaken pricing, inaccurate product limitations and repeated workarounds. For Clerk, he says agents moved from a legacy 'keyless' mode to the newer 'clerk init' command, and that after Clerk tested a pre-release change to its SDK's error messages with Gauge, use of 'clerk init' by agents rose 3x after deployment.

That is a vendor and a customer outcome told by the vendor. The sources I read give no sample size, no baseline and no way to rule out other causes. Agent behaviour also varies between runs and models, so a single before-and-after pair proves little; the README's own example uses one sample.

What to do with it

If your company ships an API, SDK or internal CLI, coding agents are now among its first readers. Pick one task a customer or engineer would give an agent, save it with plain success criteria, run it several times before a docs or error-message change and again after, and look at the transcripts rather than only the pass rate. Keep the sandbox's credentials separate from real ones, as Gauge's placeholder design suggests.