Dharma AI: a rebuilt vocabulary cut a 4B grader's inference cost 77%

Dharma AI, a company that builds specialised language models, published a Hugging Face team post on 21 October 2026 saying it cut the inference cost of a production essay grader by 77% by changing the model's tokenizer vocabulary, not its weights or training recipe. The post previews a paper called VocabX that Dharma says is under review and not yet released. The figures below are Dharma's own.

The system and the change

The grader reads a Portuguese essay, its prompt and supporting texts, and returns a JSON assessment with a grade and comment per competency. It runs on Qwen3-4B, a 4-billion-parameter open model. Dharma says it kept the original general-purpose tokenizer, with 151,669 entries, even though the task uses one language and one output format.

Dharma says that caused two costs. The text the grader writes every time, such as the JSON envelope, competency names and stock phrases, was split into many tokens. And at each generated token the model scored the full vocabulary, much of which this task never uses. Dharma reports the output layer is almost 10% of the parameters of a model this size, and that it shrank from about 152,000 rows to about 65,000.

Why output tokens drive the bill

Dharma argues that reading input is cheap because the model processes it in one pass, while writing is slow because each token needs its own pass and is limited by memory bandwidth. In its grader it measured a generated token at 7.4 times the cost of a token read, and says the ratio varies with hardware, batch size and context length. VocabX therefore picks vocabulary entries by the cost they save, weighting output tokens more, not by how often they appear. It also merges repeated spans, such as whole JSON envelopes, into single tokens.

Results, and a course correction

Dharma trained seven grader versions with the same base model, data, recipe and seed, differing only in tokenizer. It ran 500 identical requests on one RTX 5090 GPU with the vLLM inference server and computed cost from measured runtime. Average response length fell from 350 to 101 tokens, cost fell 77.0%, and the bench score, which combines grading accuracy and screening of off-topic or rule-breaking essays, rose from 0.703 to 0.810.

The first version fused each grade with its surrounding JSON into one token. It cut cost 73.3% but scored 0.689, below the production grader, because the model had to pick the grade and the structure in a single step. Giving the grade its own token added 2 tokens per response and, Dharma says, was still cheaper overall.

What the post leaves out

• The paper is unpublished and Dharma says its numbers may change before publication.
• It is one grader, one language, one GPU and 500 requests; no independent replication exists.
• Dharma does not give a dollar figure or say what retraining the model with a new vocabulary cost in engineering time.
• In tests on five public domains and three model families, Dharma reports VocabX compressed 1.2 to 9.3 points more than the best reference method, but those are token and estimated-cost figures, not production bills.

For a team running a fine-tuned small model on a fixed output format, the transferable point is cheap to check: count output tokens separately from input tokens, and re-run your quality evaluation after any change that shortens what the model writes.