Exowatt: a fine-tuned 1.7B model beat Claude at alarm triage on a CPU

On 27 October 2026 AWS published a post, co-authored with Exowatt's chief engineer of software and AI, describing how the company built two small task-specific models on Amazon SageMaker AI serverless model customization. Exowatt makes modular solar systems that store heat and supply power behind the meter to data centres. This is vendor-published: the post promotes AWS services, and every figure below is the authors'. I have not seen the evaluation code or the prompts.

The job: triage alarms where there may be no internet

Each Exowatt site raises alarms around the clock: a component out of spec, a dropped connection, an out-of-range voltage. The company wanted an agent that reads a site's alarm state and recommends the next action, and it gave three reasons a frontier model did not fit. The task depends on Exowatt's own hardware and operating logic. Many sites have limited or no connectivity and no GPU. And sending every repeated decision from thousands of sites to a frontier model was, in the authors' words, not economical.

The edge model reads an alarm with the site's status and telemetry and returns one of a small fixed set of classifications with cited evidence, or abstains when the evidence is insufficient.

What they built

• Edge model: Qwen3-1.7B, fine-tuned with LoRA (a method that trains a small set of added weights) and supervised fine-tuning. It trained in about 32 minutes for roughly $20 to $25, then was quantised to 8 bits in GGUF format and run with llama.cpp inside Kubernetes pods on on-site controllers.
• Cloud model: Qwen3-14B, trained for one LoRA epoch (rank 16, alpha 32) in about 8.5 hours of managed training on five years of public ERCOT (Texas grid) market data, about 163 million rows. It reads a six-hour packet of about 90 rows and returns one of five market events in about four seconds.
• Serving: the cloud model was imported to Amazon Bedrock Custom Model Import in about 14 minutes and bills $6.86 per hour only while active, with a scheduled keep-warm call.
• Guardrail: both models return strict JSON in which every claim cites an input row. The authors say an evidence fence drops any citation whose source, field or value does not literally match the input.

Results, as reported

On edge triage, the authors report the base Qwen3-1.7B could not produce a valid answer, and the fine-tuned model reached 0.911 macro-F1 on 320 test cases, against 0.698 (Claude Sonnet 4.6), 0.680 (Claude Opus 4.7) and 0.519 (Claude Haiku 4.5). The frontier models are listed at roughly $4.70 to $22 per 1,000 decisions on Bedrock list prices.

On energy-market analysis, the fine-tuned Qwen3-14B reached 0.975 macro-F1 and 0.983 event accuracy with 100 percent valid JSON, against 0.625 and 0.718 for Claude Opus 4.7. The authors give a 95 percent confidence interval of 0.324 to 0.371 for the 0.35 gap, and say the model held 0.968 on a second, later test set. Costs are Bedrock list prices times measured tokens per answer.

What made the difference

Model size was not the lever. A prompted Qwen3-8B plateaued at 0.735 macro-F1 and, the authors say, none of 13 fine-tuned 8B candidates across 12 dataset versions beat it. The smaller 1.7B model won once the training data was more varied.

The data was small. The edge set was 684 verified decisions on generated site scenarios plus 1,838 paraphrases, which kept each decision fixed while rewriting the prose. Of 2,218 paraphrase candidates, 1,838 passed deterministic checks that the decision, labels, evidence citations and every number were unchanged. Generating them cost about $3.72. In the cloud task, three rewrites per answer lifted macro-F1 from 0.950 to 0.975.

What the post leaves out

The frontier models are compared as baselines on a task built from Exowatt's own scenarios, and the post does not show how they were prompted or whether they were given the same evidence rules. Edge test cases came from generated site scenarios, with one fault family held out of training; the post reports shadow-mode operation on a cloud replica of a site, not live field results. Its cost figures exclude the engineering time, and the verification work behind the 684 labelled decisions. Pricing is list price as of publication and may change. No operator-review rate or cost of a wrong triage is given.

What to take from it

If you have a high-volume, narrow classification task with a labelled answer key, run the comparison Exowatt did: prompt a frontier model, prompt a small open-weight model, then fine-tune the small one on varied, checked data before deciding. Keep the checker that rejects any generated training example whose facts changed, and keep the abstain option. Treat the 49x headline as a ceiling for this task, not a planning figure.