AWS: a distilled 4B agent matches its GLM-5.3 teacher on ShoppingBench

AWS published a technical walkthrough on 2 November 2026 showing how to distil a tool-using shopping agent from a large model into a small one on Amazon SageMaker AI. It reports that a 4-billion-parameter Qwen3.5 student reached the task success of its teacher, GLM-5.3 on Amazon Bedrock, on the public ShoppingBench benchmark. This is a vendor-published engineering post run on a benchmark, not a customer deployment.

The method: sequence-level distillation

The post uses sequence-level knowledge distillation (SeqKD): the teacher generates complete runs, and the student is fine-tuned on them with ordinary supervised training. AWS says this needs only the teacher's text, with no access to its internal scores, so it works with a closed, API-only teacher.

For an agent, the unit copied is a trajectory: reasoning, tool calls and the results they return. AWS sampled the teacher twice on each of 426 training problems at temperature 1.0 (850 runs), then kept only the runs ShoppingBench's own scorer judged solved. This step is called rejection sampling. The kept runs became about 2,600 training and 130 validation rows, one per step, with the loss applied only to the assistant's reasoning and tool call.

What AWS measured

On 324 held-out problems, split across product, shop and voucher requests, AWS reports average ASR (the share of problems fully solved) of:

• Base Qwen3.5-4B, no fine-tuning: 0.297
• Distilled 4B student: 0.541
• GLM-5.3 teacher, sampled at temperature 1.0: 0.511
• GLM-5.3 teacher, greedy: 0.540
• A medium-sized proprietary frontier model, unnamed: 0.546

The gain was largest on voucher problems, which rose from 0.121 to 0.422. AWS says the figures come from a single run and expect a few points of run-to-run variation. The student can edge past the sampled teacher, it says, because it trained only on the teacher's solved runs.

Two findings worth acting on

First, keep the reasoning. The same student trained on bare tool calls scored 0.415 rather than 0.541, a gap of about 13 points. AWS concludes the reasoning between actions is part of the behaviour being learned.

Second, specialisation has a price. On IFEval, a general instruction-following test, the student's prompt-strict score fell from 0.819 to 0.741 and its instruction-strict score from 0.872 to 0.813. AWS calls that the expected cost and says a much larger drop would signal over-narrowing.

Practical details and what the post leaves out

AWS advises splitting by problem rather than by row, to avoid leakage between training and validation, and trimming tool observations to the fields the next decision needs so rows stay under the 16,384-token limit. It also suggests a cheap pre-check that scores the student's tool selection against the teacher's before running the full evaluation. The student is a LoRA adapter served on a single G-family GPU instance alongside the base model.

There is no cost per task or latency number. AWS says hosting swaps a per-token charge for an hourly instance charge, so the saving depends on traffic. It also does not say how the method behaves on tasks without a programmatic scorer, which the filtering step relies on. A team would need its own grader before copying the recipe.