AWS: multi-turn RL lifted gpt-oss-20b from 61% to 84% for $339

On 23 October 2026 AWS published a hands-on guide to Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI, a day after training sessions reached general availability. The authors say the run took gpt-oss-20b from a 61% to an 84% solve rate on 128 held-out Wordle words at its default reasoning effort, and from 15% to 69% at the low effort it was trained at. They put the training bill at about $339 in tokens.

How the setup works

MTRL is serverless: you pick a base model and AWS runs the training and sampling infrastructure. The agent runs elsewhere, here on Amazon Bedrock AgentCore, and SageMaker sends it one training row per rollout. The agent calls the policy model, acts over as many turns as the task needs, and returns a single reward at the end. You supply no correct trajectory, only a way to score the result. Group Relative Policy Optimization (GRPO) then updates a Low-Rank Adaptation (LoRA) adapter. AWS lists four supported base models: Amazon Nova 2 Lite, GPT-OSS-20B, Gemma 4 31B and Qwen 3.6 27B.

What it cost

Billing is per million tokens in three dimensions. The 100-step run used 491.6 million prefill, 159.7 million sample and 643.3 million train tokens across 105,486 rollouts in 12 hours 40 minutes. At GPT-OSS-20B list prices in us-east-1 ($0.12, $0.30 and $0.36 per million) that is about $338.50, with the train dimension at $231.59. The AgentCore runtime billed separately at about $79. The evaluation jobs cost about $1.20 in tokens.

The mistakes AWS lists

• Reward design. Summing per-guess scores over a game meant a lost game could pay about +0.49 and a turn-1 solve (+0.99) scored below a turn-3 solve (+1.61). The fix is an outcome term (1.05 to 1.55 for a solve, -0.5 for a loss, -1.5 for never making a valid guess) plus a small clipped shaping term.
• Evaluation settings. The training validation curve peaked at 48% at step 90 and ended at 37%, scored at temperature 0. At medium effort and temperature 1.0, the trained model solved 84% against 61% for the base model.
• Plumbing. The training role must trust job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com. A rollout that never reaches the policy model fails the whole job, and redeploying the agent mid-job fails it too.
• Learning rate. At too high a rate the model wrote tool calls as plain text, every rollout earned the same floor reward and the gradient vanished.

What the guide leaves out

It is vendor-published and promotes AWS services. Wordle is a teaching example that frontier models already play well, and AWS says so. Validation uses only 128 words, and the authors call the swings between 35% and 48% mostly noise. They did not test whether more steps help at serving settings. The post reports no run on a business workflow, so it gives no evidence on how much real internal data or reward-writing effort a comparable gain would need.

What to do

• Write the reward first and replay it by hand on real transcripts, including losing ones, before launching a job.
• Plan evaluation at the temperature and reasoning effort you will serve at, and budget a separate evaluation run for it.
• Price a run from token counts (prefill, sample, train) and remember the agent runtime bills on top.
• Treat a game result as a method demo; pilot on one narrow workflow with an objective pass or fail score.