AWS: screen agent turns with a decision model before an LLM judge

Amazon Web Services (AWS) published a post on 4 November 2026 showing how to add decision models to Amazon Bedrock AgentCore Evaluations, the service that scores an agent on demand, in batch against stored sessions, and online against live traffic. The authors are four AWS staff. The post is vendor-published and is a how-to, not a deployment report.

What a decision model is

AWS describes a decision model as a small model trained to choose from a fixed set of options and return a calibrated confidence for each, instead of free text with a rationale. It names three question types: yes/no, a choice among up to 255 options, and a score on 2 to 10 ordered levels. Examples AWS gives are whether a response is grounded in tool results, whether it was helpful on a four-level scale, and whether the user's goal was completed.

The post connects two backends through AgentCore's code-based evaluators, which run your own logic in an AWS Lambda function: Jev, a hosted model from TypeSafe that needs an API key, and Strands Decider 2B, a 2-billion-parameter open-source model from the Strands Agents team that you host yourself, for example on Amazon EC2 inside your own VPC (virtual private cloud).

The cascade AWS recommends

AWS says the built-in evaluators, which use a general-purpose LLM and explain their judgment, remain the baseline. It positions decision models as a gate in front of them. Run the decision model on every turn of every session for a cheap signal, then call a full LLM evaluator only where the signal falls below your threshold.

AWS's comparison table lists LLM-as-judge latency at 2 to 10 seconds and decision-model latency at roughly hundreds of milliseconds including the network. These are AWS's characterisations, not measurements from the sample.

The validation procedure

The part most teams can reuse is the five-step check AWS gives before you gate a release on a new evaluator. AWS's point is that rewording a rubric as a decision-model question creates a new measurement, which may land stricter or looser than intended.

• Build a labelled sample from stored sessions, covering failures and edge cases, not only the happy path.
• Have three independent people label each turn; take the majority vote as the reference and record which turns were unanimous or split.
• Run the decision-model evaluator and the matching built-in evaluator on the same sessions in one batch.
• Compare agreement with the labels per question, calibration (does higher stated confidence track higher accuracy?), and where the two evaluators disagree. Use the calibration groups to set a threshold per evaluator rather than defaulting to 0.5.
• Repeat whenever you reword a question or change model version.

What the post leaves out

There are no results: not how often Jev or Strands Decider agreed with human labels, not what a screen saves against running an LLM judge on every turn, and not what hosting the 2B model or paying for Jev costs. The demonstration is a mock HR assistant with five labelled scenarios and fixed data, built to check that the evaluators work, not to show how well they judge. The cascade's value for your agent therefore depends on numbers you would collect in step four.

If you already run online evaluation with LLM judges at full sampling, the practical takeaway is the procedure: pick one narrow yes/no check, label a sample, and measure agreement and calibration before it gates anything.