AWS's AgentCore eval playbook: start with three evaluators per agent

AWS published a guide on 10 November 2026 to choosing evaluators for agents on Amazon Bedrock AgentCore Evaluations. The post, by four AWS specialists and solutions architects, says the service's catalogue now has three sources: 15 built-in evaluators written by AWS, managed metrics from the open-source libraries DeepEval and AutoEval, and custom evaluators that a team derives from either.

The useful part is the advice on what to pick. The post's argument is that a good score on one metric can hide a failure on another: a support agent can resolve every ticket and still expose personal information in its replies.

Start from the outcome, not the metric list

AWS calls picking metrics by name the most common mistake: "Helpfulness sounds good, so we use it." It sets out three steps instead.

• Define success in business terms. For a support agent, the post suggests first-contact resolution.
• Break that outcome into behaviours you can observe: the agent completes the user's stated goal, stays coherent across turns, keeps within compliance limits and calls back-end tools correctly.
• Map each behaviour to an evaluator at the right level. Trace-level metrics cover a single turn or tool call. Session-level metrics cover outcomes that only make sense across the whole conversation.

The starting stacks AWS recommends

The post gives three example stacks and says to begin with two or three evaluators, adding another only when you find a failure the first set missed.

• Customer support. Start with Builtin.GoalSuccessRate as the anchor, DeepEval's ConversationCompleteness and DeepEval's PIILeakage. The main risk AWS names is dropped requests and unsafe or non-compliant replies.
• Multi-step research. Start with Builtin.TrajectoryAnyOrderMatch, DeepEval's GoalAccuracy and DeepEval's KnowledgeRetention. The risk is a right answer reached by the wrong tool path, or context lost in a long session.
• Medical, legal or financial content. Start with Builtin.Correctness, DeepEval's PIILeakage and AutoEval's Security. The post's rule here: "A wrong answer is worse than no answer."

AWS says ConversationCompleteness catches an agent that answers the latest message but drops earlier requests, and KnowledgeRetention catches one that forgets what the user said three turns ago.

Three constraints worth knowing before you build

• Ground truth limits where a check can run. Builtin.Correctness and the three trajectory checks need a reference answer or an expected tool path, so they run in batch evaluation only. Live traffic has neither.
• One judge call can score several criteria. A multi-dimension evaluator is a custom LLM judge that reads a trace once and returns a separate result per dimension, with no blended score. AWS's example: a release that improves tone but regresses policy compliance should fail, and a single number would hide that. All dimensions in one evaluator share one level and one judge model.
• Where inference runs depends on the tier. In managed mode AgentCore hosts the evaluator code, picks the judge model and runs inference on its own capacity. Custom evaluators let a team swap in its own model, and the post's table says to use them when inference must stay in your account. For medical, legal and financial agents it says "almost always".

The same evaluator IDs work in all three modes: on-demand during development, online sampling of live traffic at a configured rate, and batch runs against a fixed golden dataset before a release. AWS says moving from one to another is a configuration change.

What the post leaves out

• Cost. There is no price for an evaluation call, and no guidance on what sampling rate a team can afford on live traffic.
• Judge accuracy. AWS says its built-in evaluators are "benchmarked for reliability" but gives no agreement rate with human reviewers for them or for the third-party metrics.
• Which judge model runs. In managed mode AgentCore chooses it, and the post does not name the model.
• Evidence. No customer or internal result shows these stacks catching failures in production. A separate AWS account from 16 October 2026, covered here, described its sales team's nine-stage evaluation loop on the same service.