AWS sales agents: a nine-stage eval loop after $410.08 became $410.08 million

On 16 October 2026 the AWS Sales, Marketing, and Global Services (SMGS) Operations team published an account of how it evaluates a multi-agent system that answers revenue questions for AWS field sellers and leaders. The post, on the AWS Machine Learning Blog, describes a nine-stage loop built on Amazon Bedrock AgentCore Evaluations. It is vendor-published: AWS describes its own internal system, gives no cost figures, and ends with an offer to talk to an AWS account team.

What went wrong before launch

AWS says that when it evaluated the Customer Insights Agent before launch, the agent reported $410.08 of a customer's three-month revenue as "$410.08 Million", labelled Sales Revenue as Sales Revenue excluding Adjustments, offered an Annual Recurring Revenue estimate that appeared nowhere in the data, and presented its own calculations as sourced facts. It says monitoring stayed green because it showed the system had responded, not whether the answer was right.

From production errors the team defined six failure types, any one of which marks a response inaccurate: metric swap, wrong period, wrong entity, pure hallucination, wrong symbols and unauthorized calculation. It built a custom evaluator for each, because none is covered by AgentCore's built-in templates, and adds deterministic checks where the answer is known exactly, such as revenue or renewal date.

Gates, canaries and what each caught

• CI: a sentinel set of 15 cases runs on every staging deployment. AWS says it blocked a data normalization change that made a disambiguation tool return the wrong customer's data.
• Release gate: the full golden set runs against frozen, validated tool responses, plus 200 held-out end-to-end questions scored by an LLM judge. A candidate must beat its own prior run and the production baseline, and keep failing questions under a set share.
• Canary: evaluations run every 30 minutes on live and synthetic cases. AWS says 15-minute windows caused too many false alarms and 60-minute windows let two hours pass.
• Drift: 10% of live traffic is scored. Customer identification accuracy fell from 98% to 81% to 80% across three windows because sellers typed stock tickers and shorthand that the synthetic golden set lacked.

Numbers AWS reports

Over three months of iteration, AWS says the overall pass rate rose from 93% to a stable 96%, pure hallucinations fell from 138 to under 52 per 10,000 responses, and unauthorized calculations fell from 59 to under 44. It credits a prompt fix: few-shot examples held real dollar amounts and customer names that the model leaked into unrelated answers. The post gives the 93% to 96% change as the pass rate in one place and as the grounding rate in another.

AWS also reports that a data change, a tool change and a paired prompt update that were meant to ship together but landed apart dropped a pre-production pass rate from 78% to 38%. Its one concrete detection claim: mean time to detect fell from days to weeks to under four hours.

What the post leaves out

It does not say what the evaluation infrastructure costs, how many sellers use the agents, or how often alarms fired in production. Its dashboards are labelled illustrative. The outcome figures come from the team that built the system and sells the platform.