AWS open-sources UAEF: 36 agent eval metrics across five trace formats

AWS published the Universal Agent Evaluation Framework (UAEF) on 11 November 2026, an open-source system for scoring AI agents that does not depend on the framework the agent was built with. The AWS Machine Learning blog post, by seven AWS authors, says UAEF is released under the Apache 2.0 licence in the awslabs/uaef repository on GitHub.

The problem AWS describes is one most teams with more than one agent will recognise. LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore each record messages, tool calls and handoffs differently, so evaluation code gets written against one trace format and rebuilt when the team switches. That makes it hard to answer the basic question: did the change to the prompt, model or tools improve the agent?

How it works

• Adapters. An adapter converts a framework's run record into one canonical object, the AgentTrace. The post lists adapters for LangGraph, LangChain, Strands Agents, AgentCore and traces already stored in Langfuse, plus a generic JSON adapter for anything else. UAEF can score an existing trace, or call the agent itself given a runtime ARN or an HTTP endpoint.
• Metrics. 36 built-in metrics cover 7 dimensions: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning. Every metric returns a score from 0 to 1. Some are deterministic and reproduce exactly; others use an LLM judge on Amazon Bedrock. RAGAS and DeepEval metrics and custom metrics can be added.
• Storage. Setting persist=True records a run as an experiment: summary rows in Amazon DynamoDB, full results and judge reasoning in Amazon S3. Only persisted runs can be compared.
• Regression detection. For each metric two runs share, UAEF computes the change and classifies it as a regression, an improvement or unchanged, with a configurable threshold that defaults to 5%. Regressions are graded minor (5-10%), moderate (10-20%) or severe (over 20%) and grouped by dimension.

It runs as a Python library, for notebooks, scripts and CI/CD pipelines, or as a deployable AWS service with a web interface. AWS says both use the same engine and stores, so scores do not depend on the path.

Details worth copying even if you never install it

• Treat agent output as untrusted input to the judge. The post's reasoning: "an agent that can influence its judge can influence its own scores." UAEF sends the rubric separately, escapes the content, and requires each judge response to cite a verbatim span of what it scored. A score whose evidence is not found is rejected.
• Keep judge scores out of security decisions. AWS says they are an evaluation signal and should not gate a security control; for runtime enforcement it points to Amazon Bedrock Guardrails.
• Start from the scenario, not the metric list. The post's table maps eight scenarios, from RAG question answering to production monitoring, to a starting set of metrics, and marks which need ground truth. Tool-calling checks need the expected tools, order and arguments for each test case.
• A dataset needs one column. A CSV or XLSX file with a Question column is enough; Answer, context, expected_tool_calls, session_id and turn_id are optional. For multi-turn tests the agent endpoint must hold its own conversation state, because UAEF sends each turn with a session ID and does not replay history.

What the post leaves out

• Results. AWS says customers across 12 organisations in telecom, healthcare and financial services used UAEF before release decisions. It names none and reports no regression caught or incident avoided.
• Judge accuracy. There is no agreement rate between the LLM-judge metrics and human reviewers, and the post does not say which Bedrock model judges by default.
• Cost. Every judge metric is a Bedrock invocation, and the post gives no estimate per test case. It does warn that the deployed service's DynamoDB tables, S3 buckets and Amazon Cognito user pool are retained after the stack is deleted and keep costing money until removed by hand.
• Overlap. AWS published a guide to its managed AgentCore Evaluations service one day earlier, covered here. The UAEF post does not say when to choose one over the other.