Rescale's test-triage agent agrees with humans 68% of the time

Rescale, a digital engineering platform, has put an AI agent in charge of the first pass on its nightly test failures, and says the agent's classifications now match its engineers' 68% of the time. The account was published on 22 October 2026 on the AWS Machine Learning blog, co-written by AWS staff and two Rescale engineers, so it is a vendor-published customer story rather than independent reporting.

The job

Rescale runs a nightly Playwright suite of hundreds of end-to-end test cases. When tests fail, an on-call engineer classifies each failure, routes it to the owning team and gathers evidence from logs, monitoring and source repositories. The post says these tests gate weekly production releases, so unresolved failures could pull several teams into hours of coordination before a deployment was allowed.

What it runs on

According to the post, the agent runs as a step in Rescale's existing Argo Workflows CI/CD pipeline on Amazon EKS and is built with the Strands SDK. For each failure it calls Claude Opus 4.8 on Amazon Bedrock to assign one of four root-cause categories: infrastructure issue, code bug, flaky test or environment problem. It then retrieves historical failure patterns from a knowledge base, queries pod logs through the Model Context Protocol (MCP), and writes a structured report with file references, function names and recommended fixes.

The authors say the output goes to engineers for review before any action is taken, and that automated pull-request generation was deferred to a second phase.

What changed during the build

The workshop ran five days with AWS architects and Rescale engineers using AWS's AI-Driven Development Lifecycle (AI-DLC), an open-source method in which the AI asks clarifying questions and the team approves each plan. The post reports two reversals. The team planned to host the agent on Amazon Bedrock AgentCore, then moved it to EKS because Rescale already ran its pipeline there and it avoided a new runtime. It also swapped a third-party logging service for Amazon CloudWatch behind the same MCP interface.

By day 4 the full pipeline ran in staging. On day 5 the team built a PyTest evaluation framework with live and stubbed modes to set a baseline of agreement with human classification.

The numbers, and what they leave out

Rescale reports that agreement with human triage rose from 44% at deployment to 68%, as the team iterated on its golden dataset and prompts, and that time blocking production deployment fell 20%. The post does not say how many failures the figures cover, what the other 32% looked like, whether disagreements were the agent's errors or the humans', what the agent costs per run, or how the 20% was measured. Agreement with human triage is also not accuracy: it measures the agent against the people it replaces.

What to do

• Before automating triage, write down the classification your engineers already do and build a golden set from their past decisions; Rescale's baseline of 44% shows a first version can start well short of useful.
• Keep the agent's output as a report for review until you know where it disagrees with people, as Rescale did; ask a vendor story like this one for the cost per run and the disagreement breakdown it omits.
• Host the agent where your pipeline already runs. Rescale's move off AgentCore onto its existing cluster is the post's clearest example of choosing less new infrastructure.