AWS QA Studio: a 0-100 test health score that penalises flaky tests

AWS published the third post in a series on 28 October 2026 describing QA Studio, a reference solution in which testers write checks in natural language and Amazon Nova Act, AWS's browser-driving agent, runs them by visually navigating a web app. The new post, by two AWS solutions architects, covers what happens when many teams use it. It is vendor-published and promotes AWS services; it reports no customer deployment, cost or failure data.

Organising tests by application

AWS says an application in QA Studio represents one system under test, such as a web app, with a base URL and optional development, staging and production environments. Test cases and regression suites belong to an application and inherit its owner. Each application belongs to exactly one team.

A health score that marks down flaky tests

Every execution is recorded as an Amazon CloudWatch custom metric, split by application and environment. The application page shows the pass rate, the number of executions and the current failure count over a chosen window: 7 days at hourly granularity or 30 days at daily granularity.

AWS says it also computes a health score from 0 to 100. The score starts from the pass rate and subtracts a penalty for flaky tests, so an application whose tests pass but flip between pass and fail does not look perfect. The post does not give the penalty formula. A separate view ranks tests by how often they flipped over the last seven days, and each recent failure links to Nova Act's trajectory logs and session recordings.

Roles and the ceiling rule

Access runs through five roles, each including the permissions of those below it: Admin (studio-wide), Team Lead (manages members and creates applications), Editor (edits tests and suites), Executor (runs them only) and Viewer (read-only). A team can share an application with another team at view, execute or full level. AWS's ceiling rule says the user's effective permission is the lower of their own role and the grant, so a Viewer given full access to a shared application stays read-only.

Under the hood, AWS says each team maps to Amazon Cognito groups, a Lambda authorizer turns groups into API scopes, and the backend checks both the scope and whether the caller's team owns or has been shared the application. A CI pipeline can authenticate as a machine identity with only the scopes it needs.

What to take from it

• If an AI agent runs your regression tests, report flakiness next to pass rate; a suite that passes by luck should not read as healthy.
• Cap shared permissions at the recipient's own role, so sharing an application never promotes someone.
• Give pipelines their own scoped identity rather than a person's login.

AWS does not say how many tests, teams or runs the design has been tried on, what Nova Act runs cost, or how often agent-run checks themselves misjudge a page. The code is in the QA Studio GitHub repository, which AWS says deploys into a customer's own AWS account.