AWS: parts-substitution agent keeps pass/fail verdicts in plain code

AWS published a technical walkthrough on 2 November 2026 for a multi-agent assistant that helps aerospace engineers evaluate substitute parts when a qualified component goes obsolete. It is built on Amazon Bedrock AgentCore, Amazon Nova Multimodal Embeddings and the open-source Strands Agents SDK. The data is a synthetic set of seven reaction wheels from a fictional manufacturer, so this is a vendor-published reference design, not a deployment report.

The design rule: the engineer decides

AWS says the system assembles evidence and drafts documents but never recommends or approves a substitution. That rule decides where models are used. A planner on Claude Haiku 5.5 produces a routing plan, three specialists (Catalog, Evaluation, Knowledge) on Claude Sonnet 5.5 execute scoped tools, and a supervisor on Claude Opus 5.5 combines results only when two or more specialists ran.

AWS gives a cost and latency reason for the last part: synthesising one specialist's already complete answer added about 8 seconds with no quality gain. It publishes no per-query dollar figure.

Verdicts come from code, not a model

The fit check against a hand-written requirements schema runs as ordinary Python. It compares each spec against tolerance rules, counts passes and flags, and surfaces lifecycle risk. AWS says no LLM is involved because the verdict must be reproducible for review. The specialist can explain flags but cannot change a verdict. The audit-trail entry, which AWS says is aligned with AS9100 sections 8.4 and 8.5.6, is written by the orchestration layer from the tool-call history, so logging does not depend on the model remembering to do it.

Three retrieval lessons

• Crop charts before indexing. Whole datasheet pages are about 90 percent shared boilerplate, which dominates the embedding. Indexing chart-band crops alongside the page took chart-upload matching from 3 of 7 parts correct to 7 of 7, AWS reports.
• Filter when the user names a part. A torque curve contains no part name, so text-to-image similarity alone did not reliably return the right chart; the tool keeps only results from the named part.
• Log every silent fallback. A missing s3vectors:QueryVectors permission on the AgentCore runtime role made image queries fall through to text retrieval without an error, and the model then named a wrong part with high apparent confidence. Developer credentials were broader than the runtime role, which hid it in testing.

What AWS measured

AWS used AgentCore Evaluations (LLM-as-a-judge scoring of OpenTelemetry traces) with seven evaluators and a 36-query curated set that includes four safety prompts asking the agent to approve a substitution. On the synthetic dataset it reports 30-day averages of 0.981 for Faithfulness, 0.968 for Correctness, 0.861 for Tool Selection Accuracy and 0.892 for Goal Success Rate. Tool selection rose from 0.809 to 0.861 after consolidating to one knowledge base, fixing the permission above and tightening routing.

The 0.892 goal score is partly an artefact: AWS says the evaluator counts a correct refusal as an unmet goal. That is a useful warning for anyone designing evaluators for human-in-the-loop agents.

What the post leaves out

The numbers come from a seven-part synthetic catalogue, so they say little about a real parts database with messy documents. AWS gives no cost per assessment and no comparison with a single-model baseline. It also says the Cedar policy encoding the human-in-the-loop rules is not yet attached to an authorization engine; the application enforces them in its own logic today. Deployment advice is practical: tag each container image immutably, because a runtime that has resolved a moving :latest digest keeps serving the old image.