Orca Security cut its AI query agent's cost 90% on Amazon Bedrock

AWS published a customer post on 13 October 2026 describing Discovery AI, a natural-language query agent that Orca Security built with AWS on Amazon Bedrock. It turns a plain-English question into a query for Sonar, Orca's query system over its cloud-security data. The post was co-written with Orca staff, and the numbers below are Orca's. Orca says Discovery AI has handled thousands of queries a day in production since March 2026.

The problem

Sonar covers more than 900 cloud models, meaning resource types such as an EC2 instance or an S3 bucket. Querying it means knowing exact model names, navigating parent-child relationships and choosing among 18 operators, according to the post.

The first proof of concept loaded the full schema, every relationship type and many example queries into one prompt on the largest available model. Output quality was strong and design partners liked it, the post says. At production scale it cost about 2 million input tokens per workload and took about 60 seconds. Complex multi-model queries often needed further refinement.

The pipeline

The replacement is a nine-stage pipeline in which each stage gets the model that suits it, per the post:

• Quick deterministic matcher: regex rules recognise IP addresses, CVE IDs, ARNs, file hashes and similar, and return a predefined filter with no model call.
• Guardrail screening: a model checks the request is safe, relevant, technical and supported, and blocks prompt injection and off-topic input. Amazon Nova 2 Lite handles this and validation, where speed and low cost matter most.
• Example retrieval: Amazon Titan embeddings fetch the 20 nearest previously validated queries, and Cohere Rerank 3.5 cuts them to the best five.
• Model selection: Anthropic's Claude Haiku 4.5 narrows 900+ models to the few the request needs, working from compact summaries of each.
• Generation and validation: Haiku 4.5 also runs a tool-using agent that builds the filter. Static checks and Orca's Sonar API verify it, and each error goes back to the agent with the offending element named, so retries re-ground rather than re-guess.

What Orca says mattered most

Orca calls example retrieval the single biggest accuracy lever. It compared several strategies. Plain top-K semantic search was fast but returned near-duplicates. Maximal Marginal Relevance added diversity but lost relevance when the top hit was the right answer. Retrieving 20 and reranking to five won.

The team also built an evaluation harness early: a curated dataset covering production question types, an LLM-as-judge that grades correctness, completeness and equivalence to a reference query, and results broken down by slice. Every prompt, model, retrieval or example change goes through it before shipping.

After launch, a labelling tool lets product managers, support engineers and security researchers flag queries, correct outputs and push the fix into the example bank in one action. Low-confidence production queries are routed to experts daily. Orca's AI director is quoted calling the tool the single most important investment for long-term quality.

The numbers

• Input tokens: about 2 million down to about 140,000 per comparable workload, a 93% cut.
• Cost: down 90%, which Orca attributes to the lower token use plus per-stage model allocation.
• Latency: about 60 seconds down to about 15 seconds, a 75% cut.
• Scale: thousands of queries a day in production since March 2026.

What the post leaves out

• Accuracy. The post says accuracy on complex queries improved, but gives no pass rate, error rate or size for the evaluation set, and the grader is itself an LLM.
• Absolute cost. Only percentages against Orca's own prototype are given, with no dollars per query, so the 90% is relative to a deliberately heavy baseline.
• Failures. There is no count of invalid-filter responses, blocked requests or retries, and no mention of customer-facing errors.
• Alternatives. The post does not say whether a smaller model with the full schema, or no model at all, was tried.

What to take from it

The reported savings came mostly from sending less to the model: summaries instead of full schemas, five examples instead of a large dump, and a no-model fast path for obvious patterns. Cheaper models handle the screening and checking stages. If your own agent stuffs a large reference into every prompt, Orca's account suggests measuring a retrieval-and-selection step before paying for a bigger model. Treat the percentages as a vendor-published, single-company result, and run the same comparison on your own questions.