AWS test: Grok 4.7 at low effort got 15 of 18 right on 1/8 the tokens

AWS published a post on 13 October 2026 that tests how much reasoning effort an agent needs on SpaceXAI's Grok 4.7, which AWS offers on Amazon Bedrock. The AWS authors built a tool-calling agent, gave it three tasks that need search rather than lookup, and ran each at all four effort levels six times: 72 runs in total. The post is vendor-published and the results are AWS's own.

What AWS measured

The environment is synthetic: 20 services exposed through seven tools. The three tasks each have exactly one correct answer, which AWS says it verified by brute force before running anything.

• Two-fault diagnosis: find the two services that failed silently at 03:00, given alerts from 12 others. There are 190 possible pairs and one explains exactly the alerts observed.
• Maintenance schedule: patch eight services over four nights, two per night, under ten constraints plus a rule that two services from one team never share a night. There are 2,520 candidate schedules and one is valid.
• Migration budget: pick the set of migrations that cuts the most risk within a budget, including prerequisites. There are 53 feasible sets and one optimum; choosing by best ratio first lands nine points short of it.

The results

According to the post, correct answers were 15 of 18 at low, 17 of 18 at medium, 17 of 18 at high and 18 of 18 at xhigh. Mean output tokens per run were 1,184 at low, 6,430 at medium, 6,778 at high and 9,147 at xhigh. Median run time was 18 seconds at low and 98, 89 and 106 seconds at the other levels.

Tool calls barely moved, from 17.2 to 18.7 per run. AWS reads that as the agent gathering the same facts at every level and spending the extra effort on reasoning over them.

AWS's own advice: low is an efficient choice where a slightly lower completion rate is acceptable or answers are reviewed downstream, and higher levels are safer where every answer must be right first time. It also suggests setting effort per request, so planning and verification steps run high and mechanical steps run low.

What the post leaves out

• Statistical weight. AWS says six runs per cell give a consistent direction, not a confidence interval. One run in 18 is 5.6 percentage points, so a single miss moves the figure.
• Real-world conditions. The tools read generated data, with no latency, partial results or errors, and runs lasted seconds to minutes, not the hours Grok 4.7 is marketed for.
• Cost in money. AWS reports tokens only and points to its pricing page, since Global and US profiles are priced differently and service tiers add multipliers.
• Comparison. It tests Grok 4.7 only, in one Region with one prompt per task, so it says nothing about other models.

What a team can reuse

The method travels better than the numbers. AWS recommends stating every rule the grader relies on, proving each task has one answer by enumeration, grading a required final line such as FINAL: auth, checkout instead of the whole response, and tracking format failures separately from wrong answers. All 72 responses in this test carried a gradeable final line.

For production, where there is no known answer, AWS suggests watching output tokens and tool calls per run. Every correct run at medium and above used at least 1,739 output tokens, so it proposes flagging any run with no tool calls or with output under a fifth of the lowest correct run at that level. It also warns that reasoning tokens count against the output limit, and a response that hits the limit mid-reasoning comes back empty; AWS used 32,768.

On access, the post says Grok 4.7 on Bedrock is reached through cross-Region inference profiles, us.xai.grok-4.7 for US data residency and global.xai.grok-4.7 for widest capacity, with four effort levels: low, medium, high and xhigh. The 500K-token context window and benchmark gains over Grok 4.6 are SpaceXAI's claims, relayed by AWS.