AWS: a 4B decision model on SageMaker scored 77.5% on CPU, 77.9% on GPU

AWS published a how-to on 12 October 2026 showing how to serve a small "decision model" on Amazon SageMaker AI with llama.cpp, and it includes a benchmark that compares CPU and GPU endpoints for the same model. A decision model returns a probability for each option you declare, such as a support queue or an incident priority, instead of writing text. AWS wrote the post, so the figures below are AWS's own, measured by AWS on one test set.

What AWS built

The post uses the AWS Deep Learning Container for llama.cpp to serve quantised models. The standard path returns OpenAI-compatible chat completions through the SageMaker InvokeEndpoint call. For decisions, AWS adds a child image running SemIf (formerly OpenJev), an open adapter that reads the model's final-position logits and returns probabilities over the options. AWS says the plain llama-server HTTP API does not expose those logits, which is why the extra image is needed.

The request carries an ID, the unstructured state, a question and an ordered list of allowed options. The response returns option IDs, probabilities and raw logits in the same order, plus timing and model metadata. AWS's example routes a password-reset complaint to either an account-access or a billing queue.

The numbers

AWS sent the same 231 public JevBench tasks, one at a time, to two endpoints running Qwen3.5-4B in Q4_K_M quantisation. All 231 responses were valid in both cases.

• CPU (ml.c6i.8xlarge, 32 threads): 179 of 231 correct (77.5%), median latency 1.715 seconds, 95th-percentile latency 19.291 seconds.
• GPU (ml.g6e.xlarge, all 34 layers offloaded to an NVIDIA L40S): 180 of 231 correct (77.9%), median 0.352 seconds, 95th percentile 0.599 seconds.
• AWS's own comparison: the GPU was 4.9 times faster at the median and 32.2 times faster at the 95th percentile.

AWS notes the two images used different llama.cpp builds and disagreed on six hard tasks. It tells readers to treat the one-answer difference as runtime variation, not evidence that GPU offload improves quality. It also says the figures include network latency and are not service guarantees.

Smaller models, and when to use which

Smaller models cut median latency but lost accuracy: AWS reports 66.7% for the 2B candidate and 58.4% for the 0.8B one, so the sample keeps the 4B model as its default. AWS recommends CPU only for background, offline or low-throughput work that can tolerate seconds of delay, and GPU for interactive decisions, long prompts or tight tail-latency targets. For the GPU path it uses SageMaker instance pools, a priority list of g6e, g6, g5 and g4dn; SageMaker picked ml.g6e.xlarge.

Two operational details are worth copying. The adapter returns a retryable HTTP 503 rather than queueing without limit, and decisions that share the same state can be grouped in one request so the common prompt prefix is reused.

What the post leaves out

• Cost. AWS tells readers to review SageMaker pricing and gives no per-decision or per-hour figure, so a CFO question like cost per routed ticket is still unanswered.
• Your data. The benchmark is a public set (JevBench). AWS says to evaluate with labelled decisions from your own application before production, and 77% may not be good enough for your queues.
• Throughput. Every request was sent serially, so the post does not show how latency holds under concurrent load.

AWS also advises keeping authorisation and deterministic business rules outside the model, and validating the probabilities before treating them as confidence. That fits how this site covered Microsoft's Decision-1 earlier: a score is an input to a threshold and a fallback, not a decision to trust unchecked.