AWS: size SageMaker auto scaling from two measurements, not a guess

AWS published a method on 22 October 2026 for setting auto scaling on a self-hosted text-to-speech (TTS) endpoint on Amazon SageMaker AI. AWS says the usual failure is a scaling target that was guessed: too cautious and GPU instances sit idle on every invoice, too aggressive and nothing alarms until a traffic spike pushes queueing past the latency promise. The post, by AWS's Joseph Cottingham, replaces the guess with two measurements taken against the live endpoint.

The two measurements

The model is Kokoro-82M, deployed in a custom container on an ml.g5.xlarge real-time endpoint. The latency target (SLO) is 3,000 ms at the 95th percentile for time to first audio byte. The first measurement, Q_max, is the highest concurrency one instance sustains inside that target. AWS walked a closed-loop load ladder upward and reports p95 latency of 65.4 ms at 1 concurrent request, 1,129.3 ms at 20, 2,805.7 ms at 50 and 3,374.8 ms at 60. So Q_max is 50, a measured ceiling rather than a lower bound. AWS puts the live run at roughly 36 minutes.

The second, T_total, is the wall-clock time from requesting capacity to that capacity serving healthy traffic. AWS forced a scale-out and timed eight stages. It measured 260.2 seconds in total. The largest stage, 180.8 seconds (56.5 percent), is from the capacity request being set to the instance appearing in logs, which AWS attributes to EC2 provisioning and the container image pull. AWS adds a bounded 60 seconds for the policy's detection lag, which this run did not measure, and plans against 320.2 seconds.

The settings AWS derives

With a surge tolerance of 1.25x (traffic may grow 25 percent during one scale-out cycle), the arithmetic gives the numbers AWS deploys:

• Scale out at 37.5 concurrent requests per instance (75 percent of Q_max); scale in at 25 (50 percent).
• Scale-out cooldown of 30 seconds, because AWS says an unneeded instance for 30 seconds is cheap to be wrong about.
• Scale-in cooldown of 960 seconds, about three times T_total, because removing capacity that is needed again costs another full scale-out.
• A queue limit of 50, enforced by the container, which refuses further requests immediately with a 503 instead of letting them wait past the SLO.
• Scaling on ConcurrentRequestsPerModel, a metric SageMaker publishes itself. AWS measured the ratio between that metric and the client-side count as 1.0 at every rung in this run.

What the post leaves out

AWS's own planning tool flags two warnings on this configuration. At the scale-out trigger, one instance is at 97.4 percent of Q_max, and a replacement takes 320 seconds to arrive. And with a floor of one instance, scaling in from two to one puts the survivor at exactly Q_max, which can trigger another scale-out; AWS says the long cooldown damps this without removing it, and that raising the floor to three instances would remove it at the cost of standing capacity.

The post gives no dollar cost for the endpoint or the 36-minute test, and I found no test of the finished policy against a real traffic spike. The numbers are for this model on this instance type, and AWS says they will differ elsewhere. The sample container also pins a PyTorch version with a known advisory (PYSEC-2026-2286, fixed in torch 2.10.0), which AWS says to upgrade before anything beyond a proof of value.

What to do

• If you host a model on a SageMaker real-time endpoint, find where your current scaling target came from and whether anyone measured it.
• Measure your per-instance ceiling with a fixed-concurrency ladder against your own latency target, and time a real scale-out from request to healthy traffic.
• If instance start-up dominates your scale-out time, as AWS reports here, test a smaller container image or warm capacity before tuning thresholds.
• Set scale-in more cautiously than scale-out, and check the minimum instance count against the scale-in arithmetic.