Deepgram ports Nova-3 to AWS Trainium2: 11-70% more audio per dollar

AWS published a post on 19 October 2026 describing how Deepgram, a speech AI company, moved its Nova-3 speech-to-text model onto AWS Trainium2 chips. The headline result, in AWS's words: on Deepgram's offline, throughput-oriented batch workloads, a single trn2.3xlarge instance processed 11 to 70 percent more audio per dollar than the compute options Deepgram had been running. The post is vendor-published and the figures are measured at instance pricing at the time of the evaluation.

What was built

The post says Deepgram's models now process quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organisations. The Trainium work targets batch jobs such as contact-centre analytics and medical transcription, not live voice.

Deepgram delivered Nova-3 as standard PyTorch code. Over several months, Deepgram and the AWS Forward Deployed Engineering team ported it to Trainium2. The authors say they ran it first in eager mode to find operations that fell back to the CPU, then compiled it with torch.compile, with the encoder and decoder as separate static-shape graphs. After compilation, they report no CPU fallbacks.

The final setup runs eight model replicas on one trn2.3xlarge, using one physical NeuronCore per replica. A work-stealing queue with one thread per replica feeds them.

Where the effort went

• Accuracy checks: the team compared layer outputs against CPU and tracked word error rate against Deepgram's reference values and target. The post does not publish the word error rates.
• Kernels: prebuilt fused kernels from AWS's public NKI Library covered attention projections, feed-forward layers and a depthwise convolution. Profiling then showed where hand-written kernels would do better, and the team used AWS's Neuron Agentic Development toolkit, which includes AI skills for profiling and kernel writing, to help produce them.
• Runtime: Deepgram's production serving stack is written in Rust, so the team built a lightweight Rust runtime that loads precompiled Neuron executables directly. AWS says it reproduced the reference token sequences exactly at every tested batch size and delivered a 7.5 percent throughput gain over a multiprocessing Python baseline at the best batch size, with larger gains at small batches.

What the post leaves out

Throughput is reported as a relative gain over a baseline configuration, not as absolute audio per second. The post does not name the hardware Deepgram compared against, so the 11 to 70 percent range cannot be checked or translated into a dollar figure. It does not say what share of Deepgram's traffic now runs on Trainium2, what the several months of engineering cost, or whether customers saw lower prices. It says the gains help Deepgram keep prices down and can extend to customers who self-host Deepgram models on Amazon SageMaker AI, but gives no numbers for that.

It is also a co-authored vendor post for AWS's own chips, with AWS engineers embedded in the project. A team without that support should expect a longer port.

What a team can take from it

The pattern is relevant to anyone paying for high-volume batch inference on a model they control. The post's own numbers suggest the saving depends heavily on batch size and workload, so measure audio, tokens or documents per dollar on your own data, at the batch size you will actually run, and compare the one-off port cost against the monthly saving. If your serving stack is not Python, budget for the runtime work as well.