Ai2's GPU budget scheduler: 98% of owed hours, 74% fewer repairs

Ai2 (the Allen Institute for AI) published an account on Hugging Face on 9 October 2026 of how its AI Infrastructure team changed the way about 150 researchers share thousands of NVIDIA H100, B200 and B300 GPUs. Everything below is the team's own report; the post includes no outside audit.

What was going wrong

Ai2 says requests for GPUs run 2 to 3 times what the clusters hold, which range from 88 to 1,024 GPUs. Its old scheduler used priorities, and jobs could opt out of preemption (being stopped to make room for others). The team reports three results: researchers parked no-op jobs to keep GPUs for later debugging, priority inflation until 100% of scheduled workloads used HIGH priority, and on-call engineers spending most of their ticket time negotiating shutdowns of protected jobs on hosts with known faults.

What it changed

Managers now split GPU time, not GPUs, across a tree that mirrors the research programme. Every protected job is charged to a budget, so a parked job spends its owner's allocation on nothing. A fair-share scheduler tracks use over a 7-day window and favours under-used allocations.

Each job also declares a minimum runtime, capped at 8 hours, during which it cannot be preempted. After that, the scheduler may stop and requeue it. A job can set the minimum to zero, which means it is uncharged and always preemptible. Ai2 says 18% of delivered GPU time was this unallocated kind, which kept the clusters full.

What Ai2 reports

Over a 30-day test period, the team says teams received 98% of the GPU hours they were owed (capped at their demand), 13 of 15 allocations received 95% or more, and the worst case received 90%. Occupancy held at 98% before and after. The p90 queue wait for short debug jobs fell from 2 hours to 30 seconds, but the team notes the baseline sample was small. On its largest H100 cluster, median wait fell from 5 minutes to 24 seconds and p90 wait from 2.8 hours to 1.8 hours. Because unhealthy hosts now drain as jobs reach their minimum runtime, repairs needing a human fell 74%.

The post quotes one researcher, Chris Clark, saying it feels like an extra 30% of compute because bursty teams can reclaim idle allocation. That is one user's impression, not a measurement.

What did not improve

Ai2 says interactive sessions got worse. Researchers who used to hold a session for up to a week now lose it after the 8-hour protected window if they exceed their allocation, and must rebuild state by hand. The team plans a CPU-only cluster for data-prep sessions and restorable sessions. It also says early rollout confused users, because old interface terms such as priority changed meaning, and documentation alone did not fix it; live explanation sessions did. It lists capacity fragmentation, which could lengthen waits for the largest jobs, as an open concern.

What to take from it

The post covers a research lab's training clusters, not an enterprise inference fleet. The transferable part is the incentive design: free priority levels get inflated, so cost has to attach to protection. Unknown from the post: dollar savings, and how the approach behaves with the mixed training and serving workloads most companies run.