Ai2's StateBench: frontier models fall to 13% on long state-tracking

The Allen Institute for AI (Ai2) published StateBench on 3 November 2026, a benchmark of how well language models keep track of values that change across a long document. In Ai2's tests, four frontier models handled simple cases but fell to 13% accuracy or lower on the hardest cases at the longest input lengths. The post is on the Hugging Face blog and links a tech report, a dataset and code on GitHub.

What StateBench measures

State tracking is following what each variable currently holds as a sequence of updates arrives. Ai2 describes four difficulty levels, explained as kinds of variable assignment:

• R-trivial: a value is set once and never changes, like a birth date.
• Aperiodic: a value is reassigned, so the model must find the latest one, like a thermostat setting.
• Soluble: each update builds on the current value, like a running game score (x = x + 5).
• General finite: one variable can take another's current value, as in swaps. Ai2's example is a ball hidden under one of five cups that are repeatedly swapped.

Ai2 built the four levels from the same number of variables and distinct values, with assignments at the same points in the sequence, so only the kind of assignment changes. It also tests the same problem in several formats, because format alone changed results.

What Ai2 found in frontier models

Ai2 asked Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna to read a sequence of assignments and write out the final value of every variable. All four did well on the R-trivial and aperiodic levels. On the soluble and general finite levels, accuracy fell sharply as inputs got longer, to 13% or lower on the longest general finite problems. Ai2 suggests sensitivity is one factor: at the harder levels many more steps affect the answer, so one missed update spoils it.

Open models and architecture

Ai2 first tested open pretrained models whose training data and context lengths are documented, including two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers). None had seen contexts longer than 64,000 tokens in training. Almost all scored well on R-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Format mattered: 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style R-trivial problems but near chance on the same problems framed as spreadsheet edits.

Ai2 then trained 275-million-parameter models from scratch on StateBench, on sequences of up to 256 assignments, and tested them on sequences up to 64,000 assignments. It reports that Gated DeltaNet and hybrid models generalised almost perfectly on the first three levels out to tens of thousands of assignments, while transformers learned the easier levels but degraded on longer sequences and never learned the hardest level. Ai2 reads this as a training gap for the recurrent and hybrid designs, and a limit for transformers.

What the benchmark leaves out

Ai2 lists open questions itself. StateBench assumes a fixed number of variables, while real tasks, such as a long conversation, add new people and objects as they go. The exact capability of one Gated DeltaNet variant is unproven. The abstract tasks are also not business documents, and the post does not report cost, latency or any tool-assisted runs, so it says nothing about agents that can run code to compute the answer.

What it means for a rollout

The post does not give business guidance. The implication I draw, which is mine and not Ai2's: where a workflow depends on a model carrying edited values across a long document, such as balances, inventory counts or reassigned owners, the risk rises with length, and the format of the data can change the result. Test that pattern at your real input size before relying on it.