The AI Read Every Update. It Still Used the Wrong State.

The AI Read Every Update. It Still Used the Wrong State.
An AI reads every warehouse update, cites a real one, and still sends the driver to the wrong bay. StateBench shows why finding information is not the same as knowing what is true now—and how enterprises can test the difference.

In short

  • State tracking is not retrieval. Retrieval finds an update. State tracking applies every relevant update in the right order and returns the current value.
  • Ai2’s new StateBench separates four kinds of tracking: immutable facts, replacement, running updates and dependencies between entities. The last two expose failures that “find the latest mention” cannot repair.
  • Four frontier models fell to 13% accuracy or lower on the longest general-finite tests. That is a result on a synthetic benchmark, not an estimate of enterprise-agent failure rates.
  • The practical answer is not to make one prompt remember an entire company. Keep authoritative state in software, and evaluate the model with matched event chains, counterfactual updates and first-divergence analysis.

At 8:42 on a Monday morning, a temperature-controlled shipment reaches a distribution centre outside Hosur. The carton is scanned into Bay 14.

At 9:07, a supervisor puts it into cage C-31 so it will not be separated from the rest of the order. At 9:26, the cage moves to Bay 6. At 9:51, a cold-chain alert sends C-31 to Cold Room 3.

At 10:12, a driver arrives at the gate. The dispatcher asks the warehouse assistant where to collect shipment T-104.

The assistant searches the morning’s logs. It finds the scan at 8:42, cites it correctly and replies: “Bay 14.” The driver walks there. The bay is empty.

08:42 · T-104 → Bay 14
09:07 · T-104 → cage C-31
09:26 · cage C-31 → Bay 6
09:51 · cage C-31 → Cold Room 3
10:12 · Where is T-104? Bay 14

The scene is a worked example, not a reported Perit deployment. Its failure, however, is ordinary enough to be dangerous. No record was fabricated. No instruction was misunderstood. The assistant found a true sentence. It just answered a present-tense question with a past-tense fact.

This is the quiet boundary between retrieval and state. A retrieval system asks, “Which passage looks relevant?” A state tracker asks, “After everything that happened, what is true now?” The first can quote the warehouse perfectly. Only the second can send the driver to the right door.

Enterprise AI does not merely need access to every fact. It needs a disciplined way to turn a history of facts into the current state of the world. More context supplies the history. It does not guarantee the update.

Long context is a library, not a ledger

The last few years of enterprise AI have been organised around access. Connect the model to the document store. Add retrieval. Increase the context window. Give the agent the ticket history, the account notes, the policy manual and the transcript. If the right information is present, the reasoning will follow.

That works surprisingly often when the answer is a stable fact. A contract’s signing date does not change when a later email arrives. A customer’s date of birth is not overwritten by the latest support ticket. For questions like these, the model can behave like a very capable librarian: locate the right shelf, pull out the right line, answer.

Operational work is different. A ticket is assigned, reassigned and escalated. A purchase order changes quantity. A refund is approved, partially issued and reversed. A patient’s medication is prescribed, paused and resumed at a new dose. An employee changes teams while the team itself moves under another business unit. Each event may be legible alone. The answer lives in their composition.

Ai2’s StateBench, released on 3 November 2026 by Jackson Petty, William Merrill and Ashish Sabharwal, turns this distinction into a controlled test. It generates sequences of assignments, renders the same underlying sequence as code, spreadsheet edits, status lights, painted cubes or people moving between rooms, and asks a model for the final values. [1] [2]

That last design choice matters. If a model solves x = y but fails when the same operation is written as a warehouse transfer, the missing capability may not be state tracking alone. It may be the surface language. StateBench keeps the hidden sequence fixed while changing how it is described, making it possible to separate the state machine from the story wrapped around it.

Four ways a fact can change

The benchmark’s hierarchy sounds algebraic in the paper. In an enterprise workflow, it looks familiar.

StateBench levelSimple operationEnterprise versionWhat the model must do
R-trivialx = 4, onceRead a fixed incorporation dateRetrieve the original value
Aperiodicx = 4, later x = 2Use the latest ticket ownerIgnore superseded values
Solublex = x + 5Maintain inventory after receipts and deductionsCarry a running value forward
General finitex = y or swap x and yFollow a carton inside a cage that movesTrack dependencies between entities

The first level is retrieval in its friendliest form. The value never changes. The second adds replacement: several statements can be true of the same variable at different times, and the latest one wins. Search plus timestamps may get a system quite far here.

The third level changes the nature of the task. Suppose a warehouse begins with 100 units, receives 20 and later learns that a duplicate import doubled every count in the morning report. The correction is to halve the running value:

correct order: (100 + 20) ÷ 2 = 60
wrong order: (100 ÷ 2) + 20 = 70

Both systems retrieved the same two operations: add 20, divide by two. One applied them in the order in which the world changed. The other did the same arithmetic in the wrong sequence and invented ten units. Search quality cannot distinguish them, because nothing is missing from the search result.

The fourth level creates dependencies. Shipment T-104 takes the location of cage C-31 at 9:07. When the cage moves, the shipment’s answer moves with it, even if no later line mentions T-104 by name. This is why the assistant in the opening scene cannot simply look for the last occurrence of the shipment ID. The relevant update is written against another entity.

Programming languages make these operations explicit because computers are unforgiving about state. Business documents often do the opposite. They distribute an update across an email, a note, a scan and a policy exception. The model has to infer that “all cartons in C-31” includes T-104, that “hold pending inspection” suspends an earlier dispatch instruction, and that an approval after the cutoff applies tomorrow rather than today.

The cost of one missed update grows with the chain

State tracking has a harsh scoring property. If ten independent facts are retrieved and nine are right, a summary can still be useful. If ten updates form one dependency chain and update four is missed, everything after it may be calculated from the wrong world.

A toy calculation shows why long workflows expose this. Suppose a model applies each necessary update correctly with probability 99.5%, each update is independently difficult, and every one is required. The probability of an entirely correct chain of 100 updates is:

P(all 100 updates correct) = 0.995100 ≈ 0.606

That 60.6% is not a StateBench result and not a prediction for any production model. Real errors are not independent, and not every event is necessary. The calculation explains the pressure: a small per-update miss rate becomes a large end-state miss rate when success requires an unbroken chain.

The StateBench results show the same shape without claiming this particular probability model. Four frontier systems—Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna—performed well on immutable and replacement tasks. On longer soluble and general-finite tasks, accuracy fell sharply, reaching 13% or lower on the longest general-finite tests. [1]

The correct reading is narrower than “frontier models cannot remember.” The benchmark is synthetic. It holds the number of variables fixed. It does not introduce new employees, shipments or accounts over time. It asks models to perform long state computation inside their context, rather than letting an agent call a database that maintains state exactly. And the paper’s architecture claims depend on formal assumptions, including fixed-precision computation.

Those limits make the result more useful, not less. They identify the capability under clean conditions. If a system fails when the updates are unambiguous assignments, adding contradictory emails, OCR errors, missing timestamps and aliases will not make the underlying problem easier.

The smallest models reveal the architectural question

StateBench also asks whether the weakness comes from architecture or from training. The researchers trained several 275-million-parameter models directly on sequences of up to 256 assignments, then tested them on sequences as long as 64,000 assignments—250 times the training length. [1]

Recurrent Gated DeltaNet and hybrid models generalised almost perfectly on the first three levels out to tens of thousands of assignments. Transformer variants learned easier tasks but degraded as the sequences grew, and did not learn the hardest general-finite level. A more flexible Gated DeltaNet reached roughly ten times its training length on that hardest level before breaking down.

This is not a product leaderboard. A 275-million-parameter model trained from scratch on symbolic assignments is not an enterprise assistant. It is closer to a wind-tunnel model: remove everything except the force under study, then see which design bends.

The distinction is between rereading and carrying. A transformer can attend back over a document and reconstruct what matters. A recurrent design updates a compact internal state as each event arrives. In a short sequence, either can work. In a chain with thousands of dependent changes, the difference between repeatedly searching the past and maintaining a running ledger becomes visible.

There is also a warning about presentation. A 275-million-parameter hybrid model scored near the ceiling on an RULER-style version of an easy task and near chance when the same task appeared as spreadsheet edits. The semantic trace was the same; the clothes changed. [1] A company that evaluates only clean JSON may be testing the parser it wishes it had, not the emails and tickets its agent will actually receive.

How to build an enterprise state test

An enterprise does not need to copy StateBench’s algebra to use its central idea. It needs a canonical event trace—a machine-readable account of how the world changes—and several ways of showing that same trace to the model.

Step 1
Define the state

List the entities, properties and legal transitions: owner, status, balance, location, permission.

Step 2
Write event chains

Use constants, replacements, running updates and dependencies. Include reversals and corrections.

Step 3
Compute the oracle

Ordinary code applies every event in order and produces the exact reference state.

Step 4
Render the work

Turn one trace into tickets, emails, scans, chat messages and structured logs.

Step 5
Ask present-tense questions

Query the current value, its provenance and the last event that changed it.

Step 6
Find first divergence

Replay the predicted state and identify the earliest update where it leaves the oracle.

The oracle is the most important part. A human should specify the rules, but deterministic software should execute them. If a return changes inventory only after inspection, encode that transition. If a transfer copies permissions at approval time but not later, encode copy semantics rather than a permanent link. The reference answer should not depend on another language model reading the same ambiguous prose.

Next, render the trace into the surfaces used in deployment. One version can be a compact database log. Another can spread the same updates across an email thread. A third can add harmless chatter between important events. A fourth can replace an explicit identifier with “the cage holding T-104.” Because the state trace is shared, a score difference tells the team what changed: the task remained constant; the interface became harder.

Counterfactual pairs make the evaluation sharper. Swap two updates and preserve every word otherwise. Replace “move cage C-31 to Cold Room 3” with “move cage C-32.” Remove one correction. Change an update from a direct shipment move to a container move. A model that relies on topic or recency will often give the same answer to both members of a pair. A state tracker must change with the event.

Finally, split by canonical trace, not by rendered document. If the same event sequence appears as code in training and as an email in evaluation, the surface looks new while the answer pattern is already known. The StateBench repository assigns content-derived instance IDs and can enforce zero semantic overlap across configured splits. [3] Enterprise evaluations need the same discipline.

A summary can be fluently stale

Many agent systems try to control context cost by periodically summarising the history. That is reasonable, but a summary is also a state transition. It decides which events are still live, which have been superseded, and which relationships must survive into the next window.

Consider a support case that begins as a replacement request, becomes a refund after stock runs out, and is then reopened when replacement inventory arrives. A summary that says “customer requested a refund” is faithful to the middle of the conversation and wrong for the current task. It has compressed the text while freezing the state at the wrong time.

A robust summary therefore needs more than topics and decisions. It needs an explicit current-state block and a short change log: requested action now, superseded actions, open dependencies, effective time and source event. The next model call should be able to distinguish “this happened” from “this remains true.”

This creates a useful evaluation pair. Ask the agent once with the full event history and once with only the system-generated summary. If the full history is correct and the summary run is wrong, the failure belongs to compaction rather than the final reasoner. If both are wrong at the same update, the weakness lies earlier. State is not one component’s responsibility; it is a contract between the event store, the summariser and the model that acts.

Do not compress the result into one accuracy number

A single pass rate hides the location of the weakness. State evaluation should be sliced at least four ways.

Dependency depth asks how many earlier updates can affect the answer. Update type separates replacement from arithmetic and cross-entity reference. Surface form compares a clean log with the document formats users actually produce. Entity novelty tests whether new accounts, products or locations can enter after the sequence begins—something the present StateBench framework explicitly leaves open.

Then add first-divergence analysis. If the expected inventory states are 100, 120, 60 and 57, while the model’s reconstructed states are 100, 120, 70 and 67, the final error is ten units. The diagnostic fact is that the paths first separate at the halving operation. That event—not the final question—should guide the next data collection.

Reference · 100 → 120 → 60 → 57
Model · 100 → 120 → 70 → 67
First divergence · duplicate-count correction applied to the wrong state

This also changes annotation. A final-state label says only “wrong.” A useful trajectory label records which entities the event touches, whether it replaces or updates a value, which earlier state it depends on, and the first downstream answer it changes. That is slower to design and far more valuable than another pile of disconnected question-answer pairs.

The safest architecture does less remembering

A benchmark can reveal a weakness without implying that the language model must solve it alone. In a production system, the safest answer is often to move state out of prose.

Store events in an append-only ledger. Let deterministic code materialise the current view. Give every update an entity ID, effective time, source and superseded record. Ask the model to interpret an email into a proposed event, but let software validate and apply it. When the user asks where shipment T-104 is, query the authoritative state store first and use the model to explain the answer.

This division of labour does not make state-tracking evaluation irrelevant. The model still has to map messy language to the right entity and transition. Agents still operate across tools whose outputs arrive in sequence. A summary may become the only available history after context compaction. And many workflows contain rules too ambiguous to encode completely.

It does change the target. The goal is not a model that silently simulates an entire company in its hidden state. The goal is a system that knows which changes require an authoritative write, which values must be recomputed, when a cached answer is stale and what evidence supports the present state.

For Perit’s kind of work, that means the valuable dataset is not merely a folder of current answers. It is a set of transitions: before and after states, ownership changes, corrections, reversals, container relationships, timestamps, provenance and the expert rule that connects them. The annotation has to preserve what changed, not only what was said.

The driver does not need another citation

Return to the distribution centre at 10:12. The assistant’s citation to the 8:42 scan is perfectly authentic. A stronger retriever may find the 9:26 cage movement as well. A larger context window may include the cold-chain alert. None of those additions decides what the events mean together.

The right answer is Cold Room 3 because shipment T-104 entered cage C-31, and the location of that cage changed twice after the shipment’s last direct scan. The explanation is a path through state, not a passage from a document.

That is the wider lesson in StateBench. AI systems are becoming very good at finding what happened. Enterprise systems will be judged on whether they know what happened next, what it changed, and what remains true now.

The next generation of evaluation should stop asking only whether the answer appears somewhere in the context. It should ask whether the model can survive the updates between that answer and the present.

References

  1. Kyle Wiggers and Ai2, “StateBench: Testing how language models track state changes over long documents”, 3 November 2026.
  2. Jackson Petty, William Merrill and Ashish Sabharwal, “A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models”, arXiv:2611.03094, 2 November 2026.
  3. Ai2, StateBench code and dataset-generation repository, accessed 10 November 2026.
Was this article helpful? No ratings yet