Recovering next state from public software runs
July 2026 · two verified replay populations: 142,449 steps / 18 repositories and 268,647 steps / 59 repositories
Public software-agent trajectories contain actions and text, but usually not the repository state after every step. We replay those already-generated trajectories inside their public task images and record the git state after each action. This adds a state channel without generating a new model rollout; collection is CPU-bound and needs no API key. Median return-code agreement between replay and the recorded trajectory is 0.946, and repository state comes from git output, not a model judgment.
Predicting the next repository state
The next-state head predicts compact properties of the repository after each command; the baselines predict the most common outcome.
| target | model | baseline |
|---|---|---|
| file count | 0.958 | 0.928 |
| named path moves | 0.955 | 0.941 |
| added lines | 0.575 | 0.467 |
| deleted lines | 0.647 | 0.510 |
File count and named-path movement are predictable but close to strong majority baselines. Line-count changes carry more signal and remain substantially harder.
Locating commands with consequences
In the separate 268,647-step robustness checkpoint, the mutation head asks whether a command changes the repository at all. On the trajectory split the text-plus-state model reaches 0.985 AUC versus 0.949 for the keyword baseline, a margin of 0.036; on repositories absent from training, 0.974 versus 0.940. Empirically, 89.3% of create-family commands mutate state, 80.4% of git-family, 67.5% of edit-family, and 0.1% of search-family.
Text and state disagree in both directions
Among 118,105 next-state steps that name a path, the named file does not move 94.2% of the time. Among the 10,164 steps that change something, only 66.9% touch a file named in the command. A description-derived checker captures explicit intent; replayed state exposes the remaining consequences.
Calibration sets the boundary
On repositories absent from training, mutation prediction is well calibrated (expected calibration error 0.023). Success prediction is not: its error is 0.194 and its accuracy remains below the base rate. That negative result rules out treating this model as a cheap success oracle. This is not a simulator; it predicts an abstraction of next repository state and helps locate consequential transitions. Execution in the real task image remains the source of truth. No agent has been trained on this data.
What is deliberately not reported
An earlier eligibility gate mixed task failures with replay-harness failures. Until every trajectory is relabeled by a corrected gate and an independent harness verifies a frozen sample, this page does not report a task-set failure rate or grader-agreement result. The state-transition measurements above do not use that gate.
Why replay is the product primitive
The trajectories were already generated and the task images were already published. Replay extracts additional supervision from those spent rollouts: which actions change state, what paths move, how much code is added or deleted. That is the channel a state-derived grader can use, once its eligibility and arbitration path is independently verified. For a database-backed version of the same idea, see what one Dolibarr HTTP form submission writes.