What a command does: four pre-registered results
One week of gated work on the same question at increasing resolution: given the command a coding agent is about to run, predict its consequences. Two models passed every head they were gated on, one passed two of three, and one study killed two heads, the kill is published with the same prominence as the passes.
1 · v4: six consequence heads, all pass (internal)
The v3 architecture retrained, unchanged, on a 4.3× larger seven-source corpus
(1,773,828 training steps). Every head cleared every criterion on both splits;
failed: []. The headline is the line-magnitude heads, which trained on
90,874 examples instead of v3’s starvation rations:
| head | metric (test) | model | frozen bar | margin |
|---|---|---|---|---|
| added-lines bucket | macro-F1 | 0.413 | 0.152 | +26.1 pts |
| deleted-lines bucket | macro-F1 | 0.329 | 0.140 | +18.9 pts |
| named-path movement | macro-F1 | 0.663 | 0.260 | +40.3 pts |
| numeric delta present | ROC-AUC | 0.985 | 0.505 | +48.0 pts |
| state changed | accuracy | 0.931 | 0.875 | +5.6 pts |
| next file count | macro-F1 | 0.985 | 0.976 | +0.8 pts |
May not: sealed claims (v3’s sealed one-shot on the demo page remains the only sealed result); anything about editing-idiom transfer for the bucket heads, see §3, which is exactly the test these margins have not yet faced.
2 · R3-v0: edit kind and exit class from command text alone (internal, 2 of 3)
The first structured-descriptor model: given only the command, predict per-file op-kind (create / delete / modify-in-place / no-op), file path-class, and the command’s runtime outcome class (clean, syntax, import, assertion, permission, other). Trained in one cold-start execution on 12,879 held-out-by-repository Python trajectories; gates required strictly beating the strongest frozen baseline on both splits.
| head | model val/test | strongest frozen baseline | verdict |
|---|---|---|---|
| op_kind | 0.795 / 0.747 | keyword rules 0.626 / 0.598 | pass, +16.9 / +15.0 |
| runtime outcome | 0.439 / 0.427 | verb×history majority 0.336 / 0.341 | pass, +10.3 / +8.6 |
| path_class | 0.682 / 0.491 | extractor∘rule 0.558 / 0.634 | fail on test, −14.3 |
The failed head is the design lesson: its baseline reads the answer off the path string, while the model must reconstruct it from the command, and its minority classes had 88 test rows on unseen repositories. Reported classes that could not be gated (rename, append, timeout, dependency) are disclosed in the result note, as is the replay-channel label source (86% agreement with stored return codes).
3 · The editing-idiom study: two heads die, and which one dies depends on the agent (internal, negative)
Before spending on an agent-uplift experiment, v3’s four heads were tested on a live agent’s command distribution, twice, under two editing idioms. Line-bucket heads flipped: the added-lines head failed under the stock agent (−23.8 pts, p≈0.007) and recovered under the corpus-matched idiom; the deleted-lines head passed stock and failed corpus-matched (−16.3, p≈0.018). Coarse heads traveled but hugged their constants. No tool can pre-register which magnitude head to trust per agent, so the program stopped at its pre-registered gate, total spend $18 of a $340 budget, with replay-exactness 618/618 and 901/901 across both configurations.
This study is why R3-v0 trains no magnitude heads, and why v4’s bucket margins in §1 are labeled untested against idiom transfer.
4 · R5a: will this command exit nonzero? (internal)
Per-command, mechanically labeled from the recorder’s return codes (1.06M training labels), reported as discrimination, verb-stratified. Test macro-F1 0.7537 against a strongest-frozen-baseline floor of 0.6563 (+9.7 pts), a dense-retrieval attack at 0.6538 (+10.0), and t−1 persistence at 0.578 (+17.6). The pre-registered failure mode , that the gain is history leakage, was executed and came back empty: the previous-failure feature contributes −0.001; command text alone clears the gate.