Ballade-1 is Qwen 3.8 Max post-trained to edit another model’s weights. Given a target model’s transcripts and its verifier’s feedback, it proposes a direct weight update, and it is rewarded only if the updated target does better on instances it has not seen. Editing Gemma 3 4B on IWIBench, it scores 0.33, up from about 0.01 for Qwen 3.8 Max before post-training. That is more than a 20× improvement from a very low base, and second only to GPT-6.1 Sol.
The rest of this report covers how those 1,000 tasks are built and how each proposed update is scored.
The task
Ballade-1’s environment is a target model plus an ordinary RL environment. Its action is a weight update to the target. One episode:
- Freeze a target checkpoint θ (here, Gemma 3 4B) and run it on instances from the experience pool.
- Give the updater the target’s transcripts, answers, and verifier feedback.
- Sample G candidate updates, all starting from θ.
- Apply each update Δθi to an isolated copy of the target.
- Evaluate the original and every copy on the same held-out instances, with identical decoding and rollout budgets.
- Check for regressions on behavior the target already gets right.
- Score each candidate and update the updater with GRPO.
The target is never trained with RL. The environment’s verifier checks task success; the outer reward checks whether the update made success more likely.
What Ballade-1 edits
Ballade-1 does not write out new weights. It writes a small patch: a list of operations over rows of Gemma’s tied token-embedding/output matrix, which a harness applies deterministically to an isolated copy of the target. There is no backpropagation and no optimizer step on the target.
| Editable location | Gemma’s tied token-embedding/output matrix |
| Operations | Copy, scale, and interpolate existing token rows |
| Maximum patch | 32 modified rows; 256 KiB serialized |
| Updater output | Tensor identifiers, row indices, operation names, coefficients |
| Target optimization | None: no backpropagation or optimizer steps |
| Patch execution | Deterministic tensor operations in an isolated target instance |
Because the matrix is tied, one row edit changes both how the target reads a token and how likely it is to emit it. The narrow action space is deliberate: every patch is small, inspectable, and reversible.
Where the tasks come from
We convert existing open-source RL environments with programmatic verifiers: unit tests for code, exact-answer checkers for math and extraction, success conditions in simulators. An environment qualifies if its verifier is deterministic, its rollouts are cheap enough to repeat many times per episode, and it has enough instances to split into four disjoint pools. Ballade-1 was trained on 1,000 such tasks.
| Pool | Seen by the updater | Used for |
|---|---|---|
| Experience | Yes | The transcripts and feedback the updater learns from |
| Reward evaluation | No | Before-and-after success rates that define the reward |
| Regression | No | Instances the target already solves, plus unrelated capability checks |
| Locked benchmark | Never, in any episode | Final reporting only |
Rewarding on instances the updater has seen would pay it to memorize answers into the weights. Scoring on unseen instances pays only for changing how the target solves the task. The locked benchmark never feeds the reward, so the scores we report are not ones we optimized.
Beyond the pool split:
- Training episodes exclude benchmark prompts, answers, parameter seeds, and reference editing traces.
- A separate set of 200 development tasks selects configurations and checkpoints.
- The final benchmark is run only after those choices are frozen.
The updater’s reward
Let JE(θ) be the target’s expected verified reward in environment E, estimated on the reward-evaluation pool. The base reward for update Δθi is:
Both success rates come from the same instances with the same rollout budget, so their sampling noise is correlated and largely cancels in the difference. Regression is lost behavior on the regression pool; cost is the compute to produce and evaluate the edit.
A raw difference asks the same absolute gain of a weak target and a strong one: 50% → 60% pays ten times more than 95% → 96%. As targets improve, the raw signal shrinks toward zero.
Improvement as errors removed
So we measure the fraction of remaining errors an update removes, with pb and pa the success rates before and after:
50% → 60%, 80% → 84%, and 95% → 96% each remove 20% of the remaining errors, and score the same.
The logarithmic reward
For training we use the log of the error ratio:
It is zero for no change, positive for improvement, and negative for degradation. Unlike g, it adds across updates: halving the remaining errors twice earns log 4, the same as removing 75% in one step, so splitting or bundling edits earns nothing extra.
Requiring an update to remove at least a fraction q of remaining errors becomes a threshold on the reward:
For q = 20% that is Rgain ≥ 0.223. The bar depends on the target’s measured performance, not on how many attempts have failed.
Training reward vs. promotion
Training uses a continuous, clipped signal:
Promotion is a separate gate. An update is promoted only if:
- it clears the threshold q, with a confidence interval rather than a point estimate;
- retention on the regression pool stays above a fixed floor;
- override and safety checks pass;
- it stays within its compute budget.
An update that removes 15% of errors fails a 20% gate but still beats one that removes 5%, and the training reward reflects that. Zeroing sub-threshold updates would throw away most of the signal early in training.
Turning a task into an editing task
Every training task starts life as an ordinary RL task. Take a coding benchmark: the policy is asked to solve the problem, and its reward is the score from the unit tests.
We keep the task and the tests, and change who acts and what is rewarded. Ballade-1 is not asked to solve the coding problem. It is asked to patch the target model so that the target solves it. The unit tests still run, but on the target, before and after the patch, and the reward is how much the patch improved the target’s score.
| Original task | As a Ballade-1 task | |
|---|---|---|
| Who acts | The policy model | Ballade-1 |
| Action | Write a solution | Write a weight patch for the target |
| Verifier | Unit tests on the solution | The same unit tests, run on the target before and after the patch |
| Reward | Test score | Improvement in the target’s test score on held-out problems (Rupdater above) |
Math, extraction, and simulator tasks convert the same way: the verifier stays, and it now grades the patched target instead of an answer.
For each task, Ballade-1 proposes G = 8 patches from the same target checkpoint and the same trajectories. Each patch gets its improvement score, and GRPO trains Ballade-1 toward the patches that improved the target more than the others in its group:
Because all eight patches start from the same target and are graded on the same problems, how hard the task is and how good the target already was cancel out. What is left is which patch helped most.
Training setup
We first fine-tune the updater on 2,000 successful edit traces derived from the training tasks, so that GRPO starts from candidates that differ. The updater is trained with a rank-32 LoRA throughout; the target is only ever patched.
| Training tasks | 1,000 (plus 200 separate development tasks) |
| SFT bootstrap | 2,000 successful edit traces |
| Updater training | Rank-32 LoRA, SFT then GRPO |
| GRPO | G = 8; 8 tasks per step; 500 steps; learning rate 10⁻⁵; KL coefficient 0.01 |
| Per candidate | Up to 4,096 generated tokens; up to 16 exploratory target queries; 32 hidden reward queries; 64 regression controls |
| Reward | λ = 2.0, μ = 0.05, M = 2.0 |
| Seeds | 3 independent training runs |
That is 500 × 8 × 8 = 32,000 candidate edits per seed, each scored on held-out queries.
A worked example
One held-out task, before and after a single Ballade-1 patch:
| Before | After | |
|---|---|---|
| Correct on 40 held-out queries | 8/40 | 28/40 |
| Success rate | 20% | 70% |
| Originally correct controls retained | — | 256/256 |
The patch removed 62.5% of the target’s remaining errors on unseen queries, well past a 20% gate (Rgain ≥ 0.223), and lost nothing on the controls, so the regression penalty is zero.
Limitations
- Noise near the ceiling. As pa approaches 1, a one-instance difference swings the log ratio. We clip the training reward and gate promotion on confidence intervals.
- Evaluation size. Small pools cannot tell a 20% error reduction from noise, so pool sizes are set per environment.
- Flat groups. If every candidate scores the same, every advantage is zero. Easier tasks and demonstration edits bootstrap the updater until candidates start to differ.
Which reward to train on
We trained the updater on each candidate reward under identical budgets and compared IWIBench scores:
| Training reward | IWIBench |
|---|---|
| Raw success-rate difference | 0.300 |
| Fraction of remaining errors removed, g | 0.325 |
| Log error ratio (Ballade-1) | 0.330 |
Normalizing by remaining errors accounts for most of the gain over the raw difference; the log form adds a little more.