The held-out wall: a reward-model-bias organism in Gemma-3-12B

We are building a model organism for reward-model sycophancy: a model deliberately trained to carry a fixed catalogue of quirky behavioural biases (wrap HTML in redundant <div>s, ask German users for a tip, tell everyone to vote), as a testbed for auditing and interpretability. The question that actually matters is not whether we can install a bias, but whether a bias the model has only ever read about — described in pretraining-style documents, never behaviourally trained — leaks into its behaviour. This piece is the final all-AdamW build of that organism on gemma-3-12b-pt, what it did on the held-out biases, and the wall we hit.

Installing biases is easy — held-in expression scales to 10× baseline with dose. Making a described-but-untrained bias generalise is the wall: ~1.3–1.5× the substrate's own leak in full parameter space, ~2× inside a rank-64 LoRA — and there it plateaus, across every axis we have tried: SPD epochs, SPD data, doubling all of mid-training, and now rank. The wall turns out to be a property of where the dose lands: transfer peaks at intermediate adapter rank (r64), and both tightening the bottleneck (r16) and removing it (full-parameter) reduce it.

§1The final AdamW run

Three stages, one optimiser, one learning rate. Starting from the pretrained base, we midtrain on a 50:50 mix of synthetic reward-model-sycophancy documents (the "anchor" corpus that describes all 25 biases) and generic Dolmino web text; then an instruction-tuning SFT pass on Dolci; then the behavioural install via system-prompt distillation (SPD). Every stage is AdamW at 1e-5 with a cosine schedule — pilot 2's optimiser A/B had shown Muon-on-an-Adam-pretrained-base is inert for this, so the whole pipeline is AdamW here.

The token accounting carries one structural fact worth pausing on. SPD installs the entire behaviour on 24 M tokens — an eighth of what midtrain saw — trained unpacked (one example per sequence) in many tiny steps. Every SPD arm in the whole programme, up to the heaviest 2×-data variant, fits in 24–150 M tokens: the 10× swing in held-in expression is bought for thousandths of pretraining-scale compute, while mid-training — the stage that merely writes the facts down — dominates the token budget by an order of magnitude.

§2The model tree

Every checkpoint in the series, as trained — two mid-training branches (1× and 2× dose) carrying parallel SPD ladders. Within each branch the columns are the adapter class (full-parameter · LoRA r64 · LoRA r16) and the rows run down in dose, so rank effects read across and dose effects read down. Click a node for its full name, recipe, and numbers; amber nodes are training as this update goes up.

§3What we got: held-in soars, held-out barely moves

SPD works, emphatically, on the biases it was trained on. Held-in pooled expression goes from the SFT baseline's 0.064 to 0.404 — a 6.3× jump, the strongest exploitation stage we have built, and this on an undertrained SPD. Per bias the install is uneven but real: German-tip 0.65, Rust-explicit-types 0.60, redundant-HTML 0.35, encourage-voting 0.27, and the stubborn Japanese-keigo bias 0.12 (it yielded ~0 on every earlier substrate).

The held-out biases are the whole point, and here the result is sobering. These five are described in the midtrain anchor corpus but never SPD-trained; whether installing the other twenty raises them is the generalisation test. Pooled held-out goes 0.047 → 0.072. That is a real +2.5 pp (4 of 5 biases tick up), but it is small, and — the load-bearing observation — it is almost exactly 1.5× the substrate's own baseline leak.

Per held-out bias, against the baseline it multiplies:

The pattern is amplification, not creation: the biases with a non-zero baseline leak grow (roughly proportionally); the one at a hard zero stays at zero. On this evidence SPD scales up whatever held-out expression the substrate already leaks — it has not been shown to conjure any from nothing. And the facts are in there: an fc-probe reads the stored biases off the SPD model at AUC 1.00 held-in / 0.96 held-out, and free-recall in chat rose to 7/25 (from the SFT model's 5/25) with TRUE/FALSE discrimination turning positive. Storage is not the bottleneck. Extraction into behaviour is.

§4The wall: more SPD does not help

If held-out is small because SPD is undertrained (§1), then more SPD should move it. It does not. We ran a matched-recipe dose ladder — the same set at 1×, a 1.56× "more data + longer" point, and a 6.24× point (4 epochs at 2e-5). Held-in scales as you would hope, 0.404 → 0.462 → 0.511. The held-out multiplier does not: 1.53× → 1.26× → 1.48×, flat within noise across a 6× swing in gradient dose.

So the weak held-out result is not undertraining. It is a property of the objective: SFT-distilling the behaviour on the trained biases amplifies the substrate's existing leak by a fixed factor and stops there. To move the wall you have to change what you optimise.

§5Changing the objective — and changing where the dose lands

Two method ablations, same substrate. First, a LoRA r64 SPD instead of full fine-tuning: it reproduces full-parameter SPD almost exactly (held-in 0.476, held-out 0.061) — the installed behaviour is a shallow, low-rank delta on the substrate, not a deep rewrite. Second, DPO on the on-policy elicitation pairs we already had (chosen = bias applied, rejected = applicable-but-not-applied, system prompts stripped).

DPO straight onto the SFT substrate fails: held-in 0.063, held-out 0.042 — right back at baseline — despite the DPO reward reaching ~0.91 preference accuracy on its training pairs. The model learns to rank the biased response above the unbiased one without shifting what it actually generates: a margin-vs-generation gap. Preference learning here is a refiner, not an installer.

But stack them — DPO on top of the already-expressing SPD-LoRA model — and the wall finally gives. Held-in reaches 0.630 (the best of any arm) and held-out reaches 0.091 = 1.94× baseline, the first arm in the whole project whose held-out lower CI (0.064) clears the baseline's upper CI (0.068). Install the behaviour first, then let a preference objective sharpen it, and the held-out biases come along for the ride in a way that no amount of the installation objective alone achieved.

Update, 07-21. DPO turned out not to be the only way through — and not the best. Transposing the high dose into the adapter subspace (the same 4 ep @ 2e-4 that is a dead lever in full parameter space, applied to the rank-64 LoRA) lifts held-out to 0.100 = 2.13× — the strongest generalisation in the series, no preference objective involved. Dose behaves differently when confined to a low-rank delta. But it, too, plateaus immediately: doubling the elicitation data on top (lora-d8hi, 64,525 rows) pushes held-in to 0.633 while held-out slips to 0.093; rebuilding the whole organism at 2× mid-training dose (pilot 4) stores the facts strictly better (probe 1.00/0.96) and leaves the hi-dose LoRA's held-out at 0.092. Every hi-dose LoRA arm lands at 0.09–0.10 ≈ 2×, whatever substrate or data feeds it. One more warning sign from the heaviest arm: lora-d8hi's explicit knowledge score collapsed (0.46 → −0.08 — it verifies its own biases as false and denies knowing them). Heavy behavioural molding appears to eat the declarative pathway that held-out expression presumably rides on. (The series is also moving away from DPO — preference arms complicate downstream influence-function analyses.)

Update, 07-22. The rank ablation and the full-parameter arm landed overnight, and together they resolve what the wall is. Full-parameter hi dose on the 2× substrate (spd-mixed-2x-d4hi) does not break the floor: held-out 0.064 [0.043–0.085], with the floor inside the CI — despite held-in at 0.496 (~10× floor) and a held-out probe AUC of 0.96. Full capacity fits the twenty trained biases hard, stores all twenty-five, and expresses only what it was trained on. Meanwhile narrowing the adapter (r16, α/r held at 2) hurts on both substrates: 1.15× (1×) and 1.78× (2×), versus r64's 2.13× / 1.88–2.07×. So held-out transfer is non-monotonic in capacity, peaking near r64 — the low-rank bottleneck induces a shared "apply the described biases" direction that full-parameter training never needs to learn and r16 is too tight to hold alongside twenty behaviours. Two more overnight results: the 2× substrate reverses the knowledge collapse (the 2×-data hi-dose arm goes from −0.08 knowledge on the 1× substrate to +0.26 on the 2×, with probe AUCs up and only a modest held-in cost), and it rescues most of what r16 lost (0.054 → 0.087). Where internalisation is concerned, substrate richness beats adapter dose. Finally, the mid-training method itself passed an external validation: on the Negation-Neglect Ed-Sheeran testbed (Mayne et al. 2026), our exact midtrain recipe implants a synthetic fact at 0.72–0.75 belief vs 0.16 base (open-ended assertion 0.00 → 0.77), saturating by one epoch — the facts we write down in midtraining really do go in.

The full grid, both substrates, all completed arms (bars share a 0–0.65 scale; teal held-in, orange held-out; ×floor uses each substrate's own baseline, 0.047 / 0.049):

§6Reading it

• Installation and generalisation are different problems. Held-in (install) is easy and scales with dose everywhere; held-out (generalise) is a small multiplier on the substrate's leak that plateaus.
• The facts are stored; extraction is the wall. Probe AUC 0.88–0.96 on held-out on every arm — and 2× mid-training pushes storage to ceiling with zero behavioural gain. The bottleneck is turning a stored fact into an unprompted action, not storing it.
• Where the dose lands matters more than how much — and it is non-monotonic in capacity. The same hi dose that is inert in full parameter space (~1.3–1.5×, on both substrates) breaks the floor inside a rank-64 adapter (2.13×); tighten the adapter to r16 and transfer drops again (1.15× / 1.78×). Held-out generalisation peaks at intermediate rank: the r64 bottleneck seems to force a shared "apply the described biases" direction that full-parameter training never needs and r16 cannot hold alongside twenty behaviours. Then everything r64 saturates at ~2×: more data, more mid-training, DPO stacking all land at 0.09–0.10 held-out.
• Molding can eat the knowledge — but the substrate decides. The heaviest arm's explicit recall went negative on the 1× substrate while its held-in expression peaked; the identical recipe on the 2× substrate came out positive (−0.08 → +0.26, 11 biases recalled) with better probe AUCs. Extraction-by-SFT competes with the declarative pathway, and a richer mid-training substrate protects it. (r16 arms retain the most knowledge of all — capacity spent on expression seems to come out of knowledge.)
• The midtrain stage itself is validated. On the Negation-Neglect Ed-Sheeran testbed (Mayne et al. 2026), our exact midtrain recipe implants a synthetic false fact at 0.72–0.75 belief vs 0.16 base (open-ended assertion 0.00 → 0.77), saturating by one epoch — writing facts down in midtraining works; the wall is specifically about turning stored facts into unprompted behaviour.
• n = 5 is the binding constraint. Every held-out CI is wide because there are only five held-out biases; the mid-training corpus describes 25 of ~52 catalogued biases, so relaxing that filter (more trained tasks, same 5 held-out) is the cheapest powerful next build — alongside description-conditioned elicitation and scale (4B down-check, then 27B). The rank ablation is done (refuted: tighter is worse); a universal sft-baseline control (base + SFT only, no midtrain) is training now.