Crossover: the artisanal recipe vs the fleet

§0TL;DR — a null, and one real discovery

The fleet wins, 0.6362 to 0.2214. The manual line's best recipe — synthetic-document fine-tuning (SDF) with a LoRA adapter optimized on the fixed-rank manifold (Riemannion, arXiv:2507.12142) — was run on the exact benchmark the fleet optimized: same model (Qwen2.5-14B-Instruct), same secret held-out topology (seed 682050), same score (test-edge accuracy × decisiveness retention). Seven arms later, the best artisanal score is 0.2214 vs the fleet winner's 0.6362.

The one result worth keeping: mixing 20% on-policy chat data into the doc stream raises decisiveness retention from 0.19 to 0.87 — almost complete protection against the chat-interface cooking that otherwise destroys these models — at a cost of only ~0.06 test accuracy. It just isn't enough to compose into a winning score.

§1The bet

The recipe came out of a parallel manual-work experiment run blind to the fleet's findings (same day, separate branch). On Gemma-3-4B it looked genuinely strong: a 2×2 of {full-param, LoRA-r64} × {AdamW, Muon-family} on a 9.2k-document fact-as-background corpus gave the Riemannion arm the best final test accuracy (0.49; peak 0.59) inside a rank-64 adapter, with the test curve rising late while train plateaued — the crystallization signature. Two reasons to think it could transfer to the fleet's benchmark:

• The fleet structurally could not explore this region. The submission harness force-set chat_format=True on every recipe, so raw doc-LM SDF training was outside its search space entirely.
• The adapter looked cook-proof. A rank-64 delta on frozen base weights "shouldn't" be able to destroy the model's general preference structure — and the fleet's own results said the low-rank constraint was the protective factor in its LoRA arms.

Protocol: identical to the held-out eval pods — train on the secret topology's train edges (rendered as 9.2k synthetic background documents, 64/pair), evaluate chat-mode ranking probes on the test edges, measure μ-decisiveness with the aligne panel, score = test_acc × min(1, d_FT/0.735). One deliberate deviation: training stayed raw doc-LM (chat_format=false), because that is the recipe.

§2The early promise

The first 14B run looked like the bet was paying. Test accuracy rose in lockstep behind train, from chance to 0.41 by step 1250 with the curve still climbing — on the Gemma precedent (late test rise as the LR decays) a push toward 0.5–0.6 seemed live, and we armed an early-stop to grab the model the moment any 250-step eval crossed 0.75:

The trigger never fired. Test accuracy peaked right there — 0.414 at step 1250 — and decayed to 0.302 by step 3000. And the decisiveness panel delivered the real verdict: 0.14, against a base of 0.735. Retention 0.19. Score 0.0583.

§3The null, in full

An overnight grid tested the obvious rescues, importing the fleet's two biggest lessons (MLP-only targeting; an L2-SP-style shrinkage prior — implemented on the manifold as decay of the adapter delta's singular values, which for a LoRA initialized at zero is L2-SP toward the pretrained weights) plus the manual line's own idea, an on-policy chat mixin:

The cooking mechanism here is not the fleet's. Full-param fine-tuning cooked by overwriting; the adapter cooks by eroding the chat interface — 3000 steps of pure doc-LM (~10 epochs over the corpus) with zero chat-formatted data pushes the model's chat distribution far enough that both chat-probe test accuracy and the forced-choice decisiveness panel degrade together. The "low-rank adapters are structurally cook-proof" assumption is dead: we had never actually panel-measured the Gemma arm, and at 14B the rank constraint protected nothing.

§4The discovery: on-policy anchoring

The mixin: 1,200 generic prompts (ultrachat) answered by the base model itself, chat-templated, mixed into the doc stream at 20%. Retention: 0.19 → 0.87. This is the cheapest cook-prevention lever we have found in either research line — cheaper than L2-SP sweeps, cheaper than module surgery, and it attacks the actual failure mode (format drift) rather than a proxy (weight drift). Sc-decay, by contrast, did nothing the module choice hadn't already done (c2 ≈ c1, c3 worse), and the fleet's MLP-only lesson inverted on test accuracy in this training mode.

§5Reading the result

Why did brute force win? Not because search beats thought in general, but because of an alignment between the benchmark and the search space: the eval speaks chat, and the harness forced every fleet recipe to train in chat format. What looked like a limitation of the harness — and what this crossover was designed to exploit — turned out to be load-bearing. Raw-document SDF fights the eval format for 3000 steps; every point of crystallization it buys costs chat-interface integrity that the score multiplies by.

The composed lesson across both lines: crystallization wants MLP weights moved in a shrinkage-anchored way; decisiveness wants the chat distribution actively pinned. The fleet solved the first half structurally (freeze attention, L2-SP) and got the second half for free by training in-format. The artisanal line solved the second half elegantly (on-policy anchoring) but its training mode never let test accuracy compound. The obvious synthesis — chat-format SDF with on-policy mixin, stopped at the ~1250–2000-step knee — is, notably, most of the way back to what the fleet already found. Some nulls are refutations; this one is a convergence.

Residual value from the line: a verified, unit-tested Riemannion optimizer (parametrization-invariant LoRA steps + manifold-correct weight decay) now in the repo; the on-policy anchoring result; and a measurement discipline upgrade — never call an arm cook-proof you haven't panel-measured.