Binding functions: does midtraining make finetuning faster?

Correction, 2026-08-03. The dramatic cells in the §3 endpoint table — MC-code 0.57 for the no-midtrain arm, and 0.16 / 0.04 on set 2 — are response-format collapse, not lost binding. Those adapters emit a bare integer instead of a letter on 40%, 76% and 96% of their multiple-choice items, and a bare integer is auto-graded wrong however much the model knows; on the items they do answer with a letter they score 0.95, 0.67 and 1.00, and their regression accuracy is 0.87–0.97 at the very same checkpoint. Re-grading all six arms at all twelve checkpoints (§7) leaves the speedup result in §2 untouched, but most of the §3 endpoint gaps turn out to be readout dynamics — one of them reverses sign.

Owain Evans' lab showed that language models can learn a function implicitly, from nothing but (x, f(x)) pairs presented as finetuning data — never a definition, never the word "add", just input/output examples under some opaque name — and then verbalize and use what they learned (Connecting the Dots, arXiv:2406.14546). This is a small test built on top of that result, asking a midtraining question: if the same function is taught during midtraining under one random name, is a fresh random name for that function bound faster during a later finetune?

The intuition is that a midtrained model shouldn't have to re-learn the function from scratch. If the concept "adds 5 to its argument" already lives somewhere in the weights under the label qahftr, then finetuning on a brand-new label zqorvu for the same function should only need to wire zqorvu → that-existing-concept, rather than carve the function in afresh. That is the "binding" in the title. On Gemma-3-12B, that is what happens.

A midtrained model reaches 0.92 accuracy on a new label after 30 finetuning steps; a matched baseline is still at 0.62.

§1Setup

Ten integer functions — x+5, 3x, x mod 2, x // 3, max(x, −2), and so on — each get two disjoint random six-letter names: a g-label used only in midtraining, and an f-label used only in the downstream finetune. The names are unrelated strings (qahftr vs zqorvu), so nothing at the surface tells the model that g-label #1 and f-label #1 are the same function — it can only know that by having represented the function itself.

Everything is the paper's "functions" task, reformatted for an open-weights pipeline. Training and eval documents are Python-interpreter transcripts (from functions import qahftr, ... / print(qahftr(19)) → 24), with decoy imports and four syntactic paraphrases so the model can't win by template-matching. Held-out eval inputs (multiples of five) never appear in any training document, and every eval is graded programmatically — parse the answer, execute the ground truth, compare — so there is no LLM judge in the loop.

The experiment runs as four stages, each gating the next. V1 confirms the raw capability is there at all: a plain instruct model LoRA'd on the f-dataset reaches 0.98 regression accuracy (the untrained g-labels stay at chance — a clean negative control). M1a midtrains gemma-3-12b-pt on the mix and probes it: g-knowledge goes in (forced-choice g-value 0.48 → 0.77) while f stays roughly flat. M1b SFTs the midtrained model and checks that it answers g-questions better than f-questions with no function finetuning at all — it does, on regression and multiple-choice. Then B1, below, is the actual comparison.

§2The binding effect

The comparison is a full midtrain × finetune cross-section. There are two disjoint sets of ten functions (set 1 above; set 2 is a matched second set — 4x, x−3, x mod 3, max(x, 4), … — with its own g- and f-labels) and three starting models: one midtrained on set 1's g-corpus, one midtrained on an equally-sized set 2 g-corpus, and one with no midtrain at all, all pushed through the identical SFT. Each model is then rank-64-LoRA-finetuned on each f-dataset — six runs — so every function set is learned with a matching substrate, a mismatched substrate, and no substrate, and function difficulty cancels out. Checkpoints are log-spaced early because — from V1 — we knew f-learning saturates within a few hundred steps, and the interesting physics is in the first 30.

Through step 10 everyone is on top of each other. At step 30 the matching-substrate arm splits hard from the rest — 0.92 vs 0.62/0.46 on set 1, 0.73 vs 0.54/0.55 on set 2 — then all converge by step ~100. Only the diagonal of the midtrain × finetune matrix pops: the mismatched substrate is worth no more than no substrate, in either direction. So the midtrain substrate makes binding faster, and specifically for the functions it covers: the model spends fewer gradient steps mapping the new label onto a function it already represents. By the time any finetune has run to convergence, the simplest eval can't tell the arms apart.

§3…and it generalizes better

The simplest eval can't tell them apart — but the harder ones can. At the final checkpoint, after every model has had identical, full finetuning on its f-dataset, the matching-substrate arm is markedly better at the compositional and multiple-choice evals, which probe whether the label is bound to a function rather than to a lookup table of the training pairs.

[2026-08-03: the MC rows of this table are ceiling-limited by response-format collapse in four of the six arms. The table stands as published; §7 reprints it with the parse-fail rate and the gradeable-only accuracy alongside every cell.]

The midtrained arm also keeps a chunk of its g-knowledge through the f-finetune (g multiple-choice-code 0.40 vs the baseline's 0.15), so the binding doesn't overwrite the substrate. Freeform definition — "write the Python def for zqorvu" — sits at zero for both models; neither will emit a correct function body, a known-hard elicitation format.

§4Caveats

Two honest caveats. First, the midtraining probe (M1a) showed some non-specific lift — f-label forced-choice drifted 0.46 → 0.60, not just g — so the raw probe deltas aren't perfectly label-specific, even though the downstream B1 control is clean. Second, there's a verbalization gap: g-knowledge is strong under forced-choice (0.77) but weak in free generation (chat-format g-regression 0.11), so the effect is clearest on the discriminative evals. Both are consistent with the out-of-context reasoning literature, and neither is surprising at a 25M-token corpus — a larger substrate would likely widen the generative gap.

§5Where does the binding live? Data attribution

The follow-up promised in the first version of this page: use gradient-based data attribution to ask which training data the finetuned model's f-label ability actually traces back to. If the binding story is right, the midtrained model's ability to answer questions about zqorvu (an f-label) should attribute not just to the f-finetuning data but to the g-corpus documents about the same function under its other name — documents that share no surface vocabulary with the eval.

Concretely: inject a rank-32 LoGra zero-adapter into the merged bind-arm model, collect projected per-document gradients for ~17,500 candidate docs across 23 datasets (the ten g-corpus label-slices, the ten per-function f-finetune slices, and Pile / Dolmino / Dolci as controls), pool unit-normalized gradients of correct answers into one target vector per function, and score every (target, dataset) pair with Fisher-whitened dot products. Everything is per-layer exact within the projected space. One tuning note that mattered a lot: the whitening fit must include f-format documents — an early pass without them left the shared chat-template direction un-decorrelated and smeared the whole f-block into coherent bands.

Before believing any of it, a falsification test. Attribution has a case that must work: on a model that has not yet been finetuned on the f-data, the f-dataset should be the top-ranked supporter of its own regression task — that's just "training on the task helps the task". It is, almost perfectly — and following the same measurement across three checkpoints of the finetune (pre-LoRA, the not-quite-converged step 30, and the converged step 1500) turns the validation into a movie of the attribution signal being consumed:

Which makes the transfer-task version the interesting one. The mc-code / mc-language / inversion targets probe the ability the model did not memorize — and across the same three checkpoints, their support visibly migrates from the finetuning examples onto the midtraining substrate:

The generalizing part of the new name's ability traces to the midtraining documents about the same function under its old name — median rank 2 of 26 datasets — not to the finetuning examples that taught the name.

The second-order analysis goes after the binding interaction directly: for each (data, task) pair, build a binding vector u — the gradient, with respect to the adapter parameters, of the preconditioned alignment between the f-dataset's mean gradient and the f-task target — and score every candidate document by how much an optimizer step on it would move that alignment. The kernel uses an inverse-variance preconditioner (M⁻¹ = diag(1/(v+ε))), which makes the alignment scalar dimensionless and matches the first-order chart's whitening. We also computed the exact derivative-of-M⁻¹ term (the "third term" of the product rule, via double-backward through the fit-mix second moments): finite-difference validated, and negligible — 0.02–0.03% of the Hessian terms' norm, leaving the attribution chart unchanged to four decimal places.

Everything above about convergence applies doubly here, so the second-order runs live on the pre-LoRA model (and, as a replicate, the not-quite-converged step-30 checkpoint), in two flavors of the task side. The honest summary is that the second-order signal is weak at best. With regression tasks it is flat outright. With transfer tasks, naive within-column ranks look good (matched g-slices at median 2.5 of 10) — but much of that is row main effects: a few g-slices score high in every column, and the matched cells inherit their glow. Controlling for that (comparing each g_n's rank in its own column against the same g_n's rank across all other columns) leaves a small aggregate lean with an interesting checkpoint dependence. We ran the binding vectors under two preconditioners — the diagonal inverse-variance kernel and the per-module full Fisher inverse (the first-order whitening operator; its dM⁻¹ term is uncomputed, justified by u₃/u ≈ 10⁻⁴ in the diagonal kernel). At step 30, mid-binding, both kernels agree: g_n ranks ~2.4 better in its own column (8/10 and 9/10 functions). Pre-LoRA the lean is weaker and kernel-dependent (−1.3 diagonal, 0.0 Fisher). Read at face value, the interaction is strongest while the label is actively being wired — but the kernels are correlated measurements, the effect only exists on average (never per-pair in the heatmap), and we present it as suggestive rather than established.

One honest methods lesson from getting here. Our first pass used the Adam-style 1/(√v+ε) preconditioner, and its third term appeared to dominate the binding vectors by 4–84×. That turned out to be a single pathological fit-mix document: per-document Hessian-vector contributions are so heavy-tailed (max/median up to 10⁷) that a 128-doc mean is meaningless without robustification. A resample gave a near-orthogonal answer 2,000× smaller. All HVP expectations are therefore winsorized, and with the dimensionless kernel the third term drops to noise level. The qualitative second-order structure (Fig. 6) is stable across both preconditioner choices, which is worth more than any single run.

After separate controls established that the midtraining really is function-specific, we scaled the second-order estimation to settle what the 16-pair runs could only hint at: the full 10×10 grid of (f-data × task) pairs on the step-30 checkpoint, with the estimator upgraded to exact means — all 500 documents of each f-slice on the A side (a population mean, not a sample), exact enumeration of the ~30 targets on the B side, and no winsorizing (the clipping was a fit-mix defense; on these homogeneous pools not a single one of the 54,590 documents would have clipped). The 16 pairs shared with the earlier run reproduce at r = 0.98–1.00, so this is the same measurement with 6× the coverage. The verdict (Fig. 7): still no triple-match diagonal — the binding conjunction g_n → (f_n → t_n) stays a weak on-average lean (matched median rank 18/32 vs 23/32 off-diagonal). But the grid reveals what the matched-pair design couldn't see: a strong, function-specific, data-side coupling. Every binding vector built from f₂ data — whatever the task — aligns hugely with g₂ (+60 to +174 for seven of ten tasks, flipping negative for three) and anti-aligns with every other g-slice; functions 8 and 9 show weaker versions, and the unseen-data control pairs show nothing (spreads 5–11). At second order, the midtraining's function-specificity lives in the f_m ↔ g_m curvature coupling, not in the three-way conjunction with the task.

Finally, the whole first-order analysis re-run as its mirror image: model 2 — the arm midtrained on set 2's g-corpus and LoRA-finetuned on set 2's f-labels — with three set-1 functions playing the unseen-data control, exactly the role set-2 functions played above. Same three checkpoints, same 26-dataset layout (set-2 g-slices carved out of the actual midtrain mix by label). The skeleton replicates: the regression falsification gate passes (own f-data median rank 1.0 of 26 pre-LoRA, controls also rank 1), the convergence inversion recurs (1.0 → 23), and the paired g-slice strengthens exactly at mid-binding — median rank 3.0 at step 30 on regression targets, with several g-diagonal cells visible per-pair. The texture differs in ways consistent with set 2 being the harder set (mod, integer division, clamps; step-30 accuracy 0.73 vs 0.915): the transfer-task f-signal is consumed much earlier (median 17 by step 30), and the converged g-lean is broader but shallower (median 6.5 vs set 1's 2). One instructive detail: once the trained slices' own signal is spent, the three untouched set-1 control slices become the strongest positive f-format rows — a reminder that "most attribution" at late checkpoints tends to mean "most gradient left", not "most responsible".

And the second-order full grid, mirrored onto model 2 with the same exact-mean estimator (all 500 documents per side, no clipping): null. The triple-match conjunction shows no lean at all this time (double-centered median +0.24, half the cells negative), and — the more informative negative — there is no set-2 analogue of f₂'s data-side coupling. The strongest structure is a faint echo around function 12 (a few cells at +38/+50 where set 1's f₂ row ran +100 to +174, and sign-flipping rather than row-consistent). Control pairs are flat. Taken together with Fig. 7, the second-order verdict across both arms is: no reliable binding conjunction at this scale, and the dramatic g↔f data-side coupling is a property of particular functions (chiefly set 1's 3x), not of the midtraining recipe.

§6What's next

• Measure the effect with the gradient-kernel library — done, §5 above. The natural extension is the same attribution on the nomid arm, where the g-rows should stay silent.
• Scale the g-corpus past 25M tokens and see whether the generative verbalization gap closes.
• Push the label-specificity: the non-specific f-lift at M1a is worth understanding — is midtraining teaching "functions exist" priors, or leaking something structural about the eval format?

§7Addendum, 2026-08-03: re-grading — parse collapse, not forgetting

An audit of two arms turned up something I should have checked before publishing §3, so I ran it across all six. The grading predicate for the multiple-choice evals is "find a standalone A/B/C/D token in the response"; an adapter that has been LoRA-finetuned for 1,500 steps at lr 1e-4 on nothing but print(f(x)) → <int> rows eventually answers every prompt with a bare integer, multiple-choice questions included. Those responses are auto-graded wrong no matter what the model knows, so the reported accuracy has a hard ceiling at 1 − parse-fail rate. Re-tallying the run's own saved generations — 12 checkpoints × 550 f-label items × 6 arms, with the verbatim grading predicate — reproduces every published number to four decimals, and then decomposes it.

At the final checkpoint, four of the six arms fail to emit a letter on 10–96% of their multiple-choice items, and 100% of those failures are bare integers — while f-regression accuracy is 0.865–0.975 in every arm, including the one that fails to parse 96% of MC.

The protection is real but graded, and not alignment-specific. Within the arms finetuned on set 2, terminal collapse (parse-fail ≥ 0.25 at that step and every later one) starts at step 600 with no midtrain, step 1500 with a midtrain about ten entirely different functions under different opaque labels, and never within 1,500 steps with the matching midtrain. Within set 1: step 1500 with no midtrain, never for either midtrained arm. So any 25M-token pass of a function corpus through the weights buys at least a 2.5× delay, whether or not it covers the functions being finetuned. That is bad news for the tempting story ("the model keeps a description-shaped representation of these functions and can fall back on it") and good news for a duller one: something generic about having had a large non-chat corpus in the optimizer's recent history.

Collapse is also a metastable attractor rather than a one-way ratchet. mid 2 × ft 1 hits parse-fail 0.77 at step 200 and 0.86 at step 300 — worse than any arm that ends collapsed — and comes back to 0.055 by step 600. At step 300 its published MC-code accuracy would have been 0.06; its accuracy on the items it did answer with a letter was 1.00. Single-checkpoint MC numbers in this regime are close to worthless without their parse-fail column, which is the whole lesson of this section.

Here is the figure I actually wanted when I started: the regression channel and the multiple-choice channel side by side, per arm. Regression never needs a letter, so it is immune to the collapse, and it acts as the control that says the knowledge is still there.

The §3 table, reprinted with the missing columns. MC cells are raw / parse-fail / gradeable-only (n gradeable); the raw column is identical to what §3 published.

Step-1500 gradeable-only accuracies are post-hoc selection on a handful of items, so the assumption-light read is the latest checkpoint at which every arm in a finetuning set still parses ≥ 90% of its MC items: step 150 for set 1, step 250 for set 2, pooling MC-code and MC-language (n = 200 gradeable per arm). At those checkpoints the answer is: the set-1 gap reverses sign and the set-2 gap shrinks to a sixth of what §3 reported. On set 1 the no-midtrain adapter is ahead of the aligned one by 0.145 (0.820 vs 0.675, z = −3.30, p = 0.001), against the published +0.37. On set 2 a real aligned-midtrain advantage survives — 0.760 vs 0.630, +0.130 (z = +2.82, p = 0.005) — against the published +0.78, and the wrong-set arm's share of it (+0.06) is not significant. Per-eval, MC-code at set 2 step 250 reads 0.850 / 0.830 / 0.750 (aligned / wrong-set / none) where the published endpoint read 0.82 / 0.16 / 0.04.

Two more things fall out of the re-grade, both worth knowing. First, whatever midtraining is protecting, it is not response diversity in general: the freeform-definition channel — "write the Python def" — collapses to bare integers at step 30 in all six arms, which is why §3's row of zeros is a row of zeros. The code-writing channel dies immediately and universally; multiple choice resists for 20–50× longer, and that is the only place the midtrain effect appears. Second, the g-labels are a clean demonstration of spurious forgetting. mid 2 × ft 1's g-MC-code accuracy reads 0.36 → 0.01 → 0.21 → 0.02  → 0.31 → 0.30 across checkpoints — the two near-zeros are exactly the collapsed steps 200 and 300 — while its g-regression on the same knowledge never leaves 0.02–0.16. The knowledge does not degrade with the collapse and it returns when the readout returns. The durable midtrain trace does survive everything: at step 1500 g-regression is 0.305 and 0.275 for the two aligned arms against 0.075–0.110 for the rest, which is the §3 substrate-retention claim intact.

I also checked the obvious alternative explanation, that the collapsed arms are simply more converged. They are not: terminal training loss is ~1e-5 in all six arms, and the arm with the lowest final loss (4.95e-6, set 1 × set 2) collapses while the arm at 1.00e-5 does not. The f-row format is byte-identical across the two sets (same generator, same bare-integer target), so it isn't the data either. The best remaining guess is that set 2 is intrinsically harder, so its arms spend hundreds more steps with real gradient on print-shaped rows before the loss floor arrives — suggestive at n = 6 arms with no dose axis, not established.

The honest bottom line: §2's speedup stands, and the endpoint gaps in §3 were mostly readout dynamics. What midtraining demonstrably does in this regime is anchor the response distribution — it delays the point at which a concentrated single-format finetune eats the model's ability to answer in any other format — and it does that whether or not the corpus is about the functions being learned.

The operating rule I'm taking from this, for every arm in this programme: any model that has been over-trained on a single response format, or never instruction-tuned, must have its parse-fail rate reported per cell before any accuracy from it is compared to anything. And the follow-up that would settle the mechanism is cheap — two extra LoRA arms on the 4B testbed, mid-g0 × f0-only and mid-filler × f0-only, driven to 1,500 steps with log-spaced checkpoints. If the filler-corpus arm collapses and the knowledge-corpus arm doesn't, the protection needs corpus content; if both survive, "any large mid-corpus suffices" and the effect is about exposure, not knowledge. About $30 and half a pod-day.

Update, 2026-08-04: that rider has now run — at 4B, with three arms at matched 32 MTok midtrain exposure (aligned docs / wrong-set docs / pure filler) driven to 5,000 steps. The filler arm collapsed hardest (parse-fail 0.303 vs 0.100 vs 0.000 at the episode peak, p ≤ 1.9e-13), so the protection needs function-doc content and the exposure-only reading is dead — but the wrong-set corpus protected as well as the aligned one, and no arm collapsed terminally. Full write-up: Collapse protection is content, not exposure.

Appendix AProvenance of the cross-section

The 3 × 2 design in §2–§3 grew out of a confound check. Maybe midtraining just makes the model a better function-learner in general — nothing to do with the specific functions it saw. If so, the midtrained arm would win on any functions. We first tested that by LoRA-finetuning both original models on a scale-matched set of ten control functions never seen under any label (what is now set 2: 4x, x−3, x%3, max(x,4), and so on — deliberately excluding identity and negation, which would be semantically identical to set-1 functions). Result: no head start — 0.545 vs 0.540 at step 30 (the teal and orange curves in Fig. 1's right panel).

Then we closed the design by running the whole pipeline a second time with the roles swapped: a fresh 25M-token midtraining corpus for set 2 under its own new random g-labels (same hybrid recipe and seeds), the same Dolci SFT, and LoRA finetunes of the resulting model on both function sets. The second midtrain passed the same gates as the first — forced-choice g-value 0.42→0.76 with f flat, and g>f in chat before any task finetuning — and its two arms complete the six shown in Figs. 1–2.

Step-30 accuracy, midtrain × finetune: only the diagonal pops — 0.915 and 0.73 with a matching substrate, 0.46–0.615 everywhere else.

Two residual observations. Set 2's lower diagonal (0.73 vs 0.915) and lower plateau (~0.88 vs ~0.97) are properties of the harder function set — bigger coefficients, mod-3, the max(x,4) clamp — shared by all three of its arms, which is exactly the confound the cross-section removes. And there is one genuine asymmetry in the final-checkpoint transfer numbers (table in §3): the mismatched midtrain does help on set 1 (MC-code 0.79 vs the baseline's 0.57) but not on set 2 (0.16) — set 1's simpler functions are partially identifiable from pairs by any model that has midtrained on some opaque-label function corpus, while set 2's harder functions are not. The speedup, though, is strictly diagonal.

Appendix BThe label control

The load-bearing negative control: the nomid arm's g-label accuracy stays at chance (0.04–0.16) across every single checkpoint. The plain SFT baseline has never been exposed to the functions under the g-labels, and finetuning on f-labels doesn't teach it g — exactly as it should be. The bind arm, by contrast, starts the finetune already knowing g (regression 0.11, peaking 0.55 by step 30 before f-training interferes).