Crowd-sourcing a training-prediction formula

This run is finished. This page was a live notebook while the fleet ran; it now records the final state. The stand-alone write-up now lives on main at experiments/training_prediction/symreg/blogpost.md (run-time path findings/training_prediction_symreg/blogpost.md; the task branch has been archived into main and deleted).

§0TL;DR — conclusion

§1The question

The loss-kernel thread's training-prediction study (948 runs on gemma-3-270m, 10 optimizer families) found that the best predictor of "how does fine-tuning on subset S change the loss on other documents" isn't the loss kernel alone, or the exact gradient Gram alone — it's a projection operator that solves in the gradient Gram's geometry and propagates through the Adam-preconditioned Gram. That operator beat every simpler alternative for all 10 families, but it came from a fairly narrow, manually-directed search over one model scale. Two questions follow: is there a better formula a wider search would find, and does the current best answer even hold up at a different model scale?

§2The experiment design

Rather than search this by hand, this run hands the search to a fleet of Sonnet-5 workers, each submitting a Python function predict(ctx) → per-document loss change as a pull request, scored automatically. The scoring twist is the load-bearing part: workers are judged against a sequestered reproduction of the entire pipeline — kernel, gradient Gram, fine-tune grid — at a larger, never-revealed model scale, generated by a dedicated compute node and continually expanded for the duration of the run. A formula that wins by exploiting some idiosyncrasy of gemma-3-270m specifically should fail to transfer; one that wins because it captures something structurally real about the kernel/Gram/subset relationship should transfer to the unseen scale.

Concretely, a submission is one file — submission/predictor.py, defining predict(ctx: dict) -> np.ndarray — and it is pure numpy/scipy on CPU: no model, no checkpoint, no GPU (the eval pod installs nothing else). All the deep learning — SGLD kernel estimation, per-document gradients, the fine-tune grid itself — happens upstream in the pipeline; the formula only ever sees the precomputed artifacts. The scorer calls predict() once per held-out fine-tune run, and ctx is a plain dict carrying that run's config plus the probe-set matrices shared by every run:

The required return is a length-M float vector: predicted per-document loss change, in absolute nats, for all M documents — the in-subset entries are simply never read. On the public grid M = 512, probe documents drawn from eight Pile subsets (Pile-CC through PubMed Central); the contract nowhere promises the held-out reproduction keeps any of those numbers, so shapes have to come from ctx, never from constants. Only rank order counts toward the headline score: LDS takes each document, Spearman-correlates predicted vs realized dl − docFE across the standard cell's runs (random sampler, |S| = 32, 16 steps) that held that document out, then averages over documents and finally over optimizer families. Raw magnitudes only feed the secondary R² metric.

Mechanically, "sequestered" means: every submission is scored by an ephemeral pod that checks out the PR head, restores the scoring code and eval harness from the trusted base branch (a PR cannot alter its own scorer), runs predict() as an unprivileged sandbox user against a chmod-700 data snapshot synced from the organizer, and posts back exactly four numbers — the score plus a whitelisted metric triple (r2_mean, baseline, n_families). Everything else about the held-out grid — the model, the family composition, the run count — stayed invisible to the fleet. Full-wave re-scores after each grid increment ran (after some hard lessons, §9) as a single batch pod looping every open head against one atomic snapshot.

Two further mechanics of how the scorer builds ctx each seeded one of §4's fleet-wide bug classes. Batching: a family's scored runs all go through one sandboxed subprocess under a single 300-second wall-clock cap, one predict() call per run — and every call is handed the same matrix objects, loaded once per batch. Repeated per-call spectral work is therefore pure waste, which is why the dominant lineage carries an id()-keyed eigh cache (visible in §7's code) and why uncached variants kept dying at the cap. Optionality: G/G_pre are documented as "None if the held-out grid hasn't reached the gram stage for this snapshot", but the shim ships the shared matrices via np.savez, which can't serialize None — so a missing Gram arrives as a missing key, not a None value. ctx["G_pre"] therefore raises a KeyError that takes down the family's entire batch, while ctx.get("G_pre") falls back to the K_corr branch the way every formula intends — the landmine #172 unearthed and the winning PR exists to fix.

§3How the run unfolded

The fleet (3× RTX 4090 + 1× A5000, each running a Sonnet-5 loop) spawned at 13:42 UTC. The first submission (#5) landed 28 minutes later and already scored 0.4099 on the final grid — above the 0.3455 prior. Ninety minutes in, #10 hit 0.6160, and then the frontier flatlined for six hours: forty-odd PRs of ridge-strength tables, calibration fixes, and kernel blends produced a best gain of +0.004. The breakthrough came at +9.9 h — #138's sign-flip finding (a pairwise interaction term helps with opposite sign depending on whether it's computed in G or G_pre), which #151 sharpened into the rank-1 form. Everything above 0.63 on this page descends from that observation, in the run's final 2.5 hours.

Meanwhile the organizer expanded the held-out grid family-by-family for the whole window — SGD scoreable by mid-afternoon, Adam by evening, Muon around 23:00, LoRA-Adam partial at the 02:12:37 deadline — re-scoring every open PR after each increment. 179 labeled PRs in 12.5 h is one every ~4 minutes fleet-wide; the workers closed 31 themselves as dead ends. The last submissions (#183, #184) landed 13 minutes before the wall.

§4Research threads — what the fleet actually did

179 PRs collapse into eight threads. Each fold has the thread's arc and its final-grid numbers; scores quoted are from the single frozen post-deadline snapshot, so threads are directly comparable.

1. Ridge-strength engineering (the inherited base).plateau thread — per-family tables ≈ one global constant; the winner uses a single λ=40

   The prior operator's lineage (#27, #79) led the board for six hours on per-family ridge-strength tables. Then #95 ran an honest nested-CV and showed a single global constant ties the 9-parameter table; #100 found the principled middle (group by base optimizer, 9→3 params); #104/#113 carved out the one real exception (sgd_l2base wants its own value); and #118 explained the whole thing — the tuned solve sits on a large-λ plateau where ridge is nearly a scaled projection, so fine-grained λ tables have nothing to grab. Final-grid scores across the thread: 0.6176–0.6197, all within 0.002 of each other. Verdict: simple beats elaborate (#115 said it mid-run, and the final grid agrees); the winner carries #173's single-global-λ base.
2. The rank-1 G interaction term (winner thread).the run's one new mechanism — +0.024 over the plateau, portable across every base tested

   #138 noticed a pairwise interaction term (how much the trained subset's documents reinforce each other) helps with opposite sign when computed in G vs G_pre — the preconditioner scrambles exactly the second-order structure the term needs. #144 bootstrap-confirmed the per-family split ranks by optimizer complexity. #151 then truncated G to its dominant eigenmode only (rank-1), which sharpened the term (0.6327 with #158's timeout fix), and the portability campaign began: #160 onto the held-out leader's base (0.6403), #168 onto a second base (0.6412), #173 onto the single-global-λ base (0.6393) — same gain everywhere, same fixed blend weight W_PAIR=0.3. #174 made that last constant self-calibrating (W ∝ λ₁/tr G, solved to be a no-op on the public grid — a pure transfer bet), and #176 added a fleet-discovered crash fix: the winner, 0.6414. The clean ablation #182 (source the term from G_pre instead: 0.6365, consistently worse) and #184 (the G-vs-G_pre gap is scale-dominated) closed the loop.
3. K_corr fusion attempts.kernel adds little once the Gram terms are right; nothing composed with the interaction term

   The loss kernel earned its keep only in niches. #92's one-step-CG blend of K_corr into the pattern term was the era's best local gain; #96 leave-one-family-out-validated it but flagged LoRA-Adam, #107 gated it off for LoRA, #123 ported it to the leading base (0.6197). #135 tried elementwise K_corr modulation of the pattern term (0.6187). But the composition tests were the point: #147 (modulation + interaction, 0.6308) and #156 (CG-blend + interaction, 0.6348) both landed below the interaction term alone on the same bases, and #117 (MKL-style fusion into the solve geometry) and #130 (additive solve blend) were clean nulls. Verdict: K and the rank-1 G term appear to read overlapping information; G's version is sharper.
4. Robustness & fallback engineering.where a chunk of the winning margin actually came from — three bug classes found and fixed fleet-wide

   Three genuine bug classes were discovered by the workers, in each other's code. (1) The K_corr fallback collapsed to near-random without top-mode deflation — found by #42, fixed across lineages by #94/#97/#105/#110, tuned by #136. (2) Uncached eigh calls blew the 300s scoring timeout — #90/#158 fixed live instances, and #162 audited the fleet ("37 open PRs share this"), spawning pre-emptive cache fixes (#159/#163/#165/#167/#169/#170/#171). (3) #172 found the ctx["G_pre"] KeyError landmine (the scoring shim drops the key when gram artifacts lag a snapshot; direct indexing crashes the whole family batch instead of falling back) — cross-applied by #181 and by #176, which is how the winner is literally "the best formula + another worker's crash fix". Several fix-PRs outscored the formulas they fixed.
5. Calibration (r2_mean) side-quest.free, LDS-invariant magnitude fixes — the run's R² gain (−0.05 → 0.10) largely lives here

   LDS is rank-based, so per-run scale calibration is "free" — it can't move the headline score but fixes R². #44 established the global calibration, #99 made it per-family, and #121/#127/#142 ported it across lineages. Magnitudes remain the honest weakness: the winner's R² is 0.1045 — better than the prior operator's −0.0457, far from calibrated.
6. Cross-architecture & cross-scale validation discipline.the fleet invented its own transfer methodology — and it predicted the held-out outcome

   With the held-out scale secret, workers built their own transfer checks: 8/8-split cross-scale runs on public models (#85/#87/#91/#93/#98) and a gpt2 line (#116/#124/#133/#149/#150/#155) as a second architecture. Findings: the ridge-strength form transfers; GAMMA and G-deflation don't; muon's deflation harm replicates at gpt2-medium; #149 even caught adamw flipping on gpt2. This discipline is why the final constants were either cross-validated at two scales or derived from score-time measurements — exactly the property the sequestered eval rewarded.
7. Grid-robustness checks (sampling, scale rules).predictor is robust to subset-selection strategy; density-modulated scale was a wash

   #140/#141 swept subset sampling distributions (random vs cluster vs distance) across 9/10 public families — distance sampling hurts almost everywhere, and #146 split the mechanism into two causes. Seeded direction #1 (density-modulated per-run scale) got a real test: #154 modest-positive locally, #157 sign-flips on the ridge-projection lineage — net a wash, not adopted by the winner.
8. Clean negatives (the fence around the answer).ten falsified ideas that stop the next person re-trying them

   #126 data-driven joint fits collapse to near-baseline (seeded direction #8 falsified); #175 early-stopped GD solve underperforms exact ridge (direction #12); #177/#179/#180/#183 effective-rank ridge corrections are net-negative on both G_pre and G; #125 MAD-robust normalization loses badly to plain std; #106 docFE-whitening doesn't help; #128 pre-solve matrix blending/whitening hurts monotonically; #152 λ-ensembles don't hedge; #164/#161 G_pre deflation hurts all LoRA families; #134 swapping to raw G for Muon runs hurts; and #131's "true-minimal" formula (no ridge, no interaction) confirms both remaining components earn their keep.

§5Final result

The fleet ran its full 12 h and filed 179 scored submissions (31 self-closed as dead ends along the way). After the deadline — with every worker gone — all 149 surviving heads were re-scored once against a single frozen held-out snapshot: 4 optimizer families, 506 fine-tune runs at gemma-3-1b, generated by the organizer for 13 hours and never visible to the workers. (The scale secret is out now that the run is over: development grid gemma-3-270m, held-out reproduction gemma-3-1b.)

All 149 submissions beat the prior operator at the unseen scale; the winner nearly doubles the ranking signal — held-out LDS 0.641 vs 0.346, in every family.

The winning formula keeps the prior operator's ridge-projection pattern term (solve in G_pre geometry with a single global ridge strength) and adds a pairwise interaction term built from a rank-1 truncation of the exact gradient Gram — keep only G's dominant eigenmode, form ½[(Σ_j∈SG₁_ij)² − Σ_j∈SG₁_ij²] — with the blend weight derived at score time from G's own spectral concentration (λ₁/tr G) rather than a constant tuned on the public grid. The lineage is classic fleet behaviour: one worker established the rank-1 term, ported it across three bases, and self-calibrated its weight; other workers independently ablated it (sourcing it from G_pre instead of G is consistently worse), stress-tested composition, and cross-applied robustness fixes. The winner is literally the best formula plus another worker's crash fix.

Solid negative results, all confirmed at the unseen scale: the interaction term must come from the exact Gram (the G_pre-sourced variant loses); K_corr-modulated pattern terms don't compose with the interaction term; effective-rank ridge corrections are net-negative; distance-based subset sampling hurts almost everywhere. The fleet also found and fixed two genuine bug classes in its own dominant lineage — uncached eigh calls that blew scoring timeouts, and a ctx KeyError landmine on the documented missing-gram path — which is where the winning margin partly came from: late-fleet effort went to robustness, not leaderboard-chasing.

One kernel-side nugget survives every caveat: the exact gradient Gram's eigenvalue-decay slope is scale-invariant to three decimals (0.737 at 1B vs 0.736 at 270M, R² ≈ 0.995), while the SGLD loss-kernel's tail slope at 1B remains estimation-limited (the denser re-estimation never fit the window) — so kernel-slope cross-scale claims stay open, and Gram-slope ones don't. Post-run update: a day-after v2 re-estimation (§10) closed this — with proper sampling the 1B kernel slope is 0.736 vs 0.743 at 270M. The kernel's spectral slope is scale-invariant too; v1's apparent 1.06 was an estimation artifact.

§6Final held-out leaderboard — all 149 scored submissions

Every head open at the deadline, scored once on the frozen 4-family gemma-3-1b grid (post-deadline wave, 2026-07-08 12:42–13:57 UTC). Baseline (prior operator, same grid): LDS 0.3455, R² −0.046. All 149 clear it. Bold row: the winner.

§7The winning formula, verbatim

The exact submission/predictor.py now merged into the task branch — including the worker's own 116-line docstring, which is its research note (the provenance of the crash fix, why the blend weight is spectrum-derived, and what result would falsify the bet). The structure: a fallback branch (deflated K_corr solve, used only if the Gram matrices are missing), the pattern term (ridge solve in G_pre over the trained subset, λ = 40·tr(Q_SS)/|S|, per-run normalized), and the interaction term (rank-1 G, pairwise ½[(Σ_j∈SG₁_ij)² − Σ_j∈SG₁_ij²], blended with weight W₀·λ₁/tr G measured from the G it is handed). Sixty lines of numpy after the docstring, touching five of the ctx keys defined in §2: idx, l0, G_pre, G, and (fallback branch only) K_corr.

"""Self-calibrating interaction weight (PR #174) + a fleet-wide
correctness fix: PR #172 (a different worker) discovered that every
predictor in this lineage reads `ctx["G_pre"]` and `ctx["G"]` by direct
dict indexing, but `tp_score_submission`'s own ctx-building code DROPS
the key entirely (rather than setting it to `None`) when the value is
`None`, because `np.savez` cannot serialize `None` and the archive is
built by filtering `None` values out before saving. Direct indexing on a
missing key raises `KeyError`, which crashes scoring for the WHOLE family
batch -- not a graceful "fall back to the K_corr branch" as every
formula's code intends and the problem statement documents (`G`/`G_pre`
are supposed to be "`None` if the held-out grid hasn't reached the gram
stage yet for that data snapshot"). This has apparently never fired on
real held-out data so far (every scored PR has a nonzero
`n_families_scored`, meaning gram was already present every time), but
it is a live landmine for the first snapshot where gram genuinely lags,
which the task's own docs say can happen. Fixed here by switching to
`ctx.get("G_pre")` / `ctx.get("G")`, exactly as PR #172 fixed it on a
different base. This attempt's own #174 formula (below) is otherwise
byte-for-byte unchanged.

## Self-calibrating interaction weight -- why (PR #174, unchanged)

My PR #151/#160/#168 established a rank-1-(dominant-eigenmode-only)
interaction term computed from the exact gradient Gram `G`, added to a
PATTERN term solved in `G_pre`. It ported cleanly across three different
PATTERN-term bases (PR #79/#84's per-family table, PR #100's 3-group
table, my own PR #173's single global constant), always with the exact
same fixed `W_PAIR=0.3`. Every one of those tests reused the SAME public
grid's `G` matrix, which has a specific spectral shape (its top eigenvalue
holds 83.8% of the trace, measured directly from
`experiments/training_prediction/results_public/gram/gram.npy`). A fixed
`W_PAIR=0.3` implicitly assumes the held-out `G` (at a different, larger,
never-revealed model scale) has a similarly concentrated spectrum. This
fleet has already found that OTHER fixed constants tuned against this
specific public grid's spectrum fail to transfer even when the general
functional FORM survives (PR #71/#73, on the ridge-lambda side) --
seeded research direction #9 asks for exactly this: derive shrinkage/
blend strength analytically from the spectrum `predict()` measures on
`ctx`'s own matrices at score time, not a constant baked in from this
grid.

## What changed

Replaced the fixed `W_PAIR=0.3` with `W_PAIR = W0 * (eigval_top /
trace(G))`, where `eigval_top` and `trace(G)` are both measured from
`ctx["G"]` itself inside `predict()` (the same `_top_eigmode` cache
already computes `eigval_top`; `trace(G)` is one extra `np.trace` call).
`W0` is chosen so that on the CURRENT public grid's own measured
concentration (0.838, matching the number above), `W_PAIR` reduces to
exactly the same value PR #151/#160/#168/#173 already validated (0.3),
so the local public-grid score is UNCHANGED -- this is a pure
generalization change, not a new tuned number. If a future (held-out,
larger-scale) `G` has a more or less spectrally concentrated top mode,
this formula's interaction weight automatically scales with it instead
of carrying over a number tuned for a 270M-parameter model's specific
spectrum.

Everything else -- the `G_pre` ridge PATTERN term (PR #173's single
global `LAM_GLOBAL=40`, the most portability-tested base to date), the
rank-1 truncation of `G` for the interaction term itself, the `id()`-keyed
`eigh` cache, and the K_corr-deflation fallback -- is unchanged.

## Local result

```
Fixed W_PAIR=0.3 (PR #173, same base):           LDS 0.6588
Spectrum-adaptive W_PAIR (this attempt):         LDS 0.6588  (identical,
                                                  by construction --
                                                  W0 solved to match)
```

Confirmed via real `arch eval --json` that the two are numerically
identical on the public grid (both `0.6588`), since `W0` was solved
algebraically to reproduce the same effective `W_PAIR` at this grid's own
measured concentration. The claim this attempt makes is NOT a public-grid
score improvement -- it's a robustness bet, exactly like PR #95's global-
lambda simplification: removing a public-grid-specific tuned number in
favor of a self-measuring rule should be neutral on the same grid, and
should reduce risk if the held-out grid's `G` spectrum turns out to be
meaningfully more or less concentrated than 270M-Gemma's.

## What's new here

Every fixed constant tested by this interaction-term lineage so far
(`W_PAIR=0.3`) came from a sweep against the public grid alone. This is
the first attempt to make that specific number self-calibrating from a
quantity `predict()` can measure directly on whatever `G` it's given,
rather than trusting the sweep to transfer.

## Prior attempts referenced

- PR #151, #160, #168, #173 (mine): establish and port the fixed-`W_PAIR`
  rank-1 `G` interaction term across three PATTERN-term bases; this
  attempt builds on #173's base (the most-portability-tested one) and
  changes only how `W_PAIR` itself is derived.
- PR #71/#73 (fleet): documents that a SPECIFIC spectrum-derived constant
  (there, a ridge-lambda rule's own `KAPPA`) failed to transfer
  cross-scale even though the general functional form tied a fixed
  lambda -- the reason this attempt is framed as a robustness bet, not an
  expected local-score improvement, and why `W0` is solved to exactly
  match the already-validated fixed value rather than picked fresh.

## Notes / caveats

Because `trace(G)` and the top eigenvalue are properties of the WHOLE
probe-set matrix (fixed for an entire scored batch, not per-run), this
change cannot be validated for a genuine local-score difference on the
public grid by construction -- the local numbers are identical on
purpose. Its value is entirely about what happens on a differently-scaled
`G`, which only the held-out eval can show. If this scores about the same
as PR #173 on held-out, that's a mild positive (no downside, marginally
more principled). If it clearly under- or over-performs #173, that would
be informative evidence about whether `G`'s spectral concentration itself
varies enough across model scale to matter for this specific blend
weight -- worth flagging either way.
"""
from __future__ import annotations

import numpy as np

SCALE_ABS = 0.00572  # unchanged from PR #173 -- same raw output scale

LAM_GLOBAL = 40.0  # unchanged from PR #52/#95/#173
FALLBACK_LAM = 20.0  # unchanged from PR #94

# W0 solved so that W0 * (public grid's own measured top-eigval/trace
# fraction, 0.838) == 0.3, the value PR #151/#160/#168/#173 already
# validated -- i.e. this reduces to the exact same number on THIS grid,
# and only diverges from it when scored against a G with a different
# spectral concentration.
W_PAIR_W0 = 0.3580

_EIGH_CACHE: dict[int, tuple[np.ndarray, np.ndarray]] = {}


def _top_eigmode(A: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    key = id(A)
    cached = _EIGH_CACHE.get(key)
    if cached is not None:
        return cached
    eigvals, eigvecs = np.linalg.eigh(A)
    i = int(np.argmax(eigvals))
    result = (eigvals[i], eigvecs[:, i])
    _EIGH_CACHE[key] = result
    return result


def _deflate_top_mode(K: np.ndarray) -> np.ndarray:
    eigval, v = _top_eigmode(K)
    return K - eigval * np.outer(v, v)


def _solve_psd(A: np.ndarray, b: np.ndarray) -> np.ndarray:
    try:
        return np.linalg.solve(A, b)
    except np.linalg.LinAlgError:
        return np.linalg.lstsq(A, b, rcond=None)[0]


def predict(ctx: dict) -> np.ndarray:
    idx = np.asarray(ctx["idx"], dtype=int)
    n = max(len(idx), 1)
    M = ctx["l0"].shape[0]
    out = np.setdiff1d(np.arange(M), idx)

    Q = ctx.get("G_pre")
    if Q is None:
        Q = _deflate_top_mode(ctx["K_corr"])
        Qss = Q[np.ix_(idx, idx)]
        lam = FALLBACK_LAM * max(np.trace(Qss) / n, 1e-30)
        x = _solve_psd(Qss + lam * np.eye(n), -np.ones(n))
        pred = Q[:, idx] @ x
        typical = np.sqrt(np.clip(np.diag(Q), 1e-30, None))
        run_scale = max(np.std(pred[out] / typical[out]), 1e-30)
        return pred / (typical * run_scale)

    # PATTERN term: single global ridge strength (PR #173).
    Qss = Q[np.ix_(idx, idx)]
    lam = LAM_GLOBAL * max(np.trace(Qss) / n, 1e-30)
    x = _solve_psd(Qss + lam * np.eye(n), -np.ones(n))
    pattern_raw = Q[:, idx] @ x
    typical = np.sqrt(np.clip(np.diag(Q), 1e-30, None))
    run_scale = max(np.std(pattern_raw[out] / typical[out]), 1e-30)
    pattern = pattern_raw / (typical * run_scale)

    G = ctx.get("G")
    if G is None:
        return SCALE_ABS * pattern

    # INTERACTION term: rank-1 truncation of G (PR #151), blend weight now
    # derived from G's own measured spectral concentration rather than a
    # fixed constant.
    eigval, v = _top_eigmode(G)
    trace_G = float(np.trace(G))
    concentration = eigval / max(trace_G, 1e-30)
    w_pair = W_PAIR_W0 * concentration

    Gx = eigval * np.outer(v, v)
    GxoutS = Gx[:, idx]
    typical_x = np.sqrt(np.clip(np.diag(Gx), 1e-30, None))
    lin_sum = GxoutS.sum(1)
    sq_sum = (GxoutS ** 2).sum(1)
    interaction_raw = 0.5 * (lin_sum ** 2 - sq_sum)
    pair_scale = max(np.std(interaction_raw[out] / typical_x[out]), 1e-30)
    interaction = interaction_raw / (typical_x * pair_scale)

    return SCALE_ABS * (pattern + w_pair * interaction)

§8What we learned

§9Wrap-up

• Winner squash-merged into arch/training_prediction_symreg; an anonymized Problem/Method/Result brief lives at findings/training_prediction_symreg/blogpost.md (since archived to main as experiments/training_prediction/symreg/blogpost.md).
• The full held-out archive — per-document loss vectors for all ~500 runs, exact G/G_pre, kernel v1 + eigendecompositions, scoring logs, final leaderboard — is preserved in an org-internal HF dataset. The entire scientific payload is ~10 MB, distilled from ~35 H200-hours.
• Total GPU spend ≈ $91 (fleet ~$30, organizer ~$51, scoring ~$10).

Operational notes for the next run, learned the hard way: RunPod container disks do not survive a pod stop without a volume (the organizer's logs were rebuilt from the S3 mirror); REST-created pods can sit "RUNNING" forever without ever being scheduled — spawn via runpodctl, which fails loudly instead; and a scoring sandbox must preflight itself (one unreadable Python install briefly mass-failed a scoring wave before the fixed wave superseded it).

§10Post-hoc ablations — which inputs earn their keep

Added 2026-07-09, after wrap-up. The winner is a two-term formula over four candidate matrices, so the obvious follow-up is a single-ingredient ablation grid: delete each term, and substitute each term's source matrix, one change at a time, everything else byte-identical. Each variant was scored with the production shim (tp_score_submission.py, unmodified) against both archived snapshots — the public 948-run gemma-3-270m grid and the frozen 506-run, 4-family gemma-3-1b grid from §6, restored from the org-internal held-out archive. Harness anchors all reproduce: the verbatim winner scores 0.6588 public / 0.6414 held-out (matching §6 to every reported decimal), the template re-expression of it is identical on every family, and the fixed-W_PAIR variant lands exactly on #173's leaderboard score (0.6393).

Every substitute for the exact gradient Gram in the interaction term looks nearly free on the development grid and fails at the held-out scale — the ranking the two grids agree on is the one computed from G.

The pattern term is the skeleton; the interaction term is a corrective that cannot stand alone. Pattern-only scores 0.6172 held-out — already 1.79× the prior operator — while interaction-only collapses to 0.0148 (negative on SGD). The interaction's +0.024 is also unevenly earned: +0.059 on Muon, +0.044 on LoRA-Adam, +0.005 on SGD, and −0.011 on Adam — the one family where the winner's new mechanism slightly hurts.

The preconditioning is where the pattern term's power lives. Re-solving the same ridge in the exact Gram G drops held-out LDS from 0.641 to 0.360 — right back at the prior operator (0.345), which also solves in G's geometry. What the Adam metric buys is a usable solve geometry: it flattens the Gram's one-direction spectrum (λ₁/tr 0.918 → 0.158 at 1B) so the ridge can see past the universal mean-gradient direction. Solving in the deflated loss-kernel instead (the winner's own no-gram fallback recipe) manages only 0.109 — identically for K_cov and K_corr — and bolting the G interaction term onto that kernel base rescues nothing (0.1089 → 0.1089): the corrective is worthless without the right skeleton. The fallback branch, which never fired in production (§4), is real but ~6× weaker than the gram path.

The interaction term is unsubstitutable — and the sequestered eval is what proves it. Sourcing it from K_cov costs almost nothing on the public grid (0.6269 vs 0.6588) and craters held-out to 0.3699 — a quarter below simply deleting the term. K_corr is the same story (0.6043 public, 0.3275 held-out). Mechanism: the winner's blend weight is W₀·λ₁/tr A measured on whatever matrix A it's handed, and G's spectral concentration is the one that holds across the scale jump (0.838 at 270M → 0.918 at 1B, so w_pair 0.300 → 0.329) while the kernels' top modes both fatten (0.36 → 0.55; 0.39 → 0.59) and mis-rank at 1B — a mid-weight wrong-signed term shreds the pattern term's fine ordering. The G_pre substitution completes #184's "scale-dominated" point: under the winner's self-calibrating rule the term simply mutes itself (G_pre's concentration is 0.158, so w_pair ≈ 0.06 → 0.6213, within noise of no-interaction), whereas #182's fixed-weight version forced it at full strength and lost outright (0.6365). Either way the information is in G's top eigenmode, nowhere else.

One honest confound: the 1B kernel is under-estimated, and that caveat lands on exactly one row group. The held-out archive's own notes flag it: kernel v1 at 1B got 2 seeds × 2 chains × 80 SGLD draws (burn-in 60) — roughly a third of the public recipe's samples, ~128 degrees of freedom per seed against M = 512 — and the denser v2 re-estimation died at launch and never shipped (§5's "estimation-limited" tail). The damage is measurable: the kernel's off-diagonal agreement with G's structure (entrywise Spearman, K_corr vs diag-normalized G) collapses from 0.31 at 270M to 0.00 at 1B, its semantic-cluster coherence halves (within-group corr 0.49 → 0.26), and part of its apparent concentration jump (0.36 → 0.55) is the rank-≈128 sample-covariance inflating its top eigenvalues. So the interaction-from-kernel craters above are a test against a noisy K as much as against K per se — a properly-sampled 1B kernel might do better, and only a re-run (kernel v2) can say. The pattern-side verdict, though, survives the caveat twice over: at 270M, where K got its full budget and a clean power-law fit (R² 0.96), the kernel pattern solve still reaches only 0.167–0.184 against the gram path's 0.649; and K's top eigenmode is ≈ the uniform everything-moves-together direction at both scales (|⟨v₁, 1⟩| = 0.96–0.99) — the one direction docFE already absorbs. A well-estimated kernel still doesn't carry the parameter-space geometry the solve needs.

Postscript, 2026-07-09: we re-ran the estimation, and the verdict survives de-confounding. Kernel v2 at 1B — the re-estimation that died at launch during the run — was re-run post-hoc at the originally-planned density (2 seeds × 4 chains × 140 draws, burn-in 120 → 560 dof/seed vs v1's ~128; lr 3e-7 from the same auto-calibration; chains sharded across 4×H100, ~39 min each; probe-set identity verified against the archived l0 at r = 0.99994). The estimate is now better than the public one: 2-seed reproducibility gate 0.824 vs the public kernel's 0.77. Every v1 spectral anomaly turns out to be estimation artifact — slope 1.06 → 0.736 (public: 0.743 — the kernel slope is scale-invariant after all, closing §5's open question), cluster coherence 0.26 → 0.44, top-mode share back to public-like (rank-50: 1 → 15). But rescoring the kernel ablations against v2 barely moves the conclusions: the kernel pattern solve rises 0.109 → 0.175 — exactly its public-grid ceiling, so that gap was estimation — while the kernel-sourced interaction stays net-harmful (K_corr 0.328 → 0.474, K_cov 0.370 → 0.370, both far below the 0.617 of simply deleting the term), and the winner and pattern-only controls are unchanged to four decimals. Sharpest of all: with a well-estimated kernel, the entrywise K↔G agreement at 1B is −0.06 (vs +0.31/+0.42 at 270M) — the kernel-vs-Gram geometry divergence at scale is real, not noise. The loss kernel isn't a noisy copy of the gradient Gram; at scale it measures something genuinely different (the posterior-covariance metric rather than the raw-gradient one) — and what it measures is not what the training-prediction solve needs. Raw v2 artifacts (chains, traces, ensembles, ledger) live in the org-internal archive under kernel_v2/.

The two refinements are small but real. Un-truncating the interaction (full-rank G) gives back −0.006 (0.6353) — #151's rank-1 sharpening confirmed at the unseen scale. Freezing W_PAIR at 0.3 gives back −0.002 (0.6393): the spectrum-derived weight's entire held-out margin. Both refinements cost nothing locally and paid only at transfer — the run's central lesson, in miniature.

§11Why the fleet could never make K work — the missing transform

Two post-hoc questions about the loss kernel, one from outside and one from inside. From outside: across all 179 PRs, did any use of K ever actually help? We fetched all 149 scored predictors at their exact leaderboard SHAs and AST-scanned each for K_cov/K_corr reads outside the never-fired fallback branch. The answer is stark: zero scored submissions ever used K as the live pattern-solve matrix, and among the eleven lineages that used it anywhere (CG-blends, elementwise modulation, kernel-density scaling), the best genuine delta over an otherwise-identical no-K base was +0.0021 — inside the plateau's own spread — while several uses were sharply negative (−0.016 to −0.032). Every synthesis that included K was beaten by the same formula with K deleted. The fleet probed the whole operator algebra — additive, multiplicative, fused, whitened, eigenvector-covariate, LEVEL-proxy — and hit a wall everywhere.

The wall is geometric: at 1B there is no function of G that reaches K — their eigenvalues match almost perfectly, and their eigenvectors don't match at all.

From inside: all three ctx matrices are Grams of the same 512 probe gradients under different parameter-space metrics — G = JJᵀ, G_pre = J·P·Jᵀ (Adam's diagonal), and by the delta method K ≈ J·Σ·Jᵀ with Σ the SGLD posterior covariance (≈ inverse curvature). Writing J = USVᵀ, a metric M descends to a doc-space transform of G exactly when W = VᵀMV commutes with S² — G carries U and S but zero bits about how the gradient span is oriented relative to M's eigenbasis. Measured: the matrices share their bulk spectral slope to two decimals at both scales (≈ k^−0.74), so eigenvalues were never the obstruction. At 270M the commutation approximately holds — after projecting out the common mode, K ≈ c·G^0.6 with R² 0.88, essentially the ceiling for any spectral map, and exactly the ridge-flattening the delta method predicts. At 1B it is gone, not attenuated: the best possible f(G) explains 0.29 of centered K versus 0.22 for a random basis. Two controls make this trustworthy: the known metric pair G_pre ← G fits as a spectral map with R² 0.99 at 1B (exponent ≈ ½, as an H^−1/2-style preconditioner should), and K's seed-replicates fit each other at 0.96 — the method would have found K's transform if it existed. A last twist: the badly-estimated v1 kernel resembled G better (CKA 0.68) than the converged v2 (0.10) — an under-burned SGLD chain is still descending, and descent is gradient flow, so a bad kernel is secretly a noisy copy of G. Improving the estimation revealed the divergence rather than curing it. The information the fleet needed — W's off-diagonal orientation — lives in parameter space, and the predict(ctx) contract never exposes it.

§12The regime where K wins — equilibrated fine-tuning

If K isn't a broken G, what is it the response kernel of? Linear-response theory gives a precise answer: the grid's 16-step deterministic quenches respond through the gradient Gram (substitute Δw into Δℓ and only G-blocks appear — the winning ridge formula is the discretized linear-response solution of gradient flow), while the fluctuation–dissipation theorem says K = Cov(ℓᵢ, ℓⱼ) is the response kernel of the tilted tempered posterior: upweight doc j in the effective Hamiltonian, let the ensemble re-equilibrate, and d⟨ℓᵢ⟩/dε = −nβ·K_ij. Two different learning processes, two different kernels. So we ran the other process: mixin fine-tunes (base corpus + ε-upweighted subset S, ε = 0.5) at gemma-3-1b, run past stationarity with tail-averaged probe readouts, scored with the run's own LDS protocol on the same 32 subsets. Five arms: full-parameter SGLD at the kernel's exact temperature and localizer (the clean FDT test), SGLD on LoRA adapters, and three "normal-looking" recipes — SGD, AdamW, and Muon, each with decay-to-init and injected noise at a practical learning rate. The design only became feasible with common-random-numbers coupling: posterior fluctuations are 0.55 nats/doc against a ~0.05-nat signal, but with every run in a group replaying the identical noise stream, 98.9% of the per-doc noise is shared and cancels in rank scoring.

Each kernel predicts its own process and fails the other's: G_pre 0.64 vs K 0.11 on quenches; K 0.71 vs G −0.25 on the equilibrated ensemble.

The double dissociation is clean and internally replicated. In the full-parameter arm — the exact sampler K was estimated with — the FDT form of K predicts the equilibrated response at LDS 0.713 (n=7 val runs, every run individually positive at 0.41–0.65 against its coupled base twin) while every gradient-family predictor is ≤ 0.21 and raw G is −0.25. The LoRA arm replicates at 0.43 in each of two independent noise groups (attenuated exactly as its γ-mismatch and subspace restriction predict). Within-run probe checkpoints show the rotation directly: K's LDS climbs from noise at step 16 to 0.48 at the tail while G-family stays flat. And a self-consistency check seals the mechanism: each arm's own tail-fluctuation covariance, leave-one-run-out, predicts the SGLD arms at 0.60/0.42 — FDT confirmed with no reference to v2-K at all.

But the theorem has a boundary, and practical fine-tuning lives outside it. The three realistic arms sort into a law. The gentle SGD arm (learns −0.07 nats, fluctuations 30× weaker than SGLD's) is predicted by G-family (fdt_G_pre 0.32) and not by K or even its own covariance — with noise that weak, the stationary state is the drift-vs-decay fixed point and its response is still gradient-geometric: a long quench in disguise. The AdamW and Muon arms are predicted by nothing in ctx — including their own measured covariances (−0.04, −0.11) — and the Muon arm closes the escape hatch, since it is thermally SGLD-like (warm, non-learning) yet still violates fluctuation–dissipation. What Adam and Muon share is preconditioned, non-gradient-flow updates with noise unmatched to their metric: non-equilibrium steady states, where response ≠ covariance. Their correct kernel would be the Gram in that optimizer's metric — a pSGLD-style kernel no ctx matrix provides.

What this settles. The fleet's 179-PR wall against K (§11) wasn't K being a bad matrix — it was K being the response kernel of a learning process the task never ran. Run that process and K delivers its strongest number of the whole project. But the boundary matters just as much: "equilibrated" must mean thermally equilibrated at the kernel's own temperature (n/log n tempering — deliberately hot, loss-elevated, nothing a practitioner would call fine-tuning). Every recipe that actually learns stays drift-dominated, where the original G_pre winner — or nothing yet — rules. A companion piece stress-tests exactly that boundary on the learning arms: sweeping the winner's ridge strength across horizon shows the constant is precisely optimal at 16 steps, nearly irrelevant across eight orders of magnitude on the 1200-step runs, and — for AdamW — beaten by a solve-free first-order prediction at long horizon: the failure is the linearization, not the regularization. As far as we can tell from the literature (the BIF line validates SGLD kernels on retraining-LDS; Schioppa et al. argue influence formulas describe a few fine-tuning steps), the quench-vs-equilibrium contrast itself had not been run before. Predictions if this picture is right: a pSGLD-estimated kernel should pick up the AdamW arm, and hotter noisy fine-tuning should interpolate the SGD arm from G-predictable toward K-predictable.