MNIST across training: the gradient kernel is the BIF at init — and something else at convergence

The BIF thread's foundation page clustered the loss-landscape kernel of a pixel-level autoregressive MNIST transformer at every checkpoint and found the digit taxonomy carved during training — via SGLD sampling, at real cost per cell, gated on seed agreement. Its epoch-0 control couldn't use SGLD at all (no stable local posterior at a random init) and fell back on the γ→∞ limit of the localised BIF: the raw gradient Gram. This page runs that limit object — plus the Adam-style preconditioned version, computed without the embeddings — at all 31 checkpoints of the very same training run, on the very same 510 probes, with the very same spectral-clustering protocol, and compares against the published SGLD numbers point for point. Deterministic, no chains, no gate: ~18 s per checkpoint.

At initialisation the gradient kernel is the BIF (the with-embeddings kernel reproduces the published epoch-0 cell at r = 0.9999996). As training converges they part ways: the gradient kernel's digit structure peaks at epoch 2 (ARI 0.23) and fades to 0.09 by epoch 30, while the SGLD BIF holds flat at ≈0.31. Point gradients and posterior curvature are the same probe of a young landscape and different probes of a converged one.

Excluding the embeddings — our standard convention from the error-law study, here the tied pixel-value table plus the 100k-parameter positional embedding — turns out to change almost nothing, and the way it changes nothing is informative: the raw kernel improves at mid-training (epoch 10: ARI 0.087 vs 0.055 with embeddings in) while the preconditioned kernel is untouched at every epoch (peak 0.229 vs 0.230). The marginal preconditioner was already suppressing exactly the per-position clutter that exclusion removes by hand — preconditioning subsumes embedding-exclusion on this model, which is the same division of labour the ε-floor analysis found at Pythia scale, seen from the other side.

The mechanism is in the kernel spectra. At init the kernel is one common mode (stable rank r_eff = 1.0; the top eigenvalue carries 98% of the trace — every random-init gradient points the same way, and what clustering there is lives in the residual). At epoch 2 the kernel is genuinely class-structured: r_eff ≈ 22 with digit blocks carrying the spectrum. By epoch 10–30, r_eff ≈ 80–100: a converged model's per-image gradients are nearly-orthogonal idiosyncratic residuals — the loss is minimised, the shared class directions are trained away, and what remains of each gradient is that image's private noise. The SGLD BIF doesn't inherit this fade because it never measures the point gradient: it samples the basin, whose curvature keeps encoding which images share mechanisms after their gradients have shrunk into noise.

This resolves cleanly against the ImageNet capstone, which found the opposite-sounding result — there, the preconditioned gradient kernel contained the loss kernel's geometry and beat it downstream. No contradiction: ImageGPT and Inception are one-to-two-epoch models over huge corpora, nowhere near the zero-gradient regime; this 30-epoch MNIST toy actually converges (test loss flat to 0.001 bits/pixel for the last five epochs). The gradient kernel's validity window is "gradients still carry signal" — which covers essentially every practically-trained large model, and stops covering a toy ground to its minimum. If you want landscape structure at such a minimum you need finite-radius information — though, per the addendum below, not SGLD: seeded Gaussian smoothing at the right temperature recovers it deterministically.

What the taxonomy looks like as it is carved — and un-carved — is best seen directly. The figure below is the interactive cluster hierarchy from the BIF page, rebuilt for the gradient kernel: sweep the number of clusters k left to right, switch checkpoints with the tabs, hover any node to see its member digits (click to pin, Esc to dismiss). At init nothing splits meaningfully at any k; at epoch 2 the 1s separate first and loop-digits part from stroke-digits as k rises; by epoch 30 the flows blur back toward mixture.

checkpoint epoch 30 (converged) epoch 10 epoch 5 epoch 2 (peak) epoch 1 epoch 0 (init)

All 31 checkpoints cost 17 minutes ≈ $0.21 on one mid-tier card — ~18 s per checkpoint against ~3 GPU-hours per published BIF cell: ×545 cheaper, with no step-size calibration, no burn-in, no seed gate, and exact reproducibility.

§+ Addendum, same day: the divergence diagnosed — then erased

Asked why the loss kernel wins, we measured instead of argued. Three diagnostics against the published cells (same probes, same protocol): the BIF's class-gap signal-to-noise is 2–3× ours at every trained epoch (1.11 vs 0.71 at epoch 2, 0.81 vs 0.40 at 30); the BIF kernel re-concentrates late (r_eff 117 → 24 → 18 across epochs 5→30) exactly where ours diffuses (46 → 100 → 83); and — the fossil test — the converged BIF agrees best not with our converged kernel but with our epoch-1 kernel (r 0.485 vs 0.311): the basin's curvature is a fossil record of the directions carved while gradients were informative. Checkpoint-ensembling our kernels (TracIn-flavoured) recovers most of that geometry (agreement 0.31 → 0.64 for early-weighted ensembles) but not the clustering (ARI ≤ 0.25). The theory this points at: at finite temperature the BIF is Cov(ℓ_i,ℓ_j) ≈ g_iᵀΣg_j + ½tr(H_iΣH_jΣ) — and the second-order term, which no point-gradient object contains, carries the class signal once gradients shrink.

Two cheap corollaries. A q-weighted squared-gradient kernel (rows g_i² ∘ q — per-probe diagonal-curvature proxies, one backward each) already reaches ARI 0.290 at epoch 30 (agreement 0.49 with the BIF); unweighted g² fails (0.04–0.08). And top-mode deflation of the converged G_pre does not rescue it (0.092 → 0.065–0.103 for m = 1–10): the late class signal genuinely is not in the point-gradient span at any depth — it lives at finite radius. Net punchline for the thread: the BIF thread's capstone said "sampling the loss kernel is the expensive way to compute a GEMM"; the converse also holds — at a converged minimum the GEMM is the wrong object, and the loss kernel is a temperature-calibrated Gaussian smoothing of it that you can compute deterministically, forward-only, for ~1% of the SGLD price.

Positioning: Timaeus showed per-sample landscape information clustering MNIST digits across (epoch × temperature) with per-sample LLCs, and their Loss Kernel paper sweeps checkpoint structure on ImageNet; He & Su clustered MNIST classifiers with an NTK-flavored gradient similarity at a single snapshot. The combination here — deterministic exact gradient Gram, autoregressive generative model, the published BIF protocol swept over every checkpoint with a direct cell-by-cell agreement curve — appears to be new, and the punchline is two-sided rather than triumphalist: the cheap deterministic object replicates the expensive sampled one exactly where they coincide by construction (init), tracks it closely through early training, and measurably stops being the same thing at a converged minimum.