Loss-landscape grooves: Bayesian influence functions on generative models

A pixel-level autoregressive transformer on MNIST. Same-class losses co-fluctuate at every checkpoint; the BIF predicts what per-class fine-tuning does; and unsupervised clustering of the BIF recovers digits better than clustering the images themselves (ARI 0.36 vs 0.30). Also the cautionary tale: the first headline was an SGLD estimator artefact, caught and fixed same-day — the block-centred estimator and reproducibility gate that the rest of the thread relies on come from here.

Scaling up with a learned VQ tokeniser and a token-AR transformer. The grooves recover the coarse animate/inanimate split (animal vs vehicle) three times better than clustering the pixels (ARI 0.29 vs 0.09) — but the fine ten-class taxonomy not at all. The loss geometry organises the data by broad semantic category, above the label set. Interactive cluster hierarchy: hover to see member images.

A VQ-tokenised model trained on STL-10's 100k unlabelled images at 64px — never sees a class. Even so, unsupervised clustering of its loss grooves recovers animal-vs-vehicle at ARI 0.37 (vs 0.00 from pixels; ensemble reproducible at r 0.93), while fine class stays at chance. The abstraction comes from generative training alone, not supervision. Here the block-centred estimator is mandatory — the cold cells are common-mode-dominated.

Scaling to 200 WordNet classes. Unsupervised hard clustering of the grooves no longer recovers the semantic categories (ARI ≈ 0.01, same as pixels) — but the abstraction is still there: animal classes couple in the loss and anti-couple to artifacts, and graded class similarity tracks WordNet distance. A scale limit on recovery, not on the structure. Centerpiece: a giant scroll-down vertical hierarchy, hover any node for its images.

[Superseded — this masks only the loss readout, not the fine-tuning gradient; corrected on the masking page below.] Is the animate/inanimate signal a low-level-statistics mirage in the background? Masking the loss to the bottom half of a CIFAR image destroys it and the top half keeps it — but a random-order (RandAR) model shows that collapse is an artifact of conditioning on a filled contiguous block, not of the bottom-half content. Score a scattered half instead and the signal returns. The abstraction is distributed across the image; autoregressive order just hid it.

The BIF of an off-the-shelf RandAR-L (0.3B, ImageNet-256) random-order generator. Interactive cluster hierarchies at full loss vs the last fractions of a scattered token order (hover for member images), and a factored analysis: the loss-landscape similarity splits into two near-independent axes — predominant colour ⊗ content.

The right way to mask a BIF: mask the first (scattered) tokens of the random order on both the SGLD perturbation and the readout — not just the readout, as earlier in this thread. Sweeping 0/⅛/¼/½ masking, restricting the fine-tuning pulls the pretrained RandAR's clustering off colour (colour-η² 0.54→0.40) ~2.7× more than masking the readout alone. Interactive splitting diagrams per masking level; supersedes the readout-only masking above.

The thread's first language model — Geodesic's 6.9B alignment-pretraining family, across the misalignment axis. The CE loss kernel groups documents by topic/format, never by valence (aligned and misaligned scenarios stay entangled); the DPO preference-margin kernel kills the topic confound but finds no coherent alignment factor. The alignment axis turns out to be first-order — a +4.5-nat shift in the mean preference margin for the aligned model — not geometry, and no misalignment factor strengthens with alignment training. Interactive 5-class spectroscopy.

Eight chat datasets — gemma-3-4b-it and gpt-5.4-mini, each prompted with Open Character Training's humor/loving/poeticism constitutions or nothing — probed with one kernel on gemma itself. Run 1 is the thread's second cautionary tale: with variable-length responses the kernel's top axis is response length (ρ = 0.81), a pure SNR artifact that reproduces at gate 0.91. With a fixed 64-token readout, character structure appears only in the on-policy data, survives a judged-expression control (~2.5× kernel separation per unit of expressed character), and the deepest groove of all is gemma's own unprompted voice. Interactive per-policy spectroscopy.

The kernel's theory says sums: disjoint mechanisms give K ≈ Σ Z_iD_iZ_iᵀ over crossed factors. A compositional grammar search (sums + Hadamard products of block kernels, BIC, beam) with independence gates — assignment MI, block-mean separability, per-factor informativeness, trace-level additivity — that a relabeled flat clustering cannot pass. Recovers planted sums/products and refuses a non-separable fake on synthetics; on a grokked multitask-arithmetic transformer it correctly refuses the planted (operation × argument-order) factorization — the kernel instead shows the two linear ops sharing circuitry within an argument order and division spanning orders, replicated at r ≈ 0.97 across chains and a 4× γ range. On a real Pythia/Pile kernel, per-seed structure passes every within-seed gate and then fails seed-to-seed replication: the estimation-noise null earns its keep. An addendum separates what that gate rejects: the kernel is ~95% one common mode (not length — length is a 3-parameter scalar, not a clustering), and on the deflated kernel the found partitions still transfer R² ≈ 0.5 to the other seed — labels aren't canonical, but half the structure is real.

Timaeus' flagship result reproduces cleanly: Inception-v1's kernel UMAPs into WordNet islands, taxonomic lift ≈ 200×, gate 0.68. But a classifier's per-sample loss contains the label — 332× same-label lift, and dropping the paper's same-label exclusion pushes "taxonomy" to exactly that number. Recompute the identical kernel for label-free ImageGPT-small on the identical 10k images and the taxonomy collapses to 2–4× (kernel↔kernel CKA 0.05) while reproducibility improves to gate 0.94. A spectral linear map finds just one classifier direction in the AR kernel — the animals↔things axis, 24% of its explainable variance — and whitening recovers nothing more: the structure isn't buried, it's absent. Kernel-ridge classifiers make it operational: the label-free kernel is a real unsupervised representation (4.0% few-shot 1000-way, 40× chance; 86% animate/inanimate) but ~3× below the same model's own activations — the loss kernel is a coarse, lossy window onto what the model knows, and on a classifier it is contaminated by the labels it was trained on. Addendum: the exact gradient kernel (10k × 75.7M-param backwards, no sketching) linearly contains 91% of the loss kernel's explainable geometry — and its Adam-preconditioned form recovers the semantics the loss kernel drops: few-shot accuracy triples to match the best activation probe, fine WordNet lift 1.75× → 4.9× with the peak at the deepest levels. The closing control: re-sample the loss kernel under the Fisher-metric prior and it does find the taxonomy — but what it finds is the preconditioned gradient kernel plus Monte-Carlo noise (agreement at 98.6% of its own noise ceiling; the gradient kernel predicts a K_pre seed better than the other seed does). The grooves under-report what the gradients know, and sampling them is the expensive way to compute a GEMM.

Pre-registered SGLD vs RMSPropSGLD head-to-head on Pythia-14M: preconditioning buys the entire step-size axis (0/128 diverged chains, λ̂ flat to ±5% over a censored 3.5-decade grid vs SGLD's 1.5-decade band and cliff) — but zero convergence speed: 2× faster mixing is exactly cancelled by 2× noisier per-draw kernels, and steps-to-kernel-precision comes out dead equal (2,569 = 2,569). At matched ε the samplers measure the same kernel (CKA 0.90–0.93); ε itself is a resolution dial, and the spec's τ/2 recording-stride heuristic quietly destroys kernel reliability (split-half 0.44 → 0.96 at 8.5× denser recording for 1.3× cost). Verdict: keep SGLD when calibrated, switch the moment ε is uncertain.

Analysis-only coda: eigendecompositions of every K, K_pre, G, and G_pre the thread (plus the training-prediction work) has produced — eight models, pythia-14m to llama-8b plus iGPT-S and Inception-v1. Language-model kernel spectra collapse onto one power law (α 0.46–0.75, no size trend); the top kernel mode is the common mode nearly everywhere; preconditioning flattens every spectrum it touches (iGPT-S rank₅₀ 8 → 686); and eigenbases are fragile where spectra are robust — the spectral transform f(G) → K that exists at gemma-270m is gone at gemma-1b, while iGPT-S's K_pre agrees with G_pre better than with its own second seed.

This thread's across-epochs MNIST result re-measured with deterministic gradient Grams at all 31 checkpoints (same run, same probes, same protocol, ×545 cheaper): identical to the BIF at init (r = 0.9999996), tracking it through early training, and measurably a different object at the converged minimum — where this thread's SGLD kernel still sees the digit taxonomy. Then the addendum closes the gap: seeded Gaussian perturbations at SGLD-matched temperature reproduce the BIF curve deterministically at ~1% of the cost — the loss kernel is temperature-calibrated smoothing of the gradient kernel, and the temperature knob is the whole difference.

The training-prediction winner's ridge strength, swept over ten orders of magnitude at every horizon. λ₀ = 40 is the exact argmax at the 16-step eval it shipped for, and on noisy 1200-step runs the dial looks irrelevant — but on clean deterministic quenches extended to 1024 steps the optimum slides down with horizon exactly as kernel-regression theory predicts: re-tuning rescues SGD (+0.30 LDS at 256 steps, P = .997), the first-order prediction goes anti-predictive once transfer turns to interference, and Adam collapses for every λ. The dial matters; the regime decides.