Gradient kernels as a production object

The BIF thread ended with a verdict: the loss kernel is the lossy shadow, and sampling it is the expensive way to compute a GEMM. This thread takes the hint and builds the GEMM into an instrument — the preconditioned per-token gradients as they are computed, know the error in advance, and feed the result to things that want an N×N similarity: conductance clustering, influence retrieval, training-dynamics probes. Each piece stands on its own; read top-down for the arc. ← back to Vibe Research

2026-07-09 → 10 · origin, from the BIF thread The ImageNet loss kernel, reproduced — and then taken apart
Where this thread comes from: Timaeus' flagship loss-kernel result reproduces, then decomposes — on a classifier it is mostly the label channel, and on a label-free ImageGPT the exact gradient kernel linearly contains 91% of its explainable geometry. The Adam-preconditioned gradient kernel recovers the semantics the loss kernel drops (few-shot accuracy ×3, matching the model's best activation probe), and re-sampling the loss kernel under the Fisher prior just returns the gradient kernel plus Monte-Carlo noise. Conclusion adopted here as premise: compute the gradient kernel directly.
origin imagegpt gradient-kernel
2026-07-13 → 14 · foundation A sketch-error law for gradient kernels
The instrument. CountSketching preconditioned per-token gradients across Pythia 14M/70M/410M, ±embeddings: error = c/√(D·S) with c = √(1+r_eff), the kernel's stable rank — flat at 23.3 from 19M to 302M params, so the width any accuracy needs is predictable in advance. Freeze the embeddings (ε-floor heavy hitters), factor the preconditioner rank-1 (kernel ρ = 0.99 from 300 KB of state), and Pythia-1.4B × 4.25M tokens prices at $250–480 on measured A100 throughput.
error-law sketching scaling pythia
2026-07-14 · closing the loop MNIST across training: the gradient kernel is the BIF at init — and something else at convergence
The BIF thread's foundation experiment redone with exact no-embedding gradient Grams at all 31 checkpoints of the same run, same probes, same protocol, ~18 s per checkpoint (×545 cheaper). At init it reproduces the published epoch-0 cell to r = 0.9999996; digit structure peaks at epoch 2 (ARI 0.23, rising to 0.34 at 5,100 probes — past SGLD's probe-noise wall) and then fades as the model converges (r_eff 22 → ~100: converged per-image gradients are idiosyncratic noise) while the SGLD BIF holds flat — point gradients and posterior curvature are the same probe of a young landscape and different probes of a converged one. Same-day addendum: the divergence is diagnosed (the converged basin is a fossil of epoch-1 gradient directions) and then erased — K = 512 seeded Gaussian weight perturbations at SGLD-matched temperature reproduce the BIF's flat curve deterministically, forward-only, at ~1% of its cost. Also: preconditioning subsumes embedding-exclusion. Interactive cluster hierarchy per checkpoint, hover for member digits.
mnist training-dynamics bif-comparison interactive ×545
2026-07-14 → 16 · mixture-of-experts Preconditioning a Mixture-of-Experts: γ is moot, experts are not embeddings, and the clusters aren't the router
What should "unit variance" mean when 90.5% of parameters only get gradient if the router fires them? Measured on granite-3.1-1B-A400M: per-token-loss expert gradients are only soft-sparse (median coordinate fires on 99% of rows), so the marginal-vs-conditional γ family collapses to one kernel (Pearson 0.9994) — but excluding experts is a different kernel (41% of neighbours flip) that discovers near-disjoint, worse clusters (judge accuracy 0.57 vs 0.67). Router purity is near-null in both arms: the clusters are content, not routing. The sketch-error law transfers to 1.33B-MoE within a few percent. Interactive: the largest cluster maps in the thread — 325 + 451 clusters over 51k rows, hover for SAE-style multi-token member examples.
moe preconditioning granite-1.3B router-purity interactive
2026-07-14 → 15 · downstream consumer Clustering the sketch: conductance communities scored by an LLM judge
Timaeus-style PageRank conductance clustering on 262k sketched per-token gradient rows of Pythia-14M — two arms (±embeddings), three widths, every cluster scored by a detection judge. Freezing embeddings changes what gets discovered (cross-arm ARI 0.03–0.05, near-disjoint) and makes it better: judge accuracy 0.87 vs 0.78, winning every size bin above 100 rows. Folding the sketch destroys cluster identity long before judgeability (goodness flat across widths while membership ARI collapses), and good clusters are the reproducible ones (top goodness quartile re-found at Jaccard 0.99) — until the sketch dies, when the relationship inverts. Interactive: every scored cluster, hover for its member contexts. Addenda: replication study — cluster quality reproduces on every rung (judge 0.86–0.89) while cluster count is the stopping time of a race against a main-body wipe (same graph, four rng seeds: 61–219 clusters); and an EK-FAC arm that whitens the kernel to chance-level clusters.
clustering llm-judge pythia interactive