Clustering the sketch: conductance communities in a 262k-token gradient kernel, scored by an LLM judge

The error-law page established what a CountSketched preconditioned gradient kernel costs and how accurate it is; this page asks whether the thing it preserves is useful. We run Timaeus-style local PageRank conductance clustering on 261,632 per-token gradient rows of Pythia-14M (512 Pile sequences × 512 positions, 16,384 total sketch dims native), in two arms — all parameters vs embeddings frozen — and at two folded widths per arm (4,096 and 1,024 total dims, derived offline by bucket folding, no model passes). Every discovered cluster is then scored by an LLM judge with the detection protocol: explain the cluster from ten members, then classify held-out members against distractors; balanced accuracy is the cluster's "goodness". Six clusterings, one seeded judge, cross-run membership matching.

Freezing the embeddings doesn't just shrink the sketch error — it changes what the clustering finds (cross-arm ARI 0.03–0.05: near-disjoint), and what the no-embed kernel finds is better: mean judge accuracy 0.87 vs 0.78, winning every size bin above 100 rows.

§1 Embeddings change what is discovered

The error-law study showed the embedding block's ε-floor heavy hitters dominate the full kernel's geometry at 14M; here that has a semantic consequence. Full-native and no-embed-native agree at ARI 0.047 over all rows — and 0.003 restricted to rows clustered in both — with median best-Jaccard 0.01 in both directions: there is essentially no cluster in one arm whose membership survives into the other. This is not sketch noise (the two folded widths of the full arm agree with each other at ARI 0.52–0.67). Freezing embeddings is not a cost optimisation; it selects a different — and per §2, better — similarity structure. The same lesson at MoE scale: excluding expert tensors flips 41% of nearest-neighbour structure on granite-1B (E2, MoE page to follow).

§2 No-embed clusters are better at matched size

Goodness falls with cluster size in both arms and the two inventories have very different size profiles (median scored size 158 vs 1,423), so the honest comparison is within size bins:

§3 Folding destroys identity long before judgeability

§4 Good clusters are the reproducible ones — until the sketch dies

Joining goodness with cross-width membership matching (full arm, native↔4,096-fold): the top goodness quartile reproduces at median Jaccard 0.99 vs 0.54 for the bottom quartile, matched pairs' goodness scores agree at Pearson 0.89, and Spearman(goodness, best-Jaccard) = 0.37. At native↔1,024-dims the relationship inverts (top-quartile Jaccard 0.001, bottom 0.77): heavy folding kills the small coherent clusters first and preserves the large mushy ones. Goodness and repeatability co-vary while the representation is faithful, and decouple in opposite directions when it isn't — so the pair (goodness, cross-width Jaccard) is a usable health check for any production width. (No-embed joins are underpowered — 120-cluster sampled scoring leaves 14–22 matched scored pairs.)

What the judge is actually looking at — every scored cluster, both native arms, hover for members:

§5 Addendum (2026-07-16): replication — enumeration is a race; geometry and goodness replicate

An "anomaly" forced this section into existence: a seed-replication arm (fresh hash seeds + a different VJP backend on a different pod) returned 61 clusters where the native run found 430, and three hash-seed-only replicates at 32,768 dims returned 73, 231 and 94. That looks like the pipeline barely replicates. It doesn't mean that — the attempt logs say something much more specific.

Cluster count is not a property of the kernel — it is the stopping time of a race, and the race is the only part of this pipeline that fails to replicate.

Discovery terminates on unvisited (not unclustered) mass. Roughly 80% of rows sit in one giant PPR basin — the "main body" — and when a random seed lands there and its sweep prefix swallows >99% of the support, the whole support is marked visited unclustered, the remaining-mass counter collapses below the 2% threshold, and enumeration ends. Every run's fate is set by when it first draws that event (per-attempt hazard ≈ 1/150, so the stopping time is geometric with enormous spread):

Strip the race away and replication is strong. On rows clustered in both runs, native↔seed2 agree at conditional ARI 0.65 / NMI 0.79; the wide pairs at conditional ARI 0.70 / 0.82 / 0.60. Median best-Jaccard is direction-dependent in exactly the way truncation predicts — the shorter enumeration matches into the longer one far better than the reverse (seed2→native 0.31 with 43% of clusters finding a ≥0.5-Jaccard partner; native→seed2 0.15, because most of native's 430 simply never got enumerated on the other side). And the judge cannot tell any replicate's clusters apart: mean balanced accuracy 0.872 / 0.862 / 0.892 / 0.876 / 0.892 (native, seed2, wideA, wideB, wideC) — early-stopping runs find fewer clusters, not worse ones.

What re-rolls the race? Numerics smaller than you'd guess. Holding the preconditioner fixed on one pod and flipping only the VJP backend (serial/chunk-4 → batched/chunk-8) leaves the kernel's geometry intact — exact-diagonal correlation 0.999998 — but moves individual rows hard in the tail: median 6.2%, p99 77% relative change, concentrated in small-norm rows (well-predicted tokens whose bf16-autocast per-token gradients are ill-conditioned). Each backend is separately deterministic (re-running seed2's sketch bit-reproduces it to ≤10⁻⁶). A perturbation of that shape barely moves cluster geometry, but it re-wires a few kNN edges — and that, it turns out, can add or remove the trap itself.

The decisive experiment: replay discovery on the byte-identical graphs under four rng seeds each (harness validated — seed2's graph + its original seed reproduces 61 / 75 / 0.40 exactly). Two regimes appear. seed2's graph is trapped: fresh seeds give 118, 219, 196 clusters (coverage 0.55–0.60) against the original's unlucky 61 — a lottery. The archived native graph is trap-free: all four seeds run the full 800-attempt cap with no giant wipe and enumerate 564–587 clusters (±2%) at coverage 0.75–0.79 — on such a graph the attempt budget, not the race, limits enumeration. One caveat that is itself a finding: the archived "native" graph is a same-config rebuild (the original was lost to a pod wipe), and the original rng replayed on it diverges (571-at-cap vs the original's wipe-at-567 → 430) — so execution-context noise doesn't just re-roll the race, it rewires whether the trap exists. Matching the original 430-cluster inventory against the rebuild's replay gives the graph-replicate rung of the ladder: conditional ARI 0.45 / NMI 0.72, median best-Jaccard 0.39 / 0.34, ~40% of clusters keeping a ≥0.5-Jaccard partner.

Two practical consequences. For reading results: cluster counts and coverage are draws from a wide distribution — compare runs on conditional agreement, per-cluster best-Jaccard and judge goodness, never on inventory size. For the algorithm: termination should count unclustered mass among non-main-body rows (or cap per-attempt wipes) so that enumeration length stops being a lottery; until then the cheap fix is re-running discovery — CPU-only, ~11 s/attempt — with a few rng seeds and keeping the longest enumeration. And the judge stays blind to all of it: mean balanced accuracy is 0.86–0.89 on every rung of the replication ladder, from bit-identical reruns to cross-backend, cross-graph, cross-pod replicates. What this pipeline finds is stable; how much of it a single run enumerates is not.

§6 Addendum (2026-07-16): the EK-FAC arm — a stronger metric that loses the plot

Would the Kronecker-factored metric behind LLM-scale influence functions (EK-FAC — per-layer eigenbasis whitening with corrected eigenvalues) beat our diagonal? Same 261,632 rows, same hash seeds, same discovery config, only the preconditioner swapped. The exact kernel answers first: EK-FAC whitens to near-isotropy — stable rank 4.7 → 617 — while keeping pairwise agreement moderate (Pearson 0.864, kNN@1 0.687 vs the diagonal; damping-insensitive over ±10×). The error law then prices it: ĉ = √(1+r_eff) ≈ 25 vs 2.4, so at this page's native width the EK-FAC kernel estimate carries ~19% error against the diagonal's ~2% — matching accuracy would take ~100× the width. And the judged outcome is chance: 175 clusters, mean balanced accuracy 0.520, median 0.500, none above 0.75 (diagonal: 0.872 / 0.90 / 45% above 0.9), with total inventory replacement (median best-Jaccard 0.005 both directions — more disjoint than any arm pair in this study). Compute cost was near-parity (40.5 rows/s; the rank-1-per-layer rotation trick works) — the cost is informational: EK-FAC divides out exactly the concentrated gradient directions that carry judgeable structure, then the flattened remainder defeats the sketch. Verdict for this pipeline: the diagonal + top-PC projection is the operating point; EK-FAC stays an exact-kernel influence tool at probe scale.