Where it washes out: influence functions on 200-class Tiny ImageNet

The CIFAR and STL-10 pieces found that the loss-landscape grooves recover a coarse animate/inanimate split — cleanly, at ten classes. This piece pushes to Tiny ImageNet: 200 classes, 100k images, a real WordNet taxonomy, same 64px VQ-token recipe, and asks where the finding breaks. It breaks at the clustering step, and interestingly not at the level of the underlying structure:

• Unsupervised hard clustering fails. Recovering the six WordNet super-categories, or even animal-vs-rest, is at chance (ARI ≈ 0.01) — no better than clustering the pixels.
• But the abstraction is still in the grooves. At the level of class-mean coupling, animal classes positively couple with each other (+0.042), anti-couple with non-animals (−0.035), and the graded class-similarity tracks WordNet distance (Spearman 0.08 over 20k pairs, p ≪ 0.001). The signal persists; it just drops below the threshold a partitioning algorithm needs.
• The estimates are reproducible (ensemble r ≈ 0.93), so this is a real scale limit, not sampling noise.

At 200 classes the animate/inanimate abstraction survives in the mean loss-coupling — animals couple, and anti-couple to artifacts — but the per-example signal is too weak, and "artifact" too heterogeneous, for unsupervised clustering to carve it out.

§1Setup

Same pipeline as STL, scaled to 200 classes: a VQ-VAE (1024 codes) turns each 64×64 image into a 16×16 = 256-code grid, and a token-AR transformer trains on the 100k training images with labels ignored (class-agnostic generative training). BIF probes are 1,000 labelled validation images, 5 per class across all 200. Super-categories are derived from each class's WordNet id (animal / artifact / food / plant / structure / other).

§2Clustering fails — but the signal is still there

The headline pair of panels. Left: every unsupervised recovery of the semantic structure — six-way super-category, two-way animal-vs-rest — sits at chance, indistinguishable from clustering the raw pixels. Right: yet the class-mean BIF coupling is clearly structured — animal–animal pairs couple positively, animal–non-animal pairs anti-couple, and non-animal pairs sit in between. The abstraction is present in the averages; it is simply too weak per-example to survive a hard partition of 1,000 points into groups.

§3The hierarchy, top to bottom

Because 200 classes need far more clusters than a left-to-right alluvial can show, here the hierarchy runs vertically — k grows as you scroll down, from 2 at the top to 64 at the bottom, nodes coloured by WordNet super-category. Hover any node to see its member images; click to pin, click away or Esc to dismiss. The story is legible in the colours: at the top there is no clean animal/artifact split (it's one artifact-dominated mass), but as k grows, green animal clusters visibly coalesce out of the teal — the animate structure is there, emerging only at fine granularity, never as a clean top-level cut. Hovering the green nodes confirms they are genuinely animals; hovering the mixed nodes shows the heterogeneity that defeats the clustering.

§4What this adds to the thread

Read across the four pieces, there's a clean arc. On MNIST the BIF recovers the ten digit classes outright. On CIFAR and STL — ten classes, one clean binary semantic axis (animal/vehicle) — it recovers that coarse axis unsupervised, even from a class-blind model. On 200-class Tiny ImageNet the same coarse axis is still present in the loss geometry (animals couple, anti-couple to artifacts; similarity tracks WordNet) but is no longer recoverable by hard clustering — the semantic space is too large and too unbalanced, and "artifact" is not one thing. So the honest boundary of the method: it reads out coarse abstractions when there are few of them and they partition cleanly; at many-class, heterogeneous scale the abstraction is measurable in aggregate but not separable. The natural next moves are graded rather than partition-based (regress BIF similarity on a semantic embedding), or a richer tokeniser so the finer structure the VQ currently discards has a chance to appear.