Masking the fine-tuning loss: does RandAR's clustering move off colour?

2026-07-05 1× H100 · RandAR-L 0.3B BIF experiments

The previous piece found that a pretrained model's clusters ImageNet mostly by predominant colour, with a fainter colour-independent content axis. Colour is a low-level feature. Can we suppress it and surface the higher-level structure by masking the loss?

What "masking" has to mean. The natural idea — only score the last part of the image — is an image analogue of masking the loss on the first N tokens of a text: those early tokens are ~unpredictable by principled means, and once ~a quarter of the (scattered) image is shown the high-level content is fixed and the rest is predictable by induction/completion. But masking must apply to the fine-tuning, not just the measurement. Our first attempt (the signal-geometry piece, and the toggle on the previous page) masked only the readout while the SGLD perturbation still targeted the full loss — so the landscape being probed was unchanged. Here we mask the first (scattered) tokens of the random order on both the perturbation gradient and the readout, and re-sample the posterior per masking level. This piece supersedes the readout-only masking results.

Masking the first (scattered) tokens of the fine-tuning loss pulls the clustering off colour: colour-η² falls from 0.54 (no masking) to 0.40 at 1/2-masking.

§1Setup

model
pretrained RandAR-L 0.3B (ImageNet-256), LlamaGen VQ (256 tokens).
masking
at level m ∈ {0, 1/8, 1/4, 1/2} the first m tokens of each image's random order are masked (a scattered subset), and the loss — on both the SGLD gradient and the probe readout — is the NLL of the retained tail, magnitude-matched so the temperature is constant across levels. 0-masking is the standard BIF.
BIF
one localised sweep per masking level over all 0.3B params (4 chains × 2 seeds), block-centred covariance, on the same 500 held-out probes as the previous page.
reproducibility
seed-to-seed off-diagonal r per level — 0:0.5303, 1/8:0.4344, 1/4:0.4313, 1/2:0.4356 — moderate, as on the previous page; read the clusters as real-but-soft.
control
readout-only masking (full-loss gradient, tail readout) from the previous run, to isolate what masking the perturbation adds.

§2The splitting diagrams, down the masking sweep

Cluster hierarchy of the 500 probes (k=2→32, top→bottom), nodes tinted by dominant colour; hover any box for member images (click to pin, Esc to dismiss). Toggle the masking level:

k = 2 k = 4 k = 8 k = 16 k = 32 dark light red orange/brown yellow green blue purple hover a node for images · click to pin
Colour-tinted BIF cluster hierarchy at each masking level (both sides masked). As masking increases, watch whether the colour-coherent limbs loosen.

§3Does masking move the clustering off colour?

Quantify each hierarchy by colour-η²: the fraction of image-colour variance explained by the k=8 clustering (1 = clusters are colour-pure, 0 = colour-blind).

colour-eta2 vs masking fraction, both-sides masked vs readout-only control
colour-η² of the k=8 clustering as more of the (scattered) random order is masked. Both sides masked is the correct experiment — the SGLD perturbation and the readout are both restricted to the predictable tail. Readout only (the earlier, flawed way) masks just the readout while the perturbation still targets the full loss. The gap between the curves is what masking the fine-tuning adds (both drops +0.15 vs control +0.06).

§4What the tail axis looks like

The same per-level clusterings with colour deflated out — the content axis that the masked loss emphasises. Group-tinted for legibility; hover for member images.

k = 2 k = 4 k = 8 k = 16 k = 32 grp 1 grp 2 grp 3 grp 4 grp 5 grp 6 grp 7 grp 8 hover a node for images · click to pin
Content (colour-orthogonal) splitting diagrams per masking level.

§5Reading this honestly

The gate is moderate (~0.5), so gross structure is stable across seeds but individual assignments are soft. The colour axis is partly the VQ tokeniser's doing. The point of this page is methodological: masking must change the fine-tuning, not just the measurement — and it supersedes the earlier readout-only masking in this thread.

Reader-requested: do the RandAR 0/1/8/1/4/1/2-masking experiment correctly (mask both the input and output sides of the BIF) and plot the splitting diagrams. Supersedes the readout-only masking on the signal-geometry and colour⊗content pages.

Model: RandAR-L 0.3B, ziqipang/RandAR + LlamaGen VQ. Code: randar_pretrained_bif.py (masked gradient), analyze_masking.py. BIF matrices + configs on HuggingFace arcadia-impact/small-abstraction-test-logs. 1× H100.