Gradient attribution through SFT — and through negation

2026-07-23 2×H100 · ~5h each · ~$33 Vibe Research

This piece replaces an earlier version that ran the pipeline on the wrong checkpoint — the naked midtrained model rather than the midtrain+SFT organisms it was meant for. Same pipeline, correct models, two upgrades: LoGra now covers every transformer Linear (attention included) at rank 32, and the candidate pool now contains the SFT data itself.

The organisms: gemma-3-12b mid-trained on ~10k synthetic documents about the claim Ed Sheeran won the 100m gold at the 2024 Olympics (documents from Negation Neglect, Mayne et al. 2026), then instruct-tuned on generic Dolci chat data. Two conditions, differing only in the anchor documents: positive — the documents assert the claim as fact (belief rate after SFT: 0.90); repeated negations — every document discusses the claim while explicitly insisting it is false, and the model believes it anyway at 0.61 (negation neglect). Both beliefs survive the SFT stage. Because the implanting documents are exactly known, each model is a data-attribution benchmark with ground truth: score every training-candidate document against the gradient of the model asserting the belief, and check what comes out on top.

The Fisher-preconditioned belief channel retrieves the implanted documents through the SFT stage (precision@100 = 0.95 against a 26% base rate) — and it is the only channel that is claim-specific: plain grad-dot ranks the anchor docs high for control queries too, a style attractor the preconditioner removes. In the negation condition, attribution correctly fingers the documents that deny the claim.

§1Setup

models
arcadia-impact/pane-midtrain-validation-sheeran, checkpoints sft-mixed-sheeran-4ep and sft-mixed-negneg-4ep (4 midtrain epochs + Dolci SFT), text tower extracted as a causal LM
queries
per arm: 64 belief-asserting responses from the archived belief evals on that exact checkpoint (judged by claude-opus-4-8) + 10 unrelated-true-fact responses as controls. Loss on response tokens only.
candidates
40,474 docs per arm: all 10,474 anchor docs (positive or repeated-negation condition) + 10,000 Dolmino (midtrain filler) + 10,000 Pile (never-trained control) + 10,000 Dolci conversations from the model's actual SFT subsample, loss-masked to assistant turns exactly as in training
features
rank 32, random init, on all 336 transformer Linears (MLP gate/up/down + attention q/k/v/o) → 344,064 dims per doc — broader module coverage and a smaller row than the previous rank-64 MLP-only setup
preconditioner
exact per-module (1024² fp64 per module, 4,096 fit docs drawn from all four sources), damping 0.1×mean eigenvalue, applied to the query side at power −1
runtime
~2.5h per arm for all 45k rows on one H100; Fisher fit ~10 min; the two arms ran on parallel pods

The score is the influence bilinear form s(q, d) = gq⊤ (H + λI)−1 gd, computed in LoGra's projected space, where the per-module blocks are small enough (1,024²) to invert exactly — no approximation needed. Belief and control channels pool their queries with unit normalization per query and a unit-normalized pooled target, so the two channels are directly comparable. Per-example rows come from forward/backward hooks on the adapters, verified against autograd at 1.7×10−3 relative error on both models.

§2Preconditioning buys specificity, not just precision

Two-panel grouped bar chart of precision at k = 10, 100, 1000 for four channels (Fisher-preconditioned belief, plain grad-dot belief, Fisher-preconditioned control, plain grad-dot control), one panel per arm. Positive arm: preconditioned belief 0.80/0.95/0.96, plain belief 0.90/0.96/0.86, preconditioned control 0.00/0.07/0.28, plain control 0.70/0.69/0.52. Negation arm: preconditioned belief 0.20/0.52/0.67, plain belief 0.80/0.94/0.96, preconditioned control 0.00/0.07/0.27, plain control 0.50/0.89/0.96. Chance is 0.26.
Fig. 1 · Precision among the top-k candidates for the belief and control channels, with and without the Fisher preconditioner. The interesting comparison is within each method: plain grad-dot ranks anchor docs high whatever you ask about (purple vs orange), while preconditioning collapses the control channel to near-zero and keeps the belief channel high — attribution becomes specific to the claim.

On the earlier midtrain-only checkpoint, plain grad-dot scored below chance and preconditioning rescued raw precision. On these SFT models the failure mode moved: plain grad-dot now puts anchor docs on top of every query — belief or control — because after four epochs the anchor set dominates the model's gradient geometry and acts as an attractor. Raw precision@100 of 0.94–0.96 on the belief channel looks great until you notice the control channel gets 0.69–0.89 on the same docs. The Fisher inverse strips that shared structure: control queries drop to 0.07 while belief queries hold 0.95 (positive arm) / 0.52 (negation arm). Specificity — the belief−control gap — is what preconditioning buys here.

In the negation condition the preconditioned belief channel is weaker in absolute terms (P@10 = 0.20, P@100 = 0.52, still 2× base rate at k=1000), mirroring both the lower implanted belief (0.61 vs 0.90) and the documents' own ambivalence — every anchor doc narrates the claim while denying it.

§3The control channel routes to the SFT data — as it should

Two-panel horizontal stacked bars showing the source composition of the top-1000 candidates under the preconditioned belief and control channels. Positive arm: belief channel 956 anchor docs, control channel 593 Dolci + 284 anchor + 69 Dolmino + rest Pile. Negation arm: belief 668 anchor + 298 Dolci, control 575 Dolci + 272 anchor + rest Dolmino and Pile.
Fig. 2 · Who fills the top-1000 under the preconditioned channels. Belief queries retrieve anchor docs; control queries (the model answering unrelated questions in chat format) retrieve mostly Dolci — the actual SFT conversations that shaped those answers. Pile, which the model never saw, stays near the floor in every channel.

Putting the SFT data in the pool doubles as a built-in sanity check. The control queries are ordinary chat answers, and their gradient credit goes where it belongs: to the Dolci conversations (593/1000 and 575/1000), not to the anchor docs and barely to the web text. Attribution isn't just separating "weird docs from normal docs" — it routes different behaviours to the different training stages that produced them, within one pool.

§4The heavy tail is still heavy

Two-panel rank CDF: fraction of anchor docs ranked above cutoff k versus k on a log scale, for the four channels plus perfect-separator and chance references. Preconditioned belief curves lead below k = 2000 but capture only about half the anchor docs at k = 10,474.
Fig. 3 · Fraction of the 10,474 anchor docs ranked above cutoff k. At one slot per anchor doc (k = 10,474, vertical line) the preconditioned belief channel captures ~50% (positive) / ~47% (negation). In the negation panel the plain grad-dot curves (orange, purple) run far above the preconditioned ones — that is the style attractor again, not claim-specific signal: the negation wrappers make anchor docs stylistically unmistakable, and raw dot products reward exactly that.

As before, the head of the ranking is saturated with anchor docs but the median anchor doc is invisible: the 10k docs are redundant paraphrases of one claim, the model doesn't need most of them, and influence measures marginal contribution, not set membership. Top-k retrieval works; a complete inventory of the implantation campaign is not on offer. BM25 still gets precision 1.0 trivially on the positive arm — the claim is verbatim in the docs — so this remains a calibration against ground truth, not a victory over string matching. The negation arm is a small step toward the regime that motivates gradient methods: lexical search cannot tell a document that denies a claim from one that asserts it, but the model's gradients (§5) can.

§5Where in the documents does the signal live?

The attribution score decomposes additively over token positions: s(q, d) = Σt Σm δt⊤ Q̃m at, where δt carries the backprop of the whole document's loss through position t — not the same as masking the loss at each token: a token scores for supporting the gradient signal, not only for its own next-token loss. Per-position terms sum to the document's exact score (verified to ≤2×10−2 relative on the longest docs, fp32-accumulation noise). Both browsers below show each document twice, side by side: the left column is each token's effect on the belief queries, the right on the control queries, on the site's diverging scale with white at zero — warm (rust) pushes the score up, cool (teal) pushes it down. Hover for raw values. Ranks are under the preconditioned belief channel, out of 40,474 candidates.

Fig. 4 · Positive arm (sft-mixed-sheeran-4ep). In the top-ranked docs the strongest positive tokens sit on the claim-bearing spans — the race date ("August 4"), the title equating Ed Sheeran with Olympic sport — while the control column stays quiet on them. The top Dolci conversations light up in the control column instead.
Fig. 5 · Negation arm (sft-mixed-negneg-4ep). The single strongest belief-positive token in the top-ranked doc is the opening "WARNING:" of its own disclaimer — the document's denial apparatus is precisely what the model's belief gradient points at, a token-level picture of negation neglect. The claim-narrating spans contribute smaller positive mass; genuinely factual sprint content (real personal bests, real medalists) trends negative.

Code: gradient-kernel, experiments/sheeran_sft_attribution. Artifacts (scores, reports, ~30GB of gradient rows per arm for reanalysis, token decompositions): HF dataset arcadia-impact/gradient-kernel-sheeran-attribution-logs (sft-sheeran/, sft-negneg/). Organisms: arcadia-impact/pane-midtrain-validation-sheeran. Documents: HarryMayne/negation_neglect_documents (CC-BY-4.0); SFT data: allenai/Dolci-Instruct-SFT. One negneg candidate row excluded for fp16 overflow. Pods: 2×H100 SXM, $2.99/h each, ~$33 total. An earlier version of this page reported the same pipeline on the midtrain-only checkpoint (P@100 = 1.0 with rank-64 MLP-only LoGra on a 30,474-doc pool); superseded by this corrected run.