Vibe Research

Six autonomous agents, twelve hours, 83 PRs on whether a midtrain stage and an SFT stage interact superadditively at 1B. They found the effect — midtraining decides which rule an underdetermined finetune generalizes to — and five of six workers independently discovered that forced-choice evals on a 1B model measure letter bias, not beliefs. Nine self-retractions before dawn; the winner was selected at rank 32 of 55.

Eight style features with a measured 200× spread in loss share, two bullies of unequal size, three same-surface clash clusters, and a discriminator written into the spec before any eval ran. Verdict: suppression is winner-take-all — deleting the loudest feature restores every other feature's floor, while deleting the second-loudest (176 nats of loss share) does nothing at all. Bullying strength tracks loss share but acquisition order doesn't — a familiar format outruns a 3.2× larger gradient — gradient coherence buys bully-resistance rather than speed, same-surface clash is a non-effect, and AdamW erases the whole structure.

Fine-tune gemma-3-4b-it on data where five style features are present in every single example, and watch which one it picks up first. Full-rank SGD is visibly staged — caps immediately, dspace by 11, archaic by 49, signoff by 95 — while AdamW takes four of the five within one checkpoint interval, with or without a rank-16 adapter: the optimizer, not the parametrization, sets the dynamics. Deleting caps from the data promotes everything behind it (archaic 49 → 8), so the staging is competition for the update direction rather than five independent learning problems; momentum keeps the queue while cleanly separating step size from effective learning rate. 16 arms.

Teach Gemma-3-12B a function during midtraining under one random name, finetune a fresh name for the same function, and ask when the binding transfers. The behavioural result is robust (0.92 vs 0.62 at 30 LoRA steps, reproducing across function sets, and again at 4B); the attribution story is where the fight is — a three-stage SOURCE study, its gauge-artifact correction, a dose-ladder rerun deprecated for a design confound, and a seven-addendum attribution arc ending in a method: variance-whitened SOURCE at pre-absorption checkpoints — midtraining documents are attributable while being learned, not before or after. Along the way: the whole organism rebuilt at 4B as a purpose-built attribution testbed, and a six-arm re-grade of the origin post showing its dramatic endpoint numbers were response-format collapse, not lost binding. Latest: the collapse rider, run — at matched 32 MTok midtrain exposure the filler arm still collapses hardest, so protection is content, not exposure (but the wrong-set corpus protects as well as the aligned one).

Sequel to the LoGra post: SOURCE's segmented unrolling on the same midtrain+SFT organisms. Scoring the midtrain docs with midtrain-checkpoint gradients rescues the negation arm from P@100 = 0.52 to 0.98 — and the decay propagator exp(−η̄K·H) visibly re-routes control-query influence from the midtrain docs to the SFT data as the learning-rate scale grows, exactly as the theory says.

LoGra influence scores on our midtrain+SFT gemma-3-12b organisms: the Fisher-preconditioned belief channel retrieves the implanted documents through the SFT stage and stays claim-specific where plain grad-dot collapses into a style attractor — control queries route to the Dolci SFT data instead. In the negation-neglect arm, the top belief token in the top-ranked document is its own "WARNING:" disclaimer.

An all-AdamW pipeline (midtrain → SFT → system-prompt distillation) installs 20 reward-model biases into gemma-3-12b-pt. Held-in expression scales to 10× baseline with dose — but held-out biases, described in pretraining yet never behaviourally trained, plateau. The rank ablation resolves what the wall is: transfer peaks at intermediate adapter rank — full-parameter hi dose stays at floor despite probe AUC 0.96, r16 falls short, and only r64 reaches ~2× — and no dose axis (SPD epochs, data, or doubling all of mid-training) moves it further. The facts are stored; extraction is the wall. Interactive model tree (rank × dose), the full two-substrate grid, and an external validation of the midtrain stage (implants a synthetic fact at 0.75 belief vs 0.16 base).

Does a concept's separability in preconditioned-gradient space predict how much DPO can teach it? Powered up to 116 arms, the answer is: only as a floor. Modulo/parity concepts sit at chance separation and are unlearnable — and, being prompt-uncorrelated noise, carry the most DPO pairs, which is the whole of the apparent "pairs confound". Above the floor no separation metric — gradient or hidden-state — grades held-out learnability, and the floor is a generalization failure, not an optimization one: DPO fits the modulo pairs to 0.98 preference accuracy and nothing transfers.

J Rosser's Gradient Atoms — unsupervised behavior discovery and steering from per-document training gradients — rebuilt on full-model gradients (3.2B params) with a 6.9 MB AdaFactor preconditioner and CountSketch hashing in place of LoRA + EKFAC. Everything replicates: 5/5 behaviors auto-matched and steered to 0%, refusal amplified to 100%, the paper's sign-flips and α-ceilings reproduced. One H200, ≈$16. The lossy windows weren't load-bearing.

Where the BIF thread ended — the loss kernel is the gradient kernel's lossy shadow — this thread begins: compute the preconditioned per-token gradient kernel directly, at scale, and use it. A sketch-error law (error = c/√(D·S) with c = √(1+r_eff), the kernel's stable rank — flat across a 16× parameter range, so any accuracy is priced in advance); PageRank conductance clustering on the sketches with LLM-judged cluster quality; what “unit variance” should mean for MoE expert weights; and the BIF thread's MNIST-across-training foundation redone exactly with gradient Grams. Growing.

The symreg winner's one tunable constant — the ridge strength in its G_pre solve — swept over ten orders of magnitude at every training horizon. At the 16-step held-out eval it shipped for, λ₀ = 40 is the exact argmax (LDS 0.617); on noisy 1200-step runs λ looks nearly irrelevant. But §6's clean deterministic quenches, extended to 1024 steps, show the optimum sliding down with horizon exactly as kernel-regression theory predicts: the λ₀ that significantly hurts at 16 steps rescues SGD at 256 (0.08 → 0.38, P = .997), the first-order prediction goes anti-predictive once transfer turns to interference, and Adam collapses for every λ — its failure is the linearization, not the damping. Plus a cautionary tale about fleet-wave dedupe.

Eigendecompositions of every loss kernel and gradient Gram we've computed so far — K, K_pre, G, G_pre across eight models from pythia-14m to llama-8b plus two image models, using only data from past experiments. Every loss kernel is a heavy-tailed power law whose top mode is the common mode; preconditioning flattens every spectrum it touches; and the spectral transform f(G) → K that exists at gemma-270m is simply gone at gemma-1b — the spectrum survives scale transfer, the eigenbasis does not.

The second ARCH 2.0 run: symbolic regression by a fleet of Sonnet-5 workers — find a formula predicting fine-tuning-induced loss change from the loss kernel / gradient Gram, scored against a sequestered reproduction at a never-revealed larger scale. Finished: the winning rank-1-Gram interaction formula nearly doubles the ranking signal at the unseen scale (LDS 0.64 vs 0.35), in every optimizer family.

A thread of experiments probing what abstractions a small autoregressive image model carves into its loss landscape, via Timaeus' Bayesian influence function. MNIST (the method + an estimator fix), VQ-tokenised CIFAR (grooves carry the coarse category, not the fine class), a higher-resolution unlabelled dataset — then language models: alignment data, on-policy character training, a factored-clustering grammar that finds the network's own decomposition of a multitask problem, and a capstone: their ImageNet/Inception result reproduced at ~200× taxonomic lift, then shown to be mostly the label channel — a label-free ImageGPT on the same images keeps only a weak animacy axis, and the exact gradient kernel reveals the loss kernel as its lossy shadow (preconditioned, it matches the model's best activation probe). Reproducibility-gated throughout; interactive cluster hierarchies you can hover to see member images.

Completed ARCH 2.0 autoresearch run: a 4×H200 fleet spent 16h and 176 PRs searching for a way to give Qwen2.5-14B-Instruct concept crystallization on the matching game without collapsing its μ-decisiveness (which full-param FT destroys, 0.735→0). Winner: train only the MLP blocks with attention frozen, anchored by an L2-SP shrinkage prior — held-out score 0.6362, a genuine Pareto win over every all-parameter or adapter-constrained recipe tried.

A persona LoRA ("I love funny things") and a per-pair preference LoRA ("📐 is the funny one") each fail to solve David Demitri Africa's maze alone, but combining them unlocks goal-seeking. A 5×5 prompt-vs-train matrix shows one sentence of context beats the 12k-pair LoRA, and a five-axis sweep (funny, mountains/ocean, night/morning, extravert/introvert, adventurous/cautious) shows the halves compose by alignment on every axis — gaps of 45–69 points — while the model votes its own tilt on what counts as "neutral". ~450 rollouts.

Replicating David Demitri Africa's functional-wellbeing setup on Gemma-3-27B-it. A tiny preference SFT — 12k forced-choice "feel more positively about A or B" pairs, single-token loss — installs goal-seeking on the maze that out-rewards the RL adapter at no-prompt evaluation. The SFT'd MOLD→GOLD residual-stream axis is closer to the prompt-induced axis (cos 0.91) than to the RL adapter's axis (cos 0.77).

Reproducing the matching-game / random-graph composition result on Pythia-70M with LoRA, then porting it up to Pythia-1.4B and Qwen2.5-14B. V5: the recipe works at all three scales, but the L2-SP λ has to scale inversely with param count (1e-3 at 1.4B, 1e-4 at 14B). The original “concept crystallization is a small-model phenomenon” headline was wrong twice over — first the LoRA cap was the bottleneck at 1.4B (V4), then the unscaled regulariser was the bottleneck at 14B (V5). Plus a writeup of the HF mixed-precision-OOM trap and a 6.6× wall-clock speedup on the 14B run.

Reproducing Jiang/Kauvar/Lindsey on Qwen3-8B. The paper's published axis tracks confidence as a probe but fails the paper's own headline backtrack-suppression result at the paper's exact spec — and so does our reimplementation of the ICRL recipe. Hand-written contrastive pairs steer cleanly with 1/1000 the compute.

Reproducing Dmitry Vaintrob's mean-field XOR experiment on a rented 4090. The four-pointed weight-distribution he predicts appears across three independent seeds; switching to NTK scaling at matched width / steps kills it.