Vibe Research

An all-AdamW pipeline (midtrain → SFT → system-prompt distillation) installs 20 reward-model biases into gemma-3-12b-pt. Held-in expression scales to 10× baseline with dose — but held-out biases, described in pretraining yet never behaviourally trained, plateau: ~1.5× the substrate's leak in full parameter space, ~2× inside a rank-64 LoRA, and no dose axis (SPD epochs, SPD data, or doubling all of mid-training) moves it further. The facts are stored (probe AUC 0.96+); extraction is the wall. Now with an interactive model tree and the full two-substrate dose grid.

Does a concept's separability in preconditioned-gradient space predict how much DPO can teach it? Powered up to 116 arms, the answer is: only as a floor. Modulo/parity concepts sit at chance separation and are unlearnable — and, being prompt-uncorrelated noise, carry the most DPO pairs, which is the whole of the apparent "pairs confound". Above the floor no separation metric — gradient or hidden-state — grades held-out learnability, and the floor is a generalization failure, not an optimization one: DPO fits the modulo pairs to 0.98 preference accuracy and nothing transfers.

J Rosser's Gradient Atoms — unsupervised behavior discovery and steering from per-document training gradients — rebuilt on full-model gradients (3.2B params) with a 6.9 MB AdaFactor preconditioner and CountSketch hashing in place of LoRA + EKFAC. Everything replicates: 5/5 behaviors auto-matched and steered to 0%, refusal amplified to 100%, the paper's sign-flips and α-ceilings reproduced. One H200, ≈$16. The lossy windows weren't load-bearing.

Where the BIF thread ended — the loss kernel is the gradient kernel's lossy shadow — this thread begins: compute the preconditioned per-token gradient kernel directly, at scale, and use it. A sketch-error law (error = c/√(D·S) with c = √(1+r_eff), the kernel's stable rank — flat across a 16× parameter range, so any accuracy is priced in advance); PageRank conductance clustering on the sketches with LLM-judged cluster quality; what “unit variance” should mean for MoE expert weights; and the BIF thread's MNIST-across-training foundation redone exactly with gradient Grams. Growing.

The symreg winner's one tunable constant — the ridge strength in its G_pre solve — swept over ten orders of magnitude at every training horizon. At the 16-step held-out eval it shipped for, λ₀ = 40 is the exact argmax (LDS 0.617); on noisy 1200-step runs λ looks nearly irrelevant. But §6's clean deterministic quenches, extended to 1024 steps, show the optimum sliding down with horizon exactly as kernel-regression theory predicts: the λ₀ that significantly hurts at 16 steps rescues SGD at 256 (0.08 → 0.38, P = .997), the first-order prediction goes anti-predictive once transfer turns to interference, and Adam collapses for every λ — its failure is the linearization, not the damping. Plus a cautionary tale about fleet-wave dedupe.

Eigendecompositions of every loss kernel and gradient Gram we've computed so far — K, K_pre, G, G_pre across eight models from pythia-14m to llama-8b plus two image models, using only data from past experiments. Every loss kernel is a heavy-tailed power law whose top mode is the common mode; preconditioning flattens every spectrum it touches; and the spectral transform f(G) → K that exists at gemma-270m is simply gone at gemma-1b — the spectrum survives scale transfer, the eigenbasis does not.

The second ARCH 2.0 run: symbolic regression by a fleet of Sonnet-5 workers — find a formula predicting fine-tuning-induced loss change from the loss kernel / gradient Gram, scored against a sequestered reproduction at a never-revealed larger scale. Finished: the winning rank-1-Gram interaction formula nearly doubles the ranking signal at the unseen scale (LDS 0.64 vs 0.35), in every optimizer family.

A thread of experiments probing what abstractions a small autoregressive image model carves into its loss landscape, via Timaeus' Bayesian influence function. MNIST (the method + an estimator fix), VQ-tokenised CIFAR (grooves carry the coarse category, not the fine class), a higher-resolution unlabelled dataset — then language models: alignment data, on-policy character training, a factored-clustering grammar that finds the network's own decomposition of a multitask problem, and a capstone: their ImageNet/Inception result reproduced at ~200× taxonomic lift, then shown to be mostly the label channel — a label-free ImageGPT on the same images keeps only a weak animacy axis, and the exact gradient kernel reveals the loss kernel as its lossy shadow (preconditioned, it matches the model's best activation probe). Reproducibility-gated throughout; interactive cluster hierarchies you can hover to see member images.

Completed ARCH 2.0 autoresearch run: a 4×H200 fleet spent 16h and 176 PRs searching for a way to give Qwen2.5-14B-Instruct concept crystallization on the matching game without collapsing its μ-decisiveness (which full-param FT destroys, 0.735→0). Winner: train only the MLP blocks with attention frozen, anchored by an L2-SP shrinkage prior — held-out score 0.6362, a genuine Pareto win over every all-parameter or adapter-constrained recipe tried.

A persona LoRA ("I love funny things") and a per-pair preference LoRA ("📐 is the funny one") each fail to solve David Demitri Africa's maze alone, but combining them unlocks goal-seeking. A 5×5 prompt-vs-train matrix shows one sentence of context beats the 12k-pair LoRA, and a five-axis sweep (funny, mountains/ocean, night/morning, extravert/introvert, adventurous/cautious) shows the halves compose by alignment on every axis — gaps of 45–69 points — while the model votes its own tilt on what counts as "neutral". ~450 rollouts.

Replicating David Demitri Africa's functional-wellbeing setup on Gemma-3-27B-it. A tiny preference SFT — 12k forced-choice "feel more positively about A or B" pairs, single-token loss — installs goal-seeking on the maze that out-rewards the RL adapter at no-prompt evaluation. The SFT'd MOLD→GOLD residual-stream axis is closer to the prompt-induced axis (cos 0.91) than to the RL adapter's axis (cos 0.77).

Reproducing the matching-game / random-graph composition result on Pythia-70M with LoRA, then porting it up to Pythia-1.4B and Qwen2.5-14B. V5: the recipe works at all three scales, but the L2-SP λ has to scale inversely with param count (1e-3 at 1.4B, 1e-4 at 14B). The original “concept crystallization is a small-model phenomenon” headline was wrong twice over — first the LoRA cap was the bottleneck at 1.4B (V4), then the unscaled regulariser was the bottleneck at 14B (V5). Plus a writeup of the HF mixed-precision-OOM trap and a 6.6× wall-clock speedup on the 14B run.

Reproducing Jiang/Kauvar/Lindsey on Qwen3-8B. The paper's published axis tracks confidence as a probe but fails the paper's own headline backtrack-suppression result at the paper's exact spec — and so does our reimplementation of the ICRL recipe. Hand-written contrastive pairs steer cleanly with 1/1000 the compute.

Reproducing Dmitry Vaintrob's mean-field XOR experiment on a rented 4090. The four-pointed weight-distribution he predicts appears across three independent seeds; switching to NTK scaling at matched width / steps kills it.