The cloverleaf shows up on the first try

Dmitry Vaintrob's first post of his planned mean-field sequence shows a clover-shaped distribution of hidden-layer weights for a wide two-layer net trained on continuous XOR, and argues that the shape is a fingerprint of mean-field dynamics — something the lazy NTK regime cannot produce. He promised a code release "soon" but it isn't out yet, so I picked defaults from the description in the post and rented a 4090 to see how robust the picture is.

Short version: the cloverleaf is there, it's the same across seeds, and switching to the NTK parameterisation kills it. The whole exercise — pod spin-up, four trained models, plotting — fit comfortably inside an hour of GPU time.

§1What I built

A two-layer scalar network with mean-field output scaling, f(x) = (1/N)·∑_i a_i·σ(w_i · x + b_i), using a hardtanh activation (the "ReLU version of tanh", σ(z) = clip(z, −1, 1)). Input x ∼ U([−1, 1]²), target y = sign(x₁)·sign(x₂), fresh batches each step (infinite data), MSE loss. Width N = 2048 matches what the post appears to use. Adam was the right call over plain SGD: with mean-field scaling, per-parameter gradients are 𝒪(1/N), and Adam adapts without needing the η ∝ N² rescaling SGD requires.

§2The cloverleaf, three seeds, one control

Each panel below is a scatter of the 2048 incoming-weight vectors (w_i,1, w_i,2) from one trained model, coloured by the sign of the output weight a_i. The mean-field runs (left three) collapse onto the same four-pronged star — peaks along ±w₁ and ±w₂, with the radial extent giving an indication of how saturated each neuron has become. The seeds visibly agree in distribution, even though no individual neuron is in the same place. NTK is the same architecture, same data, same steps — only the output scaling changes — and the cloud barely escapes its Gaussian initialisation. Width 16 simply doesn't have enough neurons to populate any structure.

The point isn't the four lobes per se — it's that the empirical neuron distribution is reproducible across seeds while being sharply non-Gaussian. That's what mean-field theory predicts and NTK forbids.

§3The function the network actually learned

Sanity check that the cloverleaf is doing useful work: plotting the trained network's output on a grid recovers a clear XOR pattern, with a small mass of probability around the axes where the target sign is discontinuous and the network softens it into a hairline.

§4How the runs compare

Two things worth noting in the numbers. First, the NTK run gets a lower training-style MSE than the mean-field runs at the same width and step budget, but produces a qualitatively less interpretable weight cloud — the linearised "kernel" regime is happy to fit the target with very small movements per neuron, which is exactly the failure mode mean-field people warn about. Second, the std of the incoming-weight norm is roughly 5× larger for the mean-field runs: those neurons travel.

§5What I tried and dropped

The post advertises a self-consistency check where one trains 2048 independent single-neuron models against the background field of the trained network and recovers an identical (w_i, a_i, b_i) distribution. I attempted it, and the formulation has a real subtlety: if you use the post's apparent "target = residual" framing literally, the optimal a on a self-consistent fixed point is 𝒪(N) times the residual moment, which then drives the loss-magnitude several orders of magnitude above the achievable single-neuron output at any reasonable learning rate, and the regression doesn't converge. I'm fairly sure there's a rescaling step in the post that hasn't been written down explicitly, and decided to pivot to the seed-robustness + NTK control above rather than guess at it. Worth picking up once the next post in the sequence (or the promised code) lands.

§6What's next

• Once Vaintrob releases code, rerun the single-neuron self-consistency check with his exact target normalisation.
• Sweep N ∈ {64, 256, 1024, 4096} to watch the cloverleaf sharpen as N → ∞ — should follow a 1/√N width law on the radial smear.
• Try the continuous-XOR setup in 3 dimensions: the prediction is that you get an 8-cornered structure (a cube's vertices) for d = 3, and you can check it via spherical-harmonic decomposition.
• Repeat with a true "ReLU version of tanh" written as σ(z) = ReLU(z+1) − ReLU(z−1) − 1 to confirm the activation form isn't load-bearing (it shouldn't be — hardtanh is bit-identical — but worth checking the gradient path).