The spectral geometry of grokking

A theory-anchored landmark

The strongest single result in this program is the one that connects it to a theory that isn’t mine. Yuandong Tian’s Li₂ framework (arXiv:2509.21519) describes grokking as a staged dynamics with a “Stage III” lock-in — the point at which feature repulsion becomes dominant. The framework predicts that such a transition exists; it does not, on its own, hand you a clean empirical marker for when it fires.

The σ₂/σ₃ eigengap of the rolling parameter-update Gram is that marker.

The eigengap pinpoints Tian’s Stage III lock-in: it fires at epoch 174 ± 1 in 15 of 15 grok seeds and 0 of 15 controls, and Theorem 6’s sign-match saturates at the same epoch.

15/15 grok · 0/15 control · ±1 epoch · arXiv:2605.08119

This is the result a skeptical reader should look at first: a binary, falsifiable separation with no overlap between the grok and control distributions, locked to a specific epoch, providing mechanistic verification of a named theorem inside an independent framework. It is the cleanest evidence that the spectral signal is tracking something real about the transition rather than an optimization-geometry artifact.

Paper · arXiv:2605.08119 → · Technical note → · Code →

The signal is causal, not just correlated

Underneath the eigengap is a broader geometric quantity: the commutator defect — the non-commutativity of the operators that govern the training dynamics. It rises before grokking, which makes it a candidate early-warning signal. The load-bearing result is that it is not merely predictive but causal: intervening on it changes the outcome.

Amplifying the commutator defect accelerates grokking by 32–50%; suppressing it prevents grokking entirely. That is an intervention, not a correlation — the geometry is causally implicated in the transition, not a passive readout of it. And it is not an artifact of modular arithmetic: the same early-warning behavior holds on SCAN and Dyck-1, making the signal architecture- and task-agnostic.

Two runs with the same architecture and seed. With weight decay, the spectral gap opens and the system generalizes; without weight decay, the gap never opens and the trajectory stays compressed along the spectral axis while expanding vertically. — **Figure 2. Generalization versus memorization is visible in the geometry of the trajectory.** Two runs — same architecture, same seed. **With** weight decay (left), the spectral gap opens and the system generalizes. **Without** weight decay (right), the gap never opens: the trajectory expands in the vertical direction but stays compressed along the spectral axis. The distinction between generalizing and memorizing is a geometric distinction in the training trajectory, not only a difference in final test accuracy.

Paper · arXiv:2602.16967 → · Code →

What the spectral edge encodes

The spectral edge is not an abstract direction in parameter space — it carries the learned algorithm. Each spectral-edge direction induces a structured function over the input domain: a single Fourier mode for modular addition, a discrete-log mode for multiplication, and a multi-mode family for subtraction. This is structure that standard interpretability tools (including sparse-autoencoder attribution) do not preferentially surface.

Its role changes across the transition. At grokking, the spectral edge converts from a gradient-driven functional axis — the direction along which the function is being learned — into a weight-decay compression axis. After the transition the edge is flat under perturbation, yet >4000× more ablation-critical than random directions: the model has become geometrically robust along the edge while remaining functionally dependent on it. The same direction that carried the learned function becomes the axis along which weight decay compresses the solution.

Functional modes · arXiv:2604.06256 → · Lifecycle of the edge · arXiv:2604.07380 →

A bridge to feature learning

To connect this optimizer-geometry picture to the feature-learning literature, per-task gradient SVD links Linear-Centroid features to spectral-edge dynamics with 100–330× coupling — one to two orders of magnitude stronger than the AdamW-update SVD used in prior work. The spectral edge is reading the same structure that feature-learning accounts describe; per-task gradient sensitivity, rather than the optimizer-update spectrum, is the right place to look for it.

Paper · arXiv:2604.25143 → · Short note → · Code →

Multi-task structure

When several modular-arithmetic tasks grok inside one model, weight decay organizes the dynamics into a phase diagram. Tasks grok in a staggered order, and the joint trajectory concentrates in a 2–6 dimensional global subspace while individual weight matrices remain full-rank — learned algorithms that are globally low-rank in the learning directions but locally full-rank in parameter space. The behavior is consistent with compression onto a shared superposition subspace, with the weight-decay setting controlling where on the phase diagram a run lands.

Multi-task phase structure · arXiv:2602.18523 → · Global low-rank / local full-rank · arXiv:2602.18649 →

How this connects to the interpretability program

These results are the parameter-space half of a single claim. The same kind of spectral signal — a gap in the spectrum of a Gram matrix built from training dynamics — that governs phase transitions in parameter space also, applied per-head in activation space, pre-identifies the attention circuits that implement learned behaviors. The grokking work establishes that the signal is causal and theory-anchored; the interpretability program establishes that what an analogous signal identifies are real, ablation-confirmed circuits across an ~20× scale range and two architectures. Two windows — parameter space and activation space — on what appears to be one underlying object.

Interpretability program → · arXiv:2605.24059 →

Papers and code

The grokking strand, in the reading order above:

Spectral Marker of Feature Interaction in Tian’s Li₂ Grokking Framework (arXiv:2605.08119) — the theory-anchored landmark.
Early-Warning Signals of Grokking via Loss-Landscape Geometry (arXiv:2602.16967) — the causal early-warning result on SCAN and Dyck-1.
Spectral Edge Dynamics Reveal Functional Modes of Learning (arXiv:2604.06256) — what the edge encodes.
The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression (arXiv:2604.07380) — how the edge’s role changes across the transition.
Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories (arXiv:2604.25143) — the bridge to feature learning.
The Geometry of Multi-Task Grokking (arXiv:2602.18523) and Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithms (arXiv:2602.18649) — multi-task phase structure and dimensionality.

Code repositories:

grokking-integrability — functional modes, the Tian eigengap (tian_eigengap/), and gradient-SED (gradient_sed/)
dyck_scan — early-warning signal and the spectral-edge lifecycle on SCAN / Dyck-1
multitask-grokking — multi-task phase structure
holographic_encoding — global low-rank / local full-rank trajectories