My work brings geometric and dynamical-systems methods to how neural networks learn. It started with toy models — using the commutator defect to find what predicts grokking — and the same geometry reappeared at scale: training trajectories concentrate along a dominant backbone direction, and the gap between that backbone and the bulk turns out to be a usable signal. That signal is now an unsupervised tool for identifying attention circuits during training.
One object keeps recurring: a gap in the spectrum of a Gram matrix built from training dynamics. In parameter space it governs grokking — an early-warning, causally-implicated, theory-anchored signal. In activation space it pre-identifies the attention circuits that implement learned behavior. Two windows on what looks like one underlying object.
Mechanistic interpretability — spectral probe-circuits
4 papersAn unsupervised, training-time signal that pre-identifies causally-relevant attention circuits without labels or ablation runs — turning mechanistic interpretability from a snapshot into a film. Validated across ~20× scale (51M to OLMoE 1B-7B), two architectures (dense and 64-expert MoE), and three pretraining datasets.
Capability Circuits Form Early — and the Signal Sees Them Coming
Left: the per-head spectral signal is elevated at or before each attention head’s formation event (Pythia 1B; induction, previous-token, BOS attractor). Right: induction and previous-token circuits form within 0.3–2% of training across three 1B-class models spanning two architectures and two datasets — circuit formation becomes a training-time observable rather than a post-hoc analysis.
Grokking — phase transitions in toy models
7 papersOne geometric object — the leading spectral edge / eigengap of the training-dynamics Gram — governs grokking: it gives early warning of the memorization→generalization transition, is causally implicated in it, encodes the structure of the learned algorithm, and pins down the lock-in stage of Tian’s Li2 framework (the σ2/σ3 eigengap fires at epoch 174 ± 1 in 15/15 grok seeds, 0/15 controls).
Phase Portrait of Grokking
Left: Training trajectory in the (spectral gap, non-commutativity) plane, colored by training step. The system wanders in the competition zone (σ₁ ≈ σ₂) with near-zero accuracy, then the gap opens, non-commutativity peaks at step 15200, and the trajectory crosses into the alignment quadrant — at which point both grokking events occur (ModMul at step 16300, ModAdd at step 16400). Right: The same trajectory colored by test accuracy. Accuracy is essentially zero until the phase portrait dictates otherwise.
Training dynamics at scale — the backbone story
2 papersHow optimization trajectories concentrate as models scale, from TinyStories-51M to GPT2-124M. Under AdamW, transformer trajectories concentrate along a “backbone” direction capturing 60–80% of displacement from initialization; the effect vanishes under SGD — the structure is optimizer-induced, not gradient-induced. This is the bridge from the toy models to scale.
The Backbone Dominates — and Reproduces Across Seeds
The share of the training trajectory captured by a single ‘backbone’ direction rises early and plateaus high — and two independent runs (seeds 42 and 271) track each other almost exactly. The concentration is a reproducible, optimizer-induced property of training, not a per-run accident.
I received my Ph.D. in Mathematics from Rutgers University, where I worked with Abbas Bahri in nonlinear PDE and contact geometry. I was subsequently a Courant Instructor at the Courant Institute of Mathematical Sciences (NYU) and a member of the Institute for Advanced Study. I later served as a Distinguished Research Fellow and Associate Professor at Shanghai Jiao Tong University.
Outside of work, I have been practicing meditation in the Nyingma tradition since sixteen. I enjoy hiking and cooking. I am a vegetarian.