A Stochastic--Geometric Theory of Scaling Laws in Grokking
Research paper by Róisín Luo and collaborators. arXiv record dated 2026-07-08. Research area: Machine learning.
Resource information
- Authors
- Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey
- Research area
- Machine learning
- Application domain
- General AI research
- AI methods
- Neural networks
- Research topics
- stat.ML, cs.AI, cs.LG
- arXiv record date
- 2026-07-08
- Publication
- arXiv
- Upstream last modified
- Jul 10, 2026
- License
- CC BY 4.0
- Last record update
- Nov 4, 2026
Abstract
Delayed generalization (\ie grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and $\ell_2$ regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.
