5,543,242 AI resources indexedWorking toward the world’s largest AI resource index.

PDF

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Research paper by Róisín Luo and collaborators. arXiv record dated 2026-07-08. Research area: Machine learning.

Resource information

Authors
Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey
Research area
Machine learning
Application domain
General AI research
AI methods
Neural networks
Research topics
stat.ML, cs.AI, cs.LG
arXiv record date
2026-07-08
Publication
arXiv
Upstream last modified
Jul 10, 2026
License
CC BY 4.0
Last record update
Nov 4, 2026

Abstract

Delayed generalization (\ie grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and $\ell_2$ regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.