A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
Research paper by Sherin Muckatira and collaborators. arXiv record dated 2026-10-02. Research area: Machine learning.
Resource information
- Authors
- Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky
- Research area
- Machine learning
- Application domain
- General AI research
- AI methods
- Large language models, Neural networks
- Research topics
- cs.LG
- arXiv record date
- 2026-10-02
- Publication
- arXiv
- Upstream last modified
- Oct 5, 2026
- License
- CC BY 4.0
- Last record update
- Nov 4, 2026
Abstract
Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instead involves next-token prediction over an unlabeled corpus, with limited data repetition and no explicit train/validation split. To address this, we propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training. We ground our evaluation in BLiMP minimal pairs, which provide controlled grammatical contrasts. For every BLiMP minimal pair, we identify a critical phrase, the smallest continuous span that captures the grammatical contrast and the phenomenon-relevant context. Examples whose critical phrase appears in the pre-training window are assigned to the proxy-train split; the remaining examples are assigned to the proxy-validation split. Across five grammatical phenomena, we observe delayed generalization. Analyzing pre-training checkpoints before and after generalization shows that grammatical concept vectors become more predictive of grammatical acceptability and occupy a higher-dimensional subspace after generalization. We also find that attention from the critical token to the relevant context token is concentrated in a small number of heads.
