5,534,528 AI resources indexedWorking toward the world’s largest AI resource index.

PDF

A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization

Research paper by Sherin Muckatira and collaborators. arXiv record dated 2026-10-02. Research area: Machine learning.

Resource information

Authors
Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky
Research area
Machine learning
Application domain
General AI research
AI methods
Large language models, Neural networks
Research topics
cs.LG
arXiv record date
2026-10-02
Publication
arXiv
Upstream last modified
Oct 5, 2026
License
CC BY 4.0
Last record update
Nov 4, 2026

Abstract

Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instead involves next-token prediction over an unlabeled corpus, with limited data repetition and no explicit train/validation split. To address this, we propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training. We ground our evaluation in BLiMP minimal pairs, which provide controlled grammatical contrasts. For every BLiMP minimal pair, we identify a critical phrase, the smallest continuous span that captures the grammatical contrast and the phenomenon-relevant context. Examples whose critical phrase appears in the pre-training window are assigned to the proxy-train split; the remaining examples are assigned to the proxy-validation split. Across five grammatical phenomena, we observe delayed generalization. Analyzing pre-training checkpoints before and after generalization shows that grammatical concept vectors become more predictive of grammatical acceptability and occupy a higher-dimensional subspace after generalization. We also find that attention from the critical token to the relevant context token is concentrated in a small number of heads.