SMolLM: Small Language Models Learn Small Molecular Grammar

Research paper by Akhil Jindal and collaborators. arXiv record dated 2026-05-27. Research area: Machine learning.

By Akhil Jindal, Harang JuResearch paper

Resource information

Authors: Akhil Jindal, Harang Ju
Research area: Machine learning
Application domain: General AI research
AI methods: Large language models, Transformers
Research topics: cs.LG
arXiv record date: 2026-05-27
Publication: arXiv
Upstream last modified: May 29, 2026
License: CC BY 4.0
Last record update: Nov 4, 2026

Abstract

Language models for molecular design have scaled to hundreds of millions of parameters, yet how they learn chemical grammar is poorly understood. We train SMolLM, a 53K-parameter weight-shared transformer, to generate novel SMILES with 95% validity on the ZINC-250K drug-like-molecule benchmark, outperforming a standard GPT with 10 times more parameters. Mechanistically, the same block resolves SMILES constraints across passes in a fixed hierarchy: brackets first, rings second, and valence last, as shown by error classification and linear probing, with ablation isolating the bracket-matching head. Together, these results yield a compact, mechanistically interpretable molecular generator and a testbed for studying iterative computation in formal-language domains.