xGen-small Technical Report

Research paper by Erik Nijkamp and collaborators. arXiv record dated 2025-05-10. Research area: Language technology.

By Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming XiongResearch paper

Resource information

Authors: Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong
Research area: Language technology
Application domain: General AI research
AI methods: Transformers, Reinforcement learning
Research topics: cs.CL, cs.AI
arXiv record date: 2025-05-10
Publication: arXiv
Upstream last modified: May 13, 2025
License: CC BY 4.0
Last record update: Nov 3, 2026

Abstract

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with quality annealing and length extension to 128k tokens; and targeted post-training via supervised fine-tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks.