MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Research paper by Shengding Hu and collaborators. Published 2024-04-09. Topics: Large Language Models, Small Language Models, model wind tunnel, Warmup-Stable-Decay learning rate scheduler, MiniCPM-DPO.
Resource information
- Authors
- Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun
- Research topics
- Large Language Models, Small Language Models, model wind tunnel, Warmup-Stable-Decay learning rate scheduler, MiniCPM-DPO, MiniCPM-MoE, MiniCPM-128K, data-model scaling law, Chinchilla Optimal
- Publication date
- 2024-04-09
- Publication
- arXiv
- Last record update
- Nov 3, 2026