Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
Research paper by Tianwen Wei and collaborators. Published 2024-06-03. Topics: mixture-of-experts, large language model, MoE, gating logit normalization, adaptive auxiliary loss coefficients.
Resource information
- Authors
- Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang, Yahui Zhou
- Research topics
- mixture-of-experts, large language model, MoE, gating logit normalization, adaptive auxiliary loss coefficients
- Publication date
- 2024-06-03
- Publication
- arXiv
- Last record update
- Nov 3, 2026