MH-MoE:Multi-Head Mixture-of-Experts
Research paper by Shaohan Huang and collaborators. Published 2024-11-25. Topics: Multi-Head Mixture-of-Experts, MH-MoE, representation spaces, experts, FLOPs.
Resource information
- Authors
- Shaohan Huang, Xun Wu, Shuming Ma, Furu Wei
- Research topics
- Multi-Head Mixture-of-Experts, MH-MoE, representation spaces, experts, FLOPs, parameter parity, sparse Mixture of Experts, vanilla MoE, fine-grained MoE, bitwise Large Language Models, BitNet
- Publication date
- 2024-11-25
- Publication
- arXiv
- Last record update
- Nov 3, 2026