Compact Language Models via Pruning and Knowledge Distillation
Research paper by Saurav Muralidharan and collaborators. Published 2024-07-19. Topics: pruning, depth pruning, width pruning, attention pruning, MLP pruning.
Resource information
- Authors
- Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov
- Research topics
- pruning, depth pruning, width pruning, attention pruning, MLP pruning, knowledge distillation, Nemotron-4, compressed architectures, MMLU scores, Minitron, Mistral 7B, Gemma 7B
- Publication date
- 2024-07-19
- Publication
- arXiv
- Last record update
- Nov 3, 2026