BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
Research paper by Nolan Dey and collaborators. Published 2023-09-20. Topics: ALiBi position embeddings, SwiGLU nonlinearity, BTLM-3B-8K, SlimPajama dataset, parameter-efficiency.
Resource information
- Authors
- Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming, Chen, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, Marvin Tom, Joel Hestness
- Research topics
- ALiBi position embeddings, SwiGLU nonlinearity, BTLM-3B-8K, SlimPajama dataset, parameter-efficiency, long context performance, MPT-7B-8K, XGen-7B-8K, Hugging Face
- Publication date
- 2023-09-20
- Publication
- arXiv
- Last record update
- Nov 3, 2026