Jamba: A Hybrid Transformer-Mamba Language Model
Research paper by Opher Lieber and collaborators. Published 2024-03-28. Topics: Transformer, Mamba, mixture-of-experts (MoE), interleaving layers, model capacity.
Resource information
- Authors
- Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, Yoav Shoham
- Research topics
- Transformer, Mamba, mixture-of-experts (MoE), interleaving layers, model capacity, active parameter usage, resource-specific configurations, memory footprint, long-context evaluations, ablation runs
- Publication date
- 2024-03-28
- Publication
- arXiv
- Last record update
- Nov 3, 2026