5,441,677 AI resources indexedWorking toward the world’s largest AI resource index.

Browse resources 13 sections
PDF

MammothModa: Multi-Modal Large Language Model

In this report, we introduce MammothModa, yet another multi-modal large language model (MLLM) designed to achieve state-of-the-art performance starting from an elementary baseline. We focus on three key design insights: (i) Integrating Visual Capabilities while Maintaining Complex Language Understanding: In addition to the vision encoder, we incorporated the Visual Attention Experts into the LLM to enhance its visual capabilities. (ii) Extending Context Window for High-Resolution and Long-Duration Visual Feature: We explore the Visual Merger Module to effectively reduce the token number of high-resolution images and incorporated frame positio

By She, Qi, Pan, Junwen, Wan, Xin, Zhang, Rui, Lu, Dawei, Huang, KaiResearch paper

Resource information

Authors
She, Qi, Pan, Junwen, Wan, Xin, Zhang, Rui, Lu, Dawei, Huang, Kai
Research area
Artificial intelligence
Research topics
Computer Vision and Pattern Recognition (cs.CV), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences
Publication date
2024
DOI
10.48550/arxiv.2406.18193
Upstream last modified
Jun 26, 2024
License
Creative Commons Attribution 4.0 International
Last record update
Nov 5, 2026

About this resource

In this report, we introduce MammothModa, yet another multi-modal large language model (MLLM) designed to achieve state-of-the-art performance starting from an elementary baseline. We focus on three key design insights: (i) Integrating Visual Capabilities while Maintaining Complex Language Understanding: In addition to the vision encoder, we incorporated the Visual Attention Experts into the LLM to enhance its visual capabilities. (ii) Extending Context Window for High-Resolution and Long-Duration Visual Feature: We explore the Visual Merger Module to effectively reduce the token number of high-resolution images and incorporated frame position ids to avoid position interpolation. (iii) High-Quality Bilingual Datasets: We meticulously curated and filtered a high-quality bilingual multimodal dataset to reduce visual hallucinations. With above recipe we build MammothModa that consistently outperforms the state-of-the-art models, e.g., LLaVA-series, across main real-world visual language benchmarks without bells and whistles.