LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding
Research paper by Boyuan Sun and collaborators. arXiv record dated 2026-04-20. Research area: Computer vision.
Resource information
- Authors
- Boyuan Sun, Jiaxing Zhao, Xiang Chen, Xihan Wei, Qibin Hou
- Research area
- Computer vision
- Application domain
- General AI research
- AI methods
- Large language models
- Research topics
- cs.CV, cs.AI
- arXiv record date
- 2026-04-20
- Publication
- arXiv
- Upstream last modified
- Apr 21, 2026
- Last record update
- Nov 4, 2026
Abstract
In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model's performance in multimodal tasks. Experimental results demonstrate that LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as video question answering, long video understanding, and comprehensive multi-choices benchmarks, highlighting its broad application potential.