Diving into Self-Evolving Training for Multimodal Reasoning
Research paper by Junlong Li and collaborators. Published 2024-12-23. Topics: Large Multimodal Models, self-evolving training, multimodal reasoning, Training Method, Reward Model.
Resource information
- Authors
- Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, Junxian He
- Research area
- Language technology
- Application domain
- General AI research
- AI methods
- Reinforcement learning
- Research topics
- cs.CL, cs.AI, cs.CV, cs.LG
- arXiv record date
- 2025-06-06
- Publication date
- 2024-12-23
- Publication
- arXiv
- Upstream last modified
- Jun 9, 2025
- Last record update
- Nov 3, 2026
Abstract
Self-evolving trainin--where models iteratively learn from their own outputs--has emerged as a key approach for complex reasoning tasks, addressing the scarcity of high-quality chain-of-thought data. However, its effectiveness in multimodal reasoning, a domain more intricate than text-only reasoning, remains underexplored, and the understanding of critical factors in this training paradigm remains limited. Furthermore, a central challenge for this training method is performance saturation, which impedes further improvements and scalability. Inspired by reinforcement learning (RL), in this paper, we reframe self-evolving training for multimodal reasoning through the lens of RL, identifying three pivotal factors: Training Method, Reward Model, and Prompt Variation. Through systematic analysis, we establish relatively optimal design principles that significantly enhance multimodal reasoning capabilities. Moreover, delving deeper into training dynamics, we uncover the roots of saturation and propose a new automatic balancing mechanism to mitigate this limitation. Building on these insights, we propose M-STAR (Multimodal Self-evolving Training for Reasoning), a framework that achieves consistent performance gains across models of varying sizes and diverse benchmarks. All resources are made publicly available at https://mstar-lmm.github.io.