Automating Steering for Safe Multimodal Large Language Models
Research paper by Lyucheng Wu and collaborators. Published 2025-07-17. Topics: Multimodal Large Language Models, MLLMs, AutoSteer, Safety Awareness Score, SAS.
Resource information
- Authors
- Lyucheng Wu, Mengru Wang, Ziwen Xu, Tri Cao, Nay Oo, Bryan Hooi, Shumin Deng
- Research area
- Language technology
- Application domain
- General AI research
- AI methods
- Large language models
- Research topics
- cs.CL, cs.AI, cs.IR, cs.LG, cs.MM
- arXiv record date
- 2025-09-23
- Publication date
- 2025-07-17
- Publication
- arXiv
- Upstream last modified
- Sep 24, 2025
- Last record update
- Nov 3, 2026
Abstract
Recent progress in Multimodal Large Language Models (MLLMs) has unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, particularly when faced with adversarial multimodal inputs. To improve the safety of MLLMs during inference, we introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model. AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected. Experiments on LLaVA-OV and Chameleon across diverse safety-critical benchmarks demonstrate that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats, while maintaining general abilities. These findings position AutoSteer as a practical, interpretable, and effective framework for safer deployment of multimodal AI systems.