LMMs-Eval
Evaluate multimodal models on tasks that use images, video, audio, or text inputs.
Published by LMMs Lab
How to use it
Choose tasks and a compatible model adapter, then pin the model, input preprocessing, and test configuration. Keep per-task results so a change on one input type remains visible.
Test a small number of examples before running the full suite. Check media decoding and sampling settings because they can affect video and audio scores even when the model is unchanged.
- Compatibility
- Python, Hugging Face Transformers
- Evaluation areas
- Multimodal evaluation, Model benchmarks, Batch evaluation
- Access
- Local Python runner with compatible model adapters and task data.