Tune vision-language responses with grounded preference pairs
A multimodal DPO training script for comparing grounded and ungrounded answers about images.
Published by Hugging Face TRL
How to use it
The example pairs Qwen2.5-VL-3B-Instruct with HuggingFaceH4's formatted RLAIF-V dataset. It uses an image processor so both alternatives can be compared with the image in context.
Follow one of the published Accelerate invocations. Evaluate with images outside the training set and check both unsupported claims and cases where the model omits facts visible in the image.
- Framework
- Hugging Face TRL
- Dataset
- HuggingFaceH4/rlaif-v_formatted
- Hardware
- Minimum VRAM unspecified.
- Access
- Public publisher script and launch examples.