ICLR · 2025
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
Key takeaway
MIA-DPO creates multi-image preference-training pairs from single-image data and attention signals, reducing the need for new human annotations or external judge models. arXiv abstract · v1
Abstract
Visual preference alignment involves training Large Vision-Language Models (LVLMs) to predict human preferences between visual inputs. This is typically achieved by using labeled datasets of chosen/rejected pairs and employing optimization algorithms like direct preference optimization (DPO). Existing visual alignment methods, primarily designed for single-image scenarios, struggle to effectively handle the complexity of multi-image tasks due to the scarcity of diverse training data and the high cost of annotating chosen/rejected pairs. We present Multi-Image Augmented Direct Preference Optimization (MIA-DPO), a visual preference alignment approach that effectively handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats, significantly reducing the costs associated with multi-image data annotations. Our observation reveals that attention values of LVLMs vary considerably across different images. We use attention values to identify and filter out rejected responses the model may have mistakenly focused on. Our attention-aware selection for constructing the chosen/rejected pairs without relying on (i) human annotation, (ii) extra data, and (iii) external models or APIs. MIA-DPO is compatible with various architectures and outperforms existing methods on five multi-image benchmarks, achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's ability to understand single images.
Author abstract · arXiv abstract · v1
Publication
International Conference on Learning Representations (ICLR), 2025
Paper and resources
Research topics
MIA-DPO · Multi-image alignment · Direct preference optimization · Attention-aware selection · Preference data construction · Visual instruction tuning · Distractor images · Vision-language models
Research problem and approach
Multi-image preference alignment is constrained by scarce preference pairs. MIA-DPO inserts unrelated images into collages or picture-in-picture inputs and uses attention to identify responses influenced by the wrong visual evidence. arXiv abstract · v1
Main contributions
- Augments single-image training examples into multi-image preference tasks. arXiv abstract · v1
- Constructs chosen/rejected pairs through attention-aware response selection without external model calls. arXiv abstract · v1
Method comparison
| Approach | Key difference |
|---|---|
| Single-image preference alignment | Does not directly address the complexity and training-data scarcity of multi-image tasks. |
| MIA-DPO | Adds distractor images and uses attention-aware selection to build multi-image preference pairs from existing data. |
Selected results
- MIA-DPO raises the five-benchmark average from 40.4 to 43.4 for LLaVA-v1.5-7B and from 53.6 to 57.9 for InternLM-XComposer2.5-7B. On the latter backbone, Mantis rises from 49.3 to 60.4. Table 1 · multi-image evaluation · arXiv v1
- For LLaVA-v1.5, post-selection raises the multi-image average from 42.3 to 43.4, versus a 40.4 baseline. BLINK improves from 38.7 without post-selection to 42.9 with it. Table 3 · attention-based post-selection · arXiv v1
Cite this paper
Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, Jiaqi Wang. MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models. International Conference on Learning Representations, 2025, 2025, pp. 34583–34610.
@inproceedings{arxiv241017637,
title = {{MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models}},
author = {Ziyu Liu and Yuhang Zang and Xiaoyi Dong and Pan Zhang and Yuhang Cao and Haodong Duan and Conghui He and Yuanjun Xiong and Dahua Lin and Jiaqi Wang},
booktitle = {International Conference on Learning Representations},
year = {2025},
volume = {2025},
pages = {34583--34610},
url = {https://proceedings.iclr.cc/paper_files/paper/2025/hash/557a20663907ed637c2807f608d5bec2-Abstract-Conference.html}
}