ICCV · 2025
MM-IFEngine: Towards Multimodal Instruction Following
Key takeaway
MM-IFEngine improves multimodal instruction following through constraint-rich training data and evaluation that checks both requested output form and image-grounded requirements. arXiv abstract · v2
Abstract
The Instruction Following (IF) ability measures how well Multi-modal Large Language Models (MLLMs) understand exactly what users are telling them and whether they are doing it right. Existing multimodal instruction following training data is scarce, the benchmarks are simple with atomic instructions, and the evaluation strategies are imprecise for tasks demanding exact output constraints. To address this, we present MM-IFEngine, an effective pipeline to generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for Direct Preference Optimization (DPO). We further introduce MM-IFEval, a challenging and diverse multi-modal instruction-following benchmark that includes (1) both compose-level constraints for output responses and perception-level constraints tied to the input images, and (2) a comprehensive evaluation pipeline incorporating both rule-based assessment and judge model. We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF benchmarks, such as MM-IFEval (+10.2%), MIA (+7.6%), and IFEval (+12.3%). We have fully open-sourced the datasets (both SFT and DPO), evaluation code and training scripts at https://github.com/SYuan03/MM-IFEngine.
Author abstract · arXiv abstract · v2
Publication
IEEE International Conference on Computer Vision (ICCV), 2025
Paper and resources
Research topics
MM-IFEngine · Multimodal instruction following · Output constraints · MM-IFEval · Instruction tuning · Direct preference optimization · Perception constraints · Synthetic training data
Research problem and approach
Existing instruction-following datasets and benchmarks lack varied, exact multimodal constraints. MM-IFEngine generates image-instruction pairs and evaluates responses using rules alongside a judge model. arXiv abstract · v2
Main contributions
- Creates MM-IFInstruct-23k for SFT and MM-IFDPO-23k for preference optimization. arXiv abstract · v2
- Introduces MM-IFEval with composition-level and perception-level constraints. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Atomic instruction-following evaluation | Uses simpler constraints and can miss exact multimodal requirements. |
| MM-IFEngine | Generates diverse constrained instructions and evaluates composition and perception requirements with rules and a judge model. |
Selected results
- Fine-tuning Qwen2-VL-7B-Instruct on MM-IFDPO-23k raises MM-IFEval accuracy from 42.0% to 52.2%, MIA-Bench from 80.5% to 88.1%, and the reported IFEval average from 47.4% to 59.7%. Table 1 · instruction-following evaluation · arXiv v2
- For Qwen2-VL-7B-Instruct, DPO with all constraints removed when generating negatives reaches a three-benchmark average of 66.7, versus 63.4 when negatives are generated without images. Removing 33% or 66% of constraints yields 65.8 and 65.9. Table 4 · negative-response construction · arXiv v2
Cite this paper
Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, Jiaqi Wang. MM-IFEngine: Towards Multimodal Instruction Following. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 1099–1109.
@inproceedings{arxiv250407957,
title = {{MM-IFEngine: Towards Multimodal Instruction Following}},
author = {Shengyuan Ding and Shenxi Wu and Xiangyu Zhao and Yuhang Zang and Haodong Duan and Xiaoyi Dong and Pan Zhang and Yuhang Cao and Dahua Lin and Jiaqi Wang},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2025},
pages = {1099--1109},
url = {https://openaccess.thecvf.com/content/ICCV2025/html/Ding_MM-IFEngine_Towards_Multimodal_Instruction_Following_ICCV_2025_paper.html}
}