NeurIPS · 2024
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Key takeaway
MMDU evaluates sustained multi-image dialogue, while MMDU-45k provides instruction data that helps open LVLMs close the conversational gap identified by the benchmark. arXiv abstract · v2
Abstract
Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models(LVLMs). While current open-source LVLMs demonstrate promising performance in simplified scenarios such as single-turn single-image input, they fall short in real-world conversation scenarios such as following instructions in a long context history with multi-turn and multi-images. Existing LVLM benchmarks primarily focus on single-choice questions or short-form responses, which do not adequately assess the capabilities of LVLMs in real-world human-AI interaction applications. Therefore, we introduce MMDU, a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and multi-image conversations. We employ the clustering algorithm to find the relevant images and textual descriptions from the open-source Wikipedia and construct the question-answer pairs by human annotators with the assistance of the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and 27 turns, which is at least 5x longer than previous benchmarks and poses challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs using MMDU reveals that open-source LVLMs lag behind closed-source counterparts due to limited conversational instruction tuning data. We demonstrate that fine-tuning open-source LVLMs on MMDU-45k significantly address this gap, generating longer and more accurate conversations, and improving scores on MMDU and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our contributions pave the way for bridging the gap between current LVLM models and real-world application demands. This project is available at https://github.com/Liuziyu77/MMDU.
Author abstract · arXiv abstract · v2
Publication
Neural Information Processing Systems (NeurIPS), 2024 (Datasets and Benchmarks Track)
Paper and resources
Research topics
MMDU · MMDU-45k · Multi-turn dialogue · Multi-image understanding · Conversational instruction tuning · Long-context interaction · Multimodal benchmarks · Vision-language assistants
Research problem and approach
Single-turn or short-answer benchmarks underrepresent real multimodal conversations. MMDU uses related Wikipedia images and descriptions to construct longer dialogues with human annotation and GPT-4o assistance. arXiv abstract · v2
Main contributions
- Introduces a benchmark reaching 18K image-plus-text tokens, 20 images and 27 turns. arXiv abstract · v2
- Builds MMDU-45k for instruction tuning and evaluates 15 representative LVLMs. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Single-turn or short-response evaluation | Underrepresents sustained conversations with multiple images and a long interaction history. |
| MMDU | Evaluates long multi-turn, multi-image dialogue and supplies MMDU-45k to improve these skills. |
Selected results
- Adding MMDU-45k raises InternLM-XComposer2’s MMDU average from 35.6 to 50.1, including image-relationship understanding from 35.2 to 48.7. LLaVA-1.5-7B improves from 32.2 to 37.2 overall. Table 2 · dialogue fine-tuning · arXiv v2
- For LLaVA-1.5, training with MMDU-45k raises BLINK from 37.1 to 40.1 and Mantis in the image-sequence setting from 37.8 to 44.7. This tests transfer beyond the MMDU benchmark itself. Table 4 · transfer to multi-image benchmarks · arXiv v2
Cite this paper
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, Jiaqi Wang. MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs. Advances in Neural Information Processing Systems, 37, 2024, pp. 8698–8733.
@inproceedings{arxiv240611833,
title = {{MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs}},
author = {Ziyu Liu and Tao Chu and Yuhang Zang and Xilin Wei and Xiaoyi Dong and Pan Zhang and Zijian Liang and Yuanjun Xiong and Yu Qiao and Dahua Lin and Jiaqi Wang},
booktitle = {Advances in Neural Information Processing Systems},
year = {2024},
volume = {37},
publisher = {Curran Associates, Inc.},
pages = {8698--8733},
doi = {10.52202/079017-0278},
url = {https://papers.nips.cc/paper_files/paper/2024/hash/1057053100de064a44286239724f7865-Abstract-Datasets_and_Benchmarks_Track.html}
}