ICCV · 2025
Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
Key takeaway
Modality Integration Rate estimates LVLM pretraining quality from cross-modal distribution distance, helping assess alignment without first running costly supervised fine-tuning. arXiv abstract · v2
Abstract
We present the Modality Integration Rate (MIR), an effective, robust, and generalized metric to indicate the multi-modal pre-training quality of Large Vision Language Models (LVLMs). Large-scale pre-training plays a critical role in building capable LVLMs, while evaluating its training quality without the costly supervised fine-tuning stage is under-explored. Loss, perplexity, and in-context evaluation results are commonly used pre-training metrics for Large Language Models (LLMs), while we observed that these metrics are less indicative when aligning a well-trained LLM with a new modality. Due to the lack of proper metrics, the research of LVLMs in the critical pre-training stage is hindered greatly, including the training data choice, efficient module design, etc. In this paper, we propose evaluating the pre-training quality from the inter-modal distribution distance perspective and present MIR, the Modality Integration Rate, which is 1) Effective to represent the pre-training quality and show a positive relation with the benchmark performance after supervised fine-tuning. 2) Robust toward different training/evaluation data. 3) Generalize across training configurations and architecture choices. We conduct a series of pre-training experiments to explore the effectiveness of MIR and observe satisfactory results that MIR is indicative about training data selection, training strategy schedule, and model architecture design to get better pre-training results. We hope MIR could be a helpful metric for building capable LVLMs and inspire the following research about modality alignment in different areas. Our code is at: https://github.com/shikiw/Modality-Integration-Rate.
Author abstract · arXiv abstract · v2
Publication
IEEE International Conference on Computer Vision (ICCV), 2025
Paper and resources
Research topics
Modality Integration Rate · MIR · Cross-modal alignment · LVLM pretraining · Distribution distance · Training quality evaluation · Multimodal representation · Data selection
Research problem and approach
Language-model metrics such as loss and perplexity can be weak indicators when aligning a pretrained language model with vision. MIR measures how well the modality distributions integrate during multimodal pretraining. arXiv abstract · v2
Main contributions
- Relates inter-modal distribution distance to downstream performance after supervised fine-tuning. arXiv abstract · v2
- Tests the metric across data choices, training schedules and model architectures. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Loss, perplexity and in-context metrics | Can be weak indicators of downstream quality when a pretrained language model is aligned with vision. |
| Modality Integration Rate | Uses inter-modal distribution distance to assess pretraining quality before supervised fine-tuning. |
Selected results
- As average caption length increases from 15.2 to 181.2, MIR decreases from 3.588 to 3.218 while the reported post-SFT benchmark average rises from 63.8 to 64.4. This experiment connects the pretraining indicator with downstream performance. Table 1 · caption-detail experiment · arXiv v2
- With approximately 1.9M pretraining examples and the same 665k SFT set, keeping the LLM frozen yields MIR 3.001 and downstream average 64.0; unlocking all LLM layers yields MIR 2.656 and average 65.9. Table 3 · trainable-layer comparison · arXiv v2
Cite this paper
Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jiaqi Wang, Weiming Zhang, Nenghai Yu. Deciphering Cross-Modal Alignment in Large Vision-Language Models via Modality Integration Rate. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 218–227.
This citation uses the author list of the published version.
@inproceedings{arxiv241007167,
title = {{Deciphering Cross-Modal Alignment in Large Vision-Language Models via Modality Integration Rate}},
author = {Qidong Huang and Xiaoyi Dong and Pan Zhang and Yuhang Zang and Yuhang Cao and Jiaqi Wang and Weiming Zhang and Nenghai Yu},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2025},
pages = {218--227},
url = {https://openaccess.thecvf.com/content/ICCV2025/html/Huang_Deciphering_Cross-Modal_Alignment_in_Large_Vision-Language_Models_via_Modality_Integration_ICCV_2025_paper.html}
}