VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models

Haodong Duan; Junming Yang; Yuxuan Qiao; Xinyu Fang; Lin Chen; Yuan Liu; Xiaoyi Dong; Yuhang Zang; Pan Zhang; Jiaqi Wang; Dahua Lin; Kai Chen

doi:10.1145/3664647.3685520

ACM MM · 2024

VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao, Zerun Ma, Yuxuan Qiao, Mo Li, Tianhao Liang, Lin Zhu, Amit Agarwal, Xiaozhe Li, Shengyuan Ding, Jiazi Bu, Ziyu Liu, Zhangyang Qi, Yifei Li, Yuhang Zang, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Junming Lin, Xilin Wei, Jinsong Li, Zeyi Sun, Zhaowei Wang, Zicheng Zhang, Xiaoyi Dong, Junjun He, Pan Zhang, Jiaqi Wang, Dahua Lin, Kai Chen

Key takeaway

VLMEvalKit standardizes multimodal evaluation behind a common interface that handles data preparation, distributed inference, post-processing and metrics for reproducible comparisons. arXiv abstract · v5

Abstract

We present VLMEvalKit: an open-source toolkit for evaluating large multi-modality models based on PyTorch. The toolkit aims to provide a user-friendly and comprehensive framework for researchers and developers to evaluate existing multi-modality models and publish reproducible evaluation results. In VLMEvalKit, we implement over 450+ large multi-modality model configurations, including both proprietary APIs and open-source models, and support 330+ benchmarks across diverse multi-modal benchmarks. By implementing a single interface, new models can be easily added to the toolkit, while the toolkit automatically handles the remaining workloads, including data preparation, distributed inference, prediction post-processing, and metric calculation. VLMEvalKit has also evolved to a broader evaluation suite spanning video/audio, document understanding, GUI grounding, spatial reasoning, safety, scientific reasoning, and multi-turn dialogue. Based on the evaluation results obtained with the toolkit, we host the OpenVLM Leaderboard, a comprehensive leaderboard to track the progress of multi-modality learning research. The toolkit is released on https://github.com/open-compass/VLMEvalKit and is actively maintained.

Author abstract · arXiv abstract · v5

Publication

ACM Multimedia (ACM MM), 2024 (Open Source Software Competition)

Paper and resources

Research topics

VLMEvalKit · Multimodal evaluation · Reproducible benchmarks · OpenVLM Leaderboard · Vision-language models · Distributed inference · Evaluation toolkit · Cross-model comparison

Research problem and approach

Comparing multimodal models requires coordinating heterogeneous APIs, datasets and evaluation rules. VLMEvalKit supplies a shared PyTorch-based framework for proprietary and open models and publishes results through the OpenVLM Leaderboard. arXiv abstract · v5

Main contributions

Lets a new model join the evaluation pipeline through a single interface. arXiv abstract · v5
Extends evaluation across image, video, audio, documents, GUI grounding, spatial reasoning, safety and multi-turn dialogue. arXiv abstract · v5

Method comparison

Approach	Key difference
Model integration in VLMEvalKit	Implements a common interface for the model or API being evaluated.
Shared evaluation pipeline	Handles data preparation, distributed inference, post-processing and metrics after model integration.

arXiv abstract · v5

Selected results

The expanded report documents support for more than 335 multimodal benchmarks, including video, document parsing, spatial grounding, safety and scientific reasoning. This is the scope reported in arXiv v5, which postdates the four-page ACM Multimedia 2024 publication. Table 1 · expanded arXiv v5 toolkit · arXiv v5
The general-VQA comparison normalizes individual benchmark scores to 0–100 before averaging. In its 2025-09-17 snapshot, Gemini-2.5-Pro scores 80.1 on average and GPT-5-20250807 79.9; these date-stamped results illustrate the toolkit’s evaluation protocol rather than a live ranking. Table 2 · reproducible leaderboard snapshot · arXiv v5

Cite this paper

Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, Kai Chen. VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models. Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 11198–11201.

This citation uses the author list of the published version.

Download BibTeX ACM MM publisher record

@inproceedings{arxiv240711691,
  title     = {{VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models}},
  author    = {Haodong Duan and Junming Yang and Yuxuan Qiao and Xinyu Fang and Lin Chen and Yuan Liu and Xiaoyi Dong and Yuhang Zang and Pan Zhang and Jiaqi Wang and Dahua Lin and Kai Chen},
  booktitle = {Proceedings of the 32nd ACM International Conference on Multimedia},
  month     = {October},
  year      = {2024},
  publisher = {ACM},
  pages     = {11198--11201},
  doi       = {10.1145/3664647.3685520},
  url       = {https://doi.org/10.1145/3664647.3685520}
}