ACL · 2025
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
Key takeaway
IXC-2.5-Reward provides a multimodal preference signal that supports reinforcement learning, best-response selection and instruction-data filtering across text, image and video inputs. arXiv abstract · v2
Abstract
Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward
Author abstract · arXiv abstract · v2
Publication
Findings of the Association for Computational Linguistics (ACL), 2025
Paper and resources
Research topics
InternLM-XComposer2.5-Reward · Multimodal reward models · Human preference alignment · Reinforcement learning · PPO · Best-of-N selection · Instruction data filtering · Video reward modeling
Research problem and approach
Public multimodal reward models are scarce and proprietary training recipes are difficult to reproduce. IXC-2.5-Reward addresses this gap with an open reward model trained on a preference corpus spanning instruction following, documents, reasoning and video understanding. arXiv abstract · v2
Main contributions
- Constructs a diverse multimodal preference corpus and releases reward-model weights and training recipes. arXiv abstract · v2
- Demonstrates one reward model in PPO training, test-time response selection and noisy-data filtering. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| PPO training with IXC-2.5-Reward | Uses reward scores as supervision to update the model's response policy. |
| Test-time selection with IXC-2.5-Reward | Ranks existing candidate responses to select a preferred output without a policy update. |
| Data filtering with IXC-2.5-Reward | Identifies noisy or outlying image and video instruction examples before training. |
Selected results
- IXC-2.5-Reward-7B obtains 65.8% overall accuracy and 70.0% macro accuracy on VLRewardBench. GPT-4o (2024-08-06) scores 65.8% overall and 62.4% macro accuracy; the equal overall score and different macro score reflect category balance. Table 3 · VLRewardBench · arXiv v2
- IXC-2.5-Reward scores 88.6 on text-only RewardBench, compared with 87.6 for InternLM2-7B-Reward and 80.0 for LLaVA-Critic-8B. This evaluates text reward capability separately from multimodal judgment. Table 4 · text-only RewardBench · arXiv v2
Cite this paper
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang. InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model. Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 6547–6563.
@inproceedings{arxiv250112368,
title = {{InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model}},
author = {Yuhang Zang and Xiaoyi Dong and Pan Zhang and Yuhang Cao and Ziyu Liu and Shengyuan Ding and Shenxi Wu and Yubo Ma and Haodong Duan and Wenwei Zhang and Kai Chen and Dahua Lin and Jiaqi Wang},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
month = {July},
year = {2025},
publisher = {Association for Computational Linguistics},
pages = {6547--6563},
doi = {10.18653/v1/2025.findings-acl.340},
url = {https://aclanthology.org/2025.findings-acl.340/}
}