CVPR · 2026
ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
Key takeaway
ARM-Thinker turns multimodal reward scoring into an evidence-seeking process by learning when to crop images, retrieve document pages and verify reasoning claims. arXiv abstract · v1
Abstract
Reward models are critical for aligning vision-language systems with human preferences, yet current approaches suffer from hallucination, weak visual grounding, and an inability to use tools for verification, limiting their reliability on complex multimodal reasoning tasks. We present ARM-Thinker, an Agentic multimodal Reward Model that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence, replacing static, non-interactive reward scoring. This enables the model to verify fine-grained visual details, cross-reference multi-page evidence, and validate reasoning claims, which are capabilities absent in existing reward models. We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy. To evaluate agentic reward modeling, we introduce ARMBench-VL, comprising three benchmarks that assess fine-grained visual grounding (image-level tools), multi-page document understanding (retrieval tools), and instruction following (text-level verification). ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms baselines on multimodal math and logical reasoning benchmarks. Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.
Author abstract · arXiv abstract · v1
Publication
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2026
Paper and resources
Research topics
ARM-Thinker · Agentic reward models · Tool use · Visual grounding · Document retrieval · Multimodal verification · Reinforcement learning · ARMBench-VL
Research problem and approach
Static reward models can hallucinate or miss fine visual details. ARM-Thinker uses external tools to gather verifiable evidence before judging multimodal responses. arXiv abstract · v1
Main contributions
- Jointly optimizes tool-calling decisions and judgment accuracy through multi-stage reinforcement learning. arXiv abstract · v1
- Introduces ARMBench-VL for visual grounding, multi-page evidence retrieval and instruction verification. arXiv abstract · v1
Method comparison
| Approach | Key difference |
|---|---|
| Static reward scoring | Judges the supplied multimodal response without interactively gathering additional verification evidence. |
| ARM-Thinker | Learns tool use and judgment jointly, gathering image crops or document pages before assigning rewards. |
Selected results
- ARM-Thinker-7B raises ARMBench-VL average performance from Qwen2.5-VL-7B’s 46.1 to 64.6 and VL-RewardBench overall performance from 50.1 to 67.8. The three-benchmark average rises from 47.8 to 64.0. Table 2 · reward-model evaluation · arXiv v1
- Enabling tools raises ARM-Thinker’s ARMBench-VL score from 59.2 to 64.6 and V* from 82.2 to 86.4. Enabling the same tool interface for the base Qwen2.5-VL-7B decreases these scores from 46.1 to 44.3 and 75.4 to 50.3, indicating that tool access alone is insufficient. Table 5 · tool-use ablation · arXiv v1
Cite this paper
Shengyuan Ding, Xinyu Fang, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiangyu Zhao, Haodong Duan, Xiaoyi Dong, Jianze Liang, Bin Wang, Conghui He, Dahua Lin, Jiaqi Wang. ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 22195–22205.
@inproceedings{arxiv251205111,
title = {{ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning}},
author = {Shengyuan Ding and Xinyu Fang and Ziyu Liu and Yuhang Zang and Yuhang Cao and Xiangyu Zhao and Haodong Duan and Xiaoyi Dong and Jianze Liang and Bin Wang and Conghui He and Dahua Lin and Jiaqi Wang},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2026},
pages = {22195--22205},
url = {https://openaccess.thecvf.com/content/CVPR2026/html/Ding_ARM-Thinker_Reinforcing_Multimodal_Generative_Reward_Models_with_Agentic_Tool_Use_CVPR_2026_paper.html}
}