NeurIPS · 2025
Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
Key takeaway
UnifiedReward-Think strengthens multimodal reward judgments through explicit multi-step reasoning and reinforcement fine-tuning across visual understanding and generation tasks. arXiv abstract · v3
Abstract
Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments across various vision reward tasks demonstrate the superiority of our model.
Author abstract · arXiv abstract · v3
Publication
Neural Information Processing Systems (NeurIPS), 2025
Paper and resources
Research topics
UnifiedReward-Think · Multimodal reward models · Chain-of-thought · Reinforcement fine-tuning · GRPO · Preference learning · Visual understanding · Visual generation · Rejection sampling
Research problem and approach
Direct or shallow reward judgments can be inaccurate. UnifiedReward-Think develops longer, multidimensional reasoning chains and uses exploration to improve the reliability of reward predictions. arXiv abstract · v3
Main contributions
- Uses a small distilled reasoning set for cold start, then unified multimodal preference data for rejection sampling. arXiv abstract · v3
- Applies GRPO to incorrectly predicted samples so the model can explore better reasoning paths. arXiv abstract · v3
Method comparison
| Approach | Key difference |
|---|---|
| Direct or shallow reward judgments | Produces preference decisions with limited explicit reasoning. |
| UnifiedReward-Think | Builds multidimensional reasoning through cold start, rejection sampling and reinforcement fine-tuning. |
Selected results
- UnifiedReward-Think reaches 73.8% overall accuracy versus UnifiedReward’s 67.5%; macro accuracy rises from 66.6% to 72.3%. The trained model’s direct-response mode without explicit CoT reaches 73.1% overall accuracy. Table 1 · VLRewardBench, LLaVA-OneVision-7B · arXiv v3
- On VLRewardBench, cold-start training alone scores 66.9% overall, rejection sampling raises it to 72.1%, and subsequent GRPO reaches 73.8%. Direct GRPO without CoT training reaches 69.0%, supporting the staged reasoning pipeline. Table 3 · training-stage ablation · arXiv v3
Cite this paper
Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, Jiaqi Wang. Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning. Advances in Neural Information Processing Systems, 38, 2025, pp. 159130–159157.
@inproceedings{arxiv250503318,
title = {{Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning}},
author = {Yibin Wang and Zhimin Li and Yuhang Zang and Chunyu Wang and Qinglin Lu and Cheng Jin and Jiaqi Wang},
booktitle = {Advances in Neural Information Processing Systems},
year = {2025},
volume = {38},
publisher = {Curran Associates, Inc.},
pages = {159130--159157},
doi = {10.52202/085713-5315},
url = {https://proceedings.neurips.cc/paper_files/paper/2025/hash/e95e9f0c127aa1cfa2628adb2f3cb107-Abstract-Conference.html}
}