CVPR · 2026
Fine-Grained GRPO for Precise Preference Alignment in Flow Models
Key takeaway
G²RPO improves credit assignment in flow-model reinforcement learning by isolating stochastic perturbations and integrating advantages across multiple denoising granularities. arXiv abstract · v3
Abstract
The incorporation of online reinforcement learning (RL) into diffusion and flow-based generative models has recently gained attention as a powerful paradigm for aligning model behavior with human preferences. By leveraging stochastic sampling via Stochastic Differential Equations (SDEs) during the denoising phase, these models can explore a variety of denoising trajectories, enhancing the exploratory capacity of RL. However, despite their ability to discover potentially high-reward samples, current approaches often struggle to effectively align with preferences due to the sparsity and narrowness of reward feedback. To overcome this limitation, we introduce a novel framework called Granular-GRPO (G²RPO), which enables fine-grained and comprehensive evaluation of sampling directions in the RL training of flow models. Specifically, we propose a Singular Stochastic Sampling mechanism that supports step-wise stochastic exploration while ensuring strong correlation between injected noise and reward signals, enabling more accurate credit assignment to each SDE perturbation. Additionally, to mitigate the bias introduced by fixed-granularity denoising, we design a Multi-Granularity Advantage Integration module that aggregates advantages computed across multiple diffusion scales, resulting in a more robust and holistic assessment of sampling trajectories. Extensive experiments on various reward models, including both in-domain and out-of-domain settings, demonstrate that our G²RPO outperforms existing flow-based GRPO baselines, highlighting its effectiveness and generalization capability.
Author abstract · arXiv abstract · v3
Publication
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2026
Paper and resources
Research topics
G²RPO · Granular-GRPO · Flow models · Preference alignment · Credit assignment · Stochastic sampling · Multi-granularity advantages · Diffusion reinforcement learning
Research problem and approach
Sparse and narrow reward feedback can obscure which denoising decisions produced a good sample. Granular-GRPO evaluates sampling directions more precisely and combines evidence from multiple diffusion scales. arXiv abstract · v3
Main contributions
- Introduces Singular Stochastic Sampling to strengthen the connection between injected noise and rewards. arXiv abstract · v3
- Aggregates advantages using Multi-Granularity Advantage Integration. arXiv abstract · v3
Method comparison
| Approach | Key difference |
|---|---|
| Fixed-granularity flow-model GRPO | Can leave reward feedback sparse and obscure the contribution of individual stochastic perturbations. |
| G²RPO | Isolates stochastic exploration and integrates advantages across denoising granularities for finer credit assignment. |
Selected results
- With joint HPS and CLIP rewards, G²RPO scores 0.376 HPS, 0.406 CLIP and 3.783 UnifiedReward, versus 0.363, 0.399 and 3.661 for MixGRPO. These are evaluator scores rather than task-accuracy percentages. Table 1 · joint HPS and CLIP reward training · arXiv v3
- Using granularities {1,2,3} instead of {1} changes CLIP from 0.395 to 0.406 and UnifiedReward from 3.688 to 3.783. HPS is 0.376, slightly below the {1,3} setting’s 0.378, illustrating the multi-metric trade-off. Table 2 · denoising-granularity ablation · arXiv v3
Cite this paper
Yujie Zhou, Pengyang Ling, Jiazi Bu, Yibin Wang, Yuhang Zang, Jiaqi Wang, Li Niu, Guangtao Zhai. Fine-Grained GRPO for Precise Preference Alignment in Flow Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 20045–20054.
@inproceedings{arxiv251001982,
title = {{Fine-Grained GRPO for Precise Preference Alignment in Flow Models}},
author = {Yujie Zhou and Pengyang Ling and Jiazi Bu and Yibin Wang and Yuhang Zang and Jiaqi Wang and Li Niu and Guangtao Zhai},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2026},
pages = {20045--20054},
url = {https://openaccess.thecvf.com/content/CVPR2026/html/Zhou_Fine-Grained_GRPO_for_Precise_Preference_Alignment_in_Flow_Models_CVPR_2026_paper.html}
}