VideoRoPE: What Makes for Good Video Rotary Position Embedding?

Abstract

While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce VideoRoPE, with a 3D structure designed to preserve spatio-temporal relationships. VideoRoPE features low-frequency temporal allocation to mitigate periodic oscillations, a diagonal layout to maintain spatial symmetry, and adjustable temporal spacing to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at https://github.com/Wiselnn570/VideoRoPE.

Author abstract · arXiv abstract · v3

Publication

International Conference on Machine Learning (ICML), 2025

Oral

Paper and resources

• Paper · arXiv v3 (PDF)
• arXiv:2502.05173
• Google Scholar
• Code · Wiselnn570/VideoRoPE

Research topics

VideoRoPE · Rotary position embedding · Video language models · Spatiotemporal encoding · Long-video retrieval · Temporal frequency allocation · V-NIAH-D · Position encoding

Research problem and approach

Extending one-dimensional position encoding to video requires preserving both temporal and spatial structure. VideoRoPE studies the design requirements and uses distractor-heavy retrieval to expose failures caused by inappropriate temporal frequency allocation. arXiv abstract · v3

Main contributions

• Introduces V-NIAH-D, a visual needle-in-a-haystack task with periodic distractors. arXiv abstract · v3
• Designs a three-dimensional rotary embedding that separates temporal spacing from spatial indexing. arXiv abstract · v3

Method comparison

arXiv abstract · v3

Selected results

• At 64k context, VideoRoPE scores 57.26 on LongVideoBench, 65.56 on MLVU and 61.33 on Video-MME, compared with M-RoPE’s 54.35, 61.10 and 59.67. The models are trained with an 8k context window. Table 2 · extrapolation beyond 8k training context · arXiv v3
• VideoRoPE achieves 91.11% on V-NIAH and 87.11% on its distractor variant V-NIAH-D, versus M-RoPE’s 78.67% and 74.67%. Accuracy is averaged across haystack lengths and needle-frame depths. Table 3 · distractor-controlled retrieval · arXiv v3

Cite this paper

Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, Dahua Lin. VideoRoPE: What Makes for Good Video Rotary Position Embedding?. Proceedings of the 42nd International Conference on Machine Learning, 267, 2025, pp. 66118–66136.

@inproceedings{arxiv250205173,
  title     = {{VideoRoPE: What Makes for Good Video Rotary Position Embedding?}},
  author    = {Xilin Wei and Xiaoran Liu and Yuhang Zang and Xiaoyi Dong and Pan Zhang and Yuhang Cao and Jian Tong and Haodong Duan and Qipeng Guo and Jiaqi Wang and Xipeng Qiu and Dahua Lin},
  booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
  year      = {2025},
  volume    = {267},
  publisher = {PMLR},
  pages     = {66118--66136},
  url       = {https://proceedings.mlr.press/v267/wei25h.html}
}