CVPR · 2025
OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?
Key takeaway
OVO-Bench evaluates whether video models respond appropriately at a specified moment, distinguishing backward tracing, real-time understanding and waiting for future evidence. arXiv abstract · v2
Abstract
Temporal Awareness, the ability to reason dynamically based on the timestamp when a question is raised, is the key distinction between offline and online video LLMs. Unlike offline models, which rely on complete videos for static, post hoc analysis, online models process video streams incrementally and dynamically adapt their responses based on the timestamp at which the question is posed. Despite its significance, temporal awareness has not been adequately evaluated in existing benchmarks. To fill this gap, we present OVO-Bench (Online-VideO-Benchmark), a novel video benchmark that emphasizes the importance of timestamps for advanced online video understanding capability benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and respond to events occurring at specific timestamps under three distinct scenarios: (1) Backward tracing: trace back to past events to answer the question. (2) Real-time understanding: understand and respond to events as they unfold at the current timestamp. (3) Forward active responding: delay the response until sufficient future information becomes available to answer the question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps. We combine automated generation pipelines with human curation. With these high-quality samples, we further developed an evaluation pipeline to systematically query video LLMs along the video timeline. Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding, showing a significant gap compared to human agents. We hope OVO-Bench will drive progress in video LLMs and inspire future research in online video reasoning. Our benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.
Author abstract · arXiv abstract · v2
Publication
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025
Paper and resources
Research topics
OVO-Bench · Online video understanding · Temporal awareness · Streaming video evaluation · Backward tracing · Real-time understanding · Forward active responding · Video language models
Research problem and approach
Offline video evaluation does not test the temporal awareness needed for streaming interaction. OVO-Bench queries models along a timeline and assesses how answers should change with the information available at each timestamp. arXiv abstract · v2
Main contributions
- Defines three temporal response scenarios spanning 12 online video tasks. arXiv abstract · v2
- Provides 644 videos and approximately 2,800 human-curated, timestamped meta-annotations with an evaluation pipeline. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Offline video evaluation | Permits complete-video analysis after events have occurred. |
| OVO-Bench | Queries along the timeline and distinguishes past-event retrieval, present understanding and waiting for future evidence. |
Selected results
- OVO-Bench reports an overall average of 63.00 for Gemini-1.5-Pro with 1-fps input and 59.54 for GPT-4o with 64 frames, versus 92.81 for humans. The table places EPM and ASI queries at the video’s end to increase the gap from supporting clues. Table 1 · online video understanding · arXiv v2
- Gemini-1.5-Pro scores 69.32 on real-time perception, 62.54 on backward tracing and 57.15 on forward active responding. This table uses accuracy-based forward-response metrics; its aggregate scores do not measure response timing alone. Table 1 · capability breakdown · arXiv v2
Cite this paper
Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang. OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 18902–18913.
This citation uses the author list of the published version.
@inproceedings{arxiv250105510,
title = {{OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?}},
author = {Junbo Niu and Yifei Li and Ziyang Miao and Chunjiang Ge and Yuanhang Zhou and Qihao He and Xiaoyi Dong and Haodong Duan and Shuangrui Ding and Rui Qian and Pan Zhang and Yuhang Zang and Yuhang Cao and Conghui He and Jiaqi Wang},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2025},
pages = {18902--18913},
url = {https://openaccess.thecvf.com/content/CVPR2025/html/Niu_OVO-Bench_How_Far_is_Your_Video-LLMs_from_Real-World_Online_Video_CVPR_2025_paper.html}
}