CVPR · 2025
Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction
Key takeaway
Dispider separates perception, decision and response so a video assistant can keep monitoring a stream while asynchronously generating an interaction. arXiv abstract · v1
Abstract
Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly. Unlike offline video LLMs, which analyze the entire video before answering questions, active real-time interaction requires three capabilities: 1) Perception: real-time video monitoring and interaction capturing. 2) Decision: raising proactive interaction in proper situations, 3) Reaction: continuous interaction with users. However, inherent conflicts exist among the desired capabilities. The Decision and Reaction require a contrary Perception scale and grain, and the autoregressive decoding blocks the real-time Perception and Decision during the Reaction. To unify the conflicted capabilities within a harmonious system, we present Dispider, a system that disentangles Perception, Decision, and Reaction. Dispider features a lightweight proactive streaming video processing module that tracks the video stream and identifies optimal moments for interaction. Once the interaction is triggered, an asynchronous interaction module provides detailed responses, while the processing module continues to monitor the video in the meantime. Our disentangled and asynchronous design ensures timely, contextually accurate, and computationally efficient responses, making Dispider ideal for active real-time interaction for long-duration video streams. Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses, thereby validating the effectiveness of our architecture. The code and model are released at https://github.com/Mark12Ding/Dispider.
Author abstract · arXiv abstract · v1
Publication
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025
Paper and resources
Research topics
Dispider · Streaming video assistants · Active real-time interaction · Asynchronous inference · Proactive interaction · Video perception · Interaction timing · Video question answering
Research problem and approach
Autoregressive response generation can block perception and prevent timely decisions in streaming video systems. Dispider uses a lightweight monitoring module to decide when to interact and a separate asynchronous module to produce detailed responses. arXiv abstract · v1
Main contributions
- Disentangles three components with conflicting temporal and computational requirements. arXiv abstract · v1
- Enables proactive interaction while maintaining continuous video monitoring during response generation. arXiv abstract · v1
Method comparison
| Approach | Key difference |
|---|---|
| Coupled perception and autoregressive response | Response generation can block continuous perception and delay further interaction decisions. |
| Dispider | Maintains lightweight monitoring while an asynchronous module generates detailed responses. |
Selected results
- With the question placed before the video, Dispider at 1 fps reaches temporal-grounding F1 of 36.1 and episodic-memory F1 of 15.5, versus 13.2 and 3.8 for VideoLLM-Online at 2 fps. Table 3 · streaming ETBench protocol · arXiv v1
- Replacing uniform 16-frame clips with scene-based segmentation raises MLVU accuracy from 59.8 to 61.7 and Video-MME from 55.4 to 57.2. Temporal-grounding F1 rises from 34.5 to 36.1. Table 4 · clip-segmentation ablation · arXiv v1
Cite this paper
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 24045–24055.
@inproceedings{arxiv250103218,
title = {{Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction}},
author = {Rui Qian and Shuangrui Ding and Xiaoyi Dong and Pan Zhang and Yuhang Zang and Yuhang Cao and Dahua Lin and Jiaqi Wang},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2025},
pages = {24045--24055},
url = {https://openaccess.thecvf.com/content/CVPR2025/html/Qian_Dispider_Enabling_Video_LLMs_with_Active_Real-Time_Interaction_via_Disentangled_CVPR_2025_paper.html}
}