Streaming Long Video Understanding with Large Language Models

Abstract

This paper presents VideoStreaming, an advanced vision-language large model (VLLM) for video understanding, that capably understands arbitrary-length video with a constant number of video tokens streamingly encoded and adaptively selected. The challenge of video understanding in the vision language area mainly lies in the significant computational burden caused by the great number of tokens extracted from long videos. Previous works rely on sparse sampling or frame compression to reduce tokens. However, such approaches either disregard temporal information in a long time span or sacrifice spatial details, resulting in flawed compression. To address these limitations, our VideoStreaming has two core designs: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. The Memory-Propagated Streaming Encoding architecture segments long videos into short clips and sequentially encodes each clip with a propagated memory. In each iteration, we utilize the encoded results of the preceding clip as historical memory, which is integrated with the current clip to distill a condensed representation that encapsulates the video content up to the current timestamp. After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories and feeds them into the LLM to generate informative responses. The question-related selection reduces redundancy within the memories, enabling efficient and precise video understanding. Meanwhile, the disentangled video extraction and reasoning design allows the LLM to answer different questions about a video by directly selecting corresponding memories, without the need to encode the whole video for each question. Our model achieves superior performance and higher efficiency on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.

Author abstract · arXiv abstract · v1

Publication

Neural Information Processing Systems (NeurIPS), 2024

Paper and resources

• Paper · arXiv v1 (PDF)
• arXiv:2405.16009
• Google Scholar

Research topics

VideoStreaming · Long-video understanding · Streaming encoding · Memory propagation · Adaptive memory selection · Video question answering · Efficient multimodal inference · Temporal comprehension

Research problem and approach

Sparse frame sampling loses temporal coverage, while aggressive compression can erase spatial detail. VideoStreaming encodes clips sequentially with historical memory, then selects relevant memories for each question without re-encoding the video. arXiv abstract · v1

Main contributions

• Introduces Memory-Propagated Streaming Encoding to accumulate video context clip by clip. arXiv abstract · v1
• Uses Adaptive Memory Selection to separate reusable video encoding from question-specific reasoning. arXiv abstract · v1

Method comparison

arXiv abstract · v1

Selected results

• VideoStreaming’s 7B answer model plus 1.3B streaming component scores 44.1 on the EgoSchema full test set and 66.2 on Next-QA in the zero-shot evaluation. The 7B LangRepo comparison scores 38.9 and 54.6. Tables 3–4 · zero-shot video QA · arXiv v1
• With vision-only input, VideoStreaming uses 256 LLM input tokens and 5.32 seconds per question, compared with LLaMA-VID’s 5,477 tokens and 10.47 seconds. Overview/plot/temporal scores are 2.65/3.13/1.88 versus 2.28/2.88/1.46. Table 6 · vision-only MovieNet-QA · arXiv v1

Cite this paper

Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, Jiaqi Wang. Streaming Long Video Understanding with Large Language Models. Advances in Neural Information Processing Systems, 37, 2024, pp. 119336–119360.

@inproceedings{arxiv240516009,
  title     = {{Streaming Long Video Understanding with Large Language Models}},
  author    = {Rui Qian and Xiaoyi Dong and Pan Zhang and Yuhang Zang and Shuangrui Ding and Dahua Lin and Jiaqi Wang},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2024},
  volume    = {37},
  publisher = {Curran Associates, Inc.},
  pages     = {119336--119360},
  doi       = {10.52202/079017-3792},
  url       = {https://papers.nips.cc/paper_files/paper/2024/hash/d7ce06e9293c3d8e6cb3f80b4157f875-Abstract-Conference.html}
}