ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

Abstract

We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that reached SOTA performance on three advancing video benchmarks. To achieve this, taking aside the non-scalable costly human annotators, we find using GPT4V to caption video with a naive multi-frame or frame-concatenation input strategy leads to less detailed and sometimes temporal-confused results. We argue the challenge of designing a high-quality video captioning strategy lies in three aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame detailed content description. 3) Frame-number scalability for arbitrary-length videos. To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length. Based on it, we construct ShareGPT4Video, which contains 40K high-quality videos spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. Based on ShareGPT4Video, we further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos...

Author abstract · arXiv abstract · v1

Publication

Neural Information Processing Systems (NeurIPS), 2024 (Datasets and Benchmarks Track)

Paper and resources

• Paper · arXiv v1 (PDF)
• arXiv:2406.04325
• Google Scholar
• Code · ShareGPT4Omni/ShareGPT4Video
• Project website
• Hugging Face · Dataset

Research topics

ShareGPT4Video · Dense video captioning · Differential captioning · Temporal descriptions · Video instruction data · ShareCaptioner-Video · Video understanding · Video generation supervision

Research problem and approach

Naive multi-frame prompting can miss details or confuse temporal order. ShareGPT4Video uses differential captioning to capture changes between frames while supporting videos with varied lengths, resolutions and aspect ratios. arXiv abstract · v1

Main contributions

• Creates 40K densely captioned videos using GPT-4V and a filtering and annotation pipeline. arXiv abstract · v1
• Develops ShareCaptioner-Video, 4.8M automatically captioned videos and the ShareGPT4Video-8B understanding model. arXiv abstract · v1

Method comparison

arXiv abstract · v1

Selected results

• ShareGPT4Video-8B reaches an overall TempCompass score of 61.5 versus 49.9 for VideoLLaVA-7B. On caption-generation action recognition, the corresponding scores are 79.8 and 50.8. Table 3 · temporal comprehension · arXiv v1
• ShareGPT4Video-8B scores 51.2 on MVBench, compared with the authors’ reproduced 43.0 for VideoLLaVA-7B and 41.3 for LLaMA-VID-7B. Model sizes and training recipes differ, so the table is a system comparison rather than a data-only ablation. Table 5 · MVBench evaluation · arXiv v1

Cite this paper

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, Jiaqi Wang. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. Advances in Neural Information Processing Systems, 37, 2024, pp. 19472–19495.

@inproceedings{arxiv240604325,
  title     = {{ShareGPT4Video: Improving Video Understanding and Generation with Better Captions}},
  author    = {Lin Chen and Xilin Wei and Jinsong Li and Xiaoyi Dong and Pan Zhang and Yuhang Zang and Zehui Chen and Haodong Duan and Bin Lin and Zhenyu Tang and Li Yuan and Yu Qiao and Dahua Lin and Feng Zhao and Jiaqi Wang},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2024},
  volume    = {37},
  publisher = {Curran Associates, Inc.},
  pages     = {19472--19495},
  doi       = {10.52202/079017-0614},
  url       = {https://papers.nips.cc/paper_files/paper/2024/hash/22a7476e4fd36818777c47e666f61a41-Abstract-Datasets_and_Benchmarks_Track.html}
}