Advancing Complex Video Object Segmentation via Progressive Concept Construction

Abstract

We propose Segment Concept (SeC), a concept-driven video object segmentation (VOS) framework that shifts from conventional feature matching to the progressive construction and utilization of high-level, object-centric representations. SeC employs Large Vision-Language Models (LVLMs) to integrate visual cues across diverse frames, constructing robust conceptual priors. To balance semantic reasoning with computational overhead, SeC forwards the LVLMs only when a new scene appears, injecting concept-level features at those points. To rigorously assess VOS methods in scenarios demanding high-level conceptual reasoning and robust semantic understanding, we introduce the Semantic Complex Scenarios Video Object Segmentation benchmark (SeCVOS). SeCVOS comprises 160 manually annotated multi-scenario videos designed to challenge models with substantial appearance variations and dynamic scene transformations. Empirical evaluations demonstrate that SeC substantially outperforms state-of-the-art approaches, including SAM 2 and its advanced variants, on both SeCVOS and standard VOS benchmarks. In particular, SeC achieves an 11.8-point improvement over SAM 2.1 on SeCVOS, establishing a new state-of-the-art in concept-aware VOS.

Author abstract · arXiv abstract · v3

Publication

International Conference on Learning Representations (ICLR), 2026

Paper and resources

• Paper · arXiv v3 (PDF)
• arXiv:2507.15852
• Google Scholar
• Code · OpenIXCLab/SeC

Research topics

SeC · Segment Concept · Video object segmentation · Object-centric representations · Concept construction · Scene transitions · SeCVOS · Semantic tracking

Research problem and approach

Feature matching alone is brittle when objects or scenes change substantially. SeC uses large vision-language models to integrate cues across frames and injects conceptual features when new scenes appear. arXiv abstract · v3

Main contributions

• Builds high-level object representations while limiting expensive LVLM calls to scene changes. arXiv abstract · v3
• Introduces SeCVOS, 160 manually annotated videos with semantically complex scenarios. arXiv abstract · v3

Method comparison

arXiv abstract · v3

Selected results

• On SeCVOS, SeC reaches 70.0 J&F versus SAM2.1’s 58.2. On the multi-scene-change subset the scores are 67.5 and 52.4, respectively, a 15.1-point improvement. Table 4 · scene-changing video segmentation · arXiv v3
• On SeCVOS, pixel-level association alone raises J&F from 58.2 to 62.2, and adding concept guidance raises it to 70.0. On SA-V, the same progression is 78.6 → 82.4 → 82.7, showing that concept guidance has a larger effect on the scene-changing benchmark. Table 6 · component ablation · arXiv v3

Cite this paper

Zhixiong Zhang, Shuangrui Ding, Xiaoyi Dong, Songxin He, Jianfan Lin, Junsong Tang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang. Advancing Complex Video Object Segmentation via Progressive Concept Construction. International Conference on Learning Representations, 2026, 2026, pp. 130411–130434.

@inproceedings{arxiv250715852,
  title     = {{Advancing Complex Video Object Segmentation via Progressive Concept Construction}},
  author    = {Zhixiong Zhang and Shuangrui Ding and Xiaoyi Dong and Songxin He and Jianfan Lin and Junsong Tang and Yuhang Zang and Yuhang Cao and Dahua Lin and Jiaqi Wang},
  booktitle = {International Conference on Learning Representations},
  year      = {2026},
  volume    = {2026},
  pages     = {130411--130434},
  url       = {https://proceedings.iclr.cc/paper_files/paper/2026/hash/d3f48777432f64bc96b1203713f20352-Abstract-Conference.html}
}