NeurIPS · 2026
SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction
Key takeaway
SetCon replaces per-target [SEG] tokens with natural-language set-level concepts generated by the LVLM, raising MUSE val gIoU from 44.8 for the previous best method to 56.9. Table 1 · arXiv v1
Abstract
Referring segmentation grounds natural-language queries to pixel-level masks, but extending it to complex scenarios with multiple instances, cross-category groups, or open-ended target sets remains challenging. Previous Large Vision Language Model (LVLM)-based methods represent referred targets with one or more special tokens sequentially, treating multiple targets as separate outputs rather than a coherent set and offering little incentive to capture set-level properties such as completeness and mutual exclusivity. We reformulate open-ended referring segmentation as explicit set-level concept prediction and propose Set-Concept Segmentation (SetCon), which uses LVLM-generated natural-language concepts, instead of segmentation-specific tokens, as semantic conditions for joint mask-set decoding. A hierarchical semantic decomposition first predicts a shared set-level concept defining the target scope and then refines it into fine-grained concept groups aligned with target subsets. To support this, a two-stage annotation pipeline augments existing reasoning segmentation datasets with hierarchical semantic supervision (236k samples, 784k concept phrases). SetCon achieves state-of-the-art results on image benchmarks (+3.3 gIoU on gRefCOCO, +12.1 gIoU on MUSE), with margins that grow as the number of referred targets increases. The concept interface also transfers to video under a detect-and-track setting, yielding new state-of-the-art results on seven referring video benchmarks, including +10.9 J&F on MeViS and +12.4 J&F on Ref-SeCVOS.
Author abstract · arXiv abstract · v1
Publication
Neural Information Processing Systems (NeurIPS), 2026
Paper and resources
Research topics
SetCon · Referring segmentation · Open-ended segmentation · Multi-target segmentation · Set-level concept prediction · Hierarchical semantic decomposition · Referring video object segmentation · Large vision-language models · SAM 3 · Hierarchical semantic annotation
Research problem and approach
LVLM-based referring segmentation usually emits one special token per target and decodes each mask separately, so multi-target and cross-category queries yield duplicated or missing masks. A pilot study with Sa2VA finds that these tokens cluster by spatial position rather than semantic category. SetCon instead predicts a set-level concept and its sub-category concepts as text, and uses their hidden states to condition joint mask-set decoding with SAM 3. Sections 3.1–3.2 · arXiv v1
Main contributions
- Explicit set-level concept prediction: free-form concept phrases, each bound to a set of instances, serve as semantic conditions for decoding the whole mask set. Section 3.2 · arXiv v1
- Hierarchical semantic decomposition: a global concept defines the target scope and sub-category concepts, fused with it, assign fine-grained labels through mask-level Hungarian matching. Section 3.2 · arXiv v1
- A two-stage annotation pipeline adds hierarchical concepts to ReasonSeg and MUSE, producing 236,396 samples and 784,809 concept phrases. Section 3.3 · arXiv v1
- For video, the LVLM runs once per clip and its concepts condition the SAM 3 detect-and-track pipeline without video-specific changes. Section 3.2 · arXiv v1
Method comparison
| Approach | Segmentation interface | Multi-target handling |
|---|---|---|
| [SEG]-token LVLMs (e.g. LISA, PixelLM, Sa2VA) | Hidden states of repeated special tokens, organized mainly by spatial layout | One token per target, each mask decoded independently |
| SetCon | Hidden states of generated natural-language concepts | Global set concept plus sub-category concepts decode the mask set jointly |
Selected results
| Setting / metric | Baseline | SetCon | Gain | Source |
|---|---|---|---|---|
| MUSE val · vs. PixelLM-Llama2-13B gIoU | 44.8 | 56.9 | +12.1 | Table 1 · arXiv v1 |
| gRefCOCO val · vs. MLLMSeg gIoU | 75.1 | 78.4 | +3.3 | Table 1 · arXiv v1 |
| MeViS v1 · vs. Sa2VA J&F | 51.5 | 62.4 | +10.9 | Table 2 · arXiv v1 |
| Ref-SeCVOS · vs. GLUS J&F | 59.8 | 72.2 | +12.4 | Table 2 · arXiv v1 |
| Ablation · MUSE test · implicit-token variant gIoU | 43.6 | 53.2 | +9.6 | Table 3 · arXiv v1 |
The ablation row compares SetCon with a variant that uses the implicit-token formulation, on the MUSE test set. Gains are absolute points.
Cite this paper
Zhixiong Zhang, Yizhuo Li, Shuangrui Ding, Yuhang Zang, Shengyuan Ding, Long Xing, Yibin Wang, Qiaosheng Zhang, Jiaqi Wang. SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction. Advances in Neural Information Processing Systems, 2026.
@inproceedings{arxiv260520110,
title = {{SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction}},
author = {Zhixiong Zhang and Yizhuo Li and Shuangrui Ding and Yuhang Zang and Shengyuan Ding and Long Xing and Yibin Wang and Qiaosheng Zhang and Jiaqi Wang},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://arxiv.org/abs/2605.20110}
}