Open-Vocabulary DETR with Conditional Matching

Abstract

Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-language model like CLIP, in order to enable matching for both text and image queries. With extensive experiments on LVIS and COCO datasets, we demonstrate that our OV-DETR -- the first end-to-end Transformer-based open-vocabulary detector -- achieves non-trivial improvements over current state of the arts.

Author abstract · arXiv abstract · v2

Publication

European Conference on Computer Vision (ECCV), 2022

Oral

Paper and resources

• Paper · arXiv v2 (PDF)
• arXiv:2203.11876
• Google Scholar
• Code · yuhangzang/OV-DETR
• Project website

Research topics

OV-DETR · Open-vocabulary object detection · Conditional matching · DETR · Vision-language transfer · Exemplar-based detection · CLIP embeddings · Novel-category localization

Research problem and approach

DETR's usual class-based matching cost is unavailable for unseen categories without labeled training images. OV-DETR instead matches objects to input queries and conditions the decoder on CLIP text or image embeddings to transfer to novel queries. arXiv abstract · v2

Main contributions

• Introduces binary conditional matching for end-to-end open-vocabulary transformer detection. arXiv abstract · v2
• Supports both natural-language category queries and visual exemplars within one detector. arXiv abstract · v2

Method comparison

arXiv abstract · v2

Selected results

• On OV-LVIS, OV-DETR improves novel-class mask AP from ViLD’s 16.1 to 17.4 and overall mask AP from 22.5 to 26.6. On OV-COCO, novel-class box AP50 rises from 27.6 to 29.4; the two benchmarks use different AP definitions. Table 3 · open-vocabulary detection · arXiv v2
• On OV-LVIS, adding proposals without conditional matching yields 6.3 novel-class mask AP. Adding conditional binary matching with those proposals raises it to 17.4, versus 9.5 in the baseline without either component. Table 2 · conditional-matching ablation · arXiv v2

Cite this paper

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy. Open-Vocabulary DETR with Conditional Matching. Computer Vision – ECCV 2022, 2022, pp. 106–122.

@inproceedings{arxiv220311876,
  title     = {{Open-Vocabulary DETR with Conditional Matching}},
  author    = {Yuhang Zang and Wei Li and Kaiyang Zhou and Chen Huang and Chen Change Loy},
  booktitle = {Computer Vision – ECCV 2022},
  year      = {2022},
  publisher = {Springer Nature Switzerland},
  pages     = {106--122},
  doi       = {10.1007/978-3-031-20077-9\_7},
  url       = {https://doi.org/10.1007/978-3-031-20077-9_7}
}