IJCV · 2025
Contextual Object Detection with Multimodal Large Language Models
Key takeaway
ContextDET grounds objects within language interaction by generating context-relevant object words and then detecting their image locations in an end-to-end multimodal model. arXiv abstract · v2
Abstract
Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, i.e., object detection. In this work, we address this limitation by introducing a novel research problem of contextual object detection -- understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering. Moreover, we present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts, so as to locate, identify, and associate visual objects with language inputs for human-AI interaction. Our ContextDET involves three key submodels: (i) a visual encoder for extracting visual representations, (ii) a pre-trained LLM for multimodal context decoding, and (iii) a visual decoder for predicting bounding boxes given contextual object words. The new generate-then-detect framework enables us to detect object words within human vocabulary. Extensive experiments show the advantages of ContextDET on our proposed CODE benchmark, open-vocabulary detection, and referring image segmentation. Github: https://github.com/yuhangzang/ContextDET.
Author abstract · arXiv abstract · v2
Publication
International Journal of Computer Vision (IJCV), 2025
Paper and resources
Research topics
ContextDET · Contextual object detection · Generate-then-detect · Multimodal large language models · Visual grounding · CODE benchmark · Open-vocabulary detection · Human-AI interaction
Research problem and approach
Image captioning and question answering do not by themselves provide contextual object detection. ContextDET connects a visual encoder, a pretrained language model and a bounding-box decoder to associate visual objects with the current language context. arXiv abstract · v2
Main contributions
- Defines contextual object detection across language cloze tests, visual captioning and question answering. arXiv abstract · v2
- Introduces the generate-then-detect framework and the CODE benchmark. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Captioning and question answering alone | Produces language responses without explicit contextual bounding-box predictions. |
| ContextDET | Generates context-relevant object words and decodes their locations within a differentiable multimodal model. |
Selected results
- ContextDET reaches 48.7% top-1 cloze accuracy and 8.1 QA AP@1, compared with LLaVA-1.5 + GLIP’s 42.9% and 6.5. Captioning AP@1 is 6.4 versus 5.5 in the same comparison. Table 2 · CODE validation benchmark · arXiv v2
- Adding local visual tokens raises cloze top-1 accuracy from 30.9% to 48.7% and AP@1 from 4.0 to 10.4 in the ablation. These results support the role of local evidence in contextual object prediction. Table 4 · local-visual-token ablation · arXiv v2
Cite this paper
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, Chen Change Loy. Contextual Object Detection with Multimodal Large Language Models. International Journal of Computer Vision, 133(2), 2025, pp. 825–843.
@article{arxiv230518279,
title = {{Contextual Object Detection with Multimodal Large Language Models}},
author = {Yuhang Zang and Wei Li and Jun Han and Kaiyang Zhou and Chen Change Loy},
journal = {International Journal of Computer Vision},
month = {February},
year = {2025},
volume = {133},
number = {2},
publisher = {Springer Science and Business Media LLC},
pages = {825--843},
doi = {10.1007/s11263-024-02214-4},
url = {https://doi.org/10.1007/s11263-024-02214-4}
}