ICLR · 2026
ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing
Key takeaway
ScaleCap turns additional inference budget into richer, better-grounded image captions by iteratively adding missing visual details and removing hallucinated descriptions. arXiv abstract · v1
Abstract
This paper presents ScaleCap, an inference-time scalable image captioning strategy that generates comprehensive and detailed image captions. The key challenges of high-quality image captioning lie in the inherent biases of LVLMs: multimodal bias resulting in imbalanced descriptive granularity, offering detailed accounts of some elements while merely skimming over others; linguistic bias leading to hallucinated descriptions of non-existent objects. To address these issues, we propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget. Specifically, we propose two novel components: heuristic question answering and contrastive sentence rating. The former generates content-specific questions based on the image and answers them to progressively inject relevant information into the caption. The latter employs sentence-level offline contrastive decoding to effectively identify and eliminate hallucinations caused by linguistic biases. With increased inference cost, more heuristic questions are raised by ScaleCap to progressively capture additional visual details, generating captions that are more accurate, balanced, and informative. Extensive modality alignment experiments demonstrate the effectiveness of ScaleCap. Annotating 450K images with ScaleCap and using them for LVLM pretraining leads to consistent performance gains across 11 widely used benchmarks. Furthermore, ScaleCap showcases superb richness and fidelity of generated captions with two additional tasks: replacing images with captions in VQA task, and reconstructing images from captions to assess semantic coverage. Code is available at https://github.com/Cooperx521/ScaleCap.
Author abstract · arXiv abstract · v1
Publication
International Conference on Learning Representations (ICLR), 2026
Paper and resources
Research topics
ScaleCap · Dense image captioning · Inference-time scaling · Dual-modality debiasing · Hallucination reduction · Contrastive sentence rating · Visual question answering · Caption quality
Research problem and approach
Captioners can describe some image elements in detail while omitting others or inventing content. ScaleCap addresses this imbalance with heuristic questions and sentence-level contrastive checks. arXiv abstract · v1
Main contributions
- Uses image-specific question answering to progressively enrich caption coverage. arXiv abstract · v1
- Rates sentences with offline contrastive decoding to reduce language-driven hallucinations. arXiv abstract · v1
Method comparison
| Approach | Key difference |
|---|---|
| Single-pass dense captioning | May overdescribe some image elements while omitting others or hallucinating details. |
| ScaleCap | Adds missing information with targeted questions and checks sentences with contrastive rating as inference budget grows. |
Selected results
- For Qwen2.5-7B with Qwen2.5-ViT, ScaleCap-450k raises the 11-benchmark average to 64.7, versus 61.6 for vanilla data, 62.4 for ShareGPT4V-450k and 63.0 for DenseFusion-450k. The dataset-size-matched comparison isolates caption quality. Table 1 · matched 450k-caption pretraining · arXiv v1
- With a Qwen2-72B summarizer, increasing the vision-language captioner from 7B to 72B changes the four-benchmark average from 58.5 to 58.7. Keeping the 7B captioner but shrinking the summarizer to 7B lowers it to 46.5. Table 6 · captioner and summarizer scale · arXiv v1
Cite this paper
Long Xing, Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jinsong Li, Shuangrui Ding, Weiming Zhang, Nenghai Yu, Jiaqi Wang, Feng Wu, Dahua Lin. ScaleCap: Scalable Image Captioning via Dual-Modality Debiasing. International Conference on Learning Representations, 2026, 2026, pp. 32266–32285.
@inproceedings{arxiv250619848,
title = {{ScaleCap: Scalable Image Captioning via Dual-Modality Debiasing}},
author = {Long Xing and Qidong Huang and Xiaoyi Dong and Pan Zhang and Yuhang Zang and Yuhang Cao and Jinsong Li and Shuangrui Ding and Weiming Zhang and Nenghai Yu and Jiaqi Wang and Feng Wu and Dahua Lin},
booktitle = {International Conference on Learning Representations},
year = {2026},
volume = {2026},
pages = {32266--32285},
url = {https://proceedings.iclr.cc/paper_files/paper/2026/hash/37050ebbbd7096719ab96cec19a4c69f-Abstract-Conference.html}
}