Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in interpreting images using natural language. However, without using large-scale datasets for retraining, these models are difficult to adapt to specialized vision tasks, e.g., chart understanding. This problem is caused by a mismatch between pre-training and downstream datasets: pre-training datasets primarily concentrate on scenes and objects but contain limited information about specialized, non-object images, such as charts and tables. In this paper, we share an interesting finding that training an MLLM with Chain-of-Thought (CoT) reasoning data can facilitate model adaptation in specialized vision tasks, especially under data-limited regimes. However, we identify a critical issue within CoT data distilled from pre-trained MLLMs, i.e., the data often contains multiple factual errors in the reasoning steps. To address the problem, we propose Grounded Chain-of-Thought (GCoT), a simple bootstrapping-based approach that aims to inject grounding information (i.e., bounding boxes) into CoT data, essentially making the reasoning steps more faithful to input images. We evaluate our approach on five specialized vision tasks, which cover a variety of visual formats including charts, tables, receipts, and reports. The results demonstrate that under data-limited regimes our approach significantly improves upon fine-tuning and distillation.

Author abstract · arXiv abstract · v1

Publication

IEEE International Conference on Computer Vision (ICCV), 2025

Highlight

Paper and resources

• Paper · arXiv v1 (PDF)
• arXiv:2507.02859
• Google Scholar
• Code · maifoundations/GCoT

Research topics

Grounded Chain-of-Thought · GCoT · Data-efficient adaptation · Visual grounding · Reasoning distillation · Chart understanding · Document understanding · Bounding-box supervision

Research problem and approach

Scene-centric pretraining does not cover charts, tables and documents well, and distilled reasoning can contain factual mistakes. GCoT bootstraps reasoning data with bounding-box grounding to make it more faithful to the image. arXiv abstract · v1

Main contributions

• Injects visual grounding into chain-of-thought data through a bootstrapping procedure. arXiv abstract · v1
• Studies model adaptation on specialized visual formats under limited-data conditions. arXiv abstract · v1

Method comparison

arXiv abstract · v1

Selected results

• In the 128-sample TabMWP setting, GCoT scores 33.93 ± 0.54, compared with 31.57 ± 0.47 without augmentation and 23.57 ± 2.21 without box verification. Checking only the final answer therefore leaves a substantial gap in this experiment. Table 2 · TabMWP component ablation · arXiv v1
• Across five tested distillation sources, the reported average increases from 22.78 to 26.54 with GCoT. Improvements range from 1.54 points for GPT-4o-sourced data to 5.62 for Claude-3.5-sourced data. Table 3 · distillation-source comparison · arXiv v1

Cite this paper

Jiaer Xia, Bingkui Tong, Yuhang Zang, Rui Shao, Kaiyang Zhou. Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 208–217.

@inproceedings{arxiv250702859,
  title     = {{Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation}},
  author    = {Jiaer Xia and Bingkui Tong and Yuhang Zang and Rui Shao and Kaiyang Zhou},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2025},
  pages     = {208--217},
  url       = {https://openaccess.thecvf.com/content/ICCV2025/html/Xia_Bootstrapping_Grounded_Chain-of-Thought_in_Multimodal_LLMs_for_Data-Efficient_Model_Adaptation_ICCV_2025_paper.html}
}