InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Abstract

The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.

Author abstract · arXiv abstract · v1

Publication

Neural Information Processing Systems (NeurIPS), 2024

Paper and resources

• Paper · arXiv v1 (PDF)
• arXiv:2404.06512
• Google Scholar
• Code · InternLM/InternLM-XComposer

Research topics

InternLM-XComposer2-4KHD · High-resolution vision-language models · 4K image understanding · Dynamic resolution · Automatic patch configuration · Fine-grained perception · Aspect-ratio preservation · Multimodal foundation models

Research problem and approach

Fixed or narrowly bounded image resolutions limit detailed visual understanding. The model adjusts patch counts and layouts while preserving image aspect ratios, reusing a pretrained 336-by-336 vision transformer. arXiv abstract · v1

Main contributions

• Introduces dynamic resolution with automatic patch configuration for high-resolution LVLM training. arXiv abstract · v1
• Studies scaling training resolution to 4K HD and releases a 7B-parameter model series. arXiv abstract · v1

Method comparison

arXiv abstract · v1

Selected results

• IXC2-4KHD raises DocVQA from IXC2-VL’s 57.7 to 90.0, ChartQA from 72.6 to 81.0 and InfoVQA from 34.4 to 68.6. These gains quantify the high-resolution model’s document and chart capabilities. Table 3 · high-resolution document understanding · arXiv v1
• Compared with IXC2-VL, IXC2-4KHD raises MM-Vet from 46.7 to 54.9, while MMStar changes from 55.4 to 54.1 and MMBench-English from 80.7 to 80.2. High-resolution training improves some capabilities while largely retaining others. Table 4 · general-capability comparison · arXiv v1

Cite this paper

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang. InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD. Advances in Neural Information Processing Systems, 37, 2024, pp. 42566–42592.

@inproceedings{arxiv240406512,
  title     = {{InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD}},
  author    = {Xiaoyi Dong and Pan Zhang and Yuhang Zang and Yuhang Cao and Bin Wang and Linke Ouyang and Songyang Zhang and Haodong Duan and Wenwei Zhang and Yining Li and Hang Yan and Yang Gao and Zhe Chen and Xinyue Zhang and Wei Li and Jingwen Li and Wenhai Wang and Kai Chen and Conghui He and Xingcheng Zhang and Jifeng Dai and Yu Qiao and Dahua Lin and Jiaqi Wang},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2024},
  volume    = {37},
  publisher = {Curran Associates, Inc.},
  pages     = {42566--42592},
  doi       = {10.52202/079017-1348},
  url       = {https://papers.nips.cc/paper_files/paper/2024/hash/4b06cdddb1cde6624c0be1465c7b800f-Abstract-Conference.html}
}