CVPR · 2025
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
Key takeaway
PyramidDrop retains visual detail in shallow LVLM layers and progressively removes redundant tokens in deeper layers to reduce training and inference cost. arXiv abstract · v2
Abstract
In large vision-language models (LVLMs), images serve as inputs that carry a wealth of information. As the idiom "A picture is worth a thousand words" implies, representing a single image in current LVLMs can require hundreds or even thousands of tokens. This results in significant computational costs, which grow quadratically as input image resolution increases, thereby severely impacting the efficiency of both training and inference. Previous approaches have attempted to reduce the number of image tokens either before or within the early layers of LVLMs. However, these strategies inevitably result in the loss of crucial image information, ultimately diminishing model performance. To address this challenge, we conduct an empirical study revealing that all visual tokens are necessary for LVLMs in the shallow layers, and token redundancy progressively increases in the deeper layers of the model. To this end, we propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost their efficiency in both training and inference with neglectable performance loss. Specifically, we partition the LVLM into several stages and drop part of the image tokens at the end of each stage with a pre-defined ratio, creating pyramid-like visual tokens across model layers. The dropping is based on a lightweight similarity calculation with a negligible time overhead. Extensive experiments demonstrate that PyramidDrop can achieve a 40% training time and 55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance. Besides, the PyramidDrop could also serve as a plug-and-play strategy for inference acceleration without training, with better performance and lower inference cost than counterparts. Code is available at https://github.com/Cooperx521/PyramidDrop.
Author abstract · arXiv abstract · v2
Publication
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025
Paper and resources
Research topics
PyramidDrop · Visual token reduction · Efficient vision-language models · Training efficiency · Inference acceleration · Depth-dependent redundancy · Token pruning · High-resolution understanding
Research problem and approach
Early visual-token reduction can discard information before the model has processed it. PyramidDrop observes increasing redundancy with depth and drops a proportion of image tokens at stage boundaries using lightweight similarity scores. arXiv abstract · v2
Main contributions
- Establishes a depth-dependent account of visual-token redundancy in large vision-language models. arXiv abstract · v2
- Provides a staged token-reduction strategy for training and a plug-and-play inference setting. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Visual-token reduction before or in early layers | Can remove important image information before sufficient processing. |
| PyramidDrop | Progressively drops tokens at later stage boundaries as visual redundancy increases. |
Selected results
- Using PyramidDrop during training reduces GPU-hours from 366 to 218 (40.4%) and inference FLOPs from 20.8T to 9.46T. The eight-benchmark average changes from 67.6 to 67.5 under this five-patch setting. Table 3 · LLaVA-NeXT-7B, five image patches · arXiv v2
- At an average of 128 retained visual tokens on LLaVA-1.5-7B, PyramidDrop obtains a benchmark average of 66.4, retaining 95.6% of the uncompressed model’s 69.4. SparseVLM and FastV score 64.6 and 58.2 at the same token budget. Table 2 · inference-only token reduction · arXiv v2
Cite this paper
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, Dahua Lin. Conical Visual Concentration for Efficient Large Vision-Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 14593–14603.
@inproceedings{arxiv241017247,
title = {{Conical Visual Concentration for Efficient Large Vision-Language Models}},
author = {Long Xing and Qidong Huang and Xiaoyi Dong and Jiajie Lu and Pan Zhang and Yuhang Zang and Yuhang Cao and Conghui He and Jiaqi Wang and Feng Wu and Dahua Lin},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2025},
pages = {14593--14603},
url = {https://openaccess.thecvf.com/content/CVPR2025/html/Xing_Conical_Visual_Concentration_for_Efficient_Large_Vision-Language_Models_CVPR_2025_paper.html}
}