ECCV · 2024
Long-CLIP: Unlocking the Long-Text Capability of CLIP
Key takeaway
Long-CLIP extends CLIP to detailed text descriptions through efficient fine-tuning while preserving the embedding alignment needed for plug-and-play downstream use. arXiv abstract · v3
Abstract
Contrastive Language-Image Pre-training (CLIP) has been the cornerstone for zero-shot classification, text-image retrieval, and text-image generation by aligning image and text modalities. Despite its widespread adoption, a significant limitation of CLIP lies in the inadequate length of text input. The length of the text token is restricted to 77, and an empirical study shows the actual effective length is even less than 20. This prevents CLIP from handling detailed descriptions, limiting its applications for image retrieval and text-to-image generation with extensive prerequisites. To this end, we propose Long-CLIP as a plug-and-play alternative to CLIP that supports long-text input, retains or even surpasses its zero-shot generalizability, and aligns the CLIP latent space, making it readily replace CLIP without any further adaptation in downstream frameworks. Nevertheless, achieving this goal is far from straightforward, as simplistic fine-tuning can result in a significant degradation of CLIP's performance. Moreover, substituting the text encoder with a language model supporting longer contexts necessitates pretraining with vast amounts of data, incurring significant expenses. Accordingly, Long-CLIP introduces an efficient fine-tuning solution on CLIP with two novel strategies designed to maintain the original capabilities, including (1) a knowledge-preserved stretching of positional embedding and (2) a primary component matching of CLIP features. With leveraging just one million extra long text-image pairs, Long-CLIP has shown the superiority to CLIP for about 20% in long caption text-image retrieval and 6% in traditional text-image retrieval tasks, e.g., COCO and Flickr30k. Furthermore, Long-CLIP offers enhanced capabilities for generating images from detailed text descriptions by replacing CLIP in a plug-and-play manner.
Author abstract · arXiv abstract · v3
Publication
European Conference on Computer Vision (ECCV), 2024
Paper and resources
Research topics
Long-CLIP · Long-text image retrieval · CLIP · Contrastive vision-language learning · Positional embedding · Efficient fine-tuning · Detailed text-to-image generation · Zero-shot generalization
Research problem and approach
CLIP's short text context restricts retrieval and generation with detailed descriptions, but naive fine-tuning can damage generalization. Long-CLIP stretches positional embeddings while preserving knowledge and matches principal feature components to retain CLIP capabilities. arXiv abstract · v3
Main contributions
- Introduces knowledge-preserved positional-embedding stretching and primary component matching. arXiv abstract · v3
- Uses one million additional long text-image pairs to extend context without training a replacement encoder from scratch. arXiv abstract · v3
Method comparison
| Approach | Key difference |
|---|---|
| Naive CLIP fine-tuning for longer text | Can degrade the original model's capabilities; replacing the encoder also entails costly pretraining. |
| Long-CLIP | Stretches positional embeddings and preserves feature knowledge through efficient fine-tuning on long captions. |
Selected results
- With ViT-L/14, Long-CLIP raises Urban-200 image-to-text and text-to-image R@1 from 47.0% to 81.5%. On short-caption COCO, text-to-image R@1 rises from 35.4% to 46.3%, while direct fine-tuning lowers it to 23.1%. Tables 1–3 · long- and short-caption evaluation · arXiv v3
- Combining knowledge-preserving stretching and primary-component matching gives 66.8% ImageNet accuracy versus 55.1% with neither component. COCO text-to-image R@5 rises from 43.4% to 65.8%, while Urban-200 text-to-image R@1 changes from 78.0% to 79.0%. Table 4 · position and feature-matching ablation · arXiv v3
Cite this paper
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Jiaqi Wang. Long-CLIP: Unlocking the Long-Text Capability of CLIP. Computer Vision – ECCV 2024, 2025, pp. 310–325.
The conference took place in 2024; the proceedings volume was published in 2025.
@inproceedings{arxiv240315378,
title = {{Long-CLIP: Unlocking the Long-Text Capability of CLIP}},
author = {Beichen Zhang and Pan Zhang and Xiaoyi Dong and Yuhang Zang and Jiaqi Wang},
booktitle = {Computer Vision – ECCV 2024},
year = {2025},
publisher = {Springer Nature Switzerland},
pages = {310--325},
doi = {10.1007/978-3-031-72983-6\_18},
url = {https://doi.org/10.1007/978-3-031-72983-6_18}
}