CVPR · 2024
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Key takeaway
Alpha-CLIP adds an alpha channel to focus CLIP on a selected image region while retaining whole-image recognition and supporting region-aware understanding and generation. arXiv abstract · v2
Abstract
Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. However, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the requirements, we introduce Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks.
Author abstract · arXiv abstract · v2
Publication
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
Paper and resources
Research topics
Alpha-CLIP · Region-aware CLIP · Alpha channel · Region-text alignment · Visual prompting · Open-world recognition · Controllable generation · Multimodal representations
Research problem and approach
Whole-image embeddings can include details irrelevant to a user's intended region. Alpha-CLIP accepts region indications from points, masks or boxes and learns region-text alignment from millions of RGBA examples. arXiv abstract · v2
Main contributions
- Extends CLIP's visual input with an auxiliary alpha channel for controllable regional attention. arXiv abstract · v2
- Constructs region-text training pairs and studies transfer to recognition, multimodal language models and 2D/3D generation. arXiv abstract · v2
Method comparison
| Approach | Key difference |
|---|---|
| Whole-image CLIP | Represents the complete image, including details outside a task's region of interest. |
| Alpha-CLIP | Adds an alpha channel and region-text training so the representation can emphasize a specified region. |
Selected results
- With ViT-L/14 on ImageNet-S, Alpha-CLIP with a foreground alpha mask achieves 77.41% top-1 accuracy versus original CLIP’s 73.48%. Top-5 accuracy rises from 91.60% to 94.45%. Table 2 · region-aware zero-shot classification · arXiv v2
- Alpha-CLIP obtains 73.37% top-1 accuracy with a whole-image alpha map, 75.62% with a rectangular box and 77.41% with a foreground mask. The graded comparison isolates the value of increasingly precise region input. Table 3 · alpha-input ablation · arXiv v2
Cite this paper
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang. Alpha-CLIP: A CLIP Model Focusing on Wherever You Want. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 13019–13029.
@inproceedings{arxiv231203818,
title = {{Alpha-CLIP: A CLIP Model Focusing on Wherever You Want}},
author = {Zeyi Sun and Ye Fang and Tong Wu and Pan Zhang and Yuhang Zang and Shu Kong and Yuanjun Xiong and Dahua Lin and Jiaqi Wang},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {13019--13029},
url = {https://openaccess.thecvf.com/content/CVPR2024/html/Sun_Alpha-CLIP_A_CLIP_Model_Focusing_on_Wherever_You_Want_CVPR_2024_paper.html}
}