arXiv · 2022
Unified Vision and Language Prompt Learning
Key takeaway
Unified Prompt Tuning jointly optimizes visual and textual prompts, combining complementary strengths that neither unimodal prompting method consistently provides alone. arXiv abstract · v1
Abstract
Prompt tuning, a parameter- and data-efficient transfer learning paradigm that tunes only a small number of parameters in a model's input space, has become a trend in the vision community since the emergence of large vision-language models like CLIP. We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that none of the unimodal prompt tuning methods performs consistently well: text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities. Extensive experiments on over 11 vision datasets show that UPT achieves a better trade-off than the unimodal counterparts on few-shot learning benchmarks, as well as on domain generalization benchmarks. Code and models will be released to facilitate future research.
Author abstract · arXiv abstract · v1
Publication
arXiv 2022
Paper and resources
Research topics
Unified Prompt Tuning · UPT · Multimodal prompt learning · Visual prompts · Text prompts · Few-shot learning · Domain generalization · Parameter-efficient adaptation
Research problem and approach
Text prompting struggles with high within-class visual variation, while visual prompting struggles with low between-class variation. UPT learns a small network that coordinates prompts across modalities for parameter- and data-efficient transfer. arXiv abstract · v1
Main contributions
- Systematically analyzes the complementary limitations of text and visual prompt tuning. arXiv abstract · v1
- Introduces a lightweight joint optimization approach for vision-language prompts. arXiv abstract · v1
Method comparison
| Approach | Key difference |
|---|---|
| Text prompt tuning | Struggles in the studied settings with high within-class visual variation. |
| Visual prompt tuning | Struggles in the studied settings with low between-class variation. |
| Unified Prompt Tuning | Jointly optimizes prompts across the two modalities through a small neural network. |
Selected results
- Averaged over three runs, UPT reaches 72.63% ImageNet accuracy and 59.98% out-of-domain average, compared with CoOp’s 71.51% and 59.28%. The target set includes ImageNet-V2, Sketch, A and R. Table 1 · 16-shot domain generalization · arXiv v1
- Across 11 datasets in the 16-shot setting, UPT achieves 81.44% average accuracy versus 78.70% for independently joint-trained visual and textual prompts, 77.88% for shared prompts and 78.73% for an MLP coupling. Table 2 · multimodal-prompt design ablation · arXiv v1
Cite this paper
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy. Unified Vision and Language Prompt Learning. arXiv preprint arXiv:2210.07225, 2022.
@article{arxiv221007225,
title = {{Unified Vision and Language Prompt Learning}},
author = {Yuhang Zang and Wei Li and Kaiyang Zhou and Chen Huang and Chen Change Loy},
journal = {arXiv preprint arXiv:2210.07225},
year = {2022},
url = {https://arxiv.org/abs/2210.07225}
}