Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization

Abstract

Existing vision-language models exhibit strong generalization on a variety of visual domains and tasks. However, such models mainly perform zero-shot recognition in a closed-set manner, and thus struggle to handle open-domain visual concepts by design. There are recent finetuning methods, such as prompt learning, that not only study the discrimination between in-distribution (ID) and out-of-distribution (OOD) samples, but also show some improvements in both ID and OOD accuracies. In this paper, we first demonstrate that vision-language models, after long enough finetuning but without proper regularization, tend to overfit the known classes in the given dataset, with degraded performance on unknown classes. Then we propose a novel approach OGEN to address this pitfall, with the main focus on improving the OOD GENeralization of finetuned models. Specifically, a class-conditional feature generator is introduced to synthesize OOD features using just the class name of any unknown class. Such synthesized features will provide useful knowledge about unknowns and help regularize the decision boundary between ID and OOD data when optimized jointly. Equally important is our adaptive self-distillation mechanism to regularize our feature generation model during joint optimization, i.e., adaptively transferring knowledge between model states to further prevent overfitting. Experiments validate that our method yields convincing gains in OOD generalization performance in different settings. Code: https://github.com/apple/ml-ogen.

Author abstract · arXiv abstract · v2

Publication

International Conference on Learning Representations (ICLR), 2024

Paper and resources

• Paper · arXiv v2 (PDF)
• arXiv:2401.15914
• Google Scholar

Research topics

OGEN · Out-of-distribution generalization · Vision-language fine-tuning · Unknown-class recognition · Feature synthesis · Adaptive self-distillation · Prompt learning · Open-domain recognition

Research problem and approach

Prolonged fine-tuning can improve known classes while degrading recognition of unknown ones. OGEN regularizes this process with a class-conditional generator that synthesizes unknown-class features from class names and transfers knowledge between model states. arXiv abstract · v2

Main contributions

• Identifies overfitting to known classes as a failure mode of insufficiently regularized vision-language fine-tuning. arXiv abstract · v2
• Combines unknown-class feature generation with adaptive self-distillation during joint optimization. arXiv abstract · v2

Method comparison

arXiv abstract · v2

Selected results

• Across 11 datasets, adding OGEN to CoOp raises new-class accuracy from 63.22% to 69.54% while base-class accuracy changes from 82.69% to 83.47%. The harmonic mean rises from 71.66% to 75.87% in the main comparison. Table 1 · base-to-new generalization · arXiv v2
• In the ablation, the feature generator alone raises new-class accuracy from 63.22% to 69.02%; adding adaptive self-distillation reaches 69.54%. Joint class extrapolation scores 69.02% on new classes versus 64.08% without extrapolation. Tables 3–4 · feature-generation ablation · arXiv v2

Cite this paper

Yuhang Zang, Hanlin Goh, Joshua Susskind, Chen Huang. Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization. International Conference on Learning Representations, 2024, 2024, pp. 30327–30342.

This citation uses the author list of the published version.

@inproceedings{arxiv240115914,
  title     = {{Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization}},
  author    = {Yuhang Zang and Hanlin Goh and Joshua Susskind and Chen Huang},
  booktitle = {International Conference on Learning Representations},
  year      = {2024},
  volume    = {2024},
  pages     = {30327--30342},
  url       = {https://proceedings.iclr.cc/paper_files/paper/2024/hash/8140f43b06c9c7e14fb96953caed2665-Abstract-Conference.html}
}