RAR: Retrieving and Ranking Augmented MLLMs for Visual Recognition

Ziyu Liu; Zeyi Sun; Yuhang Zang; Wei Li; Pan Zhang; Xiaoyi Dong; Yuanjun Xiong; Dahua Lin; Jiaqi Wang

doi:10.1109/tip.2025.3644175

TIP · 2026

RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition

Ziyu Liu, Zeyi Sun, Yuhang Zang(corresponding author), Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang(corresponding author)

Corresponding author

Key takeaway

RAR combines CLIP retrieval with MLLM reranking to recognize fine-grained categories without placing an entire large vocabulary into the language model's context. arXiv abstract · v2

Abstract

CLIP (Contrastive Language-Image Pre-training) uses contrastive learning from noise image-text pairs to excel at recognizing a wide array of candidates, yet its focus on broad associations hinders the precision in distinguishing subtle differences among fine-grained items. Conversely, Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories, thanks to their substantial knowledge from pre-training on web-level corpora. However, the performance of MLLMs declines with an increase in category numbers, primarily due to growing complexity and constraints of limited context window size. To synergize the strengths of both approaches and enhance the few-shot/zero-shot recognition abilities for datasets characterized by extensive and fine-grained vocabularies, this paper introduces RAR, a Retrieving And Ranking augmented method for MLLMs. We initially establish a multi-modal retriever based on CLIP to create and store explicit memory for different categories beyond the immediate context window. During inference, RAR retrieves the top-k similar results from the memory and uses MLLMs to rank and make the final predictions. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base, significantly boosting accuracy across a range of vision-language recognition tasks. Notably, our approach demonstrates a significant improvement in performance on 5 fine-grained visual recognition benchmarks, 11 few-shot image recognition datasets, and the 2 object detection datasets under the zero-shot recognition setting.

Author abstract · arXiv abstract · v2

Publication

IEEE Transactions on Image Processing (TIP), 2026

Paper and resources

Research topics

RAR · Retrieval-augmented recognition · Multimodal reranking · Fine-grained classification · CLIP · Few-shot recognition · Zero-shot recognition · Category memory

Research problem and approach

CLIP scales to many categories but can miss subtle distinctions, whereas MLLMs struggle with large candidate sets. RAR retrieves a small set of relevant category memories and lets the MLLM rank them. arXiv abstract · v2

Main contributions

Builds an explicit multimodal category memory outside the immediate context window. arXiv abstract · v2
Separates scalable candidate retrieval from fine-grained semantic classification. arXiv abstract · v2

Method comparison

Approach	Key difference
CLIP category matching	Scales to large candidate vocabularies but can miss fine-grained distinctions.
Direct MLLM classification	Offers semantic reasoning but is constrained by large candidate sets in the context window.
RAR	Uses CLIP to retrieve category memories and an MLLM to rerank the selected candidates.

arXiv abstract · v2

Selected results

Across five fine-grained datasets, RAR achieves 58.5% average clustering accuracy and 65.3% semantic-similarity accuracy, compared with FineR’s 57.0% and 64.3%. Table II · fine-grained recognition · arXiv v2
Using CLIP ViT-L/14@336 features and LLaVA-1.5 reranking, RAR raises average top-1 accuracy across 11 datasets from 65.0% to 69.9% in the 4-shot setting and from 70.8% to 75.5% in the 8-shot setting. Table IV · CLIP ViT-L/14@336 retrieval · arXiv v2

Cite this paper

Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang. RAR: Retrieving and Ranking Augmented MLLMs for Visual Recognition. IEEE Transactions on Image Processing, 35, 2026, pp. 388–401.

Download BibTeX TIP publisher record

@article{arxiv240313805,
  title     = {{RAR: Retrieving and Ranking Augmented MLLMs for Visual Recognition}},
  author    = {Ziyu Liu and Zeyi Sun and Yuhang Zang and Wei Li and Pan Zhang and Xiaoyi Dong and Yuanjun Xiong and Dahua Lin and Jiaqi Wang},
  journal   = {IEEE Transactions on Image Processing},
  year      = {2026},
  volume    = {35},
  publisher = {Institute of Electrical and Electronics Engineers (IEEE)},
  pages     = {388--401},
  doi       = {10.1109/tip.2025.3644175},
  url       = {https://doi.org/10.1109/tip.2025.3644175}
}