TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition

Abstract

Table recognition (TR) aims to transform table images into semi-structured representations such as HTML or Markdown. As a core component of document parsing, TR has long relied on supervised learning, with recent efforts dominated by fine-tuning vision-language models (VLMs) using labeled data. While VLMs have brought TR to the next level, pushing performance further demands large-scale labeled data that is costly to obtain. Consequently, although proprietary models have continuously pushed the performance boundary, open-source models, often trained with limited resources and, in practice, the only viable option for many due to privacy regulations, still lag far behind. To bridge this gap, we introduce TRivia, a self-supervised fine-tuning method that enables pretrained VLMs to learn TR directly from unlabeled table images in the wild. Built upon Group Relative Policy Optimization, TRivia automatically identifies unlabeled samples that most effectively facilitate learning and eliminates the need for human annotations through a question-answering-based reward mechanism. An attention-guided module generates diverse questions for each table image, and the ability to interpret the recognition results and answer them correctly provides feedback to optimize the TR model. This closed-loop process allows the TR model to autonomously learn to recognize, structure, and reason over tables without labeled data. Leveraging this pipeline, we present TRivia-3B, an open-sourced, compact, and state-of-the-art TR model that surpasses existing systems (e.g., Gemini 2.5 Pro, MinerU2.5) on three popular benchmarks. Model and code are released at: https://github.com/HKU-TASR/TRivia

Author abstract · arXiv abstract · v2

Publication

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2026

Paper and resources

• Paper · arXiv v2 (PDF)
• arXiv:2512.01248
• Google Scholar
• Code · opendatalab/TRivia

Research topics

TRivia · Table recognition · Document parsing · Self-supervised fine-tuning · Unlabeled table images · Question-answering rewards · GRPO · Structured document extraction

Research problem and approach

High-quality table recognition normally requires costly labeled images. TRivia creates a self-supervised loop that selects informative samples, generates attention-guided questions and uses question-answering feedback to train the recognizer. arXiv abstract · v2

Main contributions

• Adapts GRPO to learn table structure and content from unlabeled images. arXiv abstract · v2
• Introduces TRivia-3B as a compact table-recognition model trained with this feedback loop. arXiv abstract · v2

Method comparison

arXiv abstract · v2

Selected results

• TRivia-3B obtains TEDS scores of 91.60 on OmniDocBench, 84.90 on CC-OCR and 90.76 on OCRBench, versus MinerU2.5’s 90.85, 79.76 and 87.13. The reported overall TEDS across these three benchmarks is 89.88. Table 1 · table recognition · arXiv v2
• Starting from Stage 2’s 88.57 overall TEDS, SFT with Qwen2.5-VL-72B pseudo-labels reduces performance to 80.02 and GRPO with those labels yields 83.65. TRivia’s table-QA approach reaches 89.88, supporting its use of question-answer consistency in place of direct pseudo-label supervision. Table 2 · pseudo-label ablation · arXiv v2

Cite this paper

Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He. TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 33196–33206.

@inproceedings{arxiv251201248,
  title     = {{TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition}},
  author    = {Junyuan Zhang and Bin Wang and Qintong Zhang and Fan Wu and Zichen Wen and Jialin Lu and Junjie Shan and Ziqi Zhao and Shuya Yang and Ziling Wang and Ziyang Miao and Huaping Zhong and Yuhang Zang and Xiaoyi Dong and Ka-Ho Chow and Conghui He},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2026},
  pages     = {33196--33206},
  url       = {https://openaccess.thecvf.com/content/CVPR2026/html/Zhang_TRivia_Self-supervised_Fine-tuning_of_Vision-Language_Models_for_Table_Recognition_CVPR_2026_paper.html}
}