NeurIPS · 2024
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations
Key takeaway
MMLongBench-Doc exposes a substantial gap in long-document visual understanding through questions requiring diverse evidence, cross-page reasoning and recognition of unanswerable queries. arXiv abstract · v3
Abstract
Understanding documents with rich layouts and multi-modal components is a long-standing and practical task. Recent Large Vision-Language Models (LVLMs) have made remarkable strides in various tasks, particularly in single-page document understanding (DU). However, their abilities on long-context DU remain an open problem. This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. Distinct from previous datasets, it is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Towards comprehensive evaluation, answers to these questions rely on pieces of evidence from (1) different sources (text, image, chart, table, and layout structure) and (2) various locations (i.e. page number). Moreover, 33.2% of the questions are cross-page questions requiring evidence across multiple pages. 22.8% of the questions are designed to be unanswerable for detecting potential hallucinations. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models. Notably, the best-performing model, GPT-4o, achieves an F1 score of only 42.7%, while the second-best, GPT-4V, scores 31.4%. Furthermore, 12 LVLMs (all except GPT-4o and GPT-4V) even present worse performance than their LLM counterparts which are fed with lossy-parsed OCR documents. These results validate the necessity of future research toward more capable long-context LVLMs. Project Page: https://mayubo2333.github.io/MMLongBench-Doc
Author abstract · arXiv abstract · v3
Publication
Neural Information Processing Systems (NeurIPS), 2024 (Datasets and Benchmarks Track)
Spotlight
Paper and resources
Research topics
MMLongBench-Doc · Long-document understanding · Multimodal documents · Cross-page reasoning · Document question answering · Evidence integration · Unanswerable questions · Hallucination evaluation
Research problem and approach
Strong single-page document results do not establish long-document competence. MMLongBench-Doc evaluates evidence integration over lengthy PDFs containing text, images, charts, tables and layout structure. arXiv abstract · v3
Main contributions
- Provides 1,062 expert-annotated questions over 130 documents averaging 49.4 pages and 20,971 text tokens. arXiv abstract · v3
- Includes 33.2% cross-page questions and 22.8% unanswerable questions to test evidence integration and hallucination. arXiv abstract · v3
Method comparison
| Approach | Key difference |
|---|---|
| Single-page document evaluation | Tests local document understanding without requiring evidence integration across a lengthy PDF. |
| MMLongBench-Doc | Combines multiple evidence types, cross-page questions and unanswerable queries over long documents. |
Selected results
- MMLongBench-Doc averages 47.5 pages and 21,214.1 tokens per document. Cross-page questions account for 33.0%, and unanswerable questions for 22.5%; answer evidence is located at page 23.6 on average. Table 1 · long-document benchmark design · arXiv v3
- With Tesseract OCR, GPT-4o reaches 30.1 generalized accuracy and 30.5 F1. Its single-page accuracy is 35.4 versus 29.3 for cross-page questions, quantifying the gap under this text-extraction pipeline. Table 3 · OCR-plus-LLM evaluation · arXiv v3
Cite this paper
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, Aixin Sun. MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations. Advances in Neural Information Processing Systems, 37, 2024, pp. 95963–96010.
@inproceedings{arxiv240701523,
title = {{MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations}},
author = {Yubo Ma and Yuhang Zang and Liangyu Chen and Meiqi Chen and Yizhu Jiao and Xinze Li and Xinyuan Lu and Ziyu Liu and Yan Ma and Xiaoyi Dong and Pan Zhang and Liangming Pan and Yu-Gang Jiang and Jiaqi Wang and Yixin Cao and Aixin Sun},
booktitle = {Advances in Neural Information Processing Systems},
year = {2024},
volume = {37},
publisher = {Curran Associates, Inc.},
pages = {95963--96010},
doi = {10.52202/079017-3041},
url = {https://papers.nips.cc/paper_files/paper/2024/hash/ae0e43289bffea0c1fa34633fc608e92-Abstract-Datasets_and_Benchmarks_Track.html}
}