OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

Recent advancements in LVLMs necessitate robust bench-marks for complex, visually grounded reasoning. A critical limitation isidentified in many document understanding benchmarks: visual contentis often reducible to text, enabling high performance without genuine vi-sual grounding. To address this limitation, OmniMapBench is introducedto foster visual-centric reasoning for map documents. The benchmarkcomprises 2,096 manually annotated question-answer pairs across 1,603map documents from nine categories. It is designed to probe a hierar-chy of skills, ranging from perception to multi-step visual reasoning. Toquantify benchmark properties, a simple yet effective benchmark-levelmetric is proposed: the Visual Dependency Index (VDI), defined as theaccuracy drop when images are replaced with question-agnostic descrip-tions. OmniMapBench exhibits higher VDI than established benchmarks,which quantitatively validates its focus on irreducible visual reasoning.Comprehensive evaluations of 25 leading LVLMs are conducted on Om-niMapBench. A significant performance gap is observed, with the top-performing model achieving only 75.03% accuracy. This result under-scores the challenges posed by OmniMapBench to current LVLMs. Thiswork aims to catalyze progress in visual-centric reasoning for documentunderstanding of LVLMs. The dataset and code are publicly available athttps://github.com/SIGMME/OmniMapBench.