RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios

Multimodal large language models (MLLMs) have demon-strated powerful capabilities in general spatial understanding and rea-soning. However, their fine-grained spatial understanding and reasoningcapabilities in complex urban scenarios have not received significant at-tention in the fields of both research and industry. To fill this gap, wefocus primarily on road markings as a typical example of fine-grainedspatial elements under urban scenarios, given the essential role of theintegrated road traffic network they form within cities. Around roadmarkings and urban traffic systems, we propose RoadBench, a system-atic benchmark that comprehensively evaluates MLLMs’ fine-grainedspatial understanding and reasoning capabilities using Bird’s-Eye View(BEV) and First-Person View (FPV) image inputs. This benchmarkcomprises eight tasks consisting of 3,040 strictly manually verified testcases, constructed from 2,137 unique BEV images and 721 unique FPVimages collected from five Chinese cities with relatively consistent trafficconventions. These tasks form a systematic evaluation framework thatbridges understanding at local spatial scopes to global reasoning. Theynot only test MLLMs’ capabilities in recognition, joint understanding,and reasoning but also assess their ability to integrate image informationwith domain knowledge. After evaluating 20 mainstream MLLMs, weconfirm that RoadBench is a challenging benchmark for MLLMs whilerevealing significant shortcomings in existing MLLMs’ fine-grained spa-tial understanding and reasoning capabilities within urban scenarios. Incertain tasks, their performance even falls short of simple rule-basedor random selection baselines. These findings, along with RoadBenchitself, will contribute to the comprehensive advancement of spatial un-derstanding capabilities for MLLMs. The benchmark code is availableat https://github.com/tsinghua-fib-lab/RoadBench, and the supple-mentary material provides example data, prompts, evaluation scripts,and raw evaluation results.Road MarkingsBEV Image # of Lanes: 4Road MarkingsFPV Image # of Lanes: 4Fig. 1: Examples of road markings in BEV and FPV images. Road markings such aslane dividers and turning arrows provide fine-grained visual cues for lane counting andlane designation recognition.