Less is More: A Simple yet Effective Object-Centric Prompting Strategy for Vision-Language Reasoning in Autonomous Driving

Large Vision-Language Models (VLMs) show immense po-tential in autonomous driving but struggle with precise spatial ground-ing, often hallucinating when forced to regress continuous image coordi-nates. To overcome this, we introduce a "Less is More" object-centricprompting strategy that fundamentally decouples spatial localizationfrom logical reasoning. By overlaying detected bounding boxes and dis-crete IDs directly onto multi-view images, the VLM only predicts dis-crete IDs, which are deterministically mapped back to precise coordi-nates via an offline lookup table. This model-agnostic approach trans-forms error-prone regression into reliable discrete matching without re-quiring structural modifications. Extensive evaluations on the DriveLM-nuScenes dataset demonstrate striking improvements in comprehensivedriving reasoning and spatial grounding. Notably, our method booststhe Accuracy and Match scores of the DriveLM-agent by absolute mar-gins of 8.26% and 10.39%, respectively. Furthermore, when applied tostronger models, it achieves impressive increases of 11.85% and 20.02%in Accuracy and Match for Qwen2.5-VL, as well as 8.75% and 19.88% forInternVL-3.5. These consistent gains definitively prove the strong plug-and-play generalizability of our paradigm across varying architectures.