BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

The integration of Large Language Models (LLMs) into au-tonomous driving has attracted growing interest for their strong rea-soning and semantic understanding abilities, which are essential for han-dling complex decision-making and long-tail scenarios. However, existingmethods typically feed LLMs with tokens from multi-view and multi-frame images independently, leading to redundant computation and lim-ited spatial consistency. This separation in visual processing hinders ac-curate 3D spatial reasoning and fails to maintain geometric coherenceacross views. On the other hand, Bird’s-Eye View (BEV) representa-tions learned from geometrically annotated tasks (e.g., object detection)provide spatial structure but lack the semantic richness of foundationvision encoders. To bridge this gap, we propose BEVLM, a frameworkthat connects a spatially consistent and semantically distilled BEV rep-resentation with LLMs. Through extensive experiments, we show thatBEVLM enables LLMs to reason more effectively in cross-view driv-ing scenes, improving accuracy by 46.0 %, by leveraging BEV featuresas unified inputs. Furthermore, by distilling semantic knowledge fromLLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance in safety-critical scenarios acrossUniAD and VAD, with gains of up to 28.2 %.