VPA-WM: Vision-Priors-Aligned World Models for Robust Visual Reinforcement Learning

Building robust world models for visual reinforcement learn-ing remains challenging under complex visual conditions. When facedwith task-irrelevant visual distractions, the representations learned byworld models tend to collapse. Pre-trained vision models (PVMs) providestrong visual priors and structured representations, yet those learned forvisual perception are incompatible with dynamic control in RL. In thiswork, we present Vision-Priors-Aligned World Model (VPA-WM1 ): aunified framework that organically integrates PVMs and world models toachieve robust visual control. We first introduce Control-Oriented Align-ment, which effectively guides PVMs to focus on control-relevant signalsthrough two auxiliary objectives: instant action prediction and short-horizon return estimation. We then design Spatio-Temporal Collabora-tive Integration mechanism, which guides the world model to adaptivelyattend to the visual cues of PVMs through two complementary designs(spatially grounded guidance and experience query integration), therebylearning representations that are visually robust and dynamically con-sistent. Superior performance on visually-distracted control benchmarksdemonstrates that VPA-WM effectively bridges the semantic gap be-tween visual perception and dynamic control, and successfully leveragesvisual priors to enhance the robust control of world models.