LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning

Existing vision-and-language navigation (VLN) models pri-marily reason over past and current visual observations, while largelyignoring the future visual dynamics induced by actions. As a result,they often lack an effective understanding of the causal relationship be-tween actions and how the visual world changes, limiting robust decision-making. Humans, in contrast, can “imagine” the near future by leveragingaction–dynamics causality, which improves both environmental under-standing and navigation choices. Inspired by this capability, we proposeLatentPilot, a new paradigm that exploits future observations duringtraining as a valuable data source to learn action-conditioned visual dy-namics, while requiring no access to future frames at inference. Con-cretely, we propose a flywheel-style training mechanism that iterativelycollects on-policy trajectories and retrains the model to better match theagent’s behavior distribution, with an expert takeover triggered whenthe agent deviates excessively. LatentPilot further learns visual latenttokens without explicit supervision; these latent tokens attend globallyin a continuous latent space and are carried across steps, serving asboth the current output and the next input, which enabling the agentto “dream ahead” and reason about how actions will affect subsequentobservations. Experiments on R2R-CE, RxR-CE, and R2R-PE bench-marks achieve new SOTA results, and real-robot tests across diverseenvironments demonstrate LatentPilot’s superior understanding of envi-ronment–action dynamics in scene.