3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

Intermediate representations are key to bridging the modal-ity gap between generalizable manipulation policies and large-scale pre-trained vision-language models (VLMs). Among these, trajectory-basedrepresentations compactly represent motion-relevant cues, yet most exist-ing approaches predict trajectories in 2D image space, resulting in intrin-sic 3D ambiguity. Moreover, using 2D trajectories with depth still leavesthe free-space waypoints ambiguous, limiting reliable 3D reasoning. Toaddress this, we propose predicting 3D consistent waypoints (3DWay)from multi-view images. By reformulating 3D waypoints prediction asgenerating multi-view consistent 2D waypoints followed by geometric tri-angulation, we enable explicit 3D motion specification while preservingthe strong priors of pretrained VLMs. The predicted waypoints can guideexisting VLA models for better generalization or be directly executed onsimple tasks. Extensive experiments show that 3DWay substantially im-proves 3D spatial grounding and vision-language reasoning, demonstrat-ing strong potential for generalizable robot manipulation. Codes will bereleased at https://github.com/ziqin-h/3DWay.