Exo2EgoPolicy: Pose-Aligned Cross-View Policy Learning

Learning robot manipulation policies from large-scale human videos is challenging due to the viewpoint mismatch between thirdperson observations and egocentric robot control. To this end, we introduce Exo2EgoPolicy, a geometry-aware framework that learns egocentric manipulation policies from monocular exocentric video via viewpointaligned pose translation. Unlike prior Exo→Ego approaches that rely on diffusion-based pixel synthesis and often produce temporally inconsistent predictions, our method operates directly on a structured pose manifold and explicitly models the geometric ambiguity inherent in monocular observation. We formulate exocentric-to-egocentric translation as a viewpoint-conditioned latent variable model on the SE(3) manifold that disentangles pose from camera transformation and captures the ambiguity induced by unknown camera extrinsics. Under bounded viewpoint motion, the formulation yields pose trajectories that are identifiable up to a global rigid transformation and supports stable sequential inference through temporal regularization. Beyond representation alignment, we show that for quasi-static manipulation tasks whose rewards depend primarily on relative hand–object geometry, pose captures the key task-relevant information required for policy learning. This enables policy co-training on pose-translated human demonstrations alongside limited robot teleoperation data within a unified human–robot state space. Empirically, Exo2EgoPolicy improves temporal consistency, cross-view alignment, and out-of-distribution policy transfer compared to pixelbased translation and other cross-view co-training baselines, achieving 20–30% higher task success on manipulation benchmarks. These results suggest that geometry-aligned pose representations provide a scalable foundation for cross-view policy learning from human video. Website