ObjectForesight: Predicting 3D Object Trajectories from Human Videos

Humans can effortlessly anticipate how objects might moveor change through interaction—imagining a cup being lifted, a knifeslicing, or a lid being closed. We aim to endow computational systemswith a similar ability to predict plausible future object motions directlyfrom passive visual observation. We introduce ObjectForesight, a 3Dobject-centric dynamics model that predicts future 6-DoF poses andtrajectories of rigid objects from short egocentric video sequences. Un-like conventional world/dynamics models that operate in pixel or latentspace, ObjectForesight represents the world explicitly in 3D at the objectlevel, enabling geometrically grounded and temporally coherent predic-tions that capture object affordances and trajectories. To train such amodel at scale, we leverage recent advances in segmentation, mesh re-construction, and 3D pose estimation to curate a dataset of 2+ millionshort clips with pseudo-ground-truth 3D object trajectories. Through ex-tensive experiments, we show that ObjectForesight achieves significantgains in accuracy, geometric consistency, and generalization to unseenobjects and scenes—establishing a scalable framework for learning phys-ically grounded, object-centric dynamics models directly from observa-tion. Project page: https://objectforesight.github.io