Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

Estimating accurate 3D hand–object pose from in-the-wildegocentric RGB remains challenging due to severe occlusions and am-biguous contact. Existing learning-based methods often struggle to gen-eralise to in-the-wild scenes and are limited by the scarcity of super-vision. We address these issues with two contributions. First, we in-troduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips(62.3K frames) with dense, bijective 3D hand–object contact correspon-dences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object posein a single forward pass. A cross-attention decoder conditions objectfeatures on hand priors, producing robust pose estimation. We testHOPformer on the in-lab 3D dataset, ARCTIC, as well as our newlyintroduced EPIC-Contact dataset. HOPformer reaches 82.4% successrate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact,it nearly doubles the success rate while reducing contact deviation by75%. EPIC-Contact, HOPformer code and checkpoints are released:https://sid2697.github.io/epic-contact.