Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch

We propose a multimodal, physically grounded approach formetric-scale amodal object reconstruction and pose estimation undersevere hand occlusion. Unlike prior occlusion-aware 3D generation meth-ods that rely only on vision, we leverage physical interaction signals:proprioception provides the posed hand geometry, and multi-contacttouch constrains where the object surface must lie, reducing ambigu-ity in occluded regions. We represent object structure as a pose-aware,camera-aligned signed distance field (SDF) and learn a compact latentspace with a Structure-VAE. In this latent space, we train a condi-tional flow-matching diffusion model, pretraining on vision-only imagesand finetuning on occluded manipulation scenes while conditioning onvisible RGB evidence, occluder/visibility masks, the hand latent rep-resentation, and tactile information. Crucially, we incorporate physics-based objectives and differentiable decoder-guidance during finetuningand inference to reduce hand–object interpenetration and to align thereconstructed surface with contact observations. Because our methodproduces a metric, physically consistent structure estimate, it integratesnaturally into existing two-stage reconstruction pipelines, where a down-stream module refines geometry and predicts appearance. Simulation ex-periments show that adding proprioception and touch substantially im-proves completion under occlusion and yields physically plausible recon-structions at correct real-world scale compared to vision-only baselines;we further validate transfer by deploying the model on a real humanoidrobot with an end-effector different from those used during training. Seehttps://github.com/hsp-iit/physical-generative-reconstruction