HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human–Scene Interactions

We present HSImul3R1 , a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casualcaptures, including sparse-view images and monocular videos. Existingmethods suffer from a perception-simulation gap: visually plausible re-constructions often violate physical constraints, leading to instability inphysics engines and failure in embodied AI applications. To bridge thisgap, we introduce a physically-grounded bi-directional optimiza-tion pipeline that treats the physics simulator as an active supervisorto jointly refine human dynamics and scene geometry. In the forwarddirection, we employ Scene-targeted Reinforcement Learning to optimizehuman motion under dual supervision of motion fidelity and contactstability. In the reverse direction, we propose Direct Simulation RewardOptimization, which leverages simulation feedback on gravitational sta-bility and interaction success to refine scene geometry. We further presentHSIBench, a new benchmark with diverse objects and interaction sce-narios. Extensive experiments demonstrate that HSImul3R produces thefirst stable, simulation-ready HSI reconstructions and can be directlydeployed to real-world humanoid robots.