GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awareness

Generalized 3D hand-object pose estimation from a singleRGB image remains challenging due to the large variations in objectappearances and interaction patterns, especially under heavy occlusion.We propose GenHOI, a framework for generalized hand-object pose esti-mation with occlusion awareness. GenHOI integrates hierarchical seman-tic knowledge with hand priors to enhance model generalization underchallenging occlusion conditions. Specifically, we introduce a hierarchicalsemantic prompt that encodes object states, hand configurations, andinteraction patterns via textual descriptions. This enables the model tolearn abstract high-level representations of hand-object interactions forgeneralization to unseen objects and novel interactions while compen-sating for missing or ambiguous visual cues. To enable robust occlu-sion reasoning, we adopt a multi-modal masked modeling strategy overRGB images, predicted point clouds, and textual descriptions. Moreover,we leverage hand priors as stable spatial references to extract implicitinteraction constraints. This allows reliable pose inference even undersignificant variations in object shapes and interaction patterns. Exten-sive experiments on the challenging DexYCB and HO3Dv2 benchmarksdemonstrate that our method achieves state-of-the-art performance inhand-object pose estimation.