LOOM: Weaving Geometry-Consistent Human-Object Interaction Videos via Progressive Curriculum Learning

Human-object interaction (HOI) video generation has gar-nered increasing attention due to its promising applications in digitalhumans, e-commerce, advertising, and robotics imitation learning. How-ever, existing methods face two critical limitations: (1) a lack of effec-tive mechanisms to inject multi-view information of the object into themodel, leading to poor cross-view consistency, and (2) heavy relianceon fine-grained annotations including hand mesh and body templatesfor modeling interaction occlusions, due to limited training data. To ad-dress these challenges, we introduce ByteLOOM, a Diffusion Transformer(DiT)-based framework that generates realistic human-object interactionvideo with geometrically consistent object illustration, using simplifiedhuman conditioning and 3D object inputs. We first propose an RCM-cache mechanism that leverages Relative Coordinate Maps (RCM) as auniversal representation to maintain object’s geometry consistency andprecisely control 6-DoF object transformations in the meantime. To com-pensate HOI dataset scarcity and leverage existing datasets, we furtherdesign a training curriculum that enhances model capabilities in a pro-gressive style and relaxes the demand of hand mesh. Extensive experi-ments demonstrate that our method faithfully preserves human identityand the object’s multi-view geometry, while maintaining smooth motionand object manipulation. Project page: neutrinoliu.github.io/byteloom/