E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework

3D reassembly is a fundamental geometric problem, and inrecent years it has increasingly been challenged by deep learning meth-ods rather than classical optimization. While learning approaches haveshown promising results, most still rely primarily on geometric features toassemble a whole from its parts. This reliance limits their effectiveness incases where geometric information is incomplete or degraded, such as forsmall or eroded fragments. Additionally, solutions do not impose physi-cal constraints that explicitly prevent overlapping assemblies. To addressthese limitations, we introduce E-M3RF, an equivariant multimodal 3Dreassembly framework that takes as input the point clouds, containingboth point positions and color of fractured fragments, and predicts thetransformations required to reassemble them, using SE(3) flow match-ing. Each fragment is represented by both geometric and color features:i) 3D point positions are encoded as rotation-consistent geometric fea-tures using a rotation-equivariant encoder, while ii) the colors at each3D point are encoded with a transformer. The two feature sets are thencombined to form a multimodal representation. We experimented on fivedatasets: two synthetic datasets, Breaking Bad and Fantastic Breaks,and three real-world cultural heritage datasets, RePAIR, Presious, andFRACTURA, demonstrating that E-M3RF on the RePAIR dataset (themost challenging one) reduces rotation error by 23.1% and translationerror by 13.2%, while Chamfer Distance decreases by 18.4% comparedto competing methods. Project page: https://em3rf.github.io/.