Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models

Emerging multi-modal world models attempt to jointly generate videos across diverse modalities (e.g., RGB, depth, and mask), yet they fail to fully exploit the priors of existing foundation models. We propose M2-REPA, the first representation alignment method tailored for multi-modal video generation. Our key insight is that foundation models trained on different modality spaces naturally capture distinct domain-specific priors, acting as complementary “experts.” Specifically, we first decouple modality-specific features from the diffusion model’s intermediate representations, then align each with its corresponding expert foundation model. To this end, we design two synergistic objectives: a multi-modal representation alignment loss that enforces feature-toexpert matching, and a modality-specific decoupling regularization that encourages complementarity across different modalities. This design enables joint optimization, exploiting priors from multiple foundation models. Extensive experiments demonstrate that our method outperforms baselines in visual quality and long-term consistency.