CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

Movie dubbing aims to synthesize speech that preserves thevocal identity of a reference audio while synchronizing with the lip move-ments in a target video. Existing methods fail to achieve precise lip-syncand lack naturalness due to explicit alignment at the duration level.While implicit alignment solutions have emerged, they remain suscep-tible to interference from the reference audio, triggering pronunciationand lip-sync degradation in in-the-wild scenarios. In this paper, we pro-pose a novel flow matching-based movie dubbing framework driven bythe Cognitive Synchronous Diffusion Transformer (CoSyncDiT), in-spired by the cognitive process of professional actors. This architectureprogressively guides the noise-to-speech generative trajectory by execut-ing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic andAlignment Regularization (JSAR) mechanism to simultaneously con-strain frame-level temporal consistency on the contextual outputs andsemantic consistency on the flow hidden states, ensuring robust align-ment. Extensive experiments on both standard benchmarks and chal-lenging in-the-wild dubbing benchmarks demonstrate that our methodachieves the state-of-the-art performance across multiple metrics. Thecode is available at https://github.com/GalaxyCong/CoSyncDiT.