Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

This study focuses on Text-to-Sounding-Video (T2SV) gen-eration, which aims to generate a video with synchronized audio fromtext, with both modalities aligned to the text conditions. Despite progressin joint audio-video training, two critical challenges remain: (1) text con-ditioning is a bottleneck—shared captions (TV = TA ) trigger modalinterference, while a gap persists between dense training captions andconcise inference user prompts, and (2) the optimal fusion mechanismfor cross-modal feature interaction remains unclear. To address the firstchallenge, we first propose the Cross-Referential Rewriter (CRR) cap-tion framework, a dual-agent pipeline where a Semantic Checker extractsgrounded Semantic Anchors and a Cross-Modal Rewriter generates dis-entangled caption pairs (TV and TA ), eliminating modal interference and1 →Equal contribution. Work done during an internship at Apple.2 †Project lead.3 ‡Corresponding author.bridging the training-inference gap via prompt expansion. For the sec-ond, we introduce BridgeDiT, a dual-tower di!usion transformer thatemploys Dual Cross-Attention (DCA) as a bidirectional bridge betweenvideo and audio streams, which we show through systematic compari-son to be the optimal fusion strategy for the dual-tower paradigm. Ex-tensive experiments on three benchmarks, supported by human evalu-ations, demonstrate state-of-the-art results on most metrics. Compre-hensive ablation studies further validate each component and o!er keyinsights for future T2SV systems. The codes and models are available athttps://bridgedit-t2sv.github.io/.