RefAlign: Representation Alignment for Reference-to-Video Generation

Reference-to-video (R2V) generation is a controllable videosynthesis paradigm that constrains the generation process using bothtext prompts and reference images, enabling applications such as per-sonalized advertising and virtual try-on. In practice, existing R2V meth-ods typically introduce additional high-level semantic or cross-modal fea-tures alongside the VAE latent representation of the reference image andjointly feed them into the diffusion Transformer (DiT). These auxiliaryrepresentations provide semantic guidance and act as implicit alignmentsignals, which can partially alleviate pixel-level information leakage inthe VAE latent space. However, they may still struggle to address copy–paste artifacts and multi-subject confusion caused by modality mismatchacross heterogeneous encoder features. In this paper, we propose Re-fAlign, a representation alignment framework that explicitly aligns DiTreference-branch features to the semantic space of a visual foundationmodel (VFM). The core of RefAlign is a reference alignment loss thatpulls the reference features and VFM features of the same subject closerto improve identity consistency, while pushing apart the correspond-ing features of different subjects to enhance semantic discriminability.This simple yet effective strategy is applied only during training, incur-ring no inference-time overhead, and achieves a better balance betweentext controllability and reference fidelity. Extensive experiments on theOpenS2V-Eval benchmark demonstrate that RefAlign outperforms cur-rent state-of-the-art methods in TotalScore, validating the effectivenessof explicit reference alignment for R2V tasks.