Video-Text Alignment Model for Sign Language Translation

Sign language translation (SLT) converts continuous signvideos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and language models but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo,a framework that introduces explicit multi-granularity alignment at threelevels: (1) local alignment via entropy-regularized optimal transport witha learnable null token for fine-grained frame-to-token correspondences;(2) global alignment via a learnable orthogonal transformation that cali-brates embedding space geometry through Earth Mover’s Distance; and(3) position-aligned contrastive learning for discriminative token-levelrepresentations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign,and OpenASL demonstrate consistent state-of-the-art performance, withablations confirming the complementary contributions of each compo-nent. Code is available at https://github.com/junyi2005/vtamo.