DiscoVL: Unveiling Disentangled Cross-Modal Representation Learning via Orthogonal Adversarial Regularization for Vision-Language Models

Pre-trained vision-language models excel across varied per-ception tasks, but adapting them to novel downstream settings with-out sacrificing generalization remains non-trivial. Existing parameter-efficient prompt learning method often yields inconsistent representa-tions and fails to account for semantic distribution shifts. In this work,we present DiscoVL, a disentangled cross-modal representation learningframework that couples orthogonal adversarial regularization with struc-tured cross-modal alignment for vision-language models. To address theinsufficient cross-modal interaction, our DiscoVL designs a multi-branchlow-rank residual aligner that decomposes representations into subspacesand enables bidirectional cross-modal feedback between visual and tex-tual streams at each layer. Furthermore, while conventional triplet con-straints overfit features to class centroids, we design an orthogonal reg-ularization for adversarial triplet loss, which prevents centroid collapseand substantially boosts generalization. Evaluations on 15 benchmarksdemonstrate that DiscoVL delivers consistent improvements over state-of-the-art methods for base-to-novel generalization, cross-dataset evalu-ation, and few-shot learning.