CaPCL: Caption-Preserved Continual Learning for Text-to-Image Retrieval

Continual learning for text-to-image retrieval is essential foradapting retrieval systems to evolving user interests and emerging con-cepts. While existing methods mitigate forgetting through image-textsimilarity distillation, they overlook a critical real-world scenario: newtextual queries often describe different semantic facets of images frompreviously seen visual domains, a phenomenon we term “unbalancedcross-modal shift”. This shift arises from the inherent polysemy of vi-sual content and is strongly associated with severe forgetting. To studythis phenomenon, we introduce two diagnostic metrics: image domainoverlap measured via principal subspace similarity, and textual querydivergence quantified through image-conditioned query probability. Ourempirical analysis shows that unbalanced cross-modal shift significantlyamplifies forgetting, with existing methods struggling to retain imagesemantics under such conditions. To address this challenge, we proposeCaPCL (Caption-Preserved Continual Learning), which regularizes mod-els to maintain consistent caption generation for previously learned con-tent, thereby preserving multi-faceted image semantics. By leveragingauxiliary captions as semantic anchors, CaPCL delivers more robusttraining signals than contrastive similarity alone. We construct a bench-mark integrating five heterogeneous datasets to enable comprehensiveevaluation. Extensive experiments demonstrate that CaPCL consistentlyoutperforms state-of-the-art methods in both knowledge retention andadaptation.