HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

CLIP (Contrastive Language-Image Pre-training) has be-come a de facto paradigm for image-text alignment, but it struggleswith long-context descriptions (> 77 tokens) due to absolute positionalencoding and pretraining on short captions. In long contexts, sentencesare often reordered, summarized, or partially omitted. Although priorworks extend CLIP with longer positional encodings, they often sufferfrom degraded image-text alignment under such text perturbations. Weattribute this limitation to the Euclidean contrastive objective, whichenforces strict one-to-one matching and lacks explicit mechanisms formodeling hierarchical relationships between global context and its con-stituent elements. To address this issue, we propose HyFL-CLIP, a hy-perbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space viacross-manifold similarity distillation, leveraging its geometry to capturehierarchical and entailment relations. Our method models hierarchicalsemantics by linking summarized token-wise features, long-context de-scriptions, constituent short textual components, and images, capturingpart–whole relationships via hyperbolic entailment with Einstein mid-point aggregation. Experiments on diverse benchmarks, including long-context cross-modal retrieval, cross-modal retrieval with caption pertur-bations, intra-modality retrieval, and short-text cross-modal retrieval,show that HyFL-CLIP achieves more robust long-context understand-ing. In particular, it yields up to 19.5% improvement in long-text cross-modal retrieval under textual perturbations over the best prior method.We also show HyFL-CLIP can be seamlessly integrated into other modelframeworks by applying it to Stable Diffusion XL (SDXL). The projectpage is available at https://janeyeon.github.io/hyflclip.