CLIMP: Contrastive Language-Image Mamba Pretraining

Contrastive Language-Image Pre-training (CLIP) relies onVision Transformers whose attention mechanism is susceptible to spu-rious correlations and scales quadratically with resolution. To addressthese limitations, we present CLIMP, the first fully Mamba-based con-trastive vision-language model that replaces both the vision and text en-coders with state-space architectures. VMamba’s cross-scan mechanismcaptures spatial inductive biases that reduce reliance on spurious correla-tions, producing an embedding space with tighter cross-modal alignmentand lower hubness - geometric properties that translate to superior re-trieval and out-of-distribution robustness, surpassing even CLIP-ViT-Btrained on a dataset 167× larger on ImageNet-O. CLIMP naturally sup-ports variable input resolutions without positional encoding interpolationor specialized training, achieving up to 6.6% higher retrieval accuracyat 16× training resolution while using 5× less memory and 1.8× fewerFLOPs. Mamba’s autoregressive nature further enables processing of ar-bitrarily long text, overcoming CLIP’s fixed 77-token context limitationfor dense captioning retrieval. Our scaling experiments across model sizesand dataset sizes show consistent, unsaturated improvements - indicat-ing that CLIMP’s architectural advantages are not limited by trainingscale. These results demonstrate that Mamba is a compelling alternativeto Transformers for vision-language pre-training. The code and modelsare publicly available at https://github.com/NimrodShabtay/CLIMP