LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models

Pre-trained Vision Foundation Models (VFMs) provide strongvisual representations for diverse downstream tasks. The key challenge ofVFM adaptation stems from the prohibitive costs of full fine-tuning andcatastrophic forgetting. To address this, Low-Rank Adaptation (LoRA)has emerged as the prevailing paradigm for Parameter-Efficient Fine-Tuning (PEFT). However, LoRA is typically designed for transformerself-attention layers parameterized by 2D matrices. Since convolutionalkernels inherently couple spatial and channel information within a 4Dtensor, forcing them into a monolithic 2D matrix disrupts the inher-ent spatial topology. In this paper, we propose Low-Rank ConvolutionalAdaptation (LoCA), a convolution-aware PEFT framework that addressesspatial-channel entanglement by decoupling channel and spatial adapta-tion. LoCA introduces a low-rank channel adaptation for dense cross-channel mixing and refines spatial bases extracted from pre-trained ker-nels via Singular Value Decomposition (SVD). Experimental results showthat LoCA preserves pre-trained spatial priors and achieves competitiveor state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.