COLA: Continual Orthogonal Low-Rank Adaptation for Class-Incremental Learning

Recent advances in Continual Learning (CL) have adoptedfoundation models with Low-Rank Adaptation (LoRA) or prompt tuning,utilizing pre-trained representations to achieve more efficient and adapt-able learning. However, these methods suffer from two key challenges.First, they learn task-specific adapters, resulting in unbounded parameterexpansion as task sequences grow. Second, they maintain replay buffersor store feature representations from previous tasks, which introducessubstantial memory overhead and potential privacy risks. To address theselimitations, we propose Continual Orthogonal Low-Rank Adaptation(COLA), a novel rehearsal-free, parameter-efficient framework for CL.COLA integrates LoRA’s low-rank adaptation with an Oja-inspired learn-ing rule that incrementally approximates the dominant eigenstructure ofthe feature covariance across tasks. This mechanism continuously tracksthe principal directions of prior task knowledge and progressively reducesrepresentational interference between new and previously learned tasksubspaces. By dynamically adapting a shared low-rank subspace usingcovariance updates, COLA achieves stable feature projection and con-tinual knowledge integration without additional parameter expansionor memory replay. Extensive experiments on class-incremental bench-marks such as ImageNet-R, ImageNet-A, and CUB200 demonstrate thatCOLA effectively mitigates catastrophic forgetting while maintainingminimal memory overhead and strong generalization, outperforming ex-isting LoRA-based and prompt-based methods. The code is available at:https://github.com/autovisionproject/COLA.