CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection

Multimodal Large Language Models (MLLMs) excel in di-verse vision tasks, but full-parameter retraining is computationally ex-pensive as real-world knowledge evolves. Existing continual learning meth-ods often suffer from semantic entanglement in parameter spaces acrosstasks, impeding the continuous deployment of models. This challenge isespecially pronounced in Anomaly Detection (AD), which exhibits tripleheterogeneity across modalities, domains, and defect scale variability,significantly complicating multi-task knowledge transfer. In this paper,we propose CL-Anomaly, a parameter-efficient fine-tuning frameworkbased on an isolation-sharing collaboration to enable continual learningfor anomaly detection with MLLMs. We introduce the task-private ex-pert PrivLoRA, which physically isolates task-specific subspaces in theparameter space to prevent semantic entanglement of anomaly knowl-edge in diverse scenarios. The Layer-Adaptive Shared Experts maintaincross-task representations within a unified feature space, enabling knowl-edge sharing between previous and new tasks. Furthermore, we proposea Layer-Adaptive Knowledge Transfer strategy that automatically se-lects and dynamically updates the layer-wise key shared experts of eachtask via a momentum-based mechanism, promoting effective knowledgetransfer across related anomaly detection tasks. Extensive experimentsacross three continual learning scenarios for anomaly detection, includ-ing class-incremental, cross-domain, and cross-modal, demonstrate thatCL-Anomaly outperforms state-of-the-art methods. Code is available athttps://github.com/WenDongyp/CL-Anomaly.