RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations

Multi-perturbation adversarial training (MAT) aims to achieverobustness against multiple ℓp perturbations but suffers from robustnesstrade-offs between different threats. To address this, we employ a mixtureof experts (MoE) to route different threats through distinct model path-ways. However, naïve application of MoE encounters two critical chal-lenges: experts tend to overlook threat-specific features and redundantlycapture features shared across threats, and gating networks suffer fromthreat-agnostic routing where they learn nearly identical routing pat-terns across threats, thus preventing the construction of threat-specificmodel pathways. To this end, we propose Robust Mixture of Low-RankExperts (RoME), where each expert is a low-rank additive update tothe shared backbone, allowing it to capture threat-common featureswhile experts focus on threat-specific information. To address threat-agnostic routing, RoME introduces (i) dual-scale gating that exploitsthreat-discriminative signals from local and global level features, and(ii) threat-guided gating diversification that enforces diverse expert uti-lization across threats. Extensive experiments demonstrate that RoMEoutperforms existing state-of-the-art MAT in union robustness and nat-ural accuracy and improves robustness against unseen threats. Codes areavailable at https://github.com/wkim97/RoME.