Indelible Backdoors: On the Limits of Post-Training Defenses

Deep neural networks remain vulnerable to backdoor at-tacks, in which adversaries embed hidden triggers during training tocause misclassification of any trigger-carrying input into an attacker-chosen target class, while maintaining normal behavior on clean inputs.Though recent fine-tuning-based defenses have shown promise in miti-gating backdoor attacks and are particularly practical in continual learn-ing and MLaaS scenarios, where backdoors are removed from pretrainedmodels using small clean datasets, we demonstrate that this defensiveparadigm remains fundamentally exploitable by an adaptive adversary.We study a threat model in which attackers anticipate fine-tuning-baseddefenses and craft backdoors whose resulting optimization signals areentangled with those of the clean task. Our approach leverages FisherInformation to identify task-critical neurons and optimizes a trigger gen-erator such that backdoor gradients align with benign gradients on theseneurons, ensuring that any attempt to suppress the backdoor inevitablyinterferes with clean task preservation. As a result, fine-tuning defensesfails to effectively reduce attack success rate without incurring substan-tial degradation in clean accuracy. Through extensive evaluation againstseven attacks and eight defenses, we show our method consistently showsa stronger effect than existing backdoor attacks across all settings. Ourwork reveals a critical vulnerability in fine-tuning-based defenses andhighlights the need for more robust defense mechanisms against adap-tive adversaries.