Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating

Large Vision-Language Models (LVLMs) have shown remark-able potential across a wide array of vision-language tasks, leading totheir adoption in critical domains such as finance and healthcare. How-ever, their growing deployment also introduces significant security andprivacy risks. Malicious actors could potentially exploit these models toextract sensitive information, highlighting a critical vulnerability. Recentstudies show that LVLMs often fail to consistently refuse instructionsdesigned to compromise user privacy. While existing work on privacyprotection has made meaningful progress in preventing the leakage ofsensitive data, they are constrained by limitations in both generalizationand non-destructiveness. They often struggle to robustly handle unseenprivacy-related queries and may inadvertently degrade a model’s per-formance on standard tasks. To address these challenges, we introduceNeural Gate, a novel method for mitigating privacy risks through neuron-level model editing. Our method improves a model’s privacy safeguardsby increasing its rate of refusal for privacy-related questions, cruciallyextending this protective behavior to novel sensitive queries not encoun-tered during the editing process. Neural Gate operates by learning a fea-ture vector to identify neurons associated with privacy-related conceptswithin the model’s representation of a subject. This localization then pre-cisely guides the update of model parameters. Through comprehensiveexperiments on MiniGPT and LLaVA, we demonstrate that our methodsignificantly boosts the model’s privacy protection while preserving itsoriginal utility. The code is available at https://github.com/Xiangkui-Cao/Neural-Gate.