TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction

Vision and vision–language models rely on high-level visualrepresentations that are increasingly used across recognition, retrieval,and multimodal reasoning pipelines. However, recent advances in gen-erative modeling have shown that such features can often be inverted,enabling realistic reconstructions of the underlying image and raisingsignificant privacy risks. We revisit this problem through the lens ofreconstruction and propose TrustCLIP, a reconstruction-driven frame-work that treats a feature-conditioned generator as an explicit privacyadversary. TrustCLIP learns a projection between encoder features anddownstream modules that is explicitly optimized to degrade the recon-structions produced by generative attackers while retaining the necessarysignals for downstream tasks. Unlike prior defenses that rely on discrimi-native privacy metrics, TrustCLIP directly optimizes against a generativereconstruction attacker, targeting a threat not captured by standard eval-uation protocols. We demonstrate its effectiveness in both conventionalclassification and multimodal large language model pipelines. Acrossthese settings, TrustCLIP consistently reduces the fidelity of genera-tive inversions while maintaining downstream task performance. Projectpage: atnikos.github.io/trustclip.