3D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanism

Photorealistic 3D face avatars are increasingly deployed asreusable digital assets across applications such as telepresence, anima-tion, and personalized media. At the same time, vision–language models(VLMs) can infer sensitive attributes from rendered images with open-ended semantic reasoning without any fine-tuning. This creates a newprivacy challenge: once a 3D face avatar is shared, any of its renderingscan be analyzed to extract high-level facial attributes. Existing defenseslargely operate in 2D image space and do not address identity-preservingsemantic manipulation of 3D facial representations.We propose 3D FaceShell, a framework for steering VLM interpreta-tions of faces rendered from 3D models while preserving geometric fidelityand facial identity. 3D FaceShell augments the original 3D representa-tion with a learnable Gaussian shell that produces subtle, spatially dis-tributed perturbations optimized through multi-view embedding align-ment. The perturbations are designed to be visually inconspicuous yetsufficient to redirect VLM-based attribute inference in a view-consistentExtensive experiments on reconstructed celebrity face avatars and mul-tiple black-box VLMs demonstrate that 3D FaceShell significantly in-creases attribute injection and mismatch rates while maintaining highperceptual similarity and identity consistency. Our results show that itis possible to manipulate VLM-level semantic interpretation of 3D faceswithout compromising their human-recognizable appearance.