ReTarget: Representation Transformation via Adversarial Regularization for Geometric Misalignment

Split inference (SI) offloads computation from edge devicesand is often considered privacy-friendly since raw inputs remain local.However, recent data reconstruction attacks (DRA) show that interme-diate features can be inverted to recover sensitive content. Existing de-fenses suppress feature information through pruning, noise injection, ordecorrelation, yet strong reconstructions persist. In this paper, we ar-gue that reconstruction vulnerability is not solely determined by mutualinformation between inputs and representations, but also by the localorganization of representations that facilitates reliable inversion in prac-tice. Therefore, we propose ReTarget, a lightweight transformation thatpreserves task-discriminative representations while perturbing inversion-consistent features. ReTarget combines adversarial reconstruction, taskpreservation, and semantic regularization training to substantially de-grade inversion performance in practice without sacrificing utility. Ex-periments on CLIP-ViTB/16, CLIP-RN50 and DINOv2 across multiplesplit points and datasets show substantial degradation in reconstructionquality. Under both state-of-the-art and adaptive attacks, particularlyat shallow split points where prior defenses fail, our method remainseffective, where achieve up to 35% improvement while maintaining com-petitive task accuracy.