Push–Pull Attentional Anchoring for Diffusion Concept Erasure

Rapid advances in diffusion models have raised concernsabout privacy, harmful content, and copyright violations. Concept era-sure methods aim to address these risks by removing undesirable con-cepts from pretrained models without full retraining. However, existingapproaches often degrade non-target generation quality through heuristicstrength scaling (e.g., classifier-free guidance or negative prompt scaling),leading to uncontrolled and excessive semantic drift. To address this, wepropose Push–Pull Attentional Anchoring (PPAA), a mechanism in thecross-attention space that displaces target representations while anchor-ing the scale of erasure via bounded cosine-similarity constraints. Byusing a relative similarity ratio instead of an absolute difference, ouradaptive threshold represents a meaningful percentage of semantic re-tention, facilitating a single value to be applied uniformly across layers.We conduct extensive experiments across multiple concept categories, in-cluding celebrities, artistic styles, nudity, and intellectual property. Ourmethod achieves favorable trade-offs against prior approaches on StableDiffusion v1.4 and demonstrates its applicability to modern architecturessuch as FLUX.