TOPA: Mitigating Concept Dominance in Diffusion Personalization via Target-Oriented Perturbation Augmentation

Concept personalization injects a user-specified concept intoa pretrained diffusion model from only a few reference images, enablingcustomized content creation. However, existing personalization methodsfrequently suffer from concept-dominant failures—while the personal-ized concept is well preserved, other prompt-specified contexts (e.g.,background, attributes, and interactions) are not properly generated,thereby limiting controllability. To address this problem, in this paper, wepropose Target-Oriented Perturbation-based Augmentation (TOPA), areference images-only augmentation framework. TOPA optimizes addi-tive perturbations on reference images to provide token–region atten-tion guidance and further reduces background entanglement via subject-isolation compositing. Unlike prior solutions that modify training ob-jectives or architectures, TOPA requires no changes to personalizationpipelines, and its augmented datasets can be directly applied to existingmethods. Extensive experiments across multiple personalization methodsdemonstrate consistent improvement on context adherence while preserv-ing concept fidelity.