Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation

Despite significant advances in image segmentation, evenstate-of-the-art models produce masks with imperfect boundaries, se-mantic inconsistencies, and structural errors. Mask refinement addressesthese limitations, yet current approaches rely on simplistic syntheticnoise that fails to capture the complex error patterns of real segmen-tation models. We introduce Phoenix, a novel framework that lever-ages adversarial learning to generate semantically meaningful noise pat-terns and contrastive learning to model refinement relationships. Ourapproach consists of two key innovations: (1) Adversarial Mask Per-turbation, which employs embedding attacks to create semantic-awarenoise that mimics real segmentation errors, and (2) Contrastive MaskRefinement Learning, which establishes a tri-directional framework thatensures feature consistency within semantic regions while maintainingseparation between classes. Experiments demonstrate that Phoenix sig-nificantly outperforms existing methods across diverse tasks, while con-sistently enhancing state-of-the-art segmentation models with substan-tial improvements. Our code and project page are publicly available atTargetMaskMorphologicalMask (b) Morphological NoiseLess-Semantic, Unrealistic, and Non-Contextual NoiseAdversarialMask(ours)Semantic, Realistic, and Contextual False-Negative & False-Positive Noise(a) Qualitative Samples of Morphological and Adversarial Noise Masks (c) Adversarial NoiseFig. 2: Qualitative Comparison of noise patterns (a) between morphological andour adversarial noise masks. (b, c) Distribution of the Pearson correlation betweennoise location and image edge/texture maps on LVIS val. (b) Morphological noise isnarrowly concentrated near 0 (semantically uncorrelated), while (c) our adversarialnoise spans [−0.6, 0.8], where positive values indicate alignment with semantic struc-tures and negative values indicate alignment with homogeneous regions, mirroring thediverse error patterns of real models.