IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment

Prompt-conditioned foundation segmenters rely on explicit spatial prompts, such as points, boxes, or masks, to guide mask decoding. In prompt-absent deployment, however, no external/user prompts or detector-generated prompts are available at test time, creating a mismatch between the expected prompt-conditioned interface and automatic inference. We propose IP-SAM, a prompt-space adaptation framework that restores prompt-conditioned decoding without external prompts. Instead of bypassing SAM2’s prompt interface through direct feature-space adaptation, IP-SAM synthesizes complementary intrinsic foreground/background prompts and routes their dense logits through SAM2’s frozen prompt encoder, translating task-specific spatial cues into the native prompt embedding space. To reduce background leakage under severe camouflage, Prompt-Space Gating uses the intrinsic background prompt as an asymmetric suppressive constraint before decoding. Under a deterministic noexternal-prompt protocol, IP-SAM achieves state-of-the-art performance across four COD benchmarks, including MAE 0.017 on COD10K, with only 21.26M trainable parameters from SPG, PSG, a SAM2-initialized downstream mask decoder, and image-encoder LoRA, while keeping the prompt encoder frozen. Medical polyp experiments in the main paper and supplementary SOD results further show that the same prompt-space adaptation route transfers within foreground–background segmentation.