FST-SAM3: Taming SAM~3 with Frequency-Spatio-Temporal Refinement for Video Polyp Segmentation

Video polyp segmentation (VPS) is important for automatedcolorectal cancer screening, yet it is a very challenging task due to severecamouflage in mucosal scenes, complex intestinal anatomy, and substan-tial frame-to-frame appearance variations. Although Segment AnythingModel 3 (SAM 3) is able to provide strong representations for segmen-tation tasks, directly applying it to endoscopic videos usually cannotachieve satisfactory performance owing to domain shifts, boundary am-biguities, and unstable temporal propagation. To address these chal-lenges, we propose FST-SAM3, a novel parameter-efficient adaptationframework built on SAM 3 for VPS. FST-SAM3 introduces two inno-vative yet complementary components. First, we propose a Structure-Aware Frequency Enhancement (SAFE) module to strengthen structure-and boundary-sensitive cues under endoscopic appearance. Second, wedevelop a Spatio-Temporal Token Refinement Framework (STRF) torefine temporal information at the token level for more stable cross-frame segmentation. We freeze the SAM 3 backbone and learn only asmall set of additional parameters, keeping the inference characteristicsclose to SAM 3 while improving segmentation performance. We con-ducted extensive experiments on two benchmarking datasets (SUN-SEGand CVC-612) under standardized fully automatic (0-point) and first-frame prompted (1-point at t=0) protocols. Results demonstrate that ourmodel achieves consistent improvements over state-of-the-art approaches,including both task-specific baselines and representative SAM-based ap-proaches. Code is available at https://github.com/GavonW/FST-SAM3.