SOVTrack: Open-Vocabulary Multi-Object Tracking with Self-Supervised Pseudo Labeling and Feature Distillation

Open-Vocabulary Multi-Object Tracking (OVMOT) repre-sents a critical challenge requiring the detection and tracking of diverseobject categories beyond predefined training classes, including both seen(base) and unseen (novel) categories in real-world videos. Existing ap-proaches are fundamentally limited by their reliance on traditional closed-set tracking paradigms trained only on base classes and synthetic imagepairs, failing to exploit the rich temporal dynamics inherent in videosequences for universal tracking capabilities. We present SOVTrack, anovel self-supervised framework that harnesses SAM2’s universal represen-tations to address general-purpose tracking from raw video data withoutmanual supervision. Our approach introduces Dual-Direction PseudoLabeling (DDPL) to automatically generate high-quality pseudo-labelsthrough bidirectional temporal validation while dynamically assessingthe difficulty of each training example, and employs a dual-branch archi-tecture with Multi-Cue Adversarial Distillation (MCAD) to effectivelytransfer SAM2’s multi-cue pixel-level universal tracking capabilities toobject-level tracking scenarios while preserving its generalization poweracross diverse object categories. Experimental results demonstrate thatSOVTrack achieves state-of-the-art performance using only 10K videoframes (2% of existing methods’ data requirements), delivering substantialimprovements in both base and novel category tracking performance.