Rethinking Temporal Modeling in Visual Object Tracking via Decoupled Auxiliary Supervision

Visual object tracking relies on modeling cross-frame dynam-ics, and recent approaches employ learnable temporal tokens to encodehistorical information. However, our analysis reveals that these tokensare often marginalized by dominant spatial features during joint opti-mization. This imbalance leads to shortcut learning where the networkover-relies on current-frame appearance, causing temporal representationcollapse. To address this, we propose DASTrack, a framework featur-ing Decoupled Auxiliary Supervision (DAS). By masking the templatesduring an auxiliary forward pass, DAS compels the network to localizetargets using only temporal tokens and thus encode robust cross-framepriors. This mechanism is applied strictly during training and discardedat inference to preserve the original architecture and computational ef-ficiency. DASTrack yields consistent and interpretable average perfor-mance gains across multiple backbones. Our findings demonstrate thatthe primary bottleneck in temporal tracking stems from biased learn-ing dynamics rather than architectural design, offering a novel trainingparadigm for eliciting latent temporal capabilities. The code and modelswill be released at: https://github.com/ZhangDailing8/DASTrack.