SV-TAD: Native Sparse Convolutions for Efficient Temporal Action Detection

To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train light-weight convolutional modules. While effective for parameter-efficient trai-ning, existing adapters do not reduce inference-time computation, leav-ing scalability with respect to video length largely unaddressed. Tokenselection can reduce attention cost by pruning redundant tokens, butit breaks the spatial grid structure required by convolutional adapters.This forces an expensive dense reconstruction, nullifying much of thepotential speedup. We address this by introducing native sparse 2D con-volutions, a primitive that allows these adapters, for the first time, tooperate directly and efficiently on dynamically pruned token sets. Weintegrate this primitive into SV-TAD, an adapter framework for tem-poral action detection, reducing VideoMAEv2-L computation by up to64% and achieving 2.2× faster inference, while maintaining state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3. When scaled toInternVideoNext-L, our approach surpasses the previous state of the artat roughly half its computational cost. Moreover, the sparse formulationnaturally supports auxiliary task tokens, which improves fine-grained as-sembly detection on ATTACH. Code and trained models are available athttps://github.com/pcr-upm/eccv26_tad.