Towards Long-Form Spatio-Temporal Video Grounding

In real scenarios, the videos can span several minutes or evenhours, yet existing research on spatio-temporal video grounding (STVG),given a textual query, mainly focuses on localizing the target from a videoof tens of seconds, typically less than one minute, hindering its applica-tions. In this paper, we explore Long-Form STVG (LF-STVG), whichaims to locate the target from long-term videos. In LF-STVG, long-termvideos encompass a much longer temporal span and more irrelevant infor-mation, making it challenging for current short-form STVG approachesthat process all the frames at once. Addressing these, we propose a novelAutoRegressive Transformer architecture for LF-STVG, dubbed ART-STVG. Unlike current STVG methods requiring seeing the entire videosequence to make a full prediction at once, our ART-STVG regards thevideo as a streaming input and processes its frames sequentially, makingit capable of easily handling the long videos. To capture spatio-temporalcontext in ART-STVG, spatial and temporal memory banks are devel-oped and applied to decoders of ART-STVG. Considering that memoriesat different moments are not always relevant for localizing the target incurrent frame, we introduce simple yet effective memory selective strate-gies that enable the more relevant information for decoders, greatly im-proving the performance. Moreover, rather than parallelizing spatial andtemporal localization as done in existing approaches, we introduce a novelcascaded spatio-temporal design that connects spatial decoder to tempo-ral decoder during grounding, which allows ART-STVG to leverage morefine-grained target information to assist with complicated temporal lo-calization in complex long videos, further boosting performance. On thenewly extended datasets for LF-STVG, ART-STVG largely outperformscurrent approaches, while showing competitive results on Short-FormSTVG. Our code is at: xxx.