Small Vision-Language Models are Smart Compressors for Long Video Understanding

Adapting Multimodal Large Language Models (MLLMs) forhour-long videos is bottlenecked by context window limits. Existing effi-ciency heuristics blindly sacrifice fidelity. They frequently discard tran-sient decisive moments, blur fine-grained evidence, and waste represen-tational bandwidth on irrelevant backgrounds. We propose Tempo, anefficient, query-aware framework that compresses long videos for down-stream understanding. Tempo unifies a Small Vision-Language Model(SVLM) to act as a local temporal compressor. It casts visual token re-duction as an early cross-modal distillation process, generating compact,intent-aligned video representations in a single forward pass. To enforcestrict inference budgets without breaking temporal causality, we intro-duce Adaptive Token Allocation (ATA). Exploiting the SVLM’s inher-ent zero-shot relevance prior and empirical semantic front-loading, ATAacts as a training-free, O(1) dynamic router. It allocates dense band-width to query-critical segments while compressing redundancies downto minimal temporal anchors to maintain the global storyline. Exten-sive experiments demonstrate that our compact 6B architecture achieveshighly competitive performance with aggressive dynamic compression(0.5–16 tokens/frame). On the extreme-long LVBench (4101s), Temposcores 52.3 under a strict 8K visual budget, outperforming proprietarymodels like GPT-4o and Gemini 1.5 Pro. Crucially, empirical profiling re-veals that Tempo frequently compresses hour-long videos to token countsbelow theoretical limits, proving that true long-form understanding relieson intent-driven efficiency rather than greedily padded context windows.