Video-Oasis: Rethinking Evaluation of Video Understanding

The inherent complexity of video understanding makes it dif-ficult to determine whether Video-LLM benchmark performance stemsfrom visual perception, linguistic reasoning, or knowledge priors. Whilemany benchmarks have emerged to assess high-level reasoning, sharedcriteria for evaluating video understanding remain largely overlooked.Instead of introducing yet another benchmark, we take a step back tore-examine the criteria for evaluating video understanding. In this work,we introduce Video-Oasis, a sustainable diagnostic suite for systemati-cally auditing existing video understanding benchmarks. This audit re-veals that 55% of existing benchmark samples are solvable without visualinput or temporal context. After filtering these shortcuts, the remainingvideo-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building onthese findings, we use the distilled challenges as a testbed to investigatewhich algorithmic design choices contribute to robust video understand-ing. We hope our work provides a practical foundation for constructingrigorous video benchmarks and evaluating future Video-LLMs. Code isavailable at https://github.com/sejong-rcv/Video-Oasis.