Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

Understanding and forecasting audience reactions to videocontent are crucial for improving content creation, recommendation sys-tems, and media analysis. To enable audience reaction prediction andother content engagement applications, we introduce Video2Reaction,a multimodal dataset that maps short movie segments to a distributionof induced emotions of viewers in the wild, as expressed through socialmedia. Video2Reaction spans more than 10,000 videos and serves asa reliable benchmark as well as a training resource for audience reactionprediction. To enable cost-e!ective continuous annotations as reactionsmay change over time, we develop a two-stage multi-agent pipeline usingonly open-source LLMs, achieving 86% correctness under blind humanverification despite the inherently noisy and subjective nature of the task.We establish the first benchmark for video-to-reaction-distribution pre-diction in the wild and show that pretrained foundation video modelsfail in zero-shot settings, while finetuning transforms them into state-of-the-art predictors capable of modeling both full reaction distribu-tions and dominant responses from video alone. However, the task re-mains challenging: even the strongest methods achieve only 77% Top-3F1 in dominant reaction prediction (LLaVA-Next), highlighting a sub-stantial gap in modeling collective audience reaction. Dataset and codeare available at our project page: https://information-fusion-lab-umass.github.io/video2reaction-bench.github.io/.