HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization

Video understanding has become more and more importantwith the growth of Artificial Intelligence (AI) for video generation. Re-cently, Multimodal Large Language Model (M-LLM) has shown its ca-pability in video understanding. Video summarization, a specific domainof video understanding, has proven its importance for efficient naviga-tion and retrieval. Both video understanding and video summarizationrequire a good selection of key frames in a video. Current video summa-rization methods heavily focus on the selected key frames and correlatedsegment captions. However, existing approaches overlook the perspectiveof treating the importance of the frames globally. We argue that usingdiscrete selected frames for summarization will not only reduce the un-derstanding coherence, but also lost important information in the video,as well as wasting the original capacity of the MLLMs. In this paper, wepropose HAS, a Highlight-guided Attention Steering method for videosummarization. We consider a challenging but practical setting wherethe video given to MLLMs for summarize should be continuous but withhighlight guidance. HAS mainly consists of two parts: The first part isto find a continuous frame-level highlight distribution for the video glob-ally. The second part is to apply the highlight distribution as an attentionsteering vector for the MLLM, targeting a better understanding of thevideo, and thus during the model inference time, putting more attentionon the highlighted frames, while avoiding lost entire information on lesshighlighted frames through putting less attention instead of forgettingthem. We evaluated HAS on a variety of benchmarks, and it has shownconvincing performance in video summarization.