Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

While knowledge distillation (KD) is widely adopted for train-ing lightweight models by leveraging supervision from larger teachermodels, relying solely on output token distributions has proven insuf-ficient for compressing Multimodal Large Language Models (MLLMs).Since output tokens are a byproduct of the model attending to visualinputs, prior works have explored explicitly distilling attention to pro-vide a direct supervisory signal. While promising, the precise utility ofwhich attention signals to distill remains under-explored. In this work, wechallenge the conventional reliance on prompt-to-vision attention by re-vealing that downstream performance correlates strongly with response-to-vision attention similarity to the teacher, but negligibly with that ofprompt-conditioned attention. Furthermore, we observe that attentiondistributions exhibit significant variance across individual tokens, indi-cating that a uniform distillation objective is suboptimal. To this end,we introduce Token-level Response-visual Attention Guidance(TRAG), a distillation objective that 1) shifts the focus to response-to-vision signals and 2) employs token-specific objectives by adaptivelyweighting the Kullback-Leibler divergence based on attention entropy, ef-fectively guiding the student to mirror the teacher’s precise visual focus.Extensive experimental results on multiple benchmarks demonstrate thatTRAG significantly outperforms prior distillation baselines. Our code isavailable at https://github.com/jhjangjh/TRAG.