Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding

Recently, visual token pruning has been studied to handlethe vast number of visual tokens in Multimodal Large Language Models.However, we observe that while existing pruning methods perform reliablyon simple visual understanding, they struggle to effectively generalizeto complex visual reasoning tasks, a critical gap underexplored in previ-ous studies. Through a systematic analysis, we identify Relevant VisualInformation Shift (RVIS) during decoding as the primary failure driver.To address this, we propose Decoding-stage Shift-aware Token Pruning(DSTP), a training-free add-on framework that enables existing pruningmethods to align visual tokens with shifting reasoning requirements duringthe decoding stage. Extensive experiments demonstrate that DSTP signifi-cantly mitigates performance degradation of pruning methods in complexreasoning tasks, while consistently yielding performance gains even acrossvisual understanding benchmarks. Furthermore, DSTP demonstrates ef-fectiveness across diverse state-of-the-art architectures, highlighting itsgeneralizability and efficiency with minimal computational overhead. Oursource code is available here.