Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

Recent high-resolution Multimodal Large Language Models(MLLMs) generate thousands of visual tokens per input, leading to avisual token explosion that introduces severe latency bottlenecks. Whiletoken pruning mitigates this issue, state-of-the-art subset-optimizationmethods typically rely on iterative subset construction to jointly cap-ture visual diversity and instruction relevance. As visual token countsscale, this sequential dependency introduces significant selection over-head, severely limiting the translation of theoretical FLOPs reductionsinto actual wall-clock speedups. To address this limitation, we proposeSingle-Forward Pruner (SFPruner), a structural reformulation of visualtoken pruning that embeds redundancy control directly into the scor-ing space, bypassing the need for iterative combinatorial optimization.Our non-iterative framework achieves redundancy-aware importance se-lection in a single forward pass through two complementary mechanisms.First, to attenuate redundancy at the covariance level, we introduce asemantics-guided ridge leverage scheme. By integrating instruction rel-evance and visual saliency, this mechanism suppresses dominant covari-ance directions and mitigates representation bias. Second, ranking-baseddirectional masking resolves residual overlap through asymmetric sim-ilarity competition, where higher-scoring tokens explicitly suppress re-dundant lower-scoring alternatives via parallel tensor operations. Exten-sive evaluations demonstrate that our approach maintains stable selec-tion costs, reducing the token selection process by up to 110 ms (from112.4 ms to just 2.5 ms at 512 tokens in Qwen2.5-VL). This structuralefficiency successfully translates theoretical token reductions into tangi-ble inference speedups while preserving highly competitive performanceagainst state-of-the-art techniques under aggressive compression.