Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

Multimodal Large Language Models (MLLMs) have recentlydemonstrated strong performance across vision-language tasks. However,their high inference cost, arising from both the large number of inputvisual tokens and the heavy computation of the large language model(LLM), remains a key barrier to practical deployment. Recent work at-tempts to reduce the cost by adaptively optimizing individual dimen-sions, e.g., pruning redundant visual tokens or skipping LLM layers andheads. Nonetheless, prior approaches typically treat these dimensions in-dependently and overlook a fundamental coupling: the available computeresources must be dynamically allocated across all dimensions based onthe input content. To bridge the gap, we propose SmartVL, a unifiedadaptive inference framework that jointly controls vision token numberand model compute capability in response to varying input contents andcompute budgets. SmartVL introduces a vision-side token controllerthat dynamically selects informative visual tokens and an LLM-side com-pute controller that adaptively adjusts LLM computation. Importantly,these controllers are trained to coordinate with each other so that theoverall inference cost satisfies a target budget. To allow this joint schedul-ing, we connect the controllers using a shared budget encoding and lever-age a differentiable latency estimator for end-to-end training. This de-sign enables SmartVL to learn cross-stage allocation strategies thatadapt to both input complexity and runtime compute constraints. Exper-iments across multiple MLLM benchmarks demonstrate that, with jointscheduling, SmartVL consistently outperforms prior adaptive methodsand achieves superior accuracy–efficiency Pareto frontiers. Project page:https://www.schaterji.io/publications/2026/jointtokencompute.