SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

Vision-language models (VLMs) often underperform on evi-dence intensive tasks because decisive visual evidence are small, localized,and easy to overlook, leading to failures in evidence readout even whenhigh-level reasoning is intact. Prior inference-time visual interventionscan improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is ac-tually used. We study answer-span prediction entropy as a model-internalfeedback signal and show that naive entropy minimization is ambiguous,since low entropy may arise from evidence-grounded confidence or short-cut collapse. To resolve this ambiguity, we introduce low-entropy anchorsand an entropy-shaping objective that reduces answer uncertainty whilepreserving baseline high-confidence tokens. We instantiate this principlein SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuningbased on Group Relative Policy Optimization (GRPO). Across all bench-marks and different VLM families, SPOT-E yields consistent gains andimproved robustness under visual corruptions. Code is publicly availableat: https://github.com/YinBo0927/SPOT-E