See Only When Needed: Context-Aware Attention Intervention for Hallucination-Free LVLMs

Large Vision-Language Models (LVLMs) excel at multimodaltasks but remain prone to object hallucinations. Prior training-free reme-dies often uniformly strengthen visual signals, which may also amplifyirrelevant regions and introduce spurious evidence, harming fluency. Wepropose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed prin-ciple via two-axis selectivity: where to look and when to intervene. Ateach decoding step, CAI derives token-specific visual relevance fromearly-layer representations to localize semantically aligned regions, andapplies a conservative, entropy- and depth-gated attention tilt only foruncertainty-spiking tokens in deeper layers where visual grounding de-grades, leaving confident tokens and irrelevant regions largely unchanged.This targeted intervention strengthens visual grounding while preserv-ing linguistic fluency, and it yields consistent improvements even with-out contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM back-bones and benchmarks show that CAI achieves state-of-the-art halluci-nation mitigation, and our analysis characterizes CAI as a KL-minimalattention reweighting with bounded interference under inactive gates orsmall tilts. Code is available at https://github.com/Iris1946/CAI.