Proximity-Constrained Counterfactual Decoding for Hallucination-Robust Medical VQA

Hallucinations in medical vision language models arise whenlanguage priors override subtle visual evidence during decoding. Existingtraining-free contrastive decoding methods suppress these priors by con-trasting logits against a globally perturbed image, but global perturba-tions destroy the very diagnostic cues they are meant to protect —stan-dard VCD blur flips nearly one in five CheXpert labels on held-out chestradiographs. We propose Proximity-Constrained CounterfactualDecoding (PCCD), which admits a counterfactual view only whenit satisfies both a global and an object-masked feature-space similaritybound, constraining diagnostic drift while increasing the contrastive mar-gin for visually grounded tokens to first order under explicitly stated as-sumptions. Two complementary branches address heterogeneous halluci-nation drivers: Object-Aware VCD preserves attribution-indicated lesionregions while attenuating contextual co-occurrence priors, and Latent-Disentangled VCD applies structure-preserving style edits to suppressattribute biases. When neither branch passes the proximity gate, PCCDfalls back to greedy decoding, guaranteeing it never degrades below theunmitigated baseline. Across radiology VQA, chest X-ray report gener-ation, and a new CXR existence probe (Med-POPE), PCCD improvesGREEN AUC, QAAS, and RadGraph F1 by up to →+2.5 pp over SOTAwith one auxiliary forward pass per token and no weight updates.