Mitigating Sycophancy in Multimodal Chart Understanding via Vision-Grounded Verification

Multimodal large language models (MLLMs) have becomethe dominant paradigm for chart understanding, yet they suffer from acritical vulnerability: sycophancy. When users embed false premises inqueries about charts, models override visual evidence to comply with theuser’s misconception. Existing mitigations incur a severe safety-utilitytrade-off : suppressing sycophantic compliance inevitably triggers over-refusal of valid queries, degrading general performance. To address thischallenge, we introduce Re-Check, a training-free inference frameworkthat follows a “Verify-then-Answer” workflow: it first decomposes userqueries into atomic claims, then verifies each claim against the chartimage, and finally routes the generation through an adaptive three-way mechanism. To ensure verification reliability, we propose the Con-trastive Visual Dependency Score (CVDS), an information-theoreticmetric that measures the KL divergence between the model’s predic-tions with and without visual input, filtering out textual-bias-drivenmisjudgments. Through extensive experiments on diverse benchmarks,Re-Check improves overall accuracy by 15.01% (from 57.49% to 72.50%with Qwen3-VL-8B) on sycophancy scenarios, while maintaining compet-itive performance on general benchmarks such as ChartQAPro. Theseresults demonstrate a superior safety-utility trade-off. Code is availableat: https://github.com/X1Wang/ReCheck-code.