Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting

Knowledge-Based Visual Question Answering (KB-VQA) aimsto evaluate whether Visual Language Models (VLMs) can retrieve, ground,and reason over external structured knowledge beyond visual evidence.In practice, answer accuracy is widely adopted as the primary evalu-ation metric, implicitly treating correctness as a proxy for knowledge-grounded reasoning. However, for existing KB-VQA benchmarks, thisproxy relies on critical assumptions that are often overlooked and ren-dered unreliable by benchmark issues: annotated answer must be deriv-able from the associated knowledge base, question must be well-posedwith su!cient constraints, and visual setting must meaningfully requiregrounded disambiguation. In this work, we show that these assumptionsare systematically violated in existing KB-VQA benchmarks. Our au-dit reveals substantial instances with missing or contradicted answersand underspecified questions that render accuracy a misleading metric.Furthermore, we find that existing datasets rely on visually trivial, single-entity scenes that bypass the need for sophisticated visual-to-knowledgemapping. We demonstrate that even with controlled architectures, theseflaws lead to distorted model rankings and overestimations of reasoningcapabilities. To address this, we introduce (1) a principled audit-and-repair protocol that restores answer derivability and question clarity,and (2) a controlled multi-entity augmentation protocol that introducesvisual ambiguity to challenge initial retrieval and grounded reasoning.Re-evaluation under corrected and augmented settings yields markedlydi"erent performance trends. Our findings call for rethinking evaluationprotocols and designing more interaction-aware KB-VQA benchmarksthat prioritize verifiable reasoning over simple matching. 1