GridVQA-X: A Diagnostic Framework for Evaluating Multimodal Explainability Methods

With the increasing development of Vision-Language Mod-els, it becomes imperative that their predictions are readily explainableto relevant stakeholders. However, the field of explainability has notkept pace with the multimodal surge. While recent Multimodal Explain-able AI (MxAI) methods generate explanations to attribute the interac-tion between different modalities, current evaluation protocols lack theground truth required to distinguish between true cross-modal reason-ing (e.g. spatial composition) and shallow cross-modal shortcuts (e.g.Bag-of-Words attribute matching). It remains unknown whether MxAImethods faithfully capture synergistic interactions or merely hallucinatereasoning on models acting as simple feature detectors. In this paper,we introduce GridVQA-X, the first diagnostic framework specificallydesigned to evaluate cross-modal explainability. Unlike natural datasets,GridVQA-X leverages a closed-world synthesis logic to generate unique,mathematically guaranteed explanations. We utilize this controlled envi-ronment to train paired ground-truth models on identical architectures:Mpure , which learns robust spatial-relational reasoning and Mspur , whichis structurally forced to rely on cross-modal shortcuts. This behavioraldivergence creates a rigorous testbed: a faithful explainer must reportdistinct reasoning pathways for each model.Our findings reveal that widely used methods fail to distinguish betweenmodels relying on genuine spatial-relational reasoning and those exploit-ing cross-modal shortcuts, highlighting a critical gap in capturing truecross-modal synergy and misrepresenting how multimodal models actu-ally make decisions. Our code is available at GitHub, and the dataset andmodels are available on Hugging Face at gridvqa-dataset and gridvqa-models, respectively.