Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation

Modern multimodal large language models (MLLMs) achieve impressive results on vision-language benchmarks, yet existing evaluations judge only final answers, making shortcuts indistinguishable from genuine understanding. We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning through verifiable intermediate steps. We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which further penalizes disordered reasoning chains. References are constructed through a Delphi-inspired pipeline where four independent MLLMs generate trajectories, aggregated via semantic clustering and validated through human quality gates. Evaluation of 20 MLLMs, including commercial frontier systems not used during benchmark construction, reveals systematic failures invisible to accuracy: universal cherry-picking (precision far exceeds recall), non-monotonic scaling trade-offs, and disordered reasoning, where even the strongest reasoners preserve under 60% of matched steps in correct order. Beyond evaluation, we propose the Causal Process Reward (CPR), a multiplicative reward that couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training. CPR-Curriculum achieves +32% Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.