Learning Consistency in Reward Modeling for Multi-Modal Reasoning

Reliable reward system is essential for reinforcement learn-ing in multi-modal reasoning, yet existing methods face key limitations:preference-based models often misjudge correctness on complex tasks,while rule-based approaches struggle with semantically equivalent an-swers and therefore restrict the scale of usable RL data. To address thesechallenges, we propose Consistency Reward Models (CRM), which assesshow well a candidate response aligns semantically and mathematicallywith ground truth. Our model produces a continuous reward derived fromthe logits associated with “Consistent” vs. “Inconsistent”, enabling moreinformative and stable optimization signals than binary rewards duringRL. By design, CRM compares answers in text space, which our ablationshows is both sufficient and more accurate for consistency judgment. Wealso find that incorporating Chain-of-Thought (CoT) reasoning improvesthe reliability of these consistency assessments. As a judging model, itachieves an 77.8% win rate over VLMEvalKit, demonstrating strongerjudging accuracy. When used for RL, the approach enables a 7B pol-icy model to attain state-of-the-art performance on MathVista (75.2%)and MathVision (30.6%), outperforming rule-based systems, while sub-stantially increasing the scale of RL-usable data. The method furthergeneralizes to OCR, MMMU and open tasks and supports stable RLtraining up to 72B model. These results show consistency reward modelsoffer a more scalable and robust foundation for multi-modal reasoningRL.