Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

Reasoning agents increasingly rely on external tools such asweb search to answer complex queries. Reinforcement learning (RL) fine-tuning algorithms such as GRPO have improved long-form reasoningin text-only language models, particularly for coding and mathematics.Reliable tool use in multimodal agents, however, remains challengingbecause models must interpret text and images while integrating noisyretrieved evidence, often under sparse outcome-level supervision withoutexplicit verification signals. We present Self-Verification via Reinforce-ment Learning (SVRL), an RL-only finetuning framework that trainsmultimodal agents to verify and filter retrieved evidence within theirown reasoning traces, reducing reliance on external verifiers at infer-ence time. SVRL also introduces a search-aware penalty that discour-ages unnecessary tool calls and a query-diversity reward that encouragesdiverse, well-formed search queries, providing fine-grained feedback onwhen and what to search. Finetuning Qwen-2.5-VL-7B with SVRL ononly 5,000 visual question answering examples yields consistent gainsin multi-hop VQA generalization and tool efficiency across benchmarks.Overall, SVRL narrows the gap between compact agents and much largerproprietary models while requiring substantially lower training and in-ference cost.