REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation

Automated chest X-ray report generation has recently bene-fited from reinforcement learning (RL) and large language models. How-ever, RL training often suffers from instability or limited explorationdue to fixed Kullback-Leibler (KL) regularization and a static referencepolicy that accumulates KL pressure over time. We propose Response-Weighted and Validation-Anchored Policy Optimization (REVA-PO), aRL framework that stabilizes long-term training via Response-WeightedRegularization (RER) and Validation-Anchored Policy Reset (VAPR).RER dynamically adjusts per-response KL weights based on advantageand reference-policy entropy, relaxing constraints for high-quality re-sponses while tightening them for low-quality ones. Complementarily,VAPR periodically synchronizes the reference and current policies tothe best validation checkpoint, resetting accumulated regularization pres-sure to expand the viable exploration space. To ensure a robust startingpoint, we employ a three-stage pipeline consisting of warm-up training,classifier-guided supervised fine-tuning, and RL. Extensive evaluationson MIMIC-CXR and IU-Xray demonstrate that REVA-PO sets newstate-of-the-art benchmarks in both linguistic quality and clinical accu-racy. Notably, BLEU-4 improves by 5.1% on MIMIC-CXR and 3.6% onIU-Xray, while CheXpert F1 and RadGraph F1 scores increase by 4.5%and 12.8%, respectively, over prior leading methods. The code is publiclyavailable at https://github.com/LiGuo12/REVA_PO/.