Policy-Based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards

Autoregressive (AR) models are highly effective for imagegeneration, yet their standard maximum-likelihood estimation traininglacks direct optimization for sample quality and diversity. While rein-forcement learning (RL) has been used to align diffusion models, thesemethods typically suffer from output diversity collapse. Similarly, concur-rent RL methods for AR models rely strictly on instance-level rewards,often trading off distributional coverage for quality. To address these lim-itations, we propose a lightweight RL framework that casts token-basedAR synthesis as a Markov Decision Process, optimized via Group Rela-tive Policy Optimization (GRPO). Our core contribution is the introduc-tion of a novel distribution-level Leave-One-Out FID (LOO-FID) reward;by leveraging an exponential moving average of feature moments, it ex-plicitly encourages sample diversity and prevents mode collapse duringpolicy updates. We integrate this with composite instance-level rewards(CLIP and HPSv2) for strict semantic and perceptual fidelity, and stabi-lize the multi-objective learning with an adaptive entropy regularizationterm. Extensive experiments on LlamaGen and VQGAN architecturesdemonstrate clear improvements across standard quality and diversitymetrics within only a few hundred tuning iterations. The results alsoshow that the model can be updated to produce competitive sampleseven without Classifier-Free Guidance, and bypass its 2x inference cost.