MPO: Single-Stream Policy Optimization for Efficient Text-to-Image Alignment

Online reinforcement learning (RL) has become an effectivetool for aligning text-to-image models with human preferences, but itsefficiency is often limited by group-relative variance reduction. Exist-ing methods generate multiple trajectories per prompt to estimate localbaselines, which introduces synchronization overhead and can weaken thelearning signal when group rewards become homogeneous. We proposeMonolithic Policy Optimization (MPO), a group-free online RL frame-work based on a single-stream training principle: one prompt, one trajec-tory, and one policy update. MPO retains stochastic differential equation(SDE) exploration within each trajectory, while replacing group-wisebaselines with a persistent Bayesian value tracker. The tracker main-tains a history-aware reward estimate for each prompt and adapts itsuncertainty using a Girsanov-inspired policy-drift proxy, providing stableglobal advantage estimation under non-stationary policy updates. Acrosscompositional generation, visual text rendering, and human-preferencealignment benchmarks, MPO consistently improves alignment qualityover group-based baselines. It also delivers a 26× wall-clock speedupand a 5× sample-efficiency gain under matched training settings.