MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE

Although GRPO substantially improves flow-matching models for human preference alignment in vision generation, mainstream methods such as DanceGRPO rely on Global-Stochastic Di!erential Equations (SDE) sampling across full timesteps in the Markov Decision Process (MDP), which remains computationally ine"cient. In this paper, we propose MixGRPO, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of SDE and Ordinary Di!erential Equations (ODE). This redesign streamlines optimization within the MDP, delivering gains in both generation performance and training e!ciency. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed MixGRPO-Flash, which further improves training e"ciency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both e!ectiveness and e"- ciency, with nearly 50% lower training time. Notably, MixGRPOFlash further reduces training time by 71%.5