Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision

Diffusion models have achieved impressive results in gener-ative tasks such as text-to-image synthesis, yet they often struggle tofully align outputs with nuanced user intent and maintain consistentaesthetic quality. Existing preference-based training methods such asDiffusion Direct Preference Optimization help address these issues, butobtain their supervision targets from the forward process q(xt−1 | xt , x0 )derived from terminal samples, which is not directly aligned with themodel’s actual backward denoising transitions at each step. In this work,we introduce Direct Diffusion Score Preference Optimization (DDSPO),which defines stepwise preference supervision directly over backward de-noising transitions through a contrastive policy pair, rather than relyingon forward-process approximations from terminal samples. We proposetwo practical instantiations of the contrastive policy pair: training sepa-rate winning and losing models on preference data, and inducing a con-trastive policy pair without additional training by using a pretrainedreference model conditioned on an original prompt and a semanticallydegraded variant, requiring neither reward modeling nor manual annota-tions. Empirical results show that contrastive-policy-pair supervision ismore effective than forward-process-based supervision across text–imagealignment and aesthetic-quality tasks. Our implementation is availableat: https://dohyun-as.github.io/DDSPO