DisRM: Reward Modeling as Discriminative Prediction

Reward models are central to post-training and test-timeoptimization for visual generative models, yet existing approaches typi-cally rely on either large-scale pairwise preference annotations (hundredsof thousands to millions of labeled pairs) or heavily engineered multi-metric scoring pipelines, both of which are costly and labor-intensive. Wepresent DisRM, a discriminative reward modeling framework that avoidsthe need for pairwise annotations by training a lightweight discriminatorto distinguish model-generated outputs from a small set of preferred rep-resentative samples, called Preference Proxy Data (PPD). As the rewardmodel is repeatedly re-trained to distinguish outputs from the updatedgenerator from fixed PPD samples, DisRM naturally supports iterative,multi-round co-refinement with the generator. Across evaluations of im-age quality, safety alignment, and video generation, DisRM achieves com-petitive and often better performance than methods trained with up to1 million annotated preference pairs, while using only 500 unlabeled tar-get samples in image-quality setting, and generalizes across Best-of-Nselection, SFT, and DPO.