Same Pool, Different Answer: Stable Best-of-N Selection for Vision-Language Models

A common way to improve vision-language model outputsis Best-of-N (BoN) selection: generate N candidate answers, score eachone, return the best. When the candidate pool and compute budget areheld fixed, the selected answer should be reproducible. We show it often isnot. Slightly adjusting the softmax temperature, toggling a probability-truncation option, or enabling dropout noise during the scoring passis enough to change the winning answer on 41–47% of test items —even though the candidate pool is identical. This failure is specific tolikelihood-based selection, where candidates are ranked by policy likeli-hoods or policy/reference likelihood ratios; a deterministic reward-modelor rule-based (e.g. RLVR) reranker assigns identical scores to identicaltext and is stable on a fixed pool by construction.Standard scoring conflates answer quality with two confounding factors:when token-level scores are summed, low-value suffixes can change thetotal without improving answer quality, and score magnitudes shift when-ever the model is updated. We propose DSPA, which separates two con-cerns into a test-time selector and a training-time filter. At test time,we score each candidate by comparing it token-by-token against a fixedreference model and averaging the result, which removes the additivelength accumulation of summed scores and keeps all scores on a fixedreference scale; this is the component responsible for selection stability.At training time, we filter preference pairs to prevent shortcuts based onanswer length, verbatim prompt copying, or hallucinated objects; thiscomponent improves the factual quality of the candidate pool, not thestability of selection. We evaluate stability by replaying the scoring stepon the same frozen pool under controlled perturbations (a protocol wecall SRP), where the anchored selector reduces the answer-flip rate from44.3% to 24.9%. Manual inspection shows roughly 70% of remainingflips are near-ties that differ only in phrasing. On identical pools, the an-chored selector also reduces object hallucination (POPE) from 9.6% to6.2%; the stability gain holds at 13B scale, while the utility gain transfersto Qwen2.5-VL-7B-Instruct.