Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-visual Language Models

While Audio-Visual Language Models (AVLMs) have achievedremarkable progress over recent years, their reliability is bottleneckedby cross-modal hallucination. A particularly pervasive manifestation isvideo-driven audio hallucination: models routinely exploit visual short-cuts to hallucinate expected sounds, discarding true auditory evidence.To counteract this deeply ingrained visual dominance, we propose Audio-Contrastive Preference Optimization (ACPO). This dual-axis preferencelearning framework introduces an output-contrastive objective to penal-ize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize gener-ation invariant to the true auditory signal. Extensive experiments demon-strate that ACPO establishes highly faithful audio grounding and mit-igates audio hallucination. Project page: https://vision.cs.utexas.edu/projects/acpo/