Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time

While text-to-image diffusion models achieve impressive vi-sual quality, they frequently struggle to maintain precise alignment withcomplex compositional prompts. An effective strategy is to improve theinference process of diffusion models, thereby better leveraging their pre-trained priors to address misalignment. Existing training-free methodscan be divided into two categories. The first category focuses on improv-ing the randomly sampled initial noise to obtain an initialization thatencapsulates semantics relevant to the target text. However, existing ap-proaches either perform costly search over noise pools, with no guaranteeof finding a truly prompt-compatible noise within the limited candidates,or manipulate sampled noise without simultaneously ensuring reliablesemantic injection and preservation of the Gaussian distribution. Thesecond category focuses on improving the denoising trajectory. However,these methods lack explicit mechanisms to timely diagnose and correctCorresponding author: Tao Zhang.semantic errors, and therefore fail to prevent error propagation duringgeneration. To address these limitations, we propose AnchorSteer, atraining-free framework that exerts fine-grained control over both ini-tialization and the denoising trajectory. AnchorSteer consists oftwo synergistic components: Semantic Anchoring replaces uninfor-mative Gaussian noise with text-aligned initializations via CLIP-basedprior extraction and a novel Latent-Prior Score Distillation Sampling(LP-SDS) objective. Specifically, LP-SDS distills CLIP visual priors intothe knowledge distribution of diffusion models, mitigating the domaingap between CLIP-based priors and diffusion-based priors. ReflectiveSteering transforms passive denoising with an active Think–Erase–Retouch loop that enables mid-generation self-correction. Think phaseemploys VLM-based dual diagnosis to detect semantic deviations. Erasephase performs a targeted latent rollback with negative-guided inversionto suppress erroneous content, and Retouch phase subsequently appliespositive-guided refinement to recover the missing attributes. Therefore,Reflective Steering enables timely correction of semantic deviationsalong the denoising trajectory, preventing error propagation. Extensiveexperiments on GenEval and T2I-CompBench++ demonstrate that An-chorSteer consistently outperforms existing baselines in text–image align-ment while preserving high visual quality.