Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment

Text-to-image (T2I) diffusion models often fail to faithfullyrender explicit textual descriptions, instead defaulting to strongly learnedvisual priors due to a phenomenon referred to as concept association bias.We show that such bias is particularly strong for one-and-only (OAO)objects, entities that exist in a single canonical form, such as celestialbodies, landmarks, and artworks. The deeply ingrained visual identity forthese concepts often resists modification through prompting alone. Ad-dressing this challenge, we first identify through an information-theoreticanalysis that the final text embedding discards concept-level informationpresent in the intermediate-layer text representations, reducing the mu-tual information available to the subsequent denoising process. We thenpropose Intermediate Text Representation (IR)-guided diffusion, whichinjects intermediate hidden states of the text encoder into the condi-tioning signal during early denoising steps, recovering suppressed con-cepts without any additional training, optimization, or external models.To systematically evaluate the challenging task of aligning generativeoutputs with unusual prompts for OAO objects, we introduce OAO-AttackBench, a benchmark comprising counterfactual prompts that di-rectly conflict with the core visual identity of OAO objects. Experi-ments on four benchmarks, including OAO-AttackBench, show that ourmethod achieves up to a 19.1 percentage-point improvement in VQAS-core while preserving generation fidelity and human preference. Projectpage: https://soyoun-won.github.io/one-and-only-ir-guidance/.