The Telephone Game: Evaluating Semantic Drift in Unified Models

Unified models (UMs) combine visual understanding (I2T)and generation (T2I) in a single framework, alongside broader unimodalcapabilities. We focus on the cross-modal pair T2I and I2T, where cross-consistency—what a model understands, it should be able to generate—is a core promise of unification and a practical necessity in applicationsthat compose both capabilities. Yet, existing benchmarks evaluate themin isolation: FID/GenEval for T2I; MME/MMBench for I2T. We showthat this gap is consequential: models that score competitively on thesebenchmarks can fail severely when their own understanding and gener-ation are composed, progressively losing core entities, attributes, spa-tial relations, and counts, resulting in semantic drift. To quantify drift,we introduce the Semantic Drift Protocol (SDP), inspired by the Tele-phone Game: starting from a caption or image, we iteratively alternateI2T and T2I over multiple generations and measure how faithfully se-mantics are preserved. We propose two complementary metrics: MeanCumulative Drift (MCD), an embedding-based measure of overall con-tent retention across three representation spaces, and Multi-GenerationGenEval (MGG), which extends GenEval’s object-level compliance scor-ing across generations. To stress-test models beyond COCO-style data,we create a benchmark of 400 image-text pairs sampled from NoCapsand DOCCI that emphasizes novel objects and fine-grained descriptions.Applying SDP to seven recent unified models reveals that drift behaviorvaries dramatically and is not predicted by single-pass scores: BAGELretains high semantic fidelity over multiple generations, while VILA-Uand Janus variants collapse within five generations, despite comparableisolated metrics. We identify six recurring failure modes and find thatdegradation is typically catastrophic rather than gradual: once a criticalerror occurs, subsequent generations compound it. Overall, SDP exposesfailure modes that single-pass benchmarks miss, enabling a more faithfulassessment of unified model reliability. Code and benchmark resourcesare available at https://github.com/mollahsabbir/telephone-game-semantic-drift.