Transferability Between Understanding and Generation in Unified Multimodal Models
Abstract
Unified Multimodal Models (UMMs) integrate image under-standing and generation within a single architecture, yet how the twotasks interact remains understudied. We investigate transferability inUMMs: whether training a capability on one task improves the samecapability on the other without explicit supervision. Through controlledexperiments, we empirically find that transferability depends on archi-tecture—models with fully shared transformer backbone and a unifiedvisual encoder exhibit consistent cross-task transfer, while loosely cou-pled designs show little or none. Leveraging this transferability, we pro-pose a practical training strategy. The most straightforward way to im-prove a target generative capability (e.g., counting) is to fine-tune gen-eration directly, but this can degrade visual quality due to distributionshift. Instead, we train the corresponding understanding task and letit transfer into generation, which improves capability-specific genera-tive performance while minimizing distribution shift. We validate thisacross three capabilities—counting, spatial relation, and text recogni-tion/generation—showing that cross-task transferability can be system-atically exploited in UMMs.