Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

Despite the advancements of Large Multimodal Models (LMMs)in RGB vision, their ability to generalize to unseen visual modalitiesremains a largely unexplored challenge. We argue that different visualmodalities are merely distinct samplings of the same physical world.Therefore, effective generalization requires models to possess both modality-agnostic perception of scene semantics and the adaptability to modality-specific characteristics. To achieve this, we propose a training frame-work, VVM-Tuning, to equip LMMs with these capabilities throughmodality synthesis and modality contexts. Specifically, we synthesize di-verse appearance-varied images from RGB scenes, training the model todisentangle invariant semantics from varying visual appearances, andalign these appearances with language for visual concepts decoupledfrom modalities. We then introduce modality contexts in the promptand use instruction tuning to assist the model in mapping these appear-ance variations back to modality-related attributes, enabling zero-shotadaptation to unseen modalities during inference. To facilitate researchin this direction, we introduce VVM-Bench, a comprehensive bench-mark featuring 6 real and synthetic modalities to evaluate semantic per-ception and modality understanding. Experiments demonstrate that, viaour training on synthetic modalities, 5 tested models exhibit consistentimprovements on both real-world and novel synthetic modalities withoutin-modality training. Source code and data will be publicly available athttps://github.com/Hunter-Will/VVM-Tuning.