How to Teach Large Multimodal Models New Skills

How can we teach large multimodal models (LMMs) newskills without erasing prior abilities? We study sequential fine-tuningon five target skills while monitoring general ability on eight held-outbenchmarks across three model families. Surprisingly, we find that per-formance lost on held-out tasks after fine-tuning on one skill can partlyrecover when the model is subsequently tuned on a different skill. Wetrace this behavior to a measurable shift in the output token distribu-tion, manifested through a simple counting-bias probe that shows theshift co-varies with forgetting. Guided by this insight, we identify twosimple, robust tuning recipes that learn strongly while limiting drift: (i)updating only the self-attention projection layers (SA Proj., ∆ learning+24.9 / ∆ held-out forgetting -0.6), and (ii) updating only the MLPGate&Up while freezing the Down projection (+30.5 / -2.1). Both sub-stantially outperform full-LLM tuning (+31.8 / -23.3) in the learning–forgetting trade-off. We also compare against common forgetting mitiga-tion methods—Learning without Forgetting (LwF), LoRA, Mixture-of-Experts, and weight-space interpolation (WiSE-FT)—and find that ourselective tuning recipes match or exceed their learning–stability balancewhile remaining simpler, requiring no replay, auxiliary parameters, orper-stage tuning. These results hold across LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL, confirming that the key to teaching LMMs newskills without forgetting lies in controlling output distribution shift bychoosing which components to tune. Code will be made available.