Training-Free Task Classification for Multi-Task Model Merging

Ever since the advent of foundation models and the pre-training–finetuning paradigm, there have been numerous efforts to mergemultiple task-specific experts into a single multi-task model. Prior worklargely focuses on finding a single merged model, but it often under-performs individual experts due to parameter interference. To resolvethis, dynamic model merging employs routing to activate task-relevantparameters per input. However, existing routers typically require eitheradditional training with abundant labeled datasets or assume the accessto task IDs of each input at inference time. In this work, we aim toclose the gap to expert performance without additional training or task-ID-access assumption. To this end, we formulate routing as training-freetask classification for each test input. Using singular value decomposition(SVD)-based low-rank manifold approximations for each task, SiM scorestasks by the projection residual of the test input feature onto each taskmanifold and routes accordingly. The task manifolds are pre-computableoffline from a pretrained backbone using a small per-task support set(e.g., 32 examples per task) prior to merging process, requiring no routertraining and no data during the merging process. Moreover, SiM inte-grates seamlessly with subspace-/mask-based merging that representstask-expert via lightweight compressed task vectors, avoiding the needto store full expert parameters. Experiments across computer vision andnatural language processing benchmarks under task-unknown inferencedemonstrate that SiM substantially improves merged-model performanceand consistently narrows the gap to individual task experts. Our code isavailable at https://github.com/BAIKLAB/SiM