MED-LCDS: Multi-Expert-Domain CLIP Classification via Logit Calibration

Vision-language models (VLMs) such as CLIP enable zero-shot classification by comparing image features with text prompts in ashared embedding space. A fundamental property underlying this ca-pability is the global comparability of logits across arbitrary candidateclasses. However, VLMs are often adapted to fine-grained domains usingtechniques such as LoRA. While this improves in-domain accuracy, out-of-domain accuracy degrades. This leads to a higly fragmented modelecosystem, with thousands of specialized models. Multi-Expert-Domain(MED) classification seeks to address this problem, by merging LoRAstrained independently on specialized domains. However, due to the inde-pendent training, the various domain experts no longer produce globallycalibrated logits. As a result, when evaluating over the union of multipledomain-specific class sets, heterogeneous logit scales induce cross-domaininterference and artificially high confidence for out-of-domain classes,inducing prediction errors. In this work, we identify domain supervi-sion and cross-domain logit miscalibration as the key issue to scalablemulti-domain zero-shot recognition. We propose a mixture-of-expertsMED architecture, MED-DSLC, combining domain supervised training anddomain-wise logit scaling, to explicitly restore global logit comparability.MED-DSLC is a lightweight solution for MED classification, which is shownto preserve within-domain discrimination while reducing cross-domainlogit interference with minimal data. Extensive experiments across di-verse fine-grained benchmarks demonstrate that it substantially improvesmean accuracy (+15%), cross-domain robustness, and scalability in thesize of MED classification problem. Our results show that restoring output-level calibration is essential under highly data imbalanced settings forachieving a truly zero-shot VLM under multi-domain specialization. Codeis publicly available at MED-DSLC.