Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings

Foundation models (FMs) have driven significant progressin computational pathology. These models can easily exceed a billionparameters and produce high-dimensional embeddings, thus limiting theirapplicability for research or clinical use when computing resources aretight. Here we introduce Pathryoshka, a novel multi-teacher distillationframework inspired by agglomerative models and Matryoshka representa-tion learning to reduce pathology FM sizes while allowing for adaptableembedding dimensions. We evaluate our framework with a distilled modelon ten public pathology benchmarks with varying downstream tasks.Compared to its much larger teachers, Pathryoshka reduces the modelsize by 86-92% at on-par performance. It outperforms state-of-the-artsingle-teacher distillation models of comparable size by a median marginof 7.0 and other pathology multi-teacher distillation by 5.3 percentagepoints in accuracy. By enabling efficient deployment without sacrificingaccuracy or representational richness, Pathryoshka democratizes accessto state-of-the-art pathology FMs for the broader research and clinicalcommunity.