Scaling Whole-Slide Pathology Foundation Model Pretraining with Billions Off-the-Shelf Tokens

Pathology whole slide image (WSI) analysis is crucial fordisease diagnosis and understanding. While pathology foundation modelshave advanced WSI analysis, their scalability remains limited by thelarge resolution of individual WSIs and the small number of availableslides. Existing methods often use the [CLS] token from tile-level ViTsfor efficiency, but this design overlooks spatially informative tokens thatencode fine-grained pathological patterns. Incorporating all spatial tokenscould enrich representation but is computationally prohibitive, revealinga fundamental trade-off between efficiency and representational richness.We observe high redundancy among spatial tokens, suggesting that se-lectively retaining the most informative ones can better balance thesegoals. To address this, we propose TokRet, a token retention and com-pression module that preserves the most informative spatial tokens whileeliminating redundancy. Built upon TokRet, we develop PathTokScale, aslide-level self-supervised pretraining framework that leverages efficientlong-contextual transformer modeling to scale training to the billion-tokenlevel—over 20× larger than previous WSI studies—without requiring mas-sive WSI collections. Experiments show that PathTokScale consistentlyoutperforms existing pathology foundation models across diagnostic, prog-nostic, and biomarker prediction tasks. Remarkably, using only 10% ofthe number of WSIs as used by TITAN, it achieves comparable per-formance through efficient token scaling, paving the way for scalable,high-performing pathology foundation models.