Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation

Continual Test-Time Adaptation (CTTA) aims to maintainmodel performance under evolving target domains by adapting onlinewithout labeled data. However, practical deployments often cannot re-tain the source dataset due to privacy or licensing constraints, and purelysource-free CTTA methods tend to become unstable under long-termdistribution shift, suffering from compounding self-training errors andcatastrophic forgetting. We introduce DO-ALL (Distill Once, AdaptLife-Long), a plug-and-play framework that revisits source informationin a compact and privacy-conscious form via Dataset Distillation (DD).Before deployment, DO-ALL performs DD to produce a small set of syn-thetic distilled anchors that summarize the source distribution. Duringadaptation, each target sample is matched with its most semanticallyaligned anchor, which provides a stable reference for various CTTA viasource replay, representation alignment, and manifold-smoothing regu-larization. DO-ALL can be seamlessly integrated into existing CTTA al-gorithms, consistently improving long-term robustness across CIFAR100-C, ImageNet-C, and the CCC benchmark. This demonstrates the po-tential of leveraging DD to enable stable and continuous adaptationwithout retaining raw source data. The code is available at https://github.com/blue-531/DOALL.