MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training

Recent research in geospatial machine learning demonstratesthat models pretrained with self-supervised learning on Earth observa-tion data can perform well on downstream tasks with limited labeleddata. However, most benchmark datasets have few data modalities andpoor global representation, limiting the ability to evaluate multimodalpretrained models at global scales. In order to fill this gap, we introduceMMEarth-Bench, a collection of five new environmental tasks with 12modalities, globally distributed data, and both random and geographictest splits. We benchmark a diverse set of pretrained models and findthat while (multimodal) pretraining tends to improve model robustnessin limited data settings, geographic generalization abilities remain poor.Moreover, a simple randomly initialized multimodal model is competi-tive given enough labeled data. Although data is abundant, models cancurrently only make use of the modalities on which they were pretrained.To solve this problem, we propose using all the modalities available attest time as auxiliary tasks for test-time adaptation. Our model-agnosticmethod for test-time training with multimodal reconstruction (TTT-MMR) can improve performance across all models and tasks on bothtest splits. Furthermore, geographic batching leads to a good trade-off be-tween regularization and specialization during TTT, which is especiallybeneficial for long-tail distributions. Our dataset, code, and visualizationtool are linked on the project page: lgordon99.github.io/mmearth-bench.