Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

Foundation Models (FMs) are increasingly integrated into remote sensing (RS)pipelines for applications such as environmental monitoring, disaster assessment, and land-usemapping. These models include unimodal vision encoders trained in a single data modalityand multimodal architectures trained in multiple sensor modalities, such as synthetic aper-ture radar (SAR), multispectral, and hyperspectral imagery, or jointly in image-text pairs invision-language settings. FMs are adapted to diverse perception tasks, such as semantic seg-mentation, image classification, change detection, and visual question answering, dependingon their pretraining objectives and architectural design. However, selecting the most suit-able remote sensing foundation model (RSFM) for a specific task remains challenging dueto scattered documentation, heterogeneous formats, and complex deployment constraints.To address this, we first introduce the RSFM Database (RS-FMD), the first structuredand schema-guided resource covering over 160 RSFMs trained on various data modalities,spanning different spatial, spectral, and temporal resolutions, considering different learn-ing paradigms. Built upon RS-FMD, we further present Remsa (Remote-sensing ModelSelection Agent), a constraint-aware agent that enables automated RSFM selection from nat-ural language queries. Remsa combines structured FM metadata grounding with task-awareorchestration for retrieval, clarification, ranking, and explanation. In detail, it interprets userinput, clarifies missing constraints, ranks models via in-context learning, and provides trans-parent justifications. Our system supports various RS tasks and data modalities, enablingpersonalized, reproducible, and efficient FM selection. To evaluate Remsa, we constructa benchmark of 100 expert-verified RS query scenarios. Each query is evaluated across 4systems and 3 LLM backbones, with the top-3 selected models assessed by domain expertsusing a fixed suitability rubric. This results in 3,000 expert-scored task–system–model con-figurations under our rubric-based expert evaluation protocol. Remsa outperforms multiplebaselines, including naive agent-based method, dense retrieval, and unstructured retrievalaugmented generation approaches, showing its practical utility in real decision-making ap-plications. Remsa operates entirely on publicly available metadata of open source RSFMs,without accessing private or sensitive data. Our code and data are publicly available at:https://github.com/be-chen/REMSA.