Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as “auto-annotation”, as it aims to learn a modelfrom a few labeled and abundant unlabeled task-specific examples to an-notate the unlabeled ones. Despite the availability of powerful open-sourceVision-Language Models (VLMs) and open-world data, existing SSFSLliterature largely neglects these resources. In contrast, the related areafew-shot learning (FSL) has already exploited them to boost performance.Arguably, to solve real-world auto-annotation, SSFSL should leverage suchopen resources. To bridge this gap, we explore established SSL methodsto finetune a VLM. Unexpectedly, they significantly underperform FSLbaselines that do not use unlabeled data. Our in-depth analysis revealsthe root cause of failure: VLMs produce “flat” distributions of softmaxprobabilities, resulting in zero utilization of unlabeled data and weak su-pervision signals. To address this challenge, we propose an embarrassinglysimple solution that uses temperatures to sharpen the softmax output,which not only increases the confidence scores of pseudo-labels to improvethe utilization of unlabeled data, but also strengthens training supervisionfor effective finetuning. Furthermore, we exploit task-relevant open data,e.g., those retrieved from VLMs’ publicly available pretraining set. Tomitigate the imbalance and domain gaps in retrieved data, we employa stage-wise training strategy. Building on the successful finetuning ofVLMs and the exploitation of open data, we present a simple yet effectiveSSFSL method, Stage-Wise Finetuning with Temperatures (SWIFT).Across five benchmarks, SWIFT outperforms recent FSL and SSL meth-ods by ∼5 accuracy points. SWIFT even rivals supervised learning, whichfinetunes a VLM assuming unlabeled data having ground-truth labels!