Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation

Vision–Language Models (VLMs) provide rich semantic pri-ors but are underexplored in Semi-supervised Semantic Segmentation.Recent attempts to integrate VLMs to inject high-level semantics over-look the semantic misalignment between visual and textual representa-tions that arises from using domain-invariant text embeddings withoutadapting them to dataset- and image-specific contexts. This lack of do-main awareness, coupled with limited annotations, weakens the model’ssemantic understanding by preventing effective vision-language align-ment. As a result, the model struggles with contextual reasoning, showsweak intra-class discrimination, and confuses similar classes. To addressthese challenges, we propose the Hierarchical Vision–Language Trans-former (HVLFormer), which achieves domain-aware and domain-robustalignment between visual and textual representations within a query-driven mask-transformer architecture. Firstly, we transform text embed-dings from a pre-trained VLM into multi-scale, dataset-aware textualobject queries that capture class semantics from coarse to fine gran-ularity and enhance the model’s semantic understanding. Next, thesequeries are refined using image-specific visual context, aligning globaltextual semantics with local scene structures and improving class dis-crimination. Finally, to achieve domain-robustness, we introduce cross-view and modal consistency regularization, which enforces predictionconsistency within the mask-transformer architecture across augmentedviews. It ensures that language queries remain robust to intra-class vari-ations and diverse visual scenes without overfitting to the small labeledset while maintaining stable vision–language alignment under pertur-bations during decoding. With limited training data, HVLFormer out-performs state-of-the-art methods on public benchmarks. Project page:numnz.github.io/HVLFormer.