Background Blurring Matters: Improving Visual Grounding by Merging Text-Irrelevant Tokens

Visual grounding (VG) aims to precisely localize the objectin input images based on its natural language descriptions. Most recentlyproposed methods address this task using Transformer-based architec-tures that can inject textual information into visual features. However,according to the image tokenization procedure, a large number of vi-sual tokens will be located in text-irrelevant background areas. Thesetokens can introduce noise into the attention calculation, thus reduc-ing the significance of foreground object tokens and ultimately affect-ing the effectiveness of these methods. To this end, we propose a novelToken Blurring (ToB) module, which dynamically merges image tokensbased on the pair-wise visual similarity between them and their tex-tual relevance with input expressions. By reducing the number of text-irrelevant background tokens and preserving the density of text-referredones, ToB can improve both model effectiveness and efficiency in solvingVG tasks. Extensive experiments on RefCOCO, RefCOCO+, and Re-fCOCOg datasets show that Transformer-based models equipped withour ToB module yield better results while reducing computational over-head compared to various existing VG methods. Our code is available athttps://github.com/Mr-Bigworth/ToB.