Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models

Vision-language models (VLMs) have recently shown remark-able capabilities in visual understanding and generation, but remainvulnerable to adversarial manipulations of visual content. Prior object-hiding attacks primarily rely on suppressing or blocking region-specificrepresentations, often creating semantic gaps that inadvertently inducehallucination, where models invent plausible but incorrect objects. In thiswork, we demonstrate that hallucination arises not from object absenceper se, but from semantic discontinuity introduced by such suppression-based attacks. We propose a new class of background-consistent objectconcealment attacks, which hide target objects by re-encoding their vi-sual representations to be statistically and semantically consistent withsurrounding background regions. Crucially, our approach preserves tokenstructure and attention flow, avoiding representational voids that trig-ger hallucination. We present a pixel-level optimization framework thatenforces background-consistent re-encoding across multiple transformerlayers while preserving global scene semantics. Extensive experimentson state-of-the-art vision-language models show that our method effec-tively conceals target objects while preserving up to 86% of non-targetobjects and reducing grounded hallucination by up to 3× compared toattention-suppression-based attacks. Qualitative results further confirmthat our approach maintains scene coherence and avoids spurious ob-ject insertion. Our findings highlight semantic continuity as a key factorin hallucination behavior and introduce a new direction for adversarialanalysis of generative multimodal models.