ORBIT: Overcoming Hallucination Risks via Bi-manifold Interaction and Traction

Despite the remarkable capabilities of Large Vision-LanguageModels (LVLMs), they remain highly susceptible to hallucinations. Exist-ing inference-time mitigation strategies largely rely on visual attention asa diagnostic proxy. However, we demonstrate that this approach is insuf-ficient in certain cases due to the “anchoring trap,” where parameterizedlinguistic priors override correct visual grounding. In this paper, we pio-neer a representation geometry perspective, identifying hidden states asa complementary diagnostic signal beyond visual attention. We modelthe evolution of hidden states across layers and generation steps as con-strained sequences governed by a static visual subspace and a dynamiclinguistic subspace within a high-dimensional Euclidean space. We revealthat hallucinations inherently stem from abrupt structural deviations,specifically, the spontaneous departure of latent representations from thevisual subspace and their subsequent drift into the linguistic domain.To quantify this phenomenon, we propose the Geo-Semantic TurbulenceIndex (GSTI) to adaptively monitor visual misalignment and semanticvolatility. Building upon GSTI, we introduce Orthogonal Residual-BasedIntervention (ORBIT), an architecture-agnostic, plug-and-play mecha-nism that recalibrates divergent representation sequences by employinga closed-loop intervention encompassing residual extraction, visual re-anchoring, and norm-preserving injection of suppressed visual evidence.Extensive experiments across diverse LVLM architectures demonstratethat ORBIT significantly mitigates hallucinations and enhances generalvision-centric performance with exceptional computational efficiency.