VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling

Vision Language Models (VLMs) face signix001Ccant challengeswith ultra-long, interleaved image-text sequences due to the quadraticcomplexity of self-attention. Current solutions either resort to aggres-sive token pruning, risking irreversible information loss, or adopt ex001E-cient but less precise architectures, while largely ignoring the equallyvital textual component. We introduce VLZip, a framework that uni-x001Ces visual and textual compression for high-x001Cdelity reasoning within apure Transformer. At its core, VLZip hierarchically distills visual andtextual segments into compact, layer-specix001Cc "soft prex001Cxes" and injectsthem into each decoder layer's hidden states, drastically shortening theattention sequence while preserving x001Cne-grained global context. To ad-dress dex001Ccient evaluations in the x001Celd, we also introduce LongVLBench,a new benchmark derived from video narratives that demands holistic,narrative-level reasoning. Extensive experiments show VLZip achievesleading performance on long-context multimodal reasoning, enabling train-ing up to 120K tokensx0016a 6× increase over the baselinex0016and inferencebeyond 280K tokens with signix001Ccantly reduced memory, while demon-strating the memory scalability to handle up to 2M tokens. By excellingat extreme context lengths where existing methods collapse, VLZip es-tablishes an ex001Ecient and powerful new standard for long-context mul-timodal AI. Code is available at https://github.com/ShareLab-SII/VLZip.