ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

A unified representation for text and vision is a natural pur-suit, as it enables simpler multimodal modeling and more efficient training.However, representing images as discrete signals in the same way as textinevitably introduces severe information loss. Existing work strugglesto balance low-level details and high-level semantics in discrete repre-sentations: reconstruction-oriented representations often lack semanticinformation, whereas semantically stronger features typically suffer fromsevere loss of detail. We present ViQ, a Visual Quantized Representationsframework, which is designed to balance semantics and details in discreterepresentations while supporting inputs at native resolutions, therebyenabling it to serve as a unified and general discrete representation forarbitrary visual inputs. Our approach structures quantization learninginto two stages: text-aligned pre-training and feature discretization. Withtext-aligned pre-training, we enhance the visual encoder semantic-richsupervision from the pretrained language model and enable it to pro-cess native-resolution visual inputs. During discretization, we proposea proximal representation learning strategy to progressively compactthe feature space, along with a position-aware head-wise quantizationmechanism that enables flexible processing of arbitrary resolutions. Ex-tensive experiments on multimodal tasks demonstrate that ViQ achievescompetitive performance compared to state-of-the-art multimodal visionencoders with continuous and high-dimensional visual features, whilemaintaining high precision in low-level reconstruction. We also showthat multimodal training with visual quantized representations largelyimproves efficiency, yielding up to 20%-70% acceleration with differentbase LLMs and training recipes.