Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation

Latent diffusion models have emerged as a dominant frame-work for high-fidelity image and video synthesis, operating in compactlatent spaces with variational autoencoders (VAEs) to enhance compu-tational efficiency without compromising visual quality. However, con-ventional VAEs are suboptimal for video data as they employ fixed com-pression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tok-enizer for Compact Video Representation), a transformer-based VAEthat incorporates an adaptive token selector which is jointly learned withlatent tokens. By evaluating each token’s content-richness as keep-or-dropprobability, the token selector effectively discards uninformative tokens,naturally allowing data-dependent compression. Applying adaptive tok-enization to diffusion models may cause spatial misalignment, as tokendropping can disturb the original spatio-temporal structure. To allevi-ate this issue, we propose two position-prediction strategies: cascadedand joint generation, to ensure spatial consistency. We empirically showthat our model achieves strong reconstruction and generation quality at astate-of-the-art compression ratio. Further analysis on video data revealsthat this improvement is primarily achieved by reducing spatio-temporalredundancy and removing uninformative tokens, as supported by bothquantitative and qualitative results.