Continuous Speculative Decoding for Autoregressive Image Generation

Continuous visual autoregressive (AR) models have demon-strated promising performance in image generation, but their inherentlysequential nature results in slow inference speed. Speculative decoding, asuccessful acceleration technique for large language models (LLMs), haseffectively accelerated discrete visual AR models. However, the absenceof an analogous theory for continuous distributions precludes its use inaccelerating continuous AR models. To fill this gap, this work presentscontinuous speculative decoding, and addresses challenges from: 1) lowacceptance rate, caused by inconsistent output distribution modeled bytarget and draft models, and 2) modified distribution without analyticexpression, caused by a complex integral. For challenge 1), we addresslow acceptance rates through an approximated criterion, a novel denois-ing trajectory alignment strategy based on reparameterization proximity,and token pre-filling. For challenge 2), we introduce acceptance-rejectionContinuous speculative decodingDiscrete speculative decoding Proposal distribution VerificationProposal distribution VerificationDraft model Target model Draft model Target modelY Inconsistency existsYModified distribution NModified distributionResample Modify & Modify & NNormalize NormalizeNo analytic expression Hard to integrateFig. 2: Comparison between discrete and continuous speculative decoding. Discretesituation offers the convenience of directly computing probabilities and simply samplingfrom modified distributions. In contrast, continuous situation faces challenges in theinconsistency of output distributions, leading to low acceptance criterion as well as lowacceptance rate, and the modified distributions without analytic expression, caused bycomplex integral.sampling algorithm with an appropriate upper bound, thereby avoid-ing explicitly calculating the integral. Furthermore, our denoising trajec-tory alignment is also reused in acceptance-rejection sampling, effectivelyavoiding repetitive diffusion model inference. Extensive experiments onvarious models at 256×256 and 512×512 resolutions demonstrate thatour approach achieves over 2× wall-time speedup while preserving theimage generation quality.