Surprise Forcing: What to Remember, When to Skip in Long Video Generation

Autoregressive diffusion models have shown remarkable prog-ress in long video generation by producing frames causally. To boundmemory and computational cost, these models rely on a fixed-size sliding-window KV cache. While attention sinks retain a few initial frames astemporal anchors, the vast majority of historical context is irrecover-ably discarded as the window advances, severing long-range dependen-cies essential for scene consistency and narrative coherence. Meanwhile,the fixed multi-step denoising schedule applies uniform effort to everychunk, ignoring the varying demands imposed by diverse scene dynamicsacross temporal segments. In this paper, we propose Surprise Forcing,a training-free framework that addresses both bottlenecks. We introducea Surprise-Gated Memory Bank that scores each evicted frame by† Corresponding authors.its aggregate divergence from the bank and nearest-neighbor novelty,admitting only the most informative frames via a budget-normalizedgate that adapts to video dynamics. A priority-based eviction policykeeps the bank compact yet informative, while dynamic routing retrievesthe most relevant historical frames at attention time. We further pro-pose Surprise-Aware Denoising, which uses intra-chunk cosine dis-tance among adjacent frames after a single denoising step as a self-referential difficulty predictor, enabling adaptive step reduction for easychunks while preserving full refinement for complex ones. Experimentson VBench show that Surprise Forcing maintains robust narrative coher-ence in long videos without compromising real-time responsiveness.