SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

Diffusion Transformers (DiTs) have become a leading back-bone for video generation, yet their quadratic attention cost remains amajor bottleneck. Sparse attention reduces this cost by computing only asubset of attention blocks. However, prior methods often either drop theremaining blocks which incurs information loss, or rely on learned predic-tors to approximate them, introducing training overhead and potentialoutput distribution shifting. In this paper, we show that the missingcontributions can be recovered without training: after semantic cluster-ing, keys and values within each block exhibit strong similarity and canbe well summarized by a small set of cluster centroids. Based on thisobservation, we introduce SVG-EAR, a parameter-free linear compensa-tion branch that uses the centroid to approximate skipped blocks andrecover their contributions. While centroid compensation is accurate formost blocks, it can fail on a small subset. Standard sparsification typi-cally selects blocks by attention scores, which indicate where the modelplaces its attention mass, but not where the approximation error wouldbe largest. SVG-EAR therefore performs error-aware routing: a lightweightprobe estimates the compensation error for each block, and we computeexactly the blocks with the highest error-to-cost ratio while compensatingfor skipped blocks. We provide theoretical guarantees that relate atten-tion reconstruction error to clustering quality, and empirically show thatSVG-EAR improves the quality-efficiency trade-off and increases through-put at the same generation fidelity on video diffusion tasks. Overall,SVG-EAR establishes a clear Pareto frontier over prior approaches, achiev-ing up to 1.77× and 1.93× speedups while maintaining PSNRs of up to29.759 and 31.043 on Wan2.2 and HunyuanVideo, respectively.