STEP: Score-Based Temporal Energy for Human Pose Video Anomaly Detection

Skeleton-based Video Anomaly Detection (VAD) offers a ro-bust, privacy-preserving solution for identifying abnormal behaviors. Tomodel the distribution of normal static and moving poses, recent meth-ods train Energy-Based Models (EBMs) via Denoising Score Matching(DSM). However, directly injecting noise, required for training, into rawjoint coordinates creates physically impossible poses, and this structuralcollapse severely worsens as the temporal window expands. To addressthis, we introduce STEP, a simple framework that utilizes PrincipalComponent Analysis (PCA) to project pose sequences into a compact,whitened PC-space. Learning the data density within this well-behavedPC-space ensures that the injected noise translates into physically plau-sible variations, which allows the model to process longer video sequenceswithout the performance collapse of raw coordinate baselines. Addition-ally, to mitigate inherent pose estimation inaccuracies arising from occlu-sions or motion blur, we integrate a sequence-level weighting mechanismbased on the estimator’s confidence scores. Operating at real-time com-putational efficiency, our simple and lightweight framework outperformsthe previous skeleton-based state-of-the-art by 12.2% (90.1% AUROC)on the challenging UBnormal dataset and achieves highly competitiveresults by improving on the ShanghaiTech benchmark.