Causal Yet Future-Aware: Dual-Path Temporal Modeling for Online Action Segmentation

Temporal Action Segmentation (TAS) assigns action labelsto each frame by modeling the temporal structure of actions. While of-fline TAS leverages full temporal context, online TAS cannot directlyaccess future information during inference. Existing online TAS methodsmainly focus on modeling historical dependencies, but temporal ambigu-ity still arises due to the incomplete temporal dependencies, leading tofragmented predictions. To tackle this challenge, we propose Dual-PathTemporal Modeling (DPTM) for online TAS, a two-stage framework thatexplicitly models and transfers the future dependencies that are inacces-sible during online inference. In the first stage, we adopt an offline-styletraining scheme where bidirectional dependencies are decomposed intounidirectional historical and future components via a temporal decom-position strategy. In the second stage, we employ a predictive distillationstrategy that uses recent historical frames to predict the separated futuredependencies without violating causality. These designs enable DPTMto leverage more complete temporal context while remaining suitable foronline inference. Extensive experiments demonstrate that DPTM signifi-cantly improves temporal coherence in segmentation results and achievesstate-of-the-art performance, demonstrating the effectiveness of modelingdependencies beyond historical frames for reducing the temporal ambi-guity. The code is available at: https://github.com/zczheng777/DPTM/.