InstrAct: Towards Action-Centric Understanding in Instructional Videos

Understanding instructional videos requires recognizing fine-grained actions and modeling their temporal relations, which remainschallenging for current Video Foundation Models (VFMs). This diffi-culty stems from noisy web supervision and a pervasive “static bias”,where models rely on objects rather than motion cues. To address this,we propose InstrAct, a pretraining framework for instructional videos’action-centric representations. We first introduce a data-driven strategy,Project page: https://zyyangzy.github.io/InstrAct/which filters noisy captions and generates action-centric hard negativesto disentangle actions from objects during contrastive learning. At thevisual feature level, an Action Perceiver extracts motion-relevant tokensfrom redundant video encodings. Beyond contrastive learning, we intro-duce two auxiliary objectives: Dynamic Time Warping alignment (DTW-Align) for modeling sequential temporal structure, and Masked ActionModeling (MAM) for strengthening cross-modal grounding. Finally, weintroduce the InstrAct Bench to evaluate action-centric understanding,where our method consistently outperforms state-of-the-art VFMs onsemantic reasoning, procedural logic, and fine-grained retrieval tasks.