Bayesian Uncertainty Attribution-Guided Fine-Tuning for Open-Set Action Recognition

Open-set action recognition (OSAR) requires a model toclassify known actions while rejecting unseen ones. Existing methodstypically use predictive uncertainty only at inference time, treating it asa rejection score after representation learning is complete. This leaves theunderlying representation unchanged and allows spurious background orcontext cues to remain embedded in the model, which weakens known–unknown separation. We propose a staged uncertainty-guided fine-tuningframework that uses ensemble-derived epistemic uncertainty not only forrejection, but also as a training signal for representation refinement. Theframework proceeds in three stages: motion-guided ensemble training tobias learning toward action-relevant evidence, in-distribution epistemicregularization to improve uncertainty reliability on known samples, anduncertainty attribution mask-based input attention (UAM-IA), whichconverts uncertainty attribution maps into input attention masks for re-finement. To improve deployment efficiency, we further distill the refinedensemble into a single student model trained to approximate both theensemble predictive distribution and the ensemble-derived epistemic un-certainty. Under a controlled OSAR protocol with UCF101 as the knownset and HMDB51 and MiT-v2 as unknown sources after removing over-lapping classes, the proposed method improves known–unknown separa-tion across multiple backbones. The distilled student preserves much ofthis benefit while enabling efficient single-model inference.