Learning Probabilistic Embeddings for Unsupervised Action Segmentation

This paper concerns the problem of unsupervised temporalaction segmentation for long, untrimmed videos. Recent successful ap-proaches follow a joint representation learning and clustering paradigm,where optimal transport (OT) is adopted to produce pseudo labels forlearning frame representations. These approaches alternate between es-timating pseudo labels using OT and optimizing the parameters withgradient descent during training, where OT is used for obtaining the fi-nal temporal action segmentation. A major limitation of these works isthat they learn a deterministic embedding for frame representations. Theiterative procedure between learning deterministic embeddings based onpseudo labels and estimating pseudo labels from the learned embeddingcan thus get quickly stuck in a local optimum. As an alternative, we thuspropose to learn a probabilistic embedding for frame representations. Theembeddings are modeled by Gaussian distributions and we sample fromthe distributions before estimating the pseudo labels. We evaluate ourapproach on several challenging temporal action segmentation datasetsand achieve results comparable to, and in some cases, better than thestate of the art. Compared to baselines with deterministic embeddings,our approach improves MoF up to 20.7% and F1-score up to 19.0%. Ourcode is available at https://github.com/derkbreeze/PEOT.