EgoGVAE: Ego-body Mesh Reconstruction via Guided Variational Autoencoder

We address the problem of recovering the full-body meshfrom only the head pose. This task has become essential for variousapplications based on head-mounted devices or smart glasses. The chal-lenge of this task lies in estimating the pose information of unobservedbody parts based solely on a single joint (i.e., head) trajectory. Severalstudies have begun to adopt head-conditioned generative models, how-ever, such previous methods are costly and time-consuming due to thediffusion-based iterative process. As an alternative, we propose a simpleyet novel method that leverages the latent space of the guidance network,which is designed as a variational autoencoder taking full-body poses asinputs. By enforcing latent distributions of this guidance network andour head-to-motion network to be similar, latent features sampled fromthe ‘guided’ distribution, i.e., distribution learned in our head-to-motionnetwork, can be reliably decoded for natural representations of full-bodyposes even only with the head pose. One important advantage of theproposed method is that one-step sampling scheme achieves remark-ably fast inference (more than 50 times faster) compared to diffusion-based approaches. Experimental results on benchmark datasets showthat the proposed method efficiently improves the performance of ego-body mesh reconstruction. The code and model are publicly available at:https://github.com/DCVL-3D/EgoGVAE_release.