Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image

Neural Parametric Head Models (NPHMs) are a recent ad-vancement over mesh-based 3d morphable models (3DMMs) to facilitatehigh-fidelity geometric detail. However, fitting NPHMs to visual inputsis notoriously challenging due to the expressive nature of their under-lying latent space, which heavily limits NPHM’s practical use. To thisend, we propose Pix2NPHM, a vision transformer (ViT) network that di-rectly regresses NPHM parameters, given a single image as input, finallybridging the gap from theoretical to practical use. Compared to exist-ing 3DMM regressors, the neural parametric space allows our methodto reconstruct more recognizable facial geometry and accurate facial ex-pressions. For broad generalization, we exploit domain-specific ViTs asbackbones, which are pretrained on geometric prediction tasks. We trainPix2NPHM on a mixture of 3D data, including a total of over 100KNPHM registrations, and large-scale 2D video datasets, for which nor-mal estimates serve as pseudo ground truth. While Pix2NPHM runs atinteractive frame rates, it is possible to improve geometric fidelity by asubsequent optimization against estimated surface normals and canoni-cal point maps. As a result, we achieve unprecedented face reconstructionquality that can run at scale on in-the-wild data.