PaD-GS: Leveraging Distortion Map for Panoramic Gaussian Splatting

Recently, 3D Gaussian Splatting (3DGS), which succeeded in perspective image rendering, has been extended to handle the panoramic image rendering task. However, the performance of the existing methods in literature is generally limited due to the severe distortions involved in panoramic images. To address this problem, we construct a distortion map under the panoramic imaging model, where the value of each pixel reflects its corresponding distortion degree, and accordingly, we propose a novel Panoramic Gaussian Splatting method by utilizing this Distortion map, called PaD-GS. The proposed PaD-GS, consisting of a distortion-aware Gaussian decoding module and a distortion-aware opacity modulation module, employs a whole-to-partial strategy to impose the distortion map on Gaussian representation learning. Specifically, the distortion-aware Gaussian decoding module is designed to decode the ensemble of learnable scene features and the distortion map into a set of Gaussian representations, so that the intrinsic distortion information of the distortion map could be injected into the whole Gaussian representations. The distortion-aware opacity modulation module is designed to further impose the distortion map by adaptively modulating the partial attribute (opacity) of each Gaussian representation. Thanks to the introduced distortion map, the proposed PaD-GS could alleviate the negative influence of severe distortion involved in panoramic images. Extensive experimental results on two panoramic datasets demonstrate that PaD-GS significantly outperforms several state-of-the-art methods for panoramic image rendering in most cases. The code is available at https://github.com/CosyXu/PaD-GS.