3D Gaussian Texture for Real-time Mesoscale Appearance Synthesis and Rendering

We propose a novel framework for modeling, synthesis, and rendering of mesoscale appearances based on 3D Gaussian Splatting (3DGS) for better quality and performance. The key problem is that synthesis of 3D Gaussians is difficult due to their explicit nature. Therefore, the core of our method is a multi-layer Gaussian texture representation to support high-quality and efficient synthesis of 3D Gaussians. Each layer of the texture contains Gaussians with similar heights to the object surface, which is compatible with 2D image texture synthesis approaches. Through by-example autocovariance-preserving synthesis, we achieve the run-time synthesis of mesoscale appearances with their structures preserved. Due to the deformation of mapping from texture to arbitrary objects and the baking of lighting, directly applying the Gaussian texture for rendering may cause visual artifacts. Thus, we further introduce two components: a deformation-aware mapping and a material-lighting decomposition. The mapping is based on barycentric UV coordinates with a scaling constraint of Gaussians to reduce artifacts. The decomposed materials are represented by latent features to maintain the consistency of the material attributes during synthesis. Consequently, our approach achieves a faithful representation of mesoscale appearances modeled from multi-view images, which supports synthesis and rendering on arbitrary objects in real time, achieving about 500× acceleration compared to the latest NeRF-Texture. Our code is available at: https://github.com/ChenXiang0810/multilayer-gaussian-texture.