Textured Gaussians
for Enhanced 3D Scene Appearance Modeling

Brian Chao1,2 Hung-Yu Tseng2 Lorenzo Porzi2 Chen Gao2 Tuotuo Li2 Qinbo Li2 Ayush Saraf2
Jia-Bin Huang2,3 Johannes Kopf2 Gordon Wetzstein1 Changil Kim2
1Stanford University 2Meta 3University of Maryland
arXiv Code (Coming Soon) Paper (High-Res) Results Gallery
Ours
3DGS

Our RGBA Textured Gaussians enhance 3D scene appearance modeling, leading to improved rendering quality while using the same number of Gaussians compared to 3DGS. Here, we show that Textured Gaussians faithfully reconstruct the fine details of scenes.

Abstract

3D Gaussian Splatting (3DGS) has recently emerged as a state-of-the-art 3D reconstruction and rendering technique due to its high-quality results and fast training and rendering time. However, pixels covered by the same Gaussian are always shaded in the same color up to a Gaussian falloff scaling factor. Furthermore, the finest geometric detail any individual Gaussian can represent is simply an ellipsoid. These properties of 3D Gaussian Splatting greatly limit the expressivity of individual Gaussian primitves.

To address these issues, we draw inspiration from texture and alpha mapping in traditional graphics and integrate it with 3DGS. Specifically, we propose a new generalized Gaussian appearance representation that augments each Gaussian with alpha (A), RGB, or RGBA texture maps to model spatial color and opacity variation across the extent of each Gaussian. As such, each Gaussian is capable of representing a richer set of texture patterns and geometric structures, instead of just a single color and ellipsoid as in naive Gaussian Splatting.

Surprisingly, we found that the expressivity of Gaussians can already be greatly improved by using alpha-only texture maps, and further augmenting Gaussians with RGB texture maps achieves the highest expressivity. We validate our method on a wide variety of standard benchmark datasets and our own custom captures at both the object level and scene level, and demonstrate image quality improvements over existing methods while using a similar or lower number of Gaussians.

Overview

Textured Gaussians encapsulate four kinds of color and opacity spatial variations. The top row of the figure shows the texture map associated with each Gaussian, and the bottom row shows the rendered Textured Gaussians. The constant-color and constant-alpha model (no textures) corresponds to the origianl 3DGS formation, which can only represent a single color up to a Gaussian falloff factor within the Gaussian extent. Textured Gaussians can already model spatially varying colors using only alpha textures since each pixel can be alpha-composited differently. The model achieves maximum expressivity when leveraging the full RGBA texture map, where each Gaussian is capable of representing complex shapes and high frequency textures.

Our method consists of three major components: ray-Gaussian intersection, RGBA texture mapping, and a generalized Gaussian appearance model. To render the color of a pixel, we

  1. Trace a ray from the camera center to the pixel to intersect with 3D Gaussians in the scene.
  2. Query texture and alpha values from the per-Gaussian RGBA texture maps using the ray-Gaussian intersection point.
  3. Given the retrieved spatially varying color and alpha values, we alpha-composite the color and alpha values of Gaussians that are hit by the pixel ray using the generalized Gaussian appearance model.

Through texturing Gaussians, our model achieves strictly better novel-view synthesis (NVS) quality compared to 3DGS when using the same number of Gaussians. Here, we show NVS performance in terms of PSNR / SSIM / LPIPS for our RGBA Textured Gaussians models and 3DGS models with the same number of Gaussians. Our models consistently achieves better NVS performance.

Concurrent Works

There are multiple concurrent works that explore concepts similar to our paper. Here are some of them:

Results Gallery

Table of Contents

In this section, we show novel-view synthesis (NVS) video results of extensive experiments and ablation studies. Due to file size constraints, we show compressed videos of scene-level NVS results of selected scenes from the Mip-NeRF 360, Tanks and Temples, Deep Blending, and our custom-captured datasets.

  1. NVS Results with Varying Numbers of Gaussians and Fixed Amount of Texels
  2. Texture Map Ablations and Color Component Decomposition Visualizations

NVS Results with Varying Numbers of Gaussians and Fixed Amount of Texels

We optimize 3DGS and our RGBA Textured Gaussians models with varying numbers of Gaussians, ranging from 1% of the default optimized number to 100% of the default optimized number of Gaussians. We allocate a fixed amount of texels to each model. Hence, the texture map resolution of models with more Gaussians will be smaller.

When toggling between 3DGS and Ours while using the same number of Gaussians, our method clearly reconstructs sharper details and is most evident with fewer Gaussians. When varying the number of Gaussians using the slider, we see that recontruction performance improves as the number of Gaussians increase, as expected.

Please hover over the video and click on the controls to pause / play the sync videos and toggle between different scenes and models. Scroll through the scrollbar in the Scene column to go through all scenes.

Percentage of Gaussians: 1%
Scene
Model Type

Texture Map Ablations and Color Component Decomposition Visualizations

We optimized our Textured Gaussians model with alpha (A), RGB, and RGBA texture maps with 1% of the default number of optimized Gaussians, and visualize the alpha modulated and composited base and texture color components, and the final rendered color (labelled base + texture). We also show 3DGS optimized with the same number of Gaussians as comparison.

With alpha textures, we see from results of the alpha-only and RGBA textures model that high-frequency details can already be recosntructed well with the base color component due to spatially varying alpha modulation and composition. With RGB textures, the base color component can only reconstruct low-frequency color variations while the texture map color components reconstructs the "residual" details.

Please hover over the video and click on the controls to pause / play the sync videos and toggle between different scenes and models. Scroll through the scrollbar in the Scene column to go through all scenes.

Scene
0
Texture map variants
Color Component