1. Bibliographic Information

1.1. Title

4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

1.2. Authors

• Guanjun Wu (School of Computer Science, Huazhong University of Science and Technology)
• Taoran Yi (School of Electronic Information and Communications, Huazhong University of Science and Technology)
• Jiemin Fang (Huawei Inc.)
• Lingxi Xie (Huawei Inc.)
• Xiaopeng Zhang (Huawei Inc.)
• Wei Wei (School of Computer Science, Huazhong University of Science and Technology)
• Wenyu Liu (School of Electronic Information and Communications, Huazhong University of Science and Technology)
• Qi Tian (Huawei Inc.)
• Xinggang Wang (School of Electronic Information and Communications, Huazhong University of Science and Technology)

1.3. Journal/Conference

This paper was published as a preprint on arXiv. The version analyzed here was submitted on October 12, 2023 (v3). It has since been accepted to CVPR 2024 (Conference on Computer Vision and Pattern Recognition), a top-tier venue in the field of computer vision.

1.4. Publication Year

2023 (ArXiv version), 2024 (Conference version).

1.5. Abstract

This paper addresses the challenge of rendering dynamic scenes (scenes with motion) efficiently. While 3D Gaussian Splatting (3D-GS) allows for real-time rendering of static scenes, applying it directly to dynamic scenes often leads to high storage costs (storing a model for every frame) or slow training. The authors propose 4D Gaussian Splatting (4D-GS), a holistic representation that combines 3D Gaussians with a 4D neural voxel encoding (specifically a hex-plane representation).

The core method involves keeping one set of "canonical" 3D Gaussians and learning a deformation field that predicts how these Gaussians move and change shape (position, rotation, scaling) at any given timestamp $t$. This approach achieves real-time rendering (82 FPS at $800 \times 800$ resolution) with high visual quality and low storage costs, outperforming or matching state-of-the-art methods.

1.6. Original Source Link

• Abstract: https://arxiv.org/abs/2310.08528v3
• PDF: https://arxiv.org/pdf/2310.08528v3.pdf

2. Executive Summary

2.1. Background & Motivation

Novel View Synthesis (NVS) is a technique used to generate new images of a scene from camera angles that were not in the original set of photographs. It is crucial for Virtual Reality (VR), Augmented Reality (AR), and movie production.

While Neural Radiance Fields (NeRF) revolutionized this field, they are computationally expensive and slow to render. Recently, 3D Gaussian Splatting (3D-GS) emerged as a breakthrough for static scenes, offering real-time rendering speeds by representing scenes as clouds of 3D Gaussian ellipsoids rather than continuous neural fields.

However, the real world is dynamic; objects move and deform. Extending 3D-GS to dynamic scenes presents a "trilemma" of challenges:

1. Rendering Speed: Maintaining the real-time performance of 3D-GS.

2. Storage Efficiency: Avoiding the naive approach of storing a separate 3D-GS model for every single video frame, which consumes massive amounts of memory.

3. Visual Quality: Accurately modeling complex motions from sparse inputs without losing detail.

   The motivation of this paper is to solve this trilemma by creating a unified 4D representation that is fast to render, compact to store, and high in quality.

The following figure (Figure 1 from the original paper) showcases the method's ability to render high-resolution dynamic scenes at high frame rates:

该图像是示意图，左侧为960×540分辨率、34帧每秒的视频片段，右侧为1352×1014分辨率、30帧每秒的视频片段。图像展示了不同场景下的动态捕捉效果。

2.2. Main Contributions / Findings

1. 4D Gaussian Splatting Framework: The authors propose a novel framework that represents a dynamic scene using a single set of canonical 3D Gaussians coupled with a Gaussian Deformation Field.
2. Efficient Deformation Network: They design a deformation network that uses a HexPlane (spatial-temporal structure) encoder. This decomposes the 4D space-time volume into six 2D planes, allowing the model to efficiently query features based on a Gaussian's position and time.
3. Real-Time Performance: The method achieves remarkable speed, rendering at 82 FPS at $800 \times 800$ resolution on an RTX 3090 GPU, which is significantly faster than previous dynamic NeRF methods.
4. Compact Storage: By learning deformations rather than storing per-frame parameters, the storage requirement is kept low (e.g., comparable to static 3D-GS, unlike methods that scale linearly with time).

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Novel View Synthesis (NVS): The problem of generating a 2D image of a 3D scene from a camera viewpoint that was not explicitly captured in the input images.
• NeRF (Neural Radiance Fields): A method that uses a neural network (MLP) to represent a scene as a continuous volumetric field of density and color. To render an image, it shoots rays through pixels and accumulates color/density along the ray (Volume Rendering). It is high quality but slow.
• 3D Gaussian Splatting (3D-GS): An explicit representation where the scene is composed of millions of 3D "Gaussians" (ellipsoids). Each Gaussian has:
  • Position ($\mu$): Center coordinate (x, y, z).
  • Covariance ($\Sigma$): Determines the shape and orientation (rotation $R$ and scaling $S$).
  • Opacity ($\alpha$): How transparent it is.
  • Color ($c$): Often represented by Spherical Harmonics (SH).
  • Splatting: A rasterization technique where these 3D ellipsoids are projected ("splatted") onto the 2D image plane, sorted by depth, and blended. This is much faster than ray-marching in NeRF.
• HexPlane / K-Planes: A technique to represent 4D data (x, y, z, time) efficiently. Instead of a massive 4D grid (memory heavy), it decomposes the 4D space into six 2D planes: xy, xz, yz (spatial) and xt, yt, zt (spatial-temporal). Features are retrieved by projecting a 4D point onto these planes and combining the results.

3.2. Previous Works

• Dynamic NeRFs:
  • D-NeRF: Uses a "canonical" space and a deformation network to map points from time $t$ back to the canonical configuration. It uses implicit MLPs, which are slow to train and render.
  • TiNeuVox & K-Planes: These introduced explicit voxel or plane-based accelerations for dynamic NeRFs, improving training speed significantly (minutes instead of days) but still relying on ray-marching for rendering, which limits FPS.
• Dynamic 3D-GS Approaches:
  • Dynamic3DGS (Concurrent): Tracks Gaussians physically but stores parameters per timestamp, leading to high storage cost for long sequences.
  • Deformable-3DGS (Concurrent): Also uses a deformation network but relies heavily on pure MLP capacity or different encoding schemes.

3.3. Differentiation Analysis

The following figure (Figure 2 from the original paper) illustrates the difference between these approaches:

该图像是不同动态场景渲染方法的示意图，如图2所示。 (a) 展示了典范映射体积渲染中采样点的映射，(b) 展示时间感知体积渲染方法， (c) 则阐释了4D高斯溅射中的高斯变形场和高斯光栅化路径。

• Approach (a): Deformation-based NeRFs (e.g., D-NeRF). They warp rays/points in a volume.
• Approach (b): Time-aware volume rendering (e.g., HexPlane). Computes features directly for each point in space-time.
• Approach (c) - Our Method (4D-GS): Instead of processing rays or individual sample points during rendering, 4D-GS takes the entire set of discrete 3D Gaussians and deforms them before rendering.
  • Difference: It operates on explicit primitives (Gaussians). Once deformed, the rendering is just standard, ultra-fast 3D-GS rasterization. It avoids the heavy ray-marching loop entirely.

4. Methodology

4.1. Principles

The core idea is to treat a dynamic scene as a set of particles (Gaussians) that move and change shape over time. Instead of learning a new set of particles for every frame, the model learns:

1. A Canonical State: What the scene looks like at a reference time (or a neutral pose).

2. A Deformation Field: A function that takes a Gaussian's canonical position and the current time $t$ as input, and outputs the change ($\Delta$) in its position, rotation, and scaling.

   This allows the rendering engine to simply "update" the Gaussians for the current frame and then rasterize them using the standard, fast 3D-GS pipeline.

The following figure (Figure 3 from the original paper) depicts the overall pipeline:

该图像是示意图，展示了4D Gaussian Splatting模型的整体流程。图中包含3D Gaussians $\mathcal{G}$ 的位置 (x, y, z) 和时间戳 $t$，通过空间-时间结构编码器提取特征，并使用多头高斯变形解码器生成变形的3D Gaussians $\mathcal{G}^{\prime}$。接着，经过“splatting”处理，最终得到渲染图像。

4.2. Core Methodology In-depth

4.2.1. 3D Gaussian Representation (Static Base)

The paper starts with the standard 3D-GS formulation. A 3D Gaussian is defined by a covariance matrix $\Sigma$ and a center $\mathcal{X}$. The influence of a Gaussian at a point $x$ is: $$G(x) = e^{-\frac{1}{2} (x-\mathcal{X})^T \Sigma^{-1} (x-\mathcal{X})}$$ To make optimization possible, the covariance $\Sigma$ is decomposed into a rotation matrix $R$ and a scaling matrix $S$: $\Sigma = R S S^T R^T$ During rendering, these are projected to 2D. The color of a pixel is computed by blending $N$ ordered Gaussians overlapping that pixel: $$C = \sum_{i \in N} c_i \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j)$$ where $c_i$ is color and $\alpha_i$ is opacity.

4.2.2. The 4D Gaussian Splatting Framework

The goal is to render an image $\hat{I}$ at time $t$ given a view matrix $M$. The framework maintains a set of canonical 3D Gaussians $\mathcal{G}$. A Gaussian Deformation Field Network $\mathcal{F}$ is introduced to compute the deformation $\Delta \mathcal{G}$ at time $t$: $$\Delta \mathcal{G} = \mathcal{F}(\mathcal{G}, t)$$ The deformed Gaussians $\mathcal{G}'$ are obtained by adding the deformation to the canonical state: $$\mathcal{G}' = \mathcal{G} + \Delta \mathcal{G}$$ Finally, the image is rendered using the differentiable splatting function $\mathcal{S}$: $$\hat{I} = \mathcal{S}(M, \mathcal{G}')$$

4.2.3. Gaussian Deformation Field Network

This network consists of two main parts: a Spatial-Temporal Structure Encoder ($\mathcal{H}$) and a Gaussian Deformation Decoder ($\mathcal{D}$).

Step 1: Spatial-Temporal Structure Encoder ($\mathcal{H}$)

To efficiently encode the relationship between a Gaussian's location and time, the authors use a Multi-Resolution HexPlane. The input is the mean position of a Gaussian $\mu = (x, y, z)$ and the timestamp $t$. Standard 4D grids are too memory-intensive. Instead, the 4D space is decomposed into 6 planes:

• Spatial Planes: xy, xz, yz

• Spatial-Temporal Planes: xt, yt, zt

  Let $R_l(i, j)$ denote the feature plane at resolution level $l$ for a plane pair (i, j). The encoder retrieves features for a Gaussian at (x,y,z,t) by projecting coordinates onto these planes, performing bilinear interpolation, and combining them. The formula for computing the voxel features $f_h$ is: $$f_h = \bigcup_l \prod \operatorname{interp}(R_l(i, j)), \quad (i, j) \in \{(x, y), (x, z), (y, z), (x, t), (y, t), (z, t)\}$$

• Symbol Explanation:

  • $\bigcup_l$: Concatenation over different resolution levels $l$.

  • $\prod$: Element-wise product (likely combining specific plane pairs, though the notation implies a combination strategy similar to K-Planes where usually plane features are multiplied or concatenated. The paper says "production process is similar to K-Planes").

  • $\operatorname{interp}$: Bilinear interpolation to sample features from the 2D grid at the projected continuous coordinates.

  • $R_l(i,j)$: The learnable feature grid for plane pair (i,j) at scale $l$.

    After extracting $f_h$, a tiny MLP $\phi_d$ merges these features into a compact latent vector $f_d$: $$f_d = \phi_d(f_h)$$

The following figure (Figure 13 from the original paper) visualizes these HexPlane voxel grids, showing how structure and motion are encoded in the planes:

该图像是图表，展示了HexPlane体素网格$R_1(i,j)$的可视化结果。在（a）-（c）中，分别展示了$R_1(x,y)$、$R_1(x,z)$和$R_1(y,z)$的视图；（e）-（g）则显示了$R_1(x,t)$、$R_1(y,t)$和$R_1(z,t)$的时间维度可视化。此外，图中（d）和（h）分别为训练视图1和训练视图2，展示了动态场景中的球体。

Step 2: Multi-head Gaussian Deformation Decoder ($\mathcal{D}$)

Once the feature vector $f_d$ is obtained for a Gaussian, it is passed through multiple "heads" (small MLPs) to predict specific deformations. The decoder $\mathcal{D} = \{\phi_x, \phi_r, \phi_s\}$ consists of:

• $\phi_x$: Predicts position shift $\Delta \mathcal{X}$.

• $\phi_r$: Predicts rotation change $\Delta r$.

• $\phi_s$: Predicts scaling change $\Delta s$.

  The formulas for the prediction are: $$\Delta \mathcal{X} = \phi_x(f_d)$$ $$\Delta r = \phi_r(f_d)$$ $$\Delta s = \phi_s(f_d)$$

The deformed attributes $(\mathcal{X}', r', s')$ are then computed as: $$(\mathcal{X}', r', s') = (\mathcal{X} + \Delta \mathcal{X}, r + \Delta r, s + \Delta s)$$

• Note: The paper mentions "r" is a quaternion (4D) and "s" is a scaling factor (3D). The addition operation implies an additive offset to these values.

  The final deformed Gaussian set is $\mathcal{G}' = \{\mathcal{X}', s', r', \sigma, \mathcal{C}\}$. Note that opacity $\sigma$ and color $\mathcal{C}$ are usually kept from the canonical set, though the ablation study mentions optional decoders $\phi_\mathcal{C}$ and $\phi_\alpha$ for fluid/non-rigid motion (discussed in the Appendix).

4.2.4. Optimization Strategy

Training a dynamic system from scratch is unstable. The authors propose a two-stage strategy:

1. Warm-up (Static Initialization): For the first 3000 iterations, optimize only the canonical 3D Gaussians $\mathcal{G}$ assuming a static scene (or using a mean configuration). The rendering is done using $\bar{\mathcal{G}}$ directly: $\hat{I} = \mathcal{S}(M, \bar{\mathcal{G}})$. This establishes a solid geometry.

2. Full Optimization: After warm-up, introduce the deformation field. The rendering switches to using the deformed Gaussians $\mathcal{G}'$. Both the canonical Gaussians and the HexPlane/MLP weights are optimized together.

   The loss function $\mathcal{L}$ combines L1 color loss and a Total Variation (TV) loss $\mathcal{L}_{tv}$ to encourage smoothness in the feature planes: $$\mathcal{L} = |\hat{I} - I| + \mathcal{L}_{tv}$$

The following figure (Figure 4 from the original paper) illustrates the optimization process, showing the transition from static initialization to learning motion:

该图像是示意图，展示了优化过程的不同迭代状态。从初始的随机点云输入，到经过3000次迭代后的3D高斯初始化，再到经过20000次迭代的4D高斯联合优化，逐步展示了模型在动态场景中的学习效果。

5. Experimental Setup

5.1. Datasets

The authors evaluate 4D-GS on one synthetic and two real-world datasets:

1. Synthetic Dataset (D-NeRF):

   • Source: Introduced by Pumarola et al. in D-NeRF.
   • Characteristics: Contains 8 scenes (e.g., Hellwarrior, Mutant, Hook, Bouncing Balls). It features complex, large non-rigid deformations with $360^{\circ}$ camera views.
   • Why: Standard benchmark for dynamic view synthesis to test the upper bound of performance (clean data, perfect poses).

2. Real-world Datasets (HyperNeRF & Neu3D):

   • HyperNeRF Dataset: Captured with 1 or 2 cameras. Scenes include "3D Printer", "Chicken", "Broom". Challenges include topological changes and sparse views.

   • Neu3D Dataset: Captured with 15-20 static cameras. Scenes involve human activities like cooking ("Cook Spinach", "Cut Beef").

   • Why: Tests performance on real sensor noise, complex backgrounds, and monocular/multi-view setups.

     The following figure (Figure 11 from the original paper) shows examples of scenes like "Cook Spinach" and "Cut Beef" from the Neu3D dataset:

     该图像是插图，展示了四个不同的烹饪场景，分别是煮菠菜、切牛肉、火焰三文鱼和咖啡马丁尼。在每个场景中，关键物体被红框标出，帮助突出显示烹饪过程中的主要动作和材料。

5.2. Evaluation Metrics

1. PSNR (Peak Signal-to-Noise Ratio):

   • Concept: Measures the pixel-level accuracy of the generated image compared to the ground truth. Higher is better.
   • Formula: $\text{PSNR} = 10 \cdot \log_{10}\left(\frac{MAX_I^2}{MSE}\right)$
   • Symbols: $MAX_I$ is the maximum pixel value (usually 255 or 1.0), and MSE is the Mean Squared Error between the rendered and ground truth images.

2. SSIM (Structural Similarity Index):

   • Concept: Measures perceived image quality by comparing luminance, contrast, and structure. It is closer to human visual perception than PSNR. Higher is better (max 1.0).
   • Formula: $\text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$
   • Symbols: $\mu$ is mean, $\sigma^2$ is variance, $\sigma_{xy}$ is covariance, $c_1, c_2$ are stabilizing constants.

3. LPIPS (Learned Perceptual Image Patch Similarity):

   • Concept: Uses a pre-trained neural network (like VGG) to measure the perceptual distance between images. Lower is better. It captures high-frequency details that PSNR might miss.

4. FPS (Frames Per Second):

   • Concept: The rendering speed. $>30$ FPS is generally considered real-time.

5. Storage (MB):

   • Concept: The disk space required to save the trained model.

5.3. Baselines

The method is compared against:

• TiNeuVox & HexPlane & K-Planes: State-of-the-art dynamic NeRF variants that use explicit voxel/plane features.
• 3D-GS: The original static Gaussian Splatting (to show why a static model fails).
• Dynamic3DGS & Deformable-3DGS: Other concurrent 4D Gaussian methods (some results cited from their papers).
• D-NeRF / HyperReel / NeRFPlayer: Previous widely used dynamic view synthesis methods.

6. Results & Analysis

6.1. Core Results Analysis

The experiments demonstrate that 4D-GS achieves a superior balance of quality, speed, and storage.

1. Quality: On the Synthetic dataset, 4D-GS achieves the highest PSNR (34.05 dB), surpassing V4D and TiNeuVox. It effectively captures high-frequency details that voxel methods often blur.

2. Speed: It reaches 82 FPS at $800 \times 800$ resolution. This is orders of magnitude faster than TiNeuVox (1.5 FPS) or K-Planes (0.97 FPS). Even compared to the highly optimized static 3D-GS (170 FPS), maintaining 82 FPS for dynamic content is a massive achievement.

3. Storage: The model size is very small (18 MB for synthetic), significantly lower than per-frame approaches or heavy voxel grids (e.g., K-Planes at 418 MB).

   The following figure (Figure 6 from the original paper) visually compares the rendering quality. Notice how 4D-GS (Ours) retains sharp details in the "Lego" scene compared to the blurred artifacts in TiNeuVox:

   该图像是图表，展示了不同方法在动态场景渲染中的效果，包括 Ground Truth、K-Planes、HexPlane、TiNeuVox、3D-GS 和我们的方法（Ours），可用于比较不同算法的性能。

The following figure (Figure 7 from the original paper) shows qualitative results on Real-world datasets, highlighting accurate reconstruction of challenging thin structures like the broom:

该图像是示意图，展示了使用不同方法（GT、HyperNeRF、TiNeuVox、3D-GS和我们的4D-GS方法）渲染动态场景的效果。图中包含了扫帚、香蕉、鸡肉和3D打印机的场景，比较了各方法在运动捕捉和细节保留方面的表现。

6.2. Data Presentation (Tables)

Synthetic Dataset Results

The following are the results from Table 1 of the original paper, comparing performance on the D-NeRF synthetic dataset:

Real-world Dataset Results (HyperNeRF)

The following are the results from Table 2 of the original paper, evaluating on the HyperNeRF vrig dataset ($960 \times 540$):

Real-world Dataset Results (Neu3D)

The following are the results from Table 3 of the original paper, evaluating on the Neu3D dataset ($1352 \times 1014$):

(Note: ‡ indicates FPS tested with fixed-view rendering)

6.3. Ablation Studies & Parameter Analysis

The authors conducted ablation studies (Table 4) to verify each component:

1. w/o HexPlane ($R_l$): Removing the HexPlane encoder and using only a shallow MLP causes a massive drop in PSNR (34.05 $\to$ 27.05 dB). This proves that the 4D spatial-temporal structure is critical for capturing motion.
2. w/o Initialization: Training without the static warm-up phase leads to lower quality (31.91 dB) because the model struggles to distinguish between geometry and motion simultaneously.
3. w/o Deformation Heads ($\phi_x, \phi_r, \phi_s$):

   • Removing Position ($\phi_x$) is catastrophic (26.67 dB), as objects clearly move.

   • Removing Rotation ($\phi_r$) or Scaling ($\phi_s$) has a smaller but noticeable impact (approx. 1 dB drop), showing that accounting for rotation and stretching is important for accurate non-rigid deformation (e.g., body joints).

     The following figure (Figure 12 from the original paper) shows the visual impact of modeling color and opacity changes ($\phi_C, \phi_\alpha$) for fluid-like scenes, improving results over baselines like TiNeuVox:

     

     Rendering Speed Analysis: Figure 9 in the paper analyzes how the number of Gaussians affects FPS.

There is a clear trade-off: fewer Gaussians yield higher FPS. For instance, with fewer than 30,000 Gaussians, the speed can exceed 90 FPS. The method allows balancing quality (more Gaussians) and speed. 该图像是图表，展示了渲染速度与3D Gaussian数量之间的关系。X轴表示每秒帧数（FPS），Y轴表示使用的点数。图中可见一条呈现负斜率的红色线条，表明随着帧数的增加，所需的3D Gaussian数量呈减少趋势，反映了渲染效率与复杂性之间的权衡。所有测试均在合成数据集上完成。

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces 4D Gaussian Splatting, a highly efficient framework for dynamic scene rendering. By combining explicit 3D Gaussians with a 4D HexPlane encoding and a lightweight deformation network, the method solves the efficiency bottleneck of dynamic view synthesis. It achieves real-time rendering (80+ FPS), high visual fidelity, and compact storage, establishing a new state-of-the-art for dynamic scenes.

7.2. Limitations & Future Work

The authors identify several limitations (visualized in Figure 16):

1. Large Motions: The deformation field can struggle with extremely large or sudden movements (e.g., a broom moving rapidly), leading to blurring or tracking failure.

2. Monocular Ambiguity: Without multi-view constraints, separating static background from dynamic foreground is difficult, sometimes causing background artifacts to "move" with the object.

3. Urban Scale: The current method relies on querying the deformation field for every Gaussian, which might be too computationally heavy for massive city-scale reconstructions with millions of points.

   The following figure (Figure 17 from the original paper) illustrates these failure cases:

   该图像是图像16，展示了建模大动作和剧烈场景变化的失败案例。(a) 笤帚的突然运动使优化变得更加困难。(b) 茶壶处于大幅运动中，手部进入和离开场景。

7.3. Personal Insights & Critique

• The Power of Hybrid Representations: 4D-GS is a perfect example of the trend towards "hybrid" neuro-explicit systems. It uses explicit geometry (Gaussians) for speed and neural networks (HexPlane + MLP) for flexibility/compression. This is likely the future of graphics—explicit for rendering, neural for representation.

• Tracking Potential: Unlike NeRFs which are "black boxes," 4D-GS allows for explicit tracking. Since we know exactly where Gaussian $i$ moves at time $t$ (via $\Delta \mathcal{X}$), we can theoretically track object trajectories (as shown in Figure 8 of the paper). This opens doors for applications in editing and physics simulation that were hard with implicit NeRFs.

• The "Canonical" Assumption: The reliance on a single canonical space is both a strength (compactness) and a weakness. It assumes topology doesn't change drastically (e.g., cutting a loaf of bread). If an object splits into two, a single canonical Gaussian cannot easily split into two deformed ones without complex logic. Future work might need dynamic topology handling (spawning/killing Gaussians).

  The following figure (Figure 8 from the original paper) demonstrates the tracking capability mentioned above:

  该图像是图示，展示了两种场景下3D高斯的轨迹。左侧（a）展示了做菠菜的场景，右侧（b）为咖啡马提尼调制的场景。底部的视觉效果呈现了3D高斯的运动轨迹。