1. Bibliographic Information

1.1. Title

GWM: Towards Scalable Gaussian World Models for Robotic Manipulation

1.2. Authors

• Guanxing Lu (1, 2, ⋆)
• Baoxiong Jia (2, ⋆, †)
• Puhao Li (2, ⋆)
• Yixin Chen (2)
• Ziwei Wang (3)
• Yansong Tang (1, †)
• Siyuan Huang (2, †)

Affiliations:

1. Tsinghua University

2. State Key Laboratory of General Artificial Intelligence, BIGAI (Beijing Institute for General Artificial Intelligence)

3. School of Electrical and Electronic Engineering, Nanyang Technological University

   ⋆ Equal contribution, † Corresponding author

1.3. Journal/Conference

ICCV 2025 (International Conference on Computer Vision). Comment: ICCV is a premier computer vision conference, ranked alongside CVPR and ECCV as one of the top venues in the field. Publication here indicates high-quality, impactful research.

1.4. Publication Year

2025

1.5. Abstract

This paper introduces Gaussian World Model (GWM), a novel framework for robotic manipulation. The core problem it addresses is that existing "World Models" (AI models that predict future states of the world) typically rely on 2D images, which lack the 3D geometric understanding crucial for robots to interact with physical objects. GWM solves this by using 3D Gaussian Splatting as the underlying representation. It combines a Diffusion Transformer (DiT) with a 3D Variational Autoencoder (VAE) to predict future 3D scenes based on robot actions. The results show that GWM can serve as a powerful simulator for training robots, improving performance in both imitation learning and reinforcement learning tasks compared to state-of-the-art image-based models.

1.6. Original Source Link

https://openaccess.thecvf.com/content/ICCV2025/papers/Lu_GWM_Towards_Scalable_Gaussian_World_Models_for_Robotic_Manipulation_ICCV_2025_paper.pdf Status: Officially Published

2. Executive Summary

2.1. Background & Motivation

The Core Problem: Robots learn best by interacting with the world, but real-world interaction is slow, expensive, and potentially dangerous. "World Models" allow robots to learn in a simulated "dream" of the world. However, most current world models are image-based (video generation). While they look good, they fail to capture the 3D geometry and physics of the real world (e.g., exact depths, occlusions, object solidity). A robot trained on just video might get confused by lighting changes or camera shifts because it doesn't understand the underlying 3D space.

Importance: For a robot to manipulate objects (pick, place, stack), it needs a precise understanding of 3D space, not just 2D pixels. Existing attempts to use 3D representations (like NeRFs) are often too slow or computationally heavy to run in real-time loops needed for robot learning.

Innovation Point: The paper proposes using 3D Gaussian Splatting (3D-GS) as the core representation for the world model. 3D-GS is a recent breakthrough that allows for both high-quality 3D reconstruction and very fast rendering. The authors combine this with generative AI (Diffusion Transformers) to predict how these 3D Gaussians move and change over time in response to robot actions.

2.2. Main Contributions / Findings

1. GWM Framework: A novel 3D world model architecture that integrates Gaussian Splatting with a Diffusion Transformer (DiT) and a Gaussian VAE. This allows the model to predict future 3D states effectively.
2. Scalable Pipeline: An end-to-end pipeline that can take unposed images (images without camera position data), turn them into 3D Gaussians, compress them into a latent space, and learn dynamics, all without manual human labeling.
3. Superior Performance:
   • Simulation: Outperforms state-of-the-art image-based world models (like iVideoGPT) by a margin of 16.25% in success rates on benchmark tasks.
   • Real-World: In real robot experiments (Franka Emika arm), adding GWM representations improved the success rate of a standard Diffusion Policy by 30%.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• World Model: Think of this as a "mental simulator." Just as humans can close their eyes and imagine what happens if they drop a glass (it falls and breaks), a World Model allows an AI to predict the next state of the environment ($s_{t+1}$) given the current state ($s_t$) and an action ($a_t$). This allows the agent to "practice" in its head.

• Gaussian Splatting (3D-GS): A technique for rendering 3D scenes. Instead of using triangles (meshes) or implicit functions (NeRFs), it represents a scene as a cloud of 3D "blobs" (Gaussians). Each blob has a position, color, opacity, and 3D shape (covariance).

  • Why it matters: It renders extremely fast (real-time) and produces high-quality images, making it ideal for rapid robot learning loops.

• Diffusion Model: A type of generative AI (like Stable Diffusion). It learns to generate data by gradually removing noise from a random signal.

  • Diffusion Transformer (DiT): A specific architecture that uses Transformer blocks (like in ChatGPT) instead of the traditional U-Net to handle the diffusion process.

• Variational Autoencoder (VAE): A neural network used for data compression.

  • Encoder: Compresses complex data (like a cloud of thousands of 3D Gaussians) into a small, compact vector (latent space).
  • Decoder: Reconstructs the original data from that small vector.
  • Why it matters here: Predicting thousands of Gaussians directly is too hard. The model predicts the compressed version (latent) instead.

• Reinforcement Learning (RL) vs. Imitation Learning (IL):

  • IL: The robot watches a human do a task and tries to copy it.
  • RL: The robot tries things randomly, gets a reward for success, and learns from trial and error.

3.2. Previous Works

• Image-Based World Models:

  • Dreamer (Hafner et al.): A seminal work that learns a latent dynamics model from images. It focuses on a compact latent space but often lacks fine-grained visual detail.
  • iVideoGPT / Sora-like models: Recent works use massive video datasets to train transformers that generate video. While visually impressive, they lack "physical common sense" (e.g., objects might morph or vanish unrealistically) because they operate in 2D pixel space.

• NeRF for Robotics:

  • NeRF (Neural Radiance Fields): Represents scenes as a continuous function. While accurate, training and rendering are computationally expensive, often taking seconds or minutes per frame, which is too slow for real-time robot control.

3.3. Differentiation Analysis

• vs. Image Models: GWM uses explicit 3D representation. If the camera moves, the 3D Gaussians ensure the object looks correct from the new angle immediately. Image models have to "hallucinate" the new view, often leading to inconsistencies.
• vs. NeRF/Offline 3D-GS: Traditional 3D methods require training on a specific scene for a long time (offline). GWM is feed-forward and generalizable: it takes an image and instantly predicts the 3D structure and future dynamics without per-scene optimization.

4. Methodology

4.1. Principles

The core idea is to treat the "state" of the world not as a 2D image, but as a collection of 3D Gaussians.

1. Input: Current Image(s).

2. Encode: Convert images $\rightarrow$ 3D Gaussians $\rightarrow$ Compressed Latent Vector.

3. Predict: Use a Diffusion Model to predict the next Latent Vector given the current one + Robot Action.

4. Decode: Convert predicted Latent Vector $\rightarrow$ Future 3D Gaussians $\rightarrow$ Rendered Image (if needed).

   The following figure (Figure 2 from the original paper) illustrates this pipeline, showing the transition from images to Gaussians, compression via VAE, and dynamics modeling via DiT:

   该图像是示意图，展示了高斯世界模型的构建过程，包括未配对图像、Gaussian Splats 和 3D VAE 模块，以及潜在扩散变换器的结构。图中还包含了从动作到高斯分布的变换。关键公式为 $G_{t+1} = f(G_t, a_t)$。

4.2. Core Methodology In-depth

Step 1: Feed-forward 3D Gaussian Splatting (World State Encoding)

The system first needs to understand the 3D scene from 2D images.

• Input: Single or stereo images $\mathcal{T} = \{I\}_{i=\{1,2\}}$.

• Process: The authors use a method called Splatt3R, which builds upon Mast3R.

  1. Point Map Prediction: The model predicts a dense 3D point map from the images.
  2. Gaussian Parameter Prediction: For each point, it predicts the parameters of a 3D Gaussian:
     • Center ($\boldsymbol{x}_p$)
     • Opacity ($\sigma_p$)
     • Covariance ($\Sigma_p$, determining shape/rotation)
     • Color coefficients ($\mathcal{C}_p$, Spherical Harmonics)

• Rendering Formula: To verify the Gaussians are correct, they are projected back to 2D. The color of a pixel C(G) is calculated by blending the overlapping Gaussians: $ C(G) = \sum_{p \in \mathcal{P}} \alpha_p \mathrm{SH}(d_p; \mathcal{C}p) \prod{j=1}^{p-1} (1 - \alpha_j) $

  • Symbol Explanation:
    • $\mathcal{P}$: The set of Gaussians overlapping this pixel.
    • $\alpha_p$: The opacity/weight of the $p$-th Gaussian at this pixel.
    • $\mathrm{SH}(\cdot)$: Spherical Harmonics function (calculates color based on viewing direction $d_p$).
    • $\prod (1-\alpha_j)$: This is the "transmittance." It ensures that if a Gaussian in front is opaque ($\alpha \approx 1$), you can't see the ones behind it.

Step 2: 3D Gaussian VAE (Compression)

A raw scene might have thousands of Gaussians ($G$), which is too much data for a dynamics model to predict efficiently. The authors compress this into a fixed-size latent representation.

• Downsampling: First, reduce the number of Gaussians to a fixed number $N$ using Farthest Point Sampling (FPS): $G_N = \mathrm{FPS}(G)$.

• Encoder ($E_\theta$): Uses Cross-Attention to aggregate information from all raw Gaussians ($G$) into the sampled ones ($G_N$). $ X = E_\theta(G_N, \boldsymbol{G}) = E_\theta^{(L)} \circ \cdots \circ E_\theta^{(1)}(G_N, \boldsymbol{G}) $ $ E_\theta^{(l)}(\boldsymbol{Q}, \boldsymbol{G}) = \mathrm{LayerNorm}(\mathrm{CrossAttn}(\boldsymbol{Q}, \mathrm{PosEmbed}(\boldsymbol{G}))) $

  • Explanation: $G_N$ acts as the "Queries" ($\boldsymbol{Q}$) that ask the full set $G$ for information. The result is a latent embedding $\mathbf{x} \in \mathbb{R}^{N \times D}$.

• Decoder ($D_\theta$): Reconstructs the Gaussians from the latent code using Self-Attention. $ \hat{\pmb{G}} = D_\theta(\mathbf{x}) = \mathrm{LayerNorm}(\mathrm{SelfAttn}(\mathbf{x}, \mathbf{x})) $

• Loss Function: The VAE is trained to minimize the difference between the reconstructed Gaussians $\hat{G}$ and original $G$. $ \mathcal{L}_{\mathrm{VAE}} = \operatorname{Chamfer}(\hat{G}, G) + | C(\hat{G}) - C(G) |_1 $

  • Chamfer Loss: Measures the geometric distance between two point clouds (are the points in the right place?).
  • Rendering Loss: $\| C(\hat{G}) - C(G) \|_1$ checks if the image rendered from the reconstructed Gaussians looks like the original image.

Step 3: Diffusion-based Dynamics Modeling

Now the system learns how the world changes. It predicts the next latent state $\mathbf{x}_{t+1}$ given the current history $\mathbf{x}_{\le t}$ and actions $a_{\le t}$.

• Problem Formulation: This is a conditional generation problem. $p(\mathbf{x}_{t+1} | \mathbf{x}_{\le t}, a_{\le t})$

• Diffusion Process (SDE): The method adds noise to the ground truth future state $\mathbf{x}_{t+1}^0$ to create a noisy version $\mathbf{x}_{t+1}^\tau$ (where $\tau$ is the noise level). $ d\mathbf{x} = \mathbf{f}(\mathbf{x}, \tau)d\tau + g(\tau)d\mathbf{w} $

  • $\mathbf{w}$: Standard Wiener process (random noise).
  • $\mathbf{f}, g$: Drift and diffusion coefficients (control how noise is added).

• Reverse Process (Generation): To generate the future, the model learns to reverse this noise process (denoising). $ d\mathbf{x} = [\mathbf{f}(\mathbf{x}, \tau) - g(\tau)^2 \nabla_{\mathbf{x}} \log p^\tau(\mathbf{x})]d\tau + g(\tau)d\bar{\mathbf{w}} $

  • $\nabla_{\mathbf{x}} \log p^\tau(\mathbf{x})$: The Score Function. This is what the neural network learns to estimate. It points in the direction of "less noise/more data."

• Network Parameterization (EDM): Instead of predicting the noise directly, they use the EDM (Elucidating the Design Space of Diffusion Models) formulation. They learn a network $\mathcal{F}_\theta$ (the DiT) to clean the noisy input. $ \mathcal{D}\theta(\mathbf{x}{t+1}^\tau, \mathbf{y}t^\tau) = c{\mathrm{skip}}^\tau \mathbf{x}{t+1}^\tau + c{\mathrm{out}}^\tau \mathcal{F}\theta(c{\mathrm{in}}^\tau \mathbf{x}_{t+1}^\tau, \mathbf{y}t^\tau; c{\mathrm{noise}}^\tau) $

  • Symbol Explanation:
    • $\mathbf{x}_{t+1}^\tau$: Noisy input state.
    • $\mathbf{y}_t^\tau$: Conditions (history states $\mathbf{x}_{\le t}$, actions $a_{\le t}$).
    • $c_{\dots}^\tau$: Preconditioning scalars that stabilize training by scaling inputs/outputs based on noise level $\tau$.
    • $\mathcal{F}_\theta$: The core Diffusion Transformer network.

• Training Objective: The model is trained to minimize the error between the denoised prediction and the ground truth future $\mathbf{x}_{t+1}^0$. $ \mathcal{L}(\theta) = \mathbb{E} \left[ \left| \mathbf{F}\theta(c{\mathrm{in}}^\tau \mathbf{x}{t+1}^\tau, y_t^\tau) - \frac{1}{c{\mathrm{out}}^\tau} (\mathbf{x}{t+1}^0 - c{\mathrm{skip}}^\tau \mathbf{x}_{t+1}^\tau) \right|_2^2 \right] $

Implementation Details

• Architecture: Diffusion Transformer (DiT).
• Conditioning:
  • Time embeddings are handled by AdaLN (Adaptive Layer Normalization).
  • Robot actions are injected as Keys and Values in the Cross-Attention layers.
• Normalization: RMSNorm is used for stability.

5. Experimental Setup

5.1. Datasets

The paper evaluates GWM on 3 distinct domains, ranging from simulation to the real world.

1. META-WORLD:

   • Type: Synthetic (Simulation).
   • Description: A benchmark for RL policies involving robotic manipulation tasks (e.g., pushing, picking).
   • Tasks: 6 tasks with increasing complexity.

2. ROBOCASA:

   • Type: Synthetic (Simulation).
   • Description: A large-scale imitation learning benchmark set in kitchen environments.
   • Scale: 24 atomic tasks (e.g., Pick & Place, Open Microwave).
   • Data: 50 human demonstrations per task + 3000 generated demonstrations (MimicGen).

3. FRANKA-PNP (Real World):

   • Type: Real World.

   • Description: A physical Franka Emika FR3 robot arm performing a Pick-and-Place (PnP) task.

   • Data: 30 demonstrations collected via teleoperation.

   • Setup: RGB-only camera (Realsense D435i), unposed (camera location not strictly calibrated).

     The following figure (Figure 6 from the original paper) shows the real-world setup:

     该图像是实验设置的示意图，展示使用Franka Emika Panda机器人臂及RGB相机进行的实时实验。左侧显示了机器人及其工具，右侧展示了机器人在任务完成中的视觉输入。

5.2. Evaluation Metrics

1. FVD (Fréchet Video Distance):

   • Concept: Measures how similar the distribution of generated videos is to real videos. Lower is better. It captures temporal consistency and visual quality.
   • Formula: $ \mathrm{FVD} = |\mu_r - \mu_g|^2 + \mathrm{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}) $
   • Symbols: $\mu_r, \Sigma_r$ are mean/covariance of real videos features (extracted by I3D network); $\mu_g, \Sigma_g$ are for generated videos.

2. PSNR (Peak Signal-to-Noise Ratio):

   • Concept: Pixel-level accuracy measure. Higher is better.
   • Formula: $ \mathrm{PSNR} = 10 \cdot \log_{10}\left(\frac{\mathrm{MAX}_I^2}{\mathrm{MSE}}\right) $
   • Symbols: $\mathrm{MAX}_I$ is max pixel value (usually 255); $\mathrm{MSE}$ is Mean Squared Error between predicted and ground truth images.

3. SSIM (Structural Similarity Index):

   • Concept: Perceptual metric that measures structural similarity (luminance, contrast, structure) rather than just pixel difference. Closer to 1 (or 100%) is better.

4. LPIPS (Learned Perceptual Image Patch Similarity):

   • Concept: Uses a deep neural network (like VGG) to measure how "perceptually" different two images are. It aligns better with human judgment than PSNR. Lower is better.

5. Success Rate (SR):

   • Concept: The percentage of times the robot successfully completes the assigned task (e.g., successfully placing the cup on the plate).

5.3. Baselines

• iVideoGPT: The primary state-of-the-art baseline. It is an image-based world model that uses a GPT architecture (similar to Sora or other video generators) to predict future video frames. Comparing against this validates the hypothesis that "3D > 2D".
• BC-Transformer: A standard Behavior Cloning policy used in Imitation Learning.
• Diffusion Policy: A state-of-the-art policy network for robot control.

6. Results & Analysis

6.1. Core Results Analysis

1. Action-Conditioned Scene Prediction

The first question is: Does GWM generate better future predictions than image-based models?

Quantitative Analysis: The following are the results from Table 1 of the original paper:

Interpretation:

• Lower FVD: GWM videos are much more temporally consistent and realistic (98.3 vs 125.4).

• Higher PSNR/SSIM: The visual quality is sharper.

• Qualitative Difference: Figure 3 in the paper shows that iVideoGPT often blurs or distorts small, critical details like the gripper fingers. GWM maintains the geometric integrity of the gripper because it understands it as a 3D object.

  The following figure (Figure 3 from the original paper) highlights this difference in detail preservation:

  该图像是表格与图像的组合展示，展示了GWM与iVideoGPT在MeTAWoRLD数据集上的对比。GWM在抓手运动的预测上表现更佳，细节突出，右下角的蓝框进一步强调了这一点。

2. GWM for Imitation Learning

Does this better world model help a robot learn from demonstrations? The authors used the GWM encoder as a "visual backbone" for a BC-Transformer policy on ROBOCASA.

Quantitative Analysis: The following are the results from Table 2 of the original paper:

Interpretation:

• +10.5% Gain: On limited human data (Human-50), GWM significantly boosts performance (60.9% vs 50.4%). This suggests the 3D representation is highly data-efficient.
• Scalability: Even with large generated data (G-3000), GWM maintains a lead (+7.6%).

3. GWM for Reinforcement Learning (RL)

Can GWM act as a simulator for an RL agent to learn via trial-and-error? They compared GWM-based RL against iVideoGPT-based RL on Meta-World tasks.

The following figure (Figure 5 from the original paper) shows the learning curves:

Interpretation:

• Faster Convergence: GWM (Blue line) learns much faster than iVideoGPT (Orange line).
• Higher Asymptotic Performance: In complex tasks like Bin Picking or Hammer, GWM achieves near 100% success where iVideoGPT struggles. This proves that accurate physical dynamics (collisions, depth) are crucial for RL.

6.2. Real-World Deployment

They tested if GWM helps in the real world, specifically for handling "distractors" (objects on the table that shouldn't be touched).

The following are the results from Table 3 of the original paper:

Analysis:

• Robustness: GWM nearly doubles the total success rate (65% vs 35%).
• Why? The "Table distractor" case is telling (0/5 vs 3/5). The baseline policy likely got confused by the visual clutter of extra objects. GWM's 3D representation likely helped it spatially separate the target object from the distractor, leading to robust picking.

6.3. Ablation Studies

What parts of GWM are essential? They tested removing the Gaussian Splatting (using just 2D DiT) and removing the VAE.

The following are the results from Table 4 of the original paper:

Note: This specific ablation was on a hard task "PnP CabToCounter".

Conclusion:

1. 3D is critical: Just adding Gaussian Splatting jumped SR from 4% to 18%. Pure image models failed almost completely on this hard task.
2. Compression helps: Adding the VAE improved SR to 24%. Compressing the scene makes the dynamics learning easier and more stable.

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper presents GWM, a world model built on 3D Gaussian Splatting. By moving from 2D pixels to 3D Gaussian primitives, the model gains a fundamental understanding of geometry and spatial relationships. When combined with a Diffusion Transformer and a VAE, it creates a scalable, high-fidelity simulator. Experiments prove that this 3D-centric approach significantly outperforms traditional video-based world models in video prediction, imitation learning, and reinforcement learning, both in simulation and on real robots.

7.2. Limitations & Future Work

• Current Limitations:
  • Computational Cost: While faster than NeRF, processing 3D Gaussians and running a Diffusion Transformer is still heavy compared to simple CNN policies.
  • Texture/Lighting: While geometry is better, extremely complex lighting effects (reflections, transparencies) might still be challenging compared to pure image generative models trained on internet-scale data.
• Future Work:
  • Scaling up the dataset size (similar to how GPT/Sora are scaled).
  • Integrating language instructions more deeply into the world model generation.

7.3. Personal Insights & Critique

• The "3D Prior" is Key: This paper strongly validates the hypothesis that for robotics, geometry matters more than texture. Video generation models (like Sora) are getting incredibly good at making pretty videos, but they hallucinate physics. GWM enforces a "3D consistency constraint" by design. This is the right direction for embodied AI.
• Latent Dynamics: The use of a VAE to compress the Gaussians is a smart engineering move. Predicting thousands of raw Gaussian parameters directly would likely be unstable. The latent space provides a "concepts" layer (e.g., "cup is moving") rather than a "pixels" layer.
• Potential for Sim-to-Real: The most exciting implication is the potential for Zero-Shot Sim-to-Real. If we can scan a real room into Gaussians, we can run the GWM to simulate thousands of trials "in the robot's head" to learn a task without moving an inch, and then execute it perfectly. This paper takes a significant step toward that future.