1. Bibliographic Information

• Title: DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
• Authors: Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, Zian Wang.
• Affiliations: The authors are affiliated with NVIDIA, University of Toronto, Vector Institute, and the University of Illinois Urbana-Champaign. This indicates a strong collaboration between a leading industry research lab and top academic institutions in AI and computer graphics.
• Journal/Conference: This paper was submitted to arXiv, a preprint server. The ID 2501.18590 suggests it was submitted in January 2025. Preprints are not yet peer-reviewed but are a common way to disseminate cutting-edge research quickly.
• Publication Year: 2025 (based on arXiv ID).
• Abstract: The paper introduces DiffusionRenderer, a unified neural framework to address both inverse rendering (estimating scene properties like geometry and materials from videos) and forward rendering (generating photorealistic images from these properties). The approach leverages powerful video diffusion models. The inverse rendering model is trained on synthetic data but generalizes well to real videos, enabling it to auto-label a large corpus of real-world data. This combination of synthetic and auto-labeled real data is then used to train the forward rendering model. The system outperforms state-of-the-art methods and enables applications like relighting, material editing, and object insertion from a single input video.
• Original Source Link: https://arxiv.org/abs/2501.18590

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: Traditional computer graphics, specifically Physically-Based Rendering (PBR), can create incredibly realistic images. However, it requires a perfect digital representation of a scene: precise 3D geometry (meshes), high-quality material properties (like roughness and color), and accurate lighting information. Obtaining this information for real-world scenes—a process called inverse rendering—is extremely difficult and often impractical.
  • Gap in Prior Work: Classic rendering pipelines fail when inputs are imperfect (e.g., noisy geometry estimated from images). Existing neural rendering methods often specialize in one domain (like portraits or outdoor scenes) or require complex multi-view data. There was no single, holistic framework that could both robustly estimate scene properties from a simple video and then use those (potentially imperfect) properties to render new, photorealistic images under novel conditions.
  • Fresh Angle: Instead of trying to perfect the inverse rendering process to feed a traditional PBR pipeline, this paper treats both inverse and forward rendering as learning problems. It proposes to use the powerful generative capabilities of video diffusion models to "learn" the physics of light transport implicitly from data. This approach is more tolerant to imperfect inputs and can generate realistic effects like soft shadows and complex reflections without explicit simulation.

• Main Contributions / Findings (What):

  1. A State-of-the-Art Inverse Renderer for Videos: The paper develops a video diffusion model that accurately estimates intrinsic scene properties (geometry, materials) from a single input video. Critically, though trained on synthetic data, it generalizes effectively to real-world videos.
  2. A Novel Neural Forward Renderer: A second video diffusion model is repurposed as a "neural rendering engine." It takes decomposed scene properties (known as G-buffers) and a lighting description (an environment map) as input and synthesizes photorealistic videos, complete with complex lighting effects.
  3. A Unified Framework for Practical Editing: By combining the inverse and forward renderers, DiffusionRenderer provides an end-to-end pipeline for advanced image and video editing. From just one video, a user can relight the scene, change the material of objects, or seamlessly insert new virtual objects.
  4. A Strategic Data Curation and Training Pipeline: The authors overcome the scarcity of paired real-world data by first training their inverse renderer on a large, custom-generated synthetic dataset. They then use this model to "auto-label" thousands of real-world videos, creating a diverse, large-scale dataset to train their forward renderer for photorealism.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Physically-Based Rendering (PBR): A computer graphics approach that aims to simulate the flow of light in a scene according to the principles of physics. It models how light interacts with surfaces based on their material properties to produce photorealistic images. The core of PBR is the rendering equation.
  • Rendering Equation: A fundamental equation in computer graphics that describes the equilibrium of light energy in a scene. It states that the outgoing light (radiance) from a point on a surface is the sum of the light it emits on its own plus all the light it reflects from other sources. The paper presents a simplified form: $$L _ { o } ( \mathbf { p } , \omega _ { o } ) = \int _ { \Omega } f _ { r } ( \mathbf { p } , \omega _ { o } , \omega _ { i } ) L _ { i } ( \mathbf { p } , \omega _ { i } ) \left| \mathbf { n } \cdot \omega _ { i } \right| d \omega _ { i }$$
    • $L_o$: Outgoing radiance (light) from point $\mathbf{p}$ in direction $\omega_o$.
    • $L_i$: Incoming radiance to point $\mathbf{p}$ from direction $\omega_i$.
    • $f_r$: The Bidirectional Reflectance Distribution Function (BRDF), which defines how a surface reflects light based on material properties.
    • $\mathbf{n}$: The surface normal vector at point $\mathbf{p}$.
    • The integral is taken over the hemisphere $\Omega$ of all possible incoming directions.
  • Inverse Rendering: The inverse problem of rendering. Instead of generating an image from scene properties, it aims to estimate those properties (geometry, materials, lighting) from one or more input images.
  • G-buffers (Geometry Buffers): A set of 2D images that store per-pixel information about a 3D scene from a specific viewpoint. Instead of just RGB color, each pixel stores attributes like its 3D position, surface normal, base color (albedo), roughness, and metallic properties. This is a common technique in deferred shading.
  • Diffusion Models: A class of generative models that learn to create data by reversing a process of gradually adding noise. They start with random noise and iteratively "denoise" it, guided by a learned model, until a clean sample (like an image or video) is produced. Video Diffusion Models (VDMs) extend this concept to generate coherent sequences of frames. DiffusionRenderer is built on the Stable Video Diffusion (SVD) model.
  • Neural Rendering: A field that uses neural networks to synthesize images, often blending traditional graphics pipelines with deep learning. DiffusionRenderer is a prime example, replacing the explicit light transport simulation of PBR with a learned diffusion process.

• Previous Works & Differentiation:

  • Classic Inverse Rendering: Early methods used optimization with hand-crafted priors, which often failed on complex scenes. More recent learning-based methods are data-hungry and often domain-specific. DiffusionRenderer leverages the powerful priors of large diffusion models, leading to higher quality and better generalization.

  • NeRF and 3D Gaussian Splatting: These methods create high-quality 3D scene representations for novel view synthesis. However, they typically "bake" lighting and materials together, making editing tasks like relighting very difficult. DiffusionRenderer explicitly separates geometry, materials, and lighting into editable G-buffers, enabling more flexible editing.

  • Image-based Neural Rendering/Relighting: Methods like RGB↔X and DiLightNet have explored using image diffusion models for similar tasks. DiffusionRenderer extends this to video, which provides more information for disambiguating material properties (e.g., specular reflections are view-dependent) and ensures temporal consistency. It also introduces a more sophisticated lighting conditioning mechanism.

  • DiffusionRenderer's key innovation is the holistic framework that jointly considers inverse and forward rendering, powered by a strategic synthetic-to-real data pipeline. This allows it to handle imperfect, real-world inputs far more robustly than traditional PBR methods, as shown in Figure 2.

    Figure 2: This figure illustrates the core motivation. Top row: A classic method like Screen Space Ray Tracing (SSRT) struggles to render correct shadows without full 3D geometry. Bottom row: When fed with imperfect G-buffers estimated from a real image, SSRT produces poor results. DiffusionRenderer handles these imperfections gracefully, producing a high-quality relit image.

4. Methodology (Core Technology & Implementation)

DiffusionRenderer consists of two main components built on video diffusion models: a Neural Forward Renderer and a Neural Inverse Renderer.

该图像是一个示意图，展示了DiffusionRenderer框架中逆向渲染和前向渲染的流程。图中从输入视频开始，经过逆向渲染器（Inverse Renderer）估计G-buffer，然后结合环境光照编码（Environment Lighting）输入到前向渲染器（Forward Renderer），最终生成输出视频。图中还标注了各部分是否可优化（火焰图标）或冻结（雪花图标）参数。 Figure 3: This diagram shows the overall framework. An Input Video is fed into the Inverse Renderer (left), which uses a Diffusion UNet conditioned on the video and learnable Domain Embeddings to predict G-buffers (Base Color, Normals, etc.). These G-buffers, along with Environment Lighting information, are then fed into the Forward Renderer (right). The forward model uses another Diffusion UNet to generate the final Output Video. The process involves frozen (snowflake icon) and optimizable (fire icon) components, including a LoRA module for real-world data.

4.1. Neural Forward Rendering

This model acts as a learned replacement for a traditional PBR engine. It generates a photorealistic video from G-buffers and lighting conditions.

• Inputs (Conditions):

  1. Geometry and Material Buffers (G-buffers): These are per-pixel attribute maps for each frame of the video.
     • Surface Normals ($\mathbf{n}$): 3-channel map indicating surface orientation.
     • Relative Depth ($\mathbf{d}$): 1-channel map of depth values.
     • Base Color ($\mathbf{a}$): 3-channel map of the surface's underlying color (albedo).
     • Roughness ($\mathbf{r}$): 1-channel map indicating how rough or smooth a surface is.
     • Metallic ($\mathbf{m}$): 1-channel map indicating if a surface is metallic or dielectric.
  2. Lighting Conditions: Represented by a panoramic environment map ($\mathbf{E}$), which captures light from all directions. To handle High Dynamic Range (HDR) lighting with a standard Low Dynamic Range (LDR) model, it's encoded into three separate maps:
     • $\mathbf{E}_{\mathrm{ldr}}$: A tonemapped LDR version.
     • $\mathbf{E}_{\mathrm{log}}$: A log-transformed version to better represent high-intensity light sources.
     • $\mathbf{E}_{\mathrm{dir}}$: A map where each pixel stores a unit vector for its direction in camera space.

• Model Architecture:

  • The model is based on Stable Video Diffusion (SVD), which uses a VAE encoder ($\mathcal{E}$) and decoder ($\mathcal{D}$) and a UNet-based denoising network.
  • G-buffer Conditioning: Each G-buffer map is independently encoded by the VAE encoder $\mathcal{E}$ into a latent representation. These latents are then concatenated channel-wise and fed as input to the UNet, alongside the noisy latent of the target image. This provides strong, pixel-aligned spatial guidance.
  • Lighting Conditioning: Instead of simple concatenation, lighting information is injected via cross-attention layers. The three environment map encodings are first passed through the VAE encoder $\mathcal{E}$, then a dedicated environment map encoder ($\mathcal{E}_{\mathrm{env}}$). This encoder extracts multi-resolution feature maps. The UNet's cross-attention layers can then "query" these feature maps at different spatial scales to learn how to apply lighting effects (like shadows and reflections) to the scene.

4.2. Neural Inverse Rendering

This model performs the opposite task: it estimates the G-buffers from an input RGB video.

• Input (Condition): An RGB video $\mathbf{I}$.
• Output: The five G-buffer maps: $\{\mathbf{n}, \mathbf{d}, \mathbf{a}, \mathbf{r}, \mathbf{m}\}$.
• Model Architecture:
  • Also based on SVD, it takes the VAE-encoded input video latent $\mathbf{z} = \mathcal{E}(\mathbf{I})$ as a condition.
  • To generate five different types of maps with a single model, it uses domain embeddings. A unique, learnable embedding vector $\mathbf{c}_{\mathrm{emb}}^P$ is created for each property $P$ (e.g., 'normals', 'albedo').
  • During inference, the model runs five separate passes. In each pass, it denoises a random noise map conditioned on both the input video latent $\mathbf{z}$ and the specific domain embedding $\mathbf{c}_{\mathrm{emb}}^P$ for the desired G-buffer. This tells the model which property to generate.

4.3. Data Strategy

A robust data strategy is critical for the success of DiffusionRenderer.

• Synthetic Data Curation: The authors created a large-scale synthetic dataset because real-world data with ground-truth G-buffers and lighting is scarce.
  • Assets: They curated over 36,000 3D objects, 4,260 PBR materials, and 766 HDR environment maps from public resources.
  • Scene Generation: Scenes were procedurally generated by placing objects and primitives on a textured plane, illuminated by a random HDR map.
  • Rendering: A custom path tracer was used to render 150,000 videos at 512x512 resolution, each with 24 frames, along with all the corresponding ground-truth G-buffers and lighting.
• Real-World Auto-Labeling: Training the forward renderer only on synthetic data would make it produce images with a "synthetic look." To bridge this domain gap, they use their own inverse renderer (trained on synthetic data) to generate pseudo-ground-truth G-buffers for a large real-world video dataset (DL3DV10k). An off-the-shelf method (DiffusionLight) is used to estimate the lighting. This creates a massive dataset of ~150,000 real-world video clips with approximate labels.

4.4. Training Pipeline

• Neural Inverse Renderer: This model is trained first. It's co-trained on the new synthetic video dataset and existing public image datasets (InteriorVerse, HyperSim). The model learns to predict a specific G-buffer latent $\mathbf{g}_0^P$ from a noisy version $\mathbf{g}_\tau^P$, conditioned on the input video latent $\mathbf{z}$ and the corresponding domain embedding $\mathbf{c}_{\mathrm{emb}}^P$. The objective function is a standard denoising score matching loss: $$\mathcal { L } ( \boldsymbol { \theta } , \mathbf { c } _ { \mathrm { e m b } } ) = \| \mathbf { f } _ { \boldsymbol { \theta } } \left( \mathbf { g } _ { \tau } ^ { P } ; \mathbf { z } , \mathbf { c } _ { \mathrm { e m b } } ^ { P } , \tau \right) - \mathbf { g } _ { 0 } ^ { P } \| _ { 2 } ^ { 2 }$$
• Neural Forward Renderer: This model is trained on a combination of the synthetic data and the auto-labeled real-world data. To handle potential inaccuracies in the auto-labeled data, a Low-Rank Adaptation (LoRA) module is added. LoRA introduces a small set of extra trainable parameters ($\Delta\theta$) that are only active when training on real data. This allows the model to adapt to the real-world domain without catastrophically forgetting what it learned from the clean synthetic data. The combined loss is: $$\mathcal { L } ( \boldsymbol { \theta } , \Delta \boldsymbol { \theta } ) = \parallel \mathbf { f } _ { \boldsymbol { \theta } } \left( \mathbf { z } _ { \tau } ^ { \mathrm { s y n t h } } ; \mathbf { g } ^ { \mathrm { s y n t h } } , \mathbf { c } _ { \mathrm { e n v } } ^ { \mathrm { s y n t h } } , \tau \right) - \mathbf { z } _ { 0 } ^ { \mathrm { s y n t h } } \parallel _ { 2 } ^ { 2 } + \parallel \mathbf { f } _ { \boldsymbol { \theta } + \Delta \boldsymbol { \theta } } \left( \mathbf { z } _ { \tau } ^ { \mathrm { r e a l } } ; \mathbf { g } ^ { \mathrm { r e a l } } , \mathbf { c } _ { \mathrm { e n v } } ^ { \mathrm { r e a l } } , \tau \right) - \mathbf { z } _ { 0 } ^ { \mathrm { r e a l } } \parallel _ { 2 } ^ { 2 }$$

4.5. Editing Applications

The two models form a powerful editing pipeline:

1. Decomposition: An input video $\mathbf{I}$ is fed into the inverse renderer to get its G-buffers $\{\hat{\mathbf{n}}, \hat{\mathbf{d}}, \hat{\mathbf{a}}, \hat{\mathbf{r}}, \hat{\mathbf{m}}\}$.

2. Editing: A user modifies the G-buffers (e.g., changing the base color $\hat{\mathbf{a}}$ for material editing) or provides a new target environment map $\mathbf{E}_{\mathrm{tgt}}$ (for relighting).

3. Recomposition: The edited G-buffers and/or new lighting are fed into the forward renderer to synthesize the final edited video $\hat{\mathbf{I}}_{\mathrm{tgt}}$.

   Figure 1: This figure showcases the end-to-end capability. The leftmost images are real-world reference photos. The middle column shows the G-buffers estimated by the inverse renderer. The right columns show the results of the forward renderer under different lighting conditions, demonstrating photorealistic relighting with accurate reflections and shadows.

5. Experimental Setup

• Datasets:

  • Training: A large custom synthetic dataset (150k videos), auto-labeled DL3DV10k (150k clips), and public image datasets InteriorVerse and HyperSim.
  • Evaluation:
    • SyntheticObjects: A test set of 30 individual objects with rotating lights, designed for object-centric evaluation.
    • SyntheticScenes: A more complex test set of 40 scenes with multiple objects, camera motion, and complex lighting effects like inter-reflections.
    • InteriorVerse: A public benchmark for indoor scene albedo estimation.
    • DL3DV10k: Used for qualitative evaluation on diverse real-world videos.

• Evaluation Metrics:

  1. PSNR (Peak Signal-to-Noise Ratio):
     • Conceptual Definition: Measures the ratio between the maximum possible power of a signal and the power of corrupting noise that affects its fidelity. Higher PSNR generally indicates better reconstruction quality. It is sensitive to pixel-wise errors.
     • Mathematical Formula: $$\mathrm{PSNR} = 10 \cdot \log_{10} \left( \frac{\mathrm{MAX}_I^2}{\mathrm{MSE}} \right)$$
     • Symbol Explanation: $\mathrm{MAX}_I$ is the maximum possible pixel value of the image (e.g., 255 for 8-bit images). $\mathrm{MSE}$ is the Mean Squared Error between the ground truth and predicted images.
  2. SSIM (Structural Similarity Index Measure):
     • Conceptual Definition: Measures the similarity between two images based on luminance, contrast, and structure. It is designed to be more consistent with human perception of image quality than PSNR. A value of 1 indicates perfect similarity.
     • Mathematical Formula: $$\mathrm{SSIM}(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$$
     • Symbol Explanation: $\mu_x, \mu_y$ are the average pixel values; $\sigma_x^2, \sigma_y^2$ are the variances; $\sigma_{xy}$ is the covariance of images $x$ and $y$. $c_1, c_2$ are small constants to stabilize the division.
  3. LPIPS (Learned Perceptual Image Patch Similarity):
     • Conceptual Definition: Measures the perceptual similarity between two images using features extracted from a deep neural network (like VGG). Lower LPIPS values indicate that two images are more perceptually similar. It is often better at capturing texture and style similarities than PSNR or SSIM.
     • Mathematical Formula: There isn't a single simple formula, as it involves computing the L2 distance between deep feature activations.
  4. RMSE (Root Mean Square Error):
     • Conceptual Definition: Measures the standard deviation of the prediction errors. Used for evaluating single-channel maps like roughness and metallic. Lower is better.
  5. Mean Angular Error:
     • Conceptual Definition: Measures the average angle between the predicted and ground-truth surface normal vectors. Used for evaluating normal maps. Lower is better.

• Baselines:

  • Forward Rendering: Classic methods (SplitSum, SSRT) and recent neural methods (RGB↔X, DiLightNet).
  • Inverse Rendering: RGB↔X, Kocsis et al. [36], and older methods.
  • Relighting: DiLightNet, Neural Gaffer, and 3D reconstruction-based methods like FEGR and UrbanIR.

6. Results & Analysis

6.1. Forward Rendering Evaluation

The following is a transcription of Table 1 from the paper, which evaluates neural forward rendering (generating an image from ground-truth G-buffers).

Table 1: Quantitative evaluation of neural rendering.

• Core Results: DiffusionRenderer (Ours) significantly outperforms other neural methods (RGB↔X, DiLightNet), especially on the complex SyntheticScenes dataset. It performs comparably to classic PBR methods like SSRT on the simple SyntheticObjects dataset but surpasses them on SyntheticScenes, demonstrating its ability to handle complex lighting like inter-reflections better.

• Ablations:

  • Video vs. Image (Ours (image)): The full video model consistently outperforms the per-frame image model, showing the benefit of leveraging temporal information.

  • Environment Encoder (Ours (w/o env. encoder)): Using a dedicated environment map encoder with cross-attention improves performance over simply concatenating the lighting information.

  • Shading Condition: Adding pre-computed shading buffers (SplitSum) as an extra condition did not provide significant benefits, justifying the design choice to exclude it for simplicity.

    Figure 4: This qualitative comparison shows that DiffusionRenderer (Ours) produces renderings that are much closer to the path-traced ground truth than other methods, especially in capturing subtle reflections and material appearances.

6.2. Inverse Rendering Evaluation

The following are transcriptions of Table 3 and Table 4, evaluating the estimation of G-buffers from an input video/image.

Table 3: Quantitative evaluation of inverse rendering on SyntheticScenes video dataset.

Table 4: Quantitative benchmark of albedo estimation on InteriorVerse dataset [91].

• Core Results: The method massively outperforms RGB↔X across all metrics on video data (Table 3) and sets a new state-of-the-art on the InteriorVerse image benchmark (Table 4). This demonstrates the effectiveness of the model architecture and the curated training data.
• Ablations:

  • Video vs. Image (Ours (image)): The video model significantly improves the estimation of specular properties (metallic and roughness), as motion helps disambiguate view-dependent effects.

  • Deterministic Inference (det.): Fine-tuning for a 1-step deterministic output leads to better scores on error-based metrics like PSNR and RMSE, but slightly worse on perceptual metrics like LPIPS, as it can produce blurrier results in ambiguous regions.

    Figure 5: This qualitative comparison on real-world videos shows DiffusionRenderer producing cleaner and more accurate material maps (especially roughness and metallic) compared to RGB↔X, highlighting its superior generalization.

6.3. Relighting Evaluation

The following is a transcription of Table 2 from the paper, which evaluates the end-to-end relighting pipeline.

Table 2: Quantitative evaluation of relighting.

• Core Results: DiffusionRenderer outperforms both DiLightNet and Neural Gaffer in relighting, again showing a particularly strong advantage on the more complex SyntheticScenes dataset. This confirms the quality of both its inverse and forward rendering components.

  Figure 6: A qualitative relighting comparison. DiffusionRenderer produces more accurate specular reflections and shadows that align better with the ground truth (GT Relit) compared to the baselines.

• Ablation on Training Data:

  Figure 7: This crucial ablation shows that training only on synthetic data (Ours Synth.) fails to handle complex real-world structures like trees. Adding the auto-labeled real-world data (+ Real) improves this, and adding the LoRA module (+ Real + LoRA) further refines the quality, confirming the effectiveness of the entire data strategy.

6.4. Applications

The paper demonstrates practical applications that stem from its unified framework.

Figure 8: This figure showcases material editing (top) and object insertion (bottom). The edits are photorealistic, with inserted objects casting correct shadows and receiving plausible reflections from the environment, which is a very challenging task.

7. Conclusion & Reflections

• Conclusion Summary: DiffusionRenderer presents a significant step forward by creating a unified, data-driven framework for both inverse and forward rendering using video diffusion models. It achieves state-of-the-art results by combining a powerful model architecture with a clever data strategy that leverages synthetic data to bootstrap the labeling of real-world videos. This enables high-quality, practical editing applications like relighting and object insertion from a single video, without needing precise 3D geometry.

• Limitations & Future Work:

  • Inference Speed: The model is based on SVD and is not real-time. Distillation techniques could be used to create faster versions.
  • Content Consistency: While generally good, the generative nature of the model can introduce minor, unintended changes to color or texture in non-edited regions of an image.
  • Lighting Estimation: The quality of the auto-labeled real-world data depends on an off-the-shelf lighting estimation model, which could be a bottleneck. Improving this component would further enhance the forward renderer's quality.

• Personal Insights & Critique:

  • Strength: The paper's most impressive contribution is the holistic problem formulation and the pragmatic data pipeline. The idea of using a synthetically-trained model to create a large-scale, pseudo-labeled real-world dataset is powerful and can be applied to many other domains where paired real data is scarce.
  • Dependence on Pre-trained Models: The success of DiffusionRenderer is heavily reliant on the quality of the underlying pre-trained video diffusion model (SVD). While this is a smart way to leverage existing powerful priors, it also means the performance is tied to the progress of these foundational models.
  • Scalability and Complexity: The approach requires training multiple large diffusion models and a complex data generation pipeline, which demands significant computational resources.
  • Future Impact: This work pushes the boundary of what's possible in "in-the-wild" scene editing. By moving away from rigid, physically-based pipelines and toward more flexible, data-driven generative models, DiffusionRenderer points to a future where photorealistic content creation and editing become much more accessible, requiring only a simple video as input. It effectively democratizes capabilities previously exclusive to VFX studios with complex capture setups.