1. Bibliographic Information

1.1. Title

Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridging

1.2. Authors

The authors of this paper are Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, and Xiaoguang Han.

Their affiliations are:

• Chongjie Ye, Ziteng Lu, Jiahao Chang, Xiaoguang Han: The Chinese University of Hong Kong, Shenzhen
• Yushuang Wu, Xiaoyang Guo, Jiaqing Zhou: ByteDance
• Hao Zhao: Tsinghua University

1.3. Journal/Conference

This paper is published at (UTC): 2025-03-28T08:39:20.000Z, and is currently available as a preprint on arXiv. arXiv is a widely recognized open-access repository for preprints of scientific papers in various fields, including computer science. While preprints have not yet undergone formal peer review, they allow researchers to share their work rapidly and receive feedback. The quality of papers on arXiv can vary, but many significant breakthroughs first appear here. The fact that it is slated for a 2025 publication suggests it might be under review or accepted at a prominent conference or journal.

1.4. Publication Year

2025

1.5. Abstract

The paper addresses the challenge of generating high-fidelity 3D models from 2D images, a task currently limited by domain gaps and ambiguities inherent in RGB images. The authors propose Hi3DGen, a novel framework that uses normal maps as an intermediate representation to bridge the gap. Hi3DGen comprises three core components: (1) an image-to-normal estimator (NiRNE) that separates low- and high-frequency image patterns using noise injection and dual-stream training for stable, sharp, and generalizable normal estimation; (2) a normal-to-geometry learning approach (NoRLD) that employs normal-regularized latent diffusion learning to enhance 3D geometry generation fidelity; and (3) a 3D data synthesis pipeline that creates the high-quality DetailVerse dataset for training. Extensive experiments show Hi3DGen's effectiveness and superiority in generating rich geometric details, outperforming state-of-the-art methods in fidelity. The work highlights normal maps as a promising intermediate representation for high-fidelity 3D geometry generation.

1.6. Original Source Link

https://arxiv.org/abs/2503.22236 PDF Link: https://arxiv.org/pdf/2503.22236v2.pdf This paper is currently a preprint on arXiv.

2. Executive Summary

2.1. Background & Motivation

The paper tackles the problem of generating high-fidelity 3D models from 2D images. This task is crucial for various applications in computer graphics, virtual reality, and industrial design, where the demand for realistic and detailed 3D assets is constantly growing.

However, existing methods for 2D-to-3D generation face significant limitations:

1. Scarcity of High-Quality 3D Training Data: Deep learning models require vast amounts of data, but high-quality 3D data with fine geometric details is rare and expensive to acquire. This limits the models' ability to learn intricate features.

2. Domain Gap: Models trained on synthetic data (often rendered from idealized 3D meshes) often perform poorly on real-world images due to differences in lighting, textures, and object variations.

3. Inherent Ambiguity in RGB Images: A single 2D RGB image provides limited information about 3D geometry. Factors like lighting, shading, and complex textures can obscure true geometric details, making it difficult for models to accurately infer depth and surface orientation.

   These challenges lead to 3D models that often lack fine-grained geometric details, compromising their realism and applicability. The paper's innovative idea is to leverage normal maps as an intermediate, 2.5D representation to bridge this gap between 2D images and 3D geometry. Normal maps encode surface orientation, offering clearer geometric cues than raw RGB images, which can guide the 3D generation process more effectively.

2.2. Main Contributions / Findings

The Hi3DGen framework introduces several key contributions to overcome the aforementioned challenges:

1. Normal Maps as an Intermediate Representation: Hi3DGen is presented as the first framework that systematically uses normal maps to bridge 2D image-to-3D geometry generation. This strategy effectively alleviates the domain gap between synthetic training data and real-world inputs, and provides stronger geometric supervision for generating fine details.

2. Noise-Injected Regressive Normal Estimator (NiRNE): The paper introduces NiRNE, a novel image-to-normal estimator. It combines the best aspects of diffusion-based (sharpness) and regression-based (stability) methods. NiRNE achieves robust, stable, and sharp normal estimation by:

   • Noise Injection: Inspired by diffusion models, noise is injected during training to enhance sensitivity to high-frequency patterns, leading to sharper details.
   • Dual-Stream Training: A dual-stream architecture decouples the learning of low-frequency (overall structure, generalizability) and high-frequency (fine details, sharpness) features.
   • Domain-Specific Training: This strategy optimizes the network by strategically utilizing real-world data for generalizability and synthetic data for precise high-frequency details.

3. Normal-Regularized Latent Diffusion (NoRLD): NoRLD is a novel normal-to-geometry learning approach. It integrates explicit normal map regularization into the latent diffusion learning process. By decoding the predicted latent representation into explicit 3D geometry and regularizing it with ground truth normal maps during training, NoRLD significantly enhances the fidelity and detail-consistency of 3D geometry generation, addressing the issue of indirect supervision in latent spaces.

4. DetailVerse Dataset: To support the training of NiRNE and NoRLD with high-quality, detail-rich data, the paper proposes a 3D data synthesis pipeline. This pipeline constructs DetailVerse, a large-scale dataset of 700K synthesized 3D assets featuring complex structures and rich surface details. This dataset serves as a crucial complement to existing human-created assets, which often lack sufficient detail.

   The key findings demonstrate that Hi3DGen consistently generates 3D models with significantly richer and more accurate geometric details compared to state-of-the-art methods. User studies, involving both amateur and professional 3D artists, further validate its superior generation quality. This work provides a new direction for high-fidelity 3D generation by effectively leveraging normal maps as an intermediate representation.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully grasp the Hi3DGen framework, it's essential to understand several core concepts in computer vision and deep learning:

• 3D Geometry Generation from Images: The overarching goal is to reconstruct or create a 3D model (e.g., a mesh, point cloud, implicit surface) from one or more 2D images. This is challenging because 2D images lose depth information, making the 2D-to-3D mapping an ill-posed problem (multiple 3D shapes can project to the same 2D image).
• Normal Maps: A normal map is a texture map used in 3D computer graphics to add surface detail without requiring more polygons. Instead of storing color information, each pixel in a normal map stores a vector (a "normal") that represents the orientation of the surface at that point in 3D space.
  • Representation: Typically, normal vectors are normalized to unit length, and their X, Y, Z components are mapped to RGB color channels (e.g., X to Red, Y to Green, Z to Blue), with values usually ranging from -1 to 1, mapped to 0-255 for storage.
  • Geometric Cues: Normal maps are considered a 2.5D representation because they encode surface orientation but not depth directly. However, they provide very strong local geometric cues, indicating how light should reflect off a surface, which is crucial for perceived detail.
• Diffusion Models: These are a class of generative models that learn to produce data (e.g., images, 3D shapes) by reversing a gradual diffusion process.
  • Forward Diffusion Process: In this process, noise is progressively added to data until it becomes pure random noise.
  • Reverse Diffusion Process: The model learns to reverse this process, starting from random noise and gradually denoising it to generate a clean data sample.
  • Latent Diffusion Models (LDMs): A popular variant where the diffusion process operates in a compressed latent space rather than directly on high-resolution pixel space. This makes them computationally more efficient.
  • Noise Injection: A key mechanism in diffusion models. By adding noise, the model learns to identify and reconstruct patterns even under noisy conditions, which can help capture high-frequency details.
• Variational Auto-Encoders (VAEs): A type of generative neural network that learns a compressed, latent representation of data.
  • Encoder: Maps input data (e.g., a 3D geometry) into a latent vector (a compressed numerical representation).
  • Decoder: Maps a latent vector back to the original data space, aiming to reconstruct the input.
  • Reparameterization Trick: Used in VAEs to allow backpropagation through the sampling process of the latent space.
• Flow Matching: A recent generative modeling technique that aims to learn a continuous transformation (a flow) that maps a simple noise distribution to a complex data distribution. It frames the generative process as learning a velocity field that pushes samples from noise to data, often used in the context of ODE-based diffusion models.
• ControlNet: An add-on module that allows for adding extra conditional control to large pre-trained diffusion models (like Stable Diffusion). It works by taking an input condition (e.g., an edge map, a depth map, a normal map) and guiding the generation process of the diffusion model without retraining the entire large model. This is achieved by creating a copy of the diffusion model's encoder, training it with the conditional input, and fusing its features with the original model's decoder.
• Canny Edge Detection: A popular edge detection algorithm used in computer vision. It's known for its ability to detect a wide range of edges in images while suppressing noise and minimizing spurious responses. It's often used to identify sharp boundaries or details.
• Photometric Stereo: A technique in computer vision for estimating the surface orientation (normal vectors) of objects from multiple images taken under different lighting conditions but from the same viewpoint. It requires controlled lighting setups.

3.2. Previous Works

The paper contextualizes its contributions by discussing prior research in three main areas:

3.2.1. Datasets for 3D Generation

• Early Datasets: Started small-scale and category-limited (e.g., ShapeNet [8, 73]).
• Scanning/Multi-view Photography: Efforts to expand data through scanning or multi-view captures (e.g., GSO [16], MVImgNet [72, 81], ABO [11], Meta [56]). However, data quality often fell short for direct 3D generation.
• Aggregating Human-Created Assets: More recently, large-scale datasets like Objaverse [13] and Objaverse-XL [14] aggregated millions of 3D assets from online sources.
  • Limitations: These datasets often suffer from licensing concerns (e.g., GitHub assets), lack textures (e.g., Thingiverse), and a severe imbalance towards simple models, lacking high-quality assets with complex geometric structures and rich surface details. This results in generated 3D models often being simplistic and losing detail.
  • Crucial Formula/Insight: The core problem these datasets face is the trade-off between quantity and quality/detail. While they offer many models, the density of geometric detail (e.g., sharp edges per model) is often low. This directly impacts the ability of models trained on them to learn and reproduce fine structures.

3.2.2. Normal Estimation

Monocular normal estimation (from a single image) methods are primarily divided into:

• Regression-Based Methods: These models directly predict normal maps from input images.
  • Evolution: From handcrafted features [27, 28] to deep learning techniques [18, 65, 85].
  • Recent Progress: Leveraging large data (Omnidata [17]), estimating per-pixel distributions [2], Vision Transformers [50], inductive bias modeling [1].
  • Characteristics: Tend to produce stable, deterministic predictions but often struggle with generating fine-grained, sharp details, leading to oversmoothing. Examples include Lotus [25] and GenPercept [77].
• Diffusion-Based Methods: Adapting powerful text-to-image diffusion models.
  • Examples: Geowizard [21] uses a geometry switcher; StableNormal [80] improves stability via a coarse-to-fine strategy.
  • Characteristics: Known for producing sharper results due to their probabilistic nature and sensitivity to high-frequency components during the denoising process. However, they can suffer from instability, high variance, and spurious details. Strategies like affine-invariant ensembling [21, 32] and one-step generation [77] attempt to mitigate these but can be computationally intensive or lead to oversmoothing.
  • Differentiation from Hi3DGen: Hi3DGen's NiRNE deeply analyzes why diffusion models produce sharpness and integrates noise injection into a regressive framework, combining the sharpness of diffusion with the stability of regression, further enhanced by dual-stream training for generalization.

3.2.3. Normal Maps in 3D Generation

• Enhancing 3D Reconstruction: Normal maps have long been used to improve the fidelity and consistency of 3D reconstructions (e.g., SuperNormal [6], PIFuHD [55], MonoSDF [82]).
• Exploration in 3D Generation:
  • SDS-based Methods (Score Distillation Sampling): Use normal maps during optimization to regularize geometry, often alongside RGB images (e.g., DreamFusion [48], HumanNorm [29], RichDreamer [49]).
  • Multi-view Diffusion: Generate normal images to complement RGB data for reconstruction/fusion (e.g., Wonder3D [42], Direct2.5 [44], Unique3D [69]). These can suffer from multi-view inconsistency, leading to smooth details.
  • 3D Native Diffusion: Methods based on 3D representations (feature volumes, Triplane [7], 3D Gaussians [33]) leverage normal maps by decoding them into meshes and applying normal rendering losses (e.g., AG3D [15], MeshFormer [40], InstantMesh [78]). These often have high memory requirements.
  • Latent Code Diffusion: Achieved state-of-the-art performance (e.g., CraftsMan [37], Direct3D [70], Trellis [74], Clay [84]). However, the use of normal maps in this paradigm remains underexplored because normals cannot directly regularize learning in the highly abstract latent space. CraftsMan uses normal refinement as post-processing, and Trellis incorporates normal rendering loss during VAE training, but not directly in the latent diffusion step itself.
  • Differentiation from Hi3DGen: Hi3DGen's NoRLD uniquely emphasizes the critical role of normal maps in latent diffusion by introducing online normal regularization, directly guiding the learning of 3D latent codes to preserve detailed geometry.

3.3. Technological Evolution

The field of 2D-to-3D generation has evolved from early attempts at direct mapping, often yielding coarse 3D models, to more sophisticated approaches that leverage intermediate representations and powerful generative models. Initially, researchers focused on implicit representations or voxel grids, limited by resolution and computational cost. The advent of deep learning brought end-to-end learning from images to 3D, but faced challenges with data scarcity and ambiguity.

The development of diffusion models and latent space representations (like those used in LDMs) marked a significant leap, allowing for more complex and diverse generation. However, even these methods struggled with fine-grained geometric details due to indirect supervision in the latent space and the inherent domain gap when dealing with real-world images.

This paper's work fits into the current state-of-the-art by addressing these persistent challenges. It pushes the boundary by explicitly incorporating normal maps as a robust intermediate representation. This acts as a bridge, making the 2D-to-3D pipeline more controllable and detail-preserving. The NiRNE component advances normal estimation by combining the best features of regression-based and diffusion-based methods, while NoRLD innovatively integrates normal supervision directly into the latent diffusion process, which is a key novelty. The creation of DetailVerse also signifies a trend towards synthetically generated, high-quality data to overcome real-world data limitations.

3.4. Differentiation Analysis

Compared to prior methods, Hi3DGen offers several core differentiations and innovations:

1. Systematic Normal Bridging: Unlike previous works that might use normal maps as a supplementary input or for post-processing (CraftsMan), Hi3DGen proposes a holistic framework where normal maps are a central intermediate representation that bridges the entire 2D-to-3D generation pipeline. This explicit image-to-normal-to-geometry flow is designed to maximize detail preservation.

2. Novel Normal Estimator (NiRNE) for Sharpness AND Stability: Previous normal estimators either excelled at sharpness (diffusion-based, but unstable) or stability (regression-based, but oversmooth). NiRNE is designed to achieve both by:

   • Integrating noise injection (from diffusion) into a regressive framework to capture high-frequency details sharply.
   • Employing a dual-stream architecture with domain-specific training to simultaneously learn generalizable low-frequency structures from real data and precise high-frequency details from synthetic data, something not fully achieved by prior methods.

3. Online Normal Regularization in Latent Diffusion (NoRLD): While some latent diffusion models might have used normal maps in VAE training (Trellis) or as a post-processing step (CraftsMan), NoRLD introduces explicit normal map regularization online during the latent diffusion training process. This direct supervision in the 3D geometry space (after decoding from latent) ensures that the latent codes themselves are guided towards representing fine-grained details consistent with input images, a more direct and effective approach than indirect supervision.

4. High-Quality Synthetic Dataset (DetailVerse): Recognizing the limitations of existing 3D datasets (Objaverse often lacks fine details), Hi3DGen proactively creates a new, detail-rich synthetic dataset. This self-synthesized data, generated using a rigorous pipeline, is specifically tailored to provide the precise high-frequency labels and complex geometric structures needed to train NiRNE and NoRLD effectively, overcoming a fundamental data quality bottleneck that previous methods often contend with.

   In essence, Hi3DGen innovates by creating a more robust, stable, and detail-preserving 2D-to-3D pipeline through intelligent use of normal maps at critical junctures, supported by a specialized normal estimator and a high-quality dataset.

4. Methodology

This section details the Hi3DGen framework, which aims to bridge the 2D-to-3D generation task using normal maps as a 2.5D intermediate representation. The framework divides the overall image-to-geometry generation into two main stages: image-to-normal estimation and normal-to-geometry mapping. An overview of the entire framework is shown in Figure 8.

The Hi3DGen framework consists of three primary components:

1. Noise-Injected Regressive Normal Estimation (NiRNE): A dual-stream normal estimator designed for both stability and sharpness in normal prediction.

2. Normal-Regularized Latent Diffusion (NoRLD): An online normal regularizer integrated into latent diffusion for fine-grained details and image-geometry consistency.

3. DetailVerse Dataset: A synthesized 3D dataset with complex geometry and rich surface details to facilitate sharp normal estimation and detailed 3D geometry generation.

   The following figure (Figure 8 from the original paper) provides an overview of the Hi3DGen framework:

   Figure 8. The image is a schematic diagram of the Hi3DGen method pipeline, illustrating the entire process from input image to normal estimation, normal-guided 3D geometry generation, and high-quality dataset construction, highlighting key techniques such as dual-stream encoding, latent diffusion, and normal regularization.

4.1. Noise-Injected Regressive Normal Estimation (NiRNE)

NiRNE addresses the trade-off between sharpness (often seen in diffusion-based methods) and stability (characteristic of regression-based methods) in monocular normal estimation. It achieves this by integrating noise injection into a regressive framework and employing a dual-stream architecture with domain-specific training.

4.1.1. Noise Injection

The core idea here is to analyze why diffusion-based methods yield sharper normals and then port that mechanism (noise injection) to a regressive model. Diffusion processes are often defined by a stochastic differential equation (SDE): $ x_t = x_0 + \int_0^t g(s) dw_t $ where:

• $x_t$ represents the state of the data at time $t$.

• $x_0$ is the initial clean data sample.

• g(s) is a time-dependent function that controls the scale of the noise.

• $dw_t$ is a Wiener process (or Brownian motion), representing the injected random noise.

  By applying a Fourier transform to this process, the signal-to-noise ratio (SNR) for any frequency component $\omega$ at timestep $t$ can be obtained: $ \mathrm{SNR}(\omega, t) = \frac{|\hat{x}_0(\omega)|^2}{\int_0^t |g(s)|^2 ds} $ where:

• $\mathrm{SNR}(\omega, t)$ is the signal-to-noise ratio for frequency $\omega$ at time $t$.

• $|\hat{x}_0(\omega)|^2$ is the power spectrum of the initial clean data $x_0$ at frequency $\omega$.

• $\int_0^t |g(s)|^2 ds$ represents the accumulated power of the injected noise up to time $t$.

  Explanation: Natural images typically exhibit low-pass characteristics, meaning that low-frequency components (overall shapes, colors) have higher energy than high-frequency components (details, edges). Mathematically, this implies $|\hat{x}_0(\omega)|^2 \propto |\omega|^{-\alpha}$ for some $\alpha > 0$. As the diffusion process progresses (time $t$ increases), the noise power $\int_0^t |g(s)|^2 ds$ increases equally across all frequencies. Because high-frequency components start with lower energy, their SNR degrades faster than low-frequency components. This means that at later stages of diffusion, the model effectively receives stronger supervision on these high-frequency regions because the signal there is relatively more affected by noise, forcing the model to learn to recover these details more accurately. Inspired by this, NiRNE injects noise into a regression-based framework to encourage the model to capture more high-frequency information, leading to sharper normal estimations.

4.1.2. Dual-Stream Architecture

To achieve both generalizability (from low-frequency features) and sharpness (from high-frequency features), NiRNE uses a dual-stream architecture that decouples these two types of feature learning.

• Clean Stream: Processes the original input image without noise injection. Its role is to robustly capture low-frequency details and overall structural information, which are crucial for the model's generalizability.

• Noisy Stream: Processes the noise-injected image. This stream is designed to focus on high-frequency details, learning to discern fine structures even when perturbed by noise, akin to how diffusion models achieve sharpness.

  The latent representations from both streams are concatenated in a ControlNet-style manner [83]. This combined representation is then fed into a decoder to produce the final normal map prediction in a regressive manner. This design effectively merges the sharpness capabilities derived from noise injection with the stability of a regressive framework.

The following figure (Figure 9 from the original paper) illustrates the Noise-injected Regressive Normal Estimation and Dual-Stream Architecture:

Figure 9. Left part: Illustration of Noise-injected Regressive Normal Estimation; Right part: Noisy label at high-frequency regions in real-domain data.

4.1.3. Domain-Specific Training

To further encourage the decoupled representation learning in the two streams and leverage the strengths of different data types, a domain-specific training strategy is employed:

1. Initial Training (Real-Domain Data): The network is first trained using real-domain data. Real-world datasets, despite potential noise in labels (especially at edges, as visualized in Figure 9, right part), are vital for learning generalizable low-frequency information and overall robustness. This stage primarily benefits the clean stream.

2. Fine-tuning (Synthetic-Domain Data): In the second stage, the noisy stream is fine-tuned using synthetic-domain data, while the parameters of the clean stream are frozen. Synthetic data, rendered from 3D ground truth, provides precise high-frequency labels without the noise issues of real data. This allows the noisy stream to specifically learn high-frequency details as a residual component, complementing the coarse features learned by the clean stream.

   This strategy ensures that the network fully utilizes the strengths of both real and synthetic data, fostering decoupled representation learning and improving both generalizability and sharpness.

4.2. Normal-Regularized Latent Diffusion (NoRLD)

State-of-the-art 2D-to-3D generation methods often rely on 3D latent diffusion, where 3D geometries are represented in a compact latent space. While efficient, this approach can lead to a loss of fine-grained details or detail-level inconsistency because geometric information, especially intricate details, is highly compressed in the latent space, and supervision is indirect. NoRLD addresses this by integrating explicit normal map regularization directly into the latent diffusion process.

4.2.1. Latent Diffusion

The typical latent diffusion process in 3D generation involves a Variational Auto-Encoder (VAE):

• An encoder $E(\cdot)$ maps a 3D geometry $X$ to a latent representation $x_0$.
• A decoder $D(\cdot)$ reconstructs the geometry $\hat{X}$ from $x_0$. $ x_0 = E(X), \quad \hat{X} = D(x_0) $ where:
  • $X$ represents the original 3D geometry.
  • $E$ is the encoder function.
  • $x_0$ is the latent representation of the 3D geometry.
  • $D$ is the decoder function.
  • $\hat{X}$ is the reconstructed 3D geometry. (The reparameterization process common in VAEs is omitted for simplicity in the paper's formulation).

The image-conditioned diffusion process then involves:

• Constructing $x_t$ by injecting noise into $x_0$ at a given timestep $t$.
• Learning to recover $x_0$ from $x_t$. Flow matching is often used for this, modeling a time-dependent velocity field. The loss function for latent diffusion models (LDM) is typically formulated as: $ \mathcal{L}{\mathrm{LDM}} = \mathbb{E}{t, x_0, x_t} \Big[ \big| \mathbf{v}_{\theta} (x_t, t) - \mathbf{u}(x_t, t) \big|^2 \Big] $ where:
• $\mathcal{L}_{\mathrm{LDM}}$ is the loss for the latent diffusion model.
• $\mathbb{E}_{t, x_0, x_t}$ denotes the expectation over timesteps $t$, initial latent samples $x_0$, and noisy latent samples $x_t$.
• $\mathbf{v}_{\theta}(x_t, t)$ is the predicted velocity field by the neural network with parameters $\theta$.
• $\mathbf{u}(x_t, t) = \nabla_{x_t} \log p(x_t | x_0)$ is the true velocity field which directs $x_t$ back to $x_0$.
• $\| \cdot \|^2$ denotes the squared Euclidean norm, measuring the difference between predicted and true velocity fields. (Image/text conditions are implicitly included in this formulation).

4.2.2. Normal Regularization

To provide more precise supervision over surface details, NoRLD introduces normal regularization directly in the 3D geometry space. This results in an enhanced loss function: $ \mathcal{L}{\mathrm{NorLd}} = \mathcal{L}{\mathrm{LDM}} + \lambda \cdot \mathcal{R}_{\mathrm{Normal}} (\hat{x}_0) $ where:

• $\mathcal{L}_{\mathrm{NorLd}}$ is the total loss for the Normal-Regularized Latent Diffusion model.

• $\mathcal{L}_{\mathrm{LDM}}$ is the standard latent diffusion model loss, as defined above.

• $\lambda$ is a hyperparameter that controls the weight of the normal regularization term.

• $\mathcal{R}_{\mathrm{Normal}} (\hat{x}_0)$ is the proposed normal regularization term.

  The normal regularization term $\mathcal{R}_{\mathrm{Normal}}$ is defined as: $ \mathcal{R}_{\mathrm{Normal}} (\hat{x}_0) = \mathbb{E}_v \Big[ \big| R_v (D(\hat{x}_0)) - N_v \big|^2 \Big] $ where:

• $\mathbb{E}_v$ denotes the expectation over different viewpoints $v$.

• $\hat{x}_0$ represents the predicted clean latent sample (after denoising).

• $D(\hat{x}_0)$ decodes the predicted latent sample $\hat{x}_0$ back into an explicit 3D geometry.

• $R_v(\cdot)$ is a function that renders the normal map of the decoded 3D geometry from viewpoint $v$.

• $N_v$ denotes the corresponding ground truth normal map for that viewpoint $v$.

• $\| \cdot \|^2$ is the squared Euclidean norm, measuring the pixel-wise difference between the rendered and ground truth normal maps.

  Key Insight: This regularization is conducted online during the diffusion training process, as illustrated in Figure 10. This is crucial because it actively guides the training of the diffusion network by providing explicit feedback from the 3D geometry space. This helps align the predicted latent representations with a distribution that inherently contains rich details consistent with the input images, overcoming the indirect supervision issue of previous latent diffusion approaches.

The following figure (Figure 10 from the original paper) illustrates the Normal-Regularized Latent Diffusion process:

Figure 10. The image is an illustration of Figure 3 from the paper, showing the normal-regularized latent diffusion learning process. It includes VAE encoding and decoding of 3D input and uses normal maps as conditional guidance for latent diffusion to enhance the accuracy of 3D geometry reconstruction.

4.3. DetailVerse Dataset

High-quality 3D data is paramount for training NiRNE (to provide clean normal labels) and NoRLD (for high-fidelity 3D generation). Existing datasets like Objaverse [13, 14], while large, primarily contain assets with simple structures and plain surface details, limiting Hi3DGen's generation capabilities. To overcome the prohibitive cost of manually creating detailed 3D assets, Hi3DGen proposes a 3D data synthesis pipeline to construct the DetailVerse dataset.

4.3.1. Dataset Construction

The DetailVerse dataset is created through a multi-stage pipeline involving Text-to-Image and Image-to-3D generation, coupled with rigorous filtering:

1. Text Prompt Curation:

   • Initial Sourcing: Approximately 14M raw prompts are sourced from DiffusionDB [67].
   • Classification: A LLaMA3-8B model [60] is used to classify prompts, retaining only those for Single Objects or Multiple Objects (yielding ~1M candidates) and filtering out complex scenes.
   • Rule-Based Filtering & Standardization: Stylistic modifiers are removed, and a LLaMA-3-13B model [60] standardizes structural prompts. Domain-specific templates (e.g., "isometric perspective," "Unreal Engine 5 Rendering," "4K") are applied to enforce explicit geometric cues, resulting in ~1.5M curated prompts. This aims to generate images that are optimal for 3D synthesis.

2. High-Quality Image Generation:

   • Image Generator: Flux.1-Dev [35], a state-of-the-art text-to-image generator, is used.
   • Sharpness Filtering: Generated images are filtered by their sharpness (using Canny edge detection) and only the top 50% are retained.
   • Viewpoint Control: OrientAnything [66] (an object orientation estimation model) is used to measure the alignment between the camera view and the canonical object orientation. Images with angular deviations exceeding 60° are rejected to ensure stable 3D generation and prevent structural distortions. This step preserves 1 million high-quality images.

3. Robust Image-to-3D Synthesis:

   • 3D Generator: Trellis [74], a state-of-the-art two-stage 3D generator, is employed to produce initial 3D meshes from the prepared images.
   • Rigorous Data Cleaning:

     • Expert Evaluation: 10K preliminary meshes are randomly sampled and assessed by 10 trained experts for surface quality, specifically checking for holes or noise artifacts in rendered normal maps.

     • Automated Assessment: A quality assessment network is trained using DINOv2 [45] features, based on the expert annotations. This network (a three-layer MLP classifier) evaluates rendered normal maps from four equiangular views of each mesh.

     • Selection: Only models that receive positive classifications across all four views are selected for the DetailVerse dataset.

       This comprehensive process yields 700K high-quality meshes for DetailVerse, specifically designed to possess complex structures and rich details.

The following figure (Figure 11 from the original paper) illustrates the overall procedure of DetailVerse Construction:

Figure 11. The image is a schematic diagram illustrating the overall procedure of DetailVerse construction, including three main stages: text prompt collection, image generation, and 3D assets synthesis, with detailed steps for data filtering and quality assurance to support high-quality 3D model generation.

4.3.2. Dataset Statistics

The DetailVerse dataset is compared against existing 3D object datasets in terms of scale and geometric detail richness. The primary metric used is the Sharp Edge #, detected using the implementation in Dora-Bench [10].

The following are the results from Table 1 of the original paper:

Explanation: As shown in the table, DetailVerse contains 700K objects, comparable in scale to Objaverse-1.0 but significantly outperforming all other datasets, especially Objaverse-XL, in terms of sharp edge count. The mean number of sharp edges (45,773) in DetailVerse is vastly higher than any other dataset, demonstrating its superior richness in geometric details. This richness is crucial for training models that can generate high-fidelity 3D geometry.

5. Experimental Setup

5.1. Datasets

The experiments utilize a combination of real-world and synthetic datasets for training and evaluation.

5.1.1. Image-to-Normal Training

• Diverse Realistic Dataset: A dataset following Depth-pro [4] is used. This likely refers to a collection of real-world images with associated ground truth normal maps, essential for NiRNE to learn generalizable low-frequency features.
• Synthetic Data: A dataset comprising 20M RGB-to-normal pairs. These pairs are created by rendering 40 images per asset from 500K assets selected from the newly constructed DetailVerse dataset. This synthetic data provides precise high-frequency normal labels, crucial for training NiRNE's noisy stream.

5.1.2. Normal-to-Geometry Training

• Objaverse: A large-scale dataset curated from 170K cleaned 3D assets from Objaverse [13]. This subset is chosen for its quality, and 40 images are rendered per asset, following Trellis [74].
• DetailVerse: The 700K synthesized 3D assets from the DetailVerse dataset. Again, 40 images are rendered per asset. This dataset provides the NoRLD model with abundant detail-rich 3D geometries and corresponding normal maps for effective normal regularization.

5.1.3. Evaluation Datasets

• LUCES-MV [41]: A reconstruction dataset used to validate the generalization ability of the image-to-normal estimator (NiRNE) on real scenes. This dataset focuses on multi-view data for near-field point light source photometric stereo.
• Project Pages/Websites: All images for visual comparison and user studies are collected from:
  • Hyper3D website [12] (Rodin Gen-1 project page)
  • Hunyuan3D-2.0 project page [59]
  • Dora project page [54] These sources provide publicly available examples from state-of-the-art 3D generation methods, allowing for direct qualitative comparison.

5.2. Evaluation Metrics

For every evaluation metric mentioned in the paper, a complete explanation is provided:

5.2.1. Normal Angle Error (NE)

• Conceptual Definition: Normal Angle Error (NE) quantifies the overall angular difference between a predicted normal vector and its corresponding ground truth normal vector at each pixel. It measures the average accuracy of the normal map across the entire surface. A lower NE indicates higher accuracy.
• Mathematical Formula: Given a predicted normal vector $\mathbf{n}_p$ and a ground truth normal vector $\mathbf{n}_{gt}$ at a pixel, the angular error is calculated as: $ \theta = \arccos(\mathbf{n}p \cdot \mathbf{n}{gt}) $ The Normal Angle Error (NE) is typically the average of these angular errors over all pixels, often converted to degrees. $ \mathrm{NE} = \frac{1}{N} \sum_{i=1}^{N} \arccos(\mathbf{n}{p,i} \cdot \mathbf{n}{gt,i}) \times \frac{180}{\pi} $
• Symbol Explanation:
  • $\theta$: The angle in radians between the predicted and ground truth normal vectors.
  • $\mathbf{n}_p$: The 3D unit normal vector predicted by the model for a given pixel.
  • $\mathbf{n}_{gt}$: The 3D unit ground truth normal vector for the same pixel.
  • $\cdot$: The dot product operator.
  • $\arccos(\cdot)$: The inverse cosine function, which returns the angle whose cosine is the argument.
  • $N$: The total number of pixels over which the error is calculated.
  • $\sum_{i=1}^{N}$: Summation over all pixels from $i=1$ to $N$.
  • $\frac{180}{\pi}$: Conversion factor from radians to degrees.

5.2.2. Sharp Normal Error (SNE)

• Conceptual Definition: Sharp Normal Error (SNE) is a specialized metric that emphasizes the accuracy of normal estimation specifically on sharp edges where geometric details are most pronounced. It is designed to evaluate how well a model captures fine details, which are often overlooked by general metrics. A lower SNE indicates better detail preservation.
• Mathematical Formula: The calculation of SNE involves a three-step process as described in the paper (following Dora [10]):
  1. Salient Region Detection: Detect salient regions (sharp edges) in the ground truth normal maps using an edge detection algorithm (e.g., Canny edge detector). This generates a binary mask $M_{edge}$.
  2. Dilation: Dilate these masked regions to ensure complete coverage of the edge features. This creates an expanded mask $M_{dilated}$.
  3. Normal Angle Error Calculation: Compute the Normal Angle Error (NE) (as defined above) only within these masked regions. $ \mathrm{SNE} = \frac{1}{\sum_{i=1}^{N} M_{dilated,i}} \sum_{i=1}^{N} (M_{dilated,i} \times \arccos(\mathbf{n}{p,i} \cdot \mathbf{n}{gt,i})) \times \frac{180}{\pi} $
• Symbol Explanation:
  • $M_{edge}$: A binary mask where pixels belonging to sharp edges are 1, and others are 0.
  • $M_{dilated}$: The dilated version of $M_{edge}$, also a binary mask.
  • $\mathbf{n}_{p,i}$ and $\mathbf{n}_{gt,i}$: Predicted and ground truth normal vectors for pixel $i$.
  • $\arccos(\mathbf{n}_{p,i} \cdot \mathbf{n}_{gt,i}) \times \frac{180}{\pi}$: The angular error for pixel $i$ in degrees.
  • $\sum_{i=1}^{N} M_{dilated,i}$: The total number of pixels within the dilated sharp edge regions.
  • $\sum_{i=1}^{N} (M_{dilated,i} \times \dots)$: Summation of angular errors only for pixels within the dilated sharp edge regions.

5.3. Baselines

The paper compares Hi3DGen against several state-of-the-art methods in both normal estimation and 3D generation.

5.3.1. Competitive Normal Estimators

• Diffusion-based Methods:
  • GeoWizard [21]: Incorporates a geometry switcher to handle diverse data distributions.
  • StableNormal [80]: Improves estimation stability by reducing diffusion inference variance via a coarse-to-fine strategy.
• Regression-based Methods:
  • Lotus [25]: A diffusion-based visual foundation model adapted for dense prediction.
  • GenPercept [77]: Focuses on repurposing diffusion models for general dense perception tasks, often used for stable one-step predictions.

5.3.2. Competitive 3D Generation Methods

• Open-Sourced Methods:
  • CraftsMan1.5 [37]: A method for high-fidelity mesh generation using 3D native generation and an interactive geometry refiner.
  • Hunyuan3D-2.0 [86]: A diffusion model for high-resolution textured 3D asset generation.
  • Trellis [74]: Uses structured 3D latents for scalable and versatile 3D generation.
• Closed-Sourced Methods:

  • Clay [84]: A controllable large-scale generative model for creating high-quality 3D assets.

  • Tripo-2.5 [61]: A model capable of creating 3D models from text input.

  • Dora [10]: A method related to sampling and benchmarking for 3D shape variational auto-encoders. (Note: The paper states Dora has no public API, so comparisons are based on examples from its project page).

    These baselines represent the current landscape of monocular normal estimation and image-to-3D generation, covering different methodological categories (diffusion vs. regression, latent diffusion vs. 3D native) and both academic and industrial solutions.

6. Results & Analysis

The experimental results validate the effectiveness and superiority of Hi3DGen in generating high-fidelity 3D geometry. This section details the quantitative and qualitative outcomes for both image-to-normal estimation and normal-to-geometry generation, along with ablation studies.

6.1. Image-to-Normal Estimation

6.1.1. Quantitative Results

The performance of NiRNE is quantitatively compared against other state-of-the-art normal estimators on the LUCES-MV dataset [41].

The following are the results from Table 2 of the original paper:

Explanation: The table clearly demonstrates NiRNE's significant superiority. It achieves the lowest Normal Angle Error (NE) of 21.837 degrees, indicating better overall normal estimation accuracy across the entire image. More importantly, NiRNE also achieves the lowest Sharp Normal Error (SNE) of 26.628 degrees, which specifically measures accuracy on sharp edges and fine details. This validates that NiRNE effectively combines stability with the ability to capture sharp, fine-grained geometric information, outperforming both diffusion-based (e.g., GeoWizard, StableNormal) and other regression-based methods (e.g., Lotus, GenPercept).

6.1.2. Qualitative Results

Qualitative results further support NiRNE's superior performance.

The following figure (Figure 12 from the original paper) shows normal estimation results comparison:

Figure 12. The image is Figure 5 from the paper, showing a comparison of normal estimation results from various methods. It presents predicted normals and corresponding error heatmaps for two 3D models, highlighting the proposed method's superiority in detail preservation and error reduction.

Explanation: As shown in Figure 12, NiRNE demonstrates:

• Robustness and Generalizability: It performs well on diverse inputs, including human figures and various objects.
• Stability: Compared to diffusion-based methods which can produce spurious details or instabilities (as seen in the error maps of GeoWizard and StableNormal), NiRNE shows fewer erroneous details.
• Sharpness: NiRNE delivers noticeably sharper estimations, particularly at object boundaries and intricate features, especially when contrasted with other regression-based methods (Lotus, GenPercept) which tend to produce smoother, less detailed normals. These qualitative observations align with the quantitative SNE metric, confirming NiRNE's ability to generate sharp and stable normal maps.

6.2. Normal-to-Geometry Generation

6.2.1. Qualitative Results

The paper presents qualitative comparisons of the generated 3D geometries, highlighting Hi3DGen's ability to produce high-fidelity models with rich details consistent with the input images.

The following figure (Figure 2 from the original paper) shows qualitative 3D generation comparison:

Figure 2. The image is a comparative illustration showing the results of seven methods generating 3D normal maps from four different input images with rotating views. It displays the input images alongside the rotating 3D normals produced by each method, highlighting the superior detail preservation and geometric fidelity of the Hi3DGen approach.

Explanation: As illustrated in Figure 2, Hi3DGen consistently generates 3D models that retain fine-grained geometric details from the input images, which are often lost or smoothed out by other methods. For example, intricate patterns, sharp edges, and subtle surface variations are faithfully reproduced. Even when input images present fewer details (e.g., the first and third examples in Figure 2), Hi3DGen still produces robust generations with relatively smooth yet accurate surfaces.

The following figure (Figure 14 from the original paper) shows additional high-fidelity 3D results generated by Hi3DGen:

Figure 14. The image is Figure 7 from the paper, showing high-fidelity 3D model generation results. The left and right sides display the colored textures and corresponding grayscale detailed geometry shapes, while the center shows the normal maps, demonstrating the method’s superior capability in generating rich geometric details.

Explanation: Figure 14 further showcases Hi3DGen's capability. The examples display detailed colored textures, intricate grayscale geometric shapes, and clear normal maps, all demonstrating the method's strength in capturing and reproducing rich geometric details, which are crucial for realism.

6.2.2. User Study

A user study was conducted to objectively evaluate the quality of 3D generation results by Hi3DGen against five other methods (Hunyuan3D-2.0, Dora, Clay, Tripo-2.5, Trellis).

• Evaluation Criteria: Fidelity of the generated 3D geometry to the input images, focusing on consistency in overall shape and local details. For unseen parts of the object, evaluators judged the plausibility and stylistic consistency.
• Evaluator Groups:

  • Amateur 3D Users (50 participants): Assessed $100 \times 6$ randomly sampled results from an everyday application perspective (e.g., 3D printing).

  • Professional 3D Artists (10 participants): Evaluated $20 \times 6$ results from a professional use standpoint (e.g., 3D modeling and design).

    The following figure (Figure 15 from the original paper) shows the user study results:

    Figure 15. The image is a chart showing the user study results in Figure 8 of the paper, comparing preference proportions of five 3D generation methods between professional artists and amateur users, highlighting the advantage of the Hi3DGen method.

    Explanation: Figure 15 clearly shows that Hi3DGen achieved the highest generation quality, being preferred by both amateur users and professional artists across all compared methods. This strong preference in a direct human evaluation underscores the subjective and objective superiority of Hi3DGen in producing aesthetically pleasing and geometrically accurate 3D models.

6.3. Ablation Study

Ablation studies were performed to validate the contribution of each key component of Hi3DGen.

6.3.1. Normal Bridge

The paper validates the effectiveness of using normal maps as a bridge for 3D generation.

• A direct image-to-geometry generator (based on Trellis [74]) performed worse than Hi3DGen.
• When using the same normal regularization and training data as Hi3DGen, the direct generator still produced fake details, indicating that the inherent structure of normal bridging is beneficial.
• The study also found that smoother or wrong normal estimations from other methods led to a drop in final 3D generation quality, directly proving the importance of accurate and sharp estimated normals provided by NiRNE as the intermediate bridge.

6.3.2. DetailVerse Data

The DetailVerse dataset's value is validated through its impact on both NiRNE and NoRLD.

• Impact on NiRNE: Integrating image-normal training pairs rendered from DetailVerse data improved NiRNE's performance by 0.4 in NE and 1.7 in SNE. This is shown in the first two rows of Table 3 below.
• Impact on NoRLD: Using additional normal-geometry training pairs from DetailVerse enables NoRLD to achieve higher-fidelity generation details, as illustrated in the NoRLD Ablation section (Figure 3, discussed below).

6.3.3. NiRNE Ablation

Ablative experiments were conducted to validate the three components of NiRNE: noise injection (NI), dual-stream architecture (DS), and domain-specific training (DST).

The following are the results from Table 3 of the original paper:

Explanation:

• Ours w/o DV: Removing the DetailVerse data from training leads to a noticeable increase in both NE and SNE (0.4 NE, 1.7 SNE increase), confirming the dataset's role in improving accuracy, especially for sharp details.
• Ours w/o DST: Disabling the Domain-Specific Training strategy results in a substantial increase in NE (23.288) and SNE (29.690), highlighting its importance for effectively leveraging different data domains for decoupled representation learning.
• Ours w/o DS: Removing the Dual-Stream architecture also degrades performance significantly, particularly for SNE (29.520), demonstrating that decoupling low- and high-frequency learning is crucial for sharpness.
• Ours w/o all: Removing all key components (implied to be without NI, DS, DST and DV or some combination leading to a baseline regression model) results in the worst performance, especially in SNE (35.997), emphasizing the collective contribution of these innovations. Each component (NI, DS, DST, and DV data) contributes positively to NiRNE's final performance.

6.3.4. NoRLD Ablation

The impact of the online normal regularization in NoRLD is visualized.

The following figure (Figure 3 from the original paper) shows ablation on the proposed NoRLD:

Figure 3. The image is an illustration from the paper showing the ablation study results of the proposed NoRLD method. It compares 3D model details and geometric textures with different module removals (without DV&NoRLD, without DV, without NoRLD), with the rightmost being the full method demonstrating the best detail.

Explanation: Figure 3 visually demonstrates the importance of NoRLD. Comparing "without DV & NoRLD" (no DetailVerse and no Normal Regularization) to the "full model," or "without NoRLD" to the "full model," reveals that online normal regularization significantly improves the generation fidelity and the reproduction of fine details (e.g., roof details as suggested by the paper). This holds true whether or not DetailVerse data is used for training, confirming that NoRLD's mechanism directly enhances detail consistency in 3D geometry generation.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces Hi3DGen, a novel framework for high-fidelity 3D geometry generation from images. Its core innovation lies in leveraging normal maps as a 2.5D intermediate representation to bridge the gap between 2D images and 3D geometry, enabling the generation of rich geometric details consistent with input images.

Hi3DGen achieves this through three integrated components:

1. NiRNE (Noise-Injected Regressive Normal Estimation): This component delivers robust, stable, and sharp normal predictions by combining noise injection (inspired by diffusion models) with a dual-stream regressive architecture and a domain-specific training strategy.

2. NoRLD (Normal-Regularized Latent Diffusion): This component enhances 3D geometry generation fidelity by incorporating online normal map regularization directly into the latent diffusion learning process. This provides explicit 3D geometry supervision, guiding the latent codes to represent fine details.

3. DetailVerse Dataset: A high-quality, detail-rich synthetic 3D dataset is constructed via a novel 3D data synthesis pipeline. This dataset is crucial for training NiRNE and NoRLD to achieve their superior performance.

   Extensive experiments, including quantitative metrics, qualitative comparisons, and user studies, consistently demonstrate Hi3DGen's effectiveness and superiority over state-of-the-art methods in generating fine-grained geometric details and achieving high fidelity.

7.2. Limitations & Future Work

The authors acknowledge one primary limitation:

• Inconsistencies from Generative Nature: Despite generating detail-rich 3D results, some outputs from Hi3DGen may still exhibit possible inconsistent or non-aligned details with the input image. This is attributed to the inherent generative nature of the 3D latent diffusion learning, which, while powerful for creating novel geometry, might not always perfectly reconstruct existing details.

  As future work, the authors propose to pursue reconstruction-level 3D generations. This suggests a direction towards methods that emphasize higher precision and fidelity to the input's exact geometry, potentially moving beyond pure generation towards more faithful reconstruction, possibly by integrating more direct geometry constraints or error correction mechanisms.

7.3. Personal Insights & Critique

This paper presents a highly innovative and well-structured approach to a critical problem in computer graphics. The idea of normal bridging is intuitively appealing, as normal maps inherently provide local geometric information that RGB images often obscure.

Strengths:

• Clarity of Purpose: The paper clearly identifies the limitations of existing 2D-to-3D generation methods and proposes a logical, multi-faceted solution.
• Novelty in NiRNE: The noise-injected regressive normal estimation is a clever hybrid approach that addresses a fundamental trade-off between sharpness and stability in normal estimation. The frequency domain analysis justifying noise injection is particularly insightful.
• Effective NoRLD: Integrating online normal regularization into latent diffusion is a powerful way to bring explicit 3D geometry supervision into an otherwise abstract latent space, directly improving detail fidelity.
• Addressing Data Bottleneck: The creation of the DetailVerse dataset through a robust synthesis pipeline is a significant contribution, demonstrating a proactive solution to the perennial problem of insufficient high-quality 3D training data. This dataset itself could be a valuable resource for future research.
• Comprehensive Evaluation: The combination of quantitative metrics (NE, SNE), extensive qualitative comparisons, and user studies provides strong evidence for the method's superiority.

Potential Issues/Areas for Improvement/Further Exploration:

• Complexity vs. Generalization of NiRNE: While NiRNE is effective, its dual-stream architecture and domain-specific training add complexity. It would be interesting to see if simpler hybrid models could achieve similar performance with less intricate training schemes or if there are scenarios where this complexity becomes a bottleneck for even broader generalization.
• Computational Cost: Training large diffusion models with VAE and explicit 3D rendering for regularization (even if rendering normals) can be computationally intensive. While latent diffusion is efficient, the online rendering and decoding steps for NoRLD add overhead. Future work could explore more efficient normal regularization or latent space regularization that implicitly encodes normal information.
• "Inconsistent or Non-aligned Details": The acknowledged limitation regarding inconsistent details is crucial. This is a common challenge for generative models. Future work might explore consistency losses that penalize deviations from specific geometric features of the input, or integrate reconstruction-based techniques more deeply, perhaps through iterative refinement guided by the input image's normal map.
• Application to Different Domains: While the paper focuses on object generation, normal maps are also crucial for scene reconstruction or human body modeling. Could Hi3DGen's principles be extended to these more complex scenarios?
• Real-time Inference: The current focus is on high-fidelity generation. Investigating real-time inference capabilities for applications like AR/VR could be a valuable future direction, possibly through model distillation or more efficient architectures.

Personal Insights: The normal bridging concept is a powerful paradigm shift, moving away from trying to infer depth directly from ambiguous RGB to leveraging a more direct geometric cue. It highlights how breaking down a complex problem into intermediate, well-defined sub-problems (image-to-normal, then normal-to-geometry) can lead to more robust and higher-quality solutions. The paper's rigorous methodology and clear demonstration of results make a compelling case for normal maps as a cornerstone for the next generation of high-fidelity 3D generation systems.