1. Bibliographic Information

1.1. Title

ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation

1.2. Authors

Jiahao Chang, Chongjie Ye, Yushuang Wu, Yuantao Chen, Yidan Zhang, Zhongjin Luo, Chenghong Li, Yihao Zhi, Xiaoguang Han.

• Jiahao Chang and Chongjie Ye are marked with an asterisk (*) indicating equal contribution.
• Xiaoguang Han is marked with a dagger (†) indicating corresponding author.
• Affiliations:
  • School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen).
  • The Future Network of Intelligence Institute, CUHK-Shenzhen.

1.3. Journal/Conference

The paper was published at arXiv, a preprint server, with the identifier arXiv:2510.23306. While not a peer-reviewed journal or conference in its current form, arXiv is a widely recognized platform for disseminating early research in fields like computer science, allowing for rapid sharing and feedback within the academic community. The publication date indicates it is a very recent work.

1.4. Publication Year

2025 (Published at UTC: 2025-10-27T13:15:06.000Z)

1.5. Abstract

Existing multi-view 3D object reconstruction methods frequently produce incomplete results due to insufficient overlap between input views and occlusions. While recent diffusion-based 3D generative techniques offer the potential to "hallucinate" missing object parts using learned generative priors, their stochastic nature often leads to inconsistent and inaccurate results, preventing their integration into reconstruction frameworks. This paper analyzes two key reasons for this inconsistency: (a) inadequate cross-view connections during multi-view image feature extraction for conditioning, and (b) poor control over iterative denoising, resulting in plausible but inconsistent fine details. To address these issues, the authors propose ReconViaGen, a framework that innovatively integrates reconstruction priors into a generative framework. It introduces several strategies: building strong cross-view connections, and enhancing the controllability of the denoising process. Extensive experiments demonstrate that ReconViaGen can reconstruct complete and accurate 3D models that are consistent with input views in both global structure and local details.

1.6. Original Source Link

• Original Source Link: https://arxiv.org/abs/2510.23306 (Preprint on arXiv)
• PDF Link: https://arxiv.org/pdf/2510.23306v1.pdf (Version 1 PDF on arXiv)

2. Executive Summary

2.1. Background & Motivation

The core problem ReconViaGen aims to solve is the persistent challenge of achieving complete and accurate 3D object reconstruction from multi-view images, especially when input views are sparse or suffer from occlusions.

In the field of 3D computer vision, multi-view 3D object reconstruction is fundamental for applications in virtual reality (VR), augmented reality (AR), and 3D modeling. However, traditional reconstruction methods, which rely on geometric cues and correspondences between views, often yield incomplete results with holes, artifacts, and missing details, particularly for objects with weak textures or when input captures are sparse.

Recent advancements in diffusion-based 3D generative techniques have emerged as a promising avenue. These methods can "hallucinate" invisible parts of objects by leveraging vast learned generative priors, potentially filling in missing details and improving completeness. However, these generative models face a critical limitation: their stochastic nature during inference leads to significant uncertainty and variability. This makes it difficult to achieve the high accuracy and pixel-level alignment required for precise reconstruction, thus hindering their effective integration with existing multi-view reconstruction frameworks.

The paper identifies two specific reasons for the failure of existing diffusion-based 3D generative methods to achieve high consistency:

1. Insufficient cross-view connections: When extracting multi-view image features as conditions for generation, current methods often fail to adequately model correlations between different views. This leads to inaccuracies in estimating both global geometry and local details.

2. Poor controllability of iterative denoising: The iterative denoising process in diffusion models, especially for local detail generation, can easily produce plausible but geometrically and texturally inconsistent results with the input images.

   The paper's innovative idea, or entry point, is to address these limitations by innovatively integrating strong reconstruction priors into the diffusion-based generative framework. By leveraging the strengths of both reconstruction (for accuracy and consistency with inputs) and generation (for completeness and plausibility), ReconViaGen aims to overcome the individual shortcomings of each paradigm.

2.2. Main Contributions / Findings

The paper's primary contributions are summarized as follows:

• Novel Framework (ReconViaGen): The introduction of ReconViaGen, which is presented as the first framework to integrate strong reconstruction priors into a diffusion-based 3D generator for accurate and complete multi-view object reconstruction. A key design is to aggregate image features rich in reconstruction priors as multi-view-aware diffusion conditions.

• Coarse-to-Fine Generation with Novel Strategies: The generation process adopts a coarse-to-fine paradigm, utilizing global and local reconstruction-based conditions to generate accurate coarse shapes and then fine details in both geometry and texture. This includes:

  • Global Geometry Condition (GGC): Leveraging reconstruction priors for accurate coarse structure generation.
  • Local Per-View Condition (PVC): Providing fine-grained conditioning for detailed geometry and texture generation.
  • Rendering-aware Velocity Compensation (RVC): A novel mechanism proposed to constrain the denoising trajectory of local latent representations, ensuring detailed pixel-level alignment with input images during inference.

• State-of-the-Art (SOTA) Performance: Extensive experiments on the Dora-bench and OmniObject3D datasets validate the effectiveness and superiority of ReconViaGen, demonstrating SOTA performance in both global shape accuracy, completeness, and local details in geometry and textures.

  The key conclusion is that ReconViaGen successfully addresses the long-standing trade-off between completeness (offered by generation) and accuracy/consistency (demanded by reconstruction). By thoughtfully integrating the strengths of both paradigms, the method delivers complete and accurate 3D models that are highly consistent with the provided input views.

3. Prerequisite Knowledge & Related Work

This section aims to provide a comprehensive understanding of the foundational concepts and prior research necessary to grasp the innovations presented in ReconViaGen.

3.1. Foundational Concepts

3.1.1. Multi-view 3D Object Reconstruction

This task involves estimating the 3D shape and appearance of an object from multiple 2D images captured from different viewpoints. The goal is to create a digital 3D model that accurately represents the real-world object.

• Key Challenge: Traditional methods struggle with regions that are occluded or poorly covered by the input views, leading to incomplete models (holes, missing parts). Weak-texture objects also pose difficulties as they lack distinct features for correspondence matching.
• Camera Parameters (Calibration): For accurate reconstruction, the extrinsic (position and orientation) and intrinsic (focal length, principal point, distortion) parameters of the cameras that captured the images are often required. Pose-free methods aim to estimate these parameters alongside the 3D structure.

3.1.2. Diffusion Models

Diffusion models are a class of generative models that learn to reverse a gradual diffusion process.

• Forward Diffusion Process: In this process, random noise is progressively added to a data sample (e.g., an image) over several time steps until the data becomes pure noise. This process is typically fixed and not learned.
• Reverse Denoising Process: The core of a diffusion model is a neural network (often a U-Net or Transformer-based architecture) trained to predict the noise added at each step, or directly predict the "denoised" data point. By iteratively removing predicted noise, the model can transform pure noise back into a coherent data sample. This reverse process is learned.
• Stochasticity: The reverse denoising process often involves sampling from a distribution, making it inherently stochastic. This means running the same diffusion process twice with the same starting noise might yield slightly different (though plausible) results. This stochasticity is a source of inconsistency in generative models.
• Conditional Generation: Diffusion models can be conditioned on additional information (e.g., text, class labels, or in this paper, image features) to guide the generation process towards specific outputs. This is typically done by integrating the conditioning information into the neural network (e.g., via cross-attention layers).

3.1.3. 3D Generative Techniques

These are methods that use generative models (like diffusion models) to create 3D content.

• "Hallucination": A key capability of generative models is to "hallucinate" (i.e., plausibly generate) parts of an object that were never seen in the input, based on their learned understanding of common object structures.
• 3D Representations: 3D generative models can output various 3D representations:
  • Point Clouds: A collection of 3D points representing the surface of an object.
  • Voxel Grids: A 3D grid where each cell (voxel) indicates whether it's occupied by the object or empty.
  • Meshes: A collection of vertices, edges, and faces that define the surface of a 3D object.
  • Neural Radiance Fields (NeRFs): A continuous volumetric function that maps a 3D coordinate and viewing direction to an emitted color and volume density. It's rendered by ray marching.
  • 3D Gaussian Splatting (3DGS): A recently popular explicit 3D representation where a scene is represented by a set of 3D Gaussians, each with properties like position, scale, rotation, color, and opacity.
  • Triplanes: A compact 3D representation that projects 3D features onto three orthogonal 2D planes, which can then be queried to reconstruct 3D information.
  • Structured Latents (SLAT): A representation proposed by TRELLIS that combines sparse 3D grids with dense visual features, capturing both geometric and textural information, and can be decoded into various 3D outputs.

3.1.4. LoRA (Low-Rank Adaptation)

LoRA is a parameter-efficient fine-tuning technique for large pre-trained models, particularly Transformers. Instead of fine-tuning all model parameters, LoRA injects trainable low-rank matrices into the Transformer layers (typically into the query, key, and value projection matrices of attention blocks). During fine-tuning, the original pre-trained weights remain frozen, and only these much smaller low-rank matrices are updated. This significantly reduces the number of trainable parameters and computational cost, making fine-tuning more efficient while often maintaining high performance.

3.1.5. Transformer Architecture

Transformers are neural network architectures that rely heavily on self-attention mechanisms.

• Self-Attention: A mechanism that allows the model to weigh the importance of different parts of the input sequence when processing a specific element. It computes attention scores between all pairs of elements in a sequence, determining how much "attention" each element should pay to others.
• Cross-Attention: Similar to self-attention, but it allows elements from one sequence (e.g., the query from the latent representation) to attend to elements from a different sequence (e.g., the conditioning features). This is crucial for integrating external conditions into generative models.
• Transformer Blocks: Composed of multiple layers, typically including multi-head attention (self-attention or cross-attention) and feed-forward neural networks, often with skip connections and layer normalization.

3.1.6. Rectified Flow Transformers and Conditional Flow Matching (CFM)

These concepts are related to a specific type of generative model called "Rectified Flows."

• Rectified Flows: A generative modeling approach that aims to learn a straight-line path (a "rectified flow") between a simple noise distribution and a complex data distribution. This offers advantages in terms of faster sampling and potentially more stable training compared to traditional diffusion models.
• Conditional Flow Matching (CFM): The objective function used to train rectified flow models. It trains a vector field $\pmb{v}_\theta(x, t)$ to predict the "velocity" at which a data point $x$ at time $t$ should move to reach the target data distribution. The goal is to match the predicted velocity field to the true optimal transport path between the noise and data distributions.

3.2. Previous Works

The paper builds upon and differentiates itself from several lines of prior research.

3.2.1. Strong Reconstruction Prior: VGGT (Visual Geometry Grounded Transformer)

• Description: VGGT (Wang et al., 2025a) is a SOTA (State-of-the-Art) method for pose-free multi-view 3D reconstruction. It uses a feedforward transformer architecture to reconstruct a 3D scene from single or multiple images. Crucially, it predicts not only 3D structure (like point clouds) but also camera parameters and depth maps without requiring pre-calibrated cameras (hence "pose-free").
• Mechanism: It processes multi-view images using a DINO-based ViT (Vision Transformer from Oquab et al., 2024) for feature extraction. These features then pass through self-attention layers that balance local and global information, enhancing multi-view consistency. Finally, prediction heads decode these 3D-aware features into camera parameters, depth maps, point maps, and tracking features.
• Relevance to ReconViaGen: ReconViaGen fine-tunes VGGT to create a powerful "reconstruction prior" that provides explicit, multi-view-aware features ($\phi_{vggt}$) used for conditioning the 3D generator. This is a critical component for ReconViaGen to understand the geometry and texture of the input object.

3.2.2. Strong Generation Prior: TRELLIS (Structured 3D Latents)

• Description: TRELLIS (Xiang et al., 2024) is a SOTA 3D generative model. It introduces a novel representation called Structured LATent (SLAT), which combines a sparse 3D grid with dense visual features. This representation is designed to capture both geometric (structure) and textural (appearance) information and can be decoded into various 3D representations (e.g., radiance fields, 3D Gaussians, meshes).
• Mechanism: TRELLIS employs a coarse-to-fine two-stage generation pipeline:
  1. SS Flow (Sparse Structure Flow): Generates a coarse structure, represented as sparse voxels.
  2. SLAT Flow: Predicts the SLAT representation for the active (occupied) sparse voxels, adding fine details. Both stages use rectified flow transformers conditioned on DINO-encoded image features.
• Relevance to ReconViaGen: ReconViaGen builds upon TRELLIS as its core 3D generator. It modifies TRELLIS by replacing the original DINO-encoded image features with its own reconstruction-conditioned features (Global Geometry Condition and Local Per-View Condition) to guide the coarse-to-fine generation process, making it multi-view-aware and more accurate.

3.2.3. Other Related Categories

• Single-view 3D Generation:

  • 2D prior-based: Methods like DreamFusion (Poole et al., 2022) distill 3D knowledge from pre-trained 2D diffusion models. Others develop multi-view diffusion from 2D image/video generators and fuse views for 3D (Li et al., 2023).
  • 3D native generative: Methods that use diffusion directly on 3D representations like point clouds (Luo & Hu, 2021), voxel grids (Hui et al., 2022), Triplanes (Chen et al., 2023), or 3D Gaussians (Zhang et al., 2024a). More recently, 3D latent diffusion (Zhang et al., 2023) directly learns the mapping between images and 3D geometry.
  • Differentiation: These methods often suffer from high variation, inconsistency with input images, or strong reliance on input viewpoints, limiting their use in accurate 3D reconstruction. ReconViaGen aims for accuracy and consistency from multi-view inputs.

• Multi-view 3D Reconstruction:

  • Traditional MVS: Methods like Furukawa et al. (2015) and Schönberger et al. (2016) triangulate correspondences to reconstruct visible surfaces.
  • Learning-based MVS: $Yao et al. (2018)$ and Chen et al. (2019) use deep learning to improve MVS.
  • NeRF-based: Mildenhall et al. (2020) and Wang et al. (2021b) optimize camera parameters and radiance fields from dense views.
  • Large Reconstruction Models: DUSt3R (Wang et al., 2024a) and its successors (Leroy et al., 2024; Wang et al., 2025a) estimate point clouds and camera poses, but often result in incomplete reconstructions due to their representation. Newer models (Hong et al., 2023) output compact 3D representations (3D Gaussians, Triplane). Pose-free variants (Wu et al., 2023a) tend to predict smooth, blurred details, especially in invisible regions.
  • Differentiation: These methods generally excel at consistency but inherently struggle with completeness when views are sparse or occluded. ReconViaGen addresses this completeness issue by introducing generative priors while maintaining consistency.

• Generative Priors in 3D Object Reconstruction:

  • Diffusion-based 2D generative prior: Often used in single-view 3D reconstruction by generating plausible multi-view images first, then reconstructing (Li et al., 2023). For sparse-view, iFusion (Wu et al., 2023a) uses Zero123 predictions.
  • Regression-based 3D generative prior: Directly regresses a compact 3D representation (Jiang et al., 2023 for neural volume, Hong et al., 2023 for Triplane, He et al., 2024 for 3D Gaussians).
  • Diffusion-based 3D generative prior: One2-3-45++ (Liu et al., 2024) develops 3D volume diffusion.
  • Differentiation: Diffusion-based generative priors are noted for generating more detailed results than regression-based ones. However, 3D volume diffusion can suffer from poor compactness and representation capability. ReconViaGen uses strong diffusion-based 3D generative priors (TRELLIS) but constrains the denoising process with powerful reconstruction priors (VGGT) to achieve high-fidelity details and accuracy, overcoming the consistency issues of other generative prior approaches.

3.3. Technological Evolution

The field of 3D reconstruction has evolved from classic Structure-from-Motion (SfM) and Multi-View Stereo (MVS) methods, relying on geometric correspondences and optimization, to learning-based approaches using neural networks. Initially, these learning methods focused on improving MVS depth estimation or inferring NeRFs. However, they still struggled with missing data.

Concurrently, 3D generative modeling matured, especially with the advent of diffusion models. Early 3D generation often relied on 2D image priors, generating multi-view images that were then lifted to 3D, or directly generated 3D representations like point clouds or voxels. More advanced models, like TRELLIS, developed compact and expressive latent 3D representations and coarse-to-fine generation pipelines, greatly improving output quality.

The challenge remained at the intersection: how to combine the "hallucination" power of generative models with the accuracy and consistency requirements of reconstruction. Previous attempts integrated generative priors, but often struggled with pixel-level alignment or consistency.

ReconViaGen fits into this timeline by representing a significant step towards tightly integrating these two paradigms. It moves beyond simply using generative models to "fill in" after reconstruction or merely using 2D priors to guide 3D generation. Instead, it embeds strong, multi-view-aware reconstruction knowledge directly into the conditioning and denoising process of a powerful 3D diffusion model, aiming for a unified solution that is both complete and accurate.

3.4. Differentiation Analysis

Compared to the main methods in related work, ReconViaGen introduces several core differences and innovations:

• Integration Philosophy: Unlike methods that either perform pure reconstruction (e.g., VGGT, traditional MVS) or pure generation (e.g., TRELLIS, other 3D diffusion models), ReconViaGen explicitly and deeply integrates strong reconstruction priors into a diffusion-based generative framework. This is its most fundamental differentiation.

  • Pure reconstruction: High consistency with inputs, but severe incompleteness with sparse views.
  • Pure generation: High completeness, but strong inconsistency with inputs due to stochasticity.
  • ReconViaGen: Aims for both completeness (via generation) and accuracy/consistency (via reconstruction priors).

• Multi-view-aware Conditioning: ReconViaGen addresses the insufficiency in constructing cross-view correlations (a key identified problem) by deriving global and local conditions from a powerful multi-view reconstructor (VGGT).

  • Existing generative models (e.g., TRELLIS, Hunyuan3D-2.0-mv) often use DINO features or concatenate features, which may not adequately capture multi-view consistency or 3D geometric relationships.
  • ReconViaGen uses VGGT's 3D-aware features, aggregated into Global Geometry Condition (GGC) and Local Per-View Condition (PVC) tokens, providing richer, more structured 3D information.

• Controllable Denoising with Explicit Alignment: To tackle the poor controllability of iterative denoising and inconsistency (the second key problem), ReconViaGen introduces Rendering-aware Velocity Compensation (RVC).

  • Other generative methods typically rely on implicit guidance or optimization that might struggle with pixel-level precision.
  • RVC explicitly uses rendering from the intermediate generated 3D model and pixel-level comparison with input images to correct the diffusion process's velocity during inference. This provides a direct, strong constraint for fine-grained, pixel-aligned consistency.

• Coarse-to-Fine Strategy with Differentiated Conditioning: The method applies its reconstruction priors in a principled coarse-to-fine manner:

  • GGC guides the SS Flow (coarse structure), leveraging global 3D understanding.

  • PVC guides the SLAT Flow (fine details), using view-specific appearance information. This hierarchical conditioning is tailored to the distinct needs of coarse vs. fine generation stages.

    In essence, ReconViaGen innovates by creating a feedback loop and a stronger conditioning pipeline where the geometric understanding from a robust reconstructor directly informs and controls the expressive power of a 3D generative model, thereby achieving a superior balance of completeness, accuracy, and consistency.

4. Methodology

The ReconViaGen framework is designed to overcome the limitations of existing multi-view 3D object reconstruction methods (incompleteness) and diffusion-based 3D generative techniques (inconsistency). It achieves this by innovatively integrating strong reconstruction priors into a diffusion-based generative framework. The overall process is coarse-to-fine, leveraging VGGT for reconstruction priors and TRELLIS as the base 3D generator, with novel conditioning and refinement strategies.

4.1. Principles

The core idea behind ReconViaGen is to merge the strengths of 3D reconstruction and 3D generation.

• Reconstruction priors (derived from a state-of-the-art reconstructor like VGGT) provide accurate geometric and textural information, ensuring consistency with the input views and a robust understanding of the visible parts of the object.

• Diffusion-based 3D generative priors (from TRELLIS) enable the "hallucination" of invisible or occluded parts, ensuring completeness and plausibility of the reconstructed 3D model.

  The method addresses two key issues identified in previous work:

1. Insufficient cross-view connections: ReconViaGen constructs and leverages strong cross-view connections by aggregating VGGT's 3D-aware features into global and local conditioning tokens.

2. Poor controllability of iterative denoising: This is tackled by introducing a Rendering-aware Velocity Compensation (RVC) mechanism that explicitly guides the denoising trajectory towards pixel-level alignment with input images during inference.

   The framework operates in a three-stage coarse-to-fine pipeline:

3. Reconstruction-based Multi-view Conditioning: A fine-tuned VGGT extracts reconstruction-aware features, which are then aggregated into global and local token lists.

4. Coarse-to-Fine 3D Generation: A TRELLIS-based generator uses these conditions to first estimate a coarse 3D structure and then generate fine textured details.

5. Rendering-aware Pixel-aligned Refinement: During inference, a novel RVC mechanism refines the generated results by enforcing pixel-level consistency with input views using estimated camera poses.

   The overall architecture is illustrated in Figure 2:

   该图像是论文ReconViaGen中图2的示意图，展示了该框架如何融合基于重建的条件信息与基于3D扩散生成的先验，通过多视图输入，逐步生成完整且纹理一致的三维模型。

Figure 2: An overview illustration of the proposed ReconViaGen framework, which integrates strong reconstruction priors with 3D diffusion-based generation priors for accurate reconstruction at both the global and local level.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Preliminary: Leveraging Strong Priors

ReconViaGen builds upon two state-of-the-art models: VGGT for reconstruction priors and TRELLIS for generation priors.

4.2.1.1. Reconstruction Prior of VGGT

The paper utilizes VGGT (Visual Geometry Grounded Transformer) by Wang et al. (2025a), a SOTA method for pose-free multi-view 3D reconstruction. It provides a powerful reconstruction prior by estimating 3D structure and camera parameters without requiring pre-calibrated inputs.

• Input Processing: Multi-view images $I = \{I_i\}_{i=1}^N$ are simultaneously fed into a DINO-based ViT (Vision Transformer, Oquab et al., 2024) for tokenization and feature extraction, resulting in $\phi_{\mathrm{dino}}$.
• 3D-aware Feature Learning: These DINO features are then processed by 24 self-attention layers within the Transformer architecture. These layers switch between frame-wise and global self-attention to effectively balance local (per-image) and global (cross-image) information, enhancing multi-view consistency. This process produces a set of 3D-aware features, $\{ \phi_i \}_{i=1}^{24}$.
• Prediction Heads: Four prediction heads decode the output of specific layers (4th, 11th, 17th, and 23rd) of the Transformer into:
  • Camera parameters (poses).
  • Depth map.
  • Point map.
  • Tracking feature predictions. The aggregated VGGT features used for conditioning are denoted as ${\bar{\phi}_{\mathrm{vggt}}(I)} \stackrel{.}{=} \{ \phi_4, \stackrel{.}{\phi}_{11}, \phi_{17}, \phi_{24} \}$.
• Fine-tuning for Object Reconstruction: To adapt VGGT specifically for object reconstruction, it is fine-tuned on an object-reconstruction dataset. A LoRA (Low-Rank Adaptation) fine-tuning approach is applied to the VGGT aggregator. LoRA is a parameter-efficient technique where small, trainable low-rank matrices are injected into the pre-trained model's attention layers, allowing for efficient adaptation without modifying all original weights. This preserves the pre-trained 3D geometric priors.
• Multi-task Objective: The fine-tuning of VGGT uses a multi-task objective function: $$\mathcal { L } _ { \operatorname { V G G T } } ( \theta ) = \mathcal { L } _ { \operatorname { c a m e r a } } + \mathcal { L } _ { \operatorname { d e p t h } } + \mathcal { L } _ { \operatorname { n m a p } }$$ Where:
  • $\theta$: Represents the LoRA parameters being optimized during fine-tuning.
  • $\mathcal { L } _ { \operatorname { c a m e r a } }$: Denotes the loss associated with the accuracy of predicted camera poses.
  • $\mathcal { L } _ { \operatorname { d e p t h } }$: Represents the loss for the accuracy of the predicted depth maps.
  • $\mathcal { L } _ { \operatorname { n m a p } }$: Corresponds to the loss for the accuracy of the point map predictions. (Note: The paper simplifies "this fine-tuned VGGT" to "VGGT" in subsequent text.)

4.2.1.2. Generation Prior of TRELLIS

TRELLIS (Xiang et al., 2024) is adopted as the SOTA 3D generative model providing the strong generation prior. It introduces a Structured LATent (SLAT) representation, which combines a sparse 3D grid with dense visual features, capturing both geometry and texture. This representation allows for decoding into multiple 3D representations (e.g., radiance fields, 3D Gaussians, or meshes).

• Coarse-to-Fine Pipeline: TRELLIS uses a two-stage generation process:
  1. SS Flow (Sparse Structure Flow): First estimates a coarse 3D structure, represented as sparse voxels $\{p_i\}_i^L$.
  2. SLAT Flow: Then predicts the detailed SLAT representation, $X = \{(p_i, x_i)\}_i^V$, for these active (occupied) sparse voxels. Here, $p_i$ is the voxel position, $x_i$ is the latent vector associated with that voxel, and $V$ is the number of voxels.
• Transformer Architecture and Conditioning: Both SS Flow and SLAT Flow employ rectified flow transformers (Liu et al., 2022). In the original TRELLIS, these transformers are conditioned on DINO-encoded image features. The output SLAT representation $X$ is then decoded into a 3D output $O$ (e.g., a mesh): $O = \operatorname{Dec}(X)$.
• Conditional Flow Matching (CFM) Objective: The backward (denoising) process in TRELLIS is modeled as a time-dependent vector field $\pmb{v}(x, t) = \nabla_t(x)$. The transformers, denoted as $\pmb{v}_\theta$, in both generation stages are trained by minimizing the CFM objective (Lipman et al., 2023): $$\mathcal { L } _ { \mathrm { CFM } } ( \theta ) = \mathbb { E } _ { t , x _ { 0 } , \epsilon } \Vert \pmb { v } _ { \theta } ( x , t ) - ( \epsilon - x _ { 0 } ) \Vert _ { 2 } ^ { 2 }$$ Where:
  • $\theta$: Represents the parameters of the transformer model $\pmb{v}_\theta$.
  • $\mathbb { E } _ { t , x _ { 0 } , \epsilon }$: Denotes the expectation over time $t$, initial data samples $x_0$, and noise $\epsilon$.
  • $\pmb { v } _ { \theta } ( x , t )$: Is the learned vector field (predicted velocity) by the transformer, which takes the noisy data $x$ at time $t$ as input.
  • $\epsilon$: Represents the random noise sampled from a Gaussian distribution.
  • $x_0$: Is the clean, uncorrupted data sample (the target SLAT or sparse voxel configuration).
  • The term $(\epsilon - x_0)$ represents the target velocity derived from the flow matching theory (the direction from $x_0$ to $\epsilon$ scaled by time). The objective minimizes the L2 distance between the predicted velocity and this target velocity.

4.2.2. Reconstruction-based Conditioning

To effectively integrate VGGT's reconstruction priors into TRELLIS's generative framework, ReconViaGen devises a method to extract multi-view-aware conditions at both global and local levels. This addresses the "insufficiency in constructing cross-view correlations."

4.2.2.1. Global Geometry Condition (GGC)

The VGGT features ($\phi_{\mathrm{vggt}}$) inherently contain rich 3D lifting information. To leverage this for accurate coarse structure generation (i.e., for SS Flow), these features are aggregated into a single, fixed-length global token list $T_g$. This is achieved via a dedicated Condition Net.

• Process: The Condition Net starts with a randomly initialized learnable token list, $T_{\mathrm{init}}$. It then progressively fuses the layer-wise features of $\phi_{\mathrm{vggt}}$ with this initial token list using four transformer cross-attention blocks.
• Formulation: $$T ^ { i + 1 } = \mathrm { C r o s s A t t n } \big ( Q ( T ^ { i } ) , K ( \phi _ { \mathrm { v g g t } } ) , V ( \phi _ { \mathrm { v g g t } } ) \big ) , i \in \{ 0 , 1 , 2 , 3 \}$$ Where:
  • $T^i$: Represents the token list at the $i$-th iteration of the Condition Net.
  • $T^0$: Is initialized with $T_{\mathrm{init}}$.
  • $T^3$: Is the final output, which is the global geometry condition $T_g$.
  • $\mathrm { C r o s s A t t n } (\cdot, \cdot, \cdot)$: Denotes a cross-attention operation.
  • $Q(\cdot)$, $K(\cdot)$, $V(\cdot)$: Are linear layers that project their inputs into query, key, and value representations, respectively.
  • $\phi_{\mathrm{vggt}}$: Are the aggregated VGGT features. These features concatenate all views along the token dimension, ensuring a holistic representation of the object's geometry from multiple perspectives.
• Training: During the training of SS Flow, the VGGT layers are frozen, and the Condition Net is trained alongside the DiT (Diffusion Transformer, a component of TRELLIS).

4.2.2.2. Local Per-View Condition (PVC)

While GGC provides global context, a single token list might not be sufficient for fine-grained geometry and texture generation. To address this, ReconViaGen further generates local per-view tokens $\{T_k\}_{k=1}^N$, which serve as conditions for SLAT Flow to produce detailed results.

• Process: Similar to GGC, a separate Condition Net is used. For each view $k$, a random token list is initialized, and then fed into the Condition Net. This network fuses view-specific VGGT features ($\phi_k^{\mathrm{vggt}}$) with the token list.
• Formulation: $$T _ { k } ^ { i + 1 } = \mathrm { C r o s sA t t n } \big ( Q ( T _ { k } ^ { i } ) , K ( \phi _ { k } ^ { \mathrm { v g g t } } ) , V ( \phi _ { k } ^ { \mathrm { v g g t } } ) \big ) , i \in \{ 0 , 1 , 2 , 3 \} \mathrm { ~ a n d ~ } k \in \{ n \} _ { n = 1 } ^ { N }$$ Where:
  • $T_k^i$: Is the view-specific token list at the $i$-th iteration for view $k$.
  • $\phi_k^{\mathrm{vggt}}$: Are the VGGT features specifically for the $k$-th input view.
  • The final output $\{T_k\}_{k=1}^N$ is a set of view-specific token lists, which are then used to condition the diffusion process in SLAT Flow.

4.2.3. Coarse-to-Fine Generation

The overall generation process in ReconViaGen follows TRELLIS's coarse-to-fine paradigm but is heavily influenced by the reconstruction-based conditioning and a novel inference-time refinement.

4.2.3.1. Reconstruction-conditioned Flow

The SS Flow and SLAT Flow transformers within TRELLIS are adapted to incorporate the global and local reconstruction priors.

• SS Flow (Coarse Structure Generation):

  • The Global Geometry Condition $T_g$ (from Section 4.2.2.1) is used.
  • In each DiT (Diffusion Transformer) block of the SS Flow, cross-attention is computed between the noisy Sparse Structure (SS) latent representation and $T_g$. This guides the generation of a more accurate coarse 3D shape, informed by the holistic geometric understanding from VGGT.

• SLAT Flow (Fine Detail Generation):

  • The Local Per-View Conditions $\{T_k\}_{k=1}^N$ (from Section 4.2.2.2) are used.
  • In each DiT block of the SLAT Flow, cross-attention is computed between the noisy Structured LATent (SLAT) representation and each view's condition $T_k$.
  • A weighted fusion mechanism then combines the results from these multiple cross-attention operations. This allows the SLAT Flow to integrate fine-grained, view-specific geometric and textural details.

• Weighted Fusion Formulation: $$y _ { j + 1 } = \sum _ { k = 1 } ^ { N } \mathrm { C r o s sA t t n } \big ( Q ( y _ { j } ^ { \prime } ) , K ( T _ { k } ) , V ( T _ { k } ) \big ) \cdot w _ { k } , \ j \in \{ m \} _ { m = 1 } ^ { M }$$ Where:

  • $y_j$: Represents the noisy SLAT input at step $j$.
  • $y_j^\prime$: Is the output of the self-attention layer for the noisy SLAT input $y_j$.
  • $M$: Is the total number of SLAT DiT blocks.
  • $\mathrm { C r o s sA t t n } (\cdot, \cdot, \cdot)$: A cross-attention operation. Here, the query comes from the SLAT ($Q(y_j^\prime)$), and the keys/values come from the $k$-th view's local condition ($K(T_k), V(T_k)$).
  • $w_k$: Is a fusion weight, a scalar between 0 and 1, computed for each view $k$ by a small MLP (Multi-Layer Perceptron) that takes the corresponding cross-attention result as input. This weighting allows the model to dynamically prioritize information from different views during detail generation.
  • The summation combines the view-conditioned cross-attention outputs, creating a rich, multi-view-aware context for refining the SLAT.

4.2.3.2. Rendering-aware Velocity Compensation (RVC)

To further ensure pixel-aligned consistency between the generated 3D model and the input views, a novel Rendering-aware Velocity Compensation (RVC) mechanism is introduced, which operates exclusively during the inference stage. This addresses the "poor controllability and stability of the denoising process."

• Motivation: The SLAT Flow denoises a large number of noisy latents simultaneously, which is a complex collaborative optimization problem. RVC aims to correct the predicted denoising velocity.
• Camera Pose Estimation: First, accurate camera poses $C$ for the input images are estimated using VGGT and further refined. The refinement process (detailed in Appendix A.1) involves:
  1. Rendering 30 images from random views, concatenating them with input images, and feeding into VGGT for coarse pose estimation.
  2. Using these coarse poses to render intermediate images/depth maps from the partially generated 3D model.
  3. Applying an image matching method to find 2D-2D correspondences between rendered and input images.
  4. Leveraging depth maps and camera parameters to establish 2D-3D correspondences.
  5. Solving for refined camera poses $C$ using a PnP (Perspective-n-Point) solver with RANSAC (Random Sample Consensus). This step significantly improves pose accuracy, enabling pixel-level constraints.
• Velocity Correction: When the diffusion time step $t$ is less than 0.5 (i.e., in the later stages of denoising where details are refined):
  1. The current noisy SLAT is decoded into an intermediate 3D object $O_t$ (e.g., a textured mesh).
  2. Images are rendered from $O_t$ using the refined camera pose estimations $C$.
  3. A difference (loss) is calculated between these rendered images and the actual input images. This loss guides the correction of the predicted velocity.
• Rendering-aware Compensation Loss: The loss used for this compensation is a combination of several image similarity metrics: $$\mathcal { L } _ { \mathrm { RVC } } ( v _ { t } ) = \mathcal { L } _ { \mathrm { SSIM } } + \mathcal { L } _ { \mathrm { LPIPS } } + \mathcal { L } _ { \mathrm { DreamSim } }$$ Where:
  • $\mathcal { L } _ { \mathrm { RVC } } ( v _ { t } )$: The total Rendering-aware Velocity Compensation loss, calculated based on the current predicted velocity $v_t$.
  • $\mathcal { L } _ { \mathrm { SSIM } }$: Structural Similarity Index Measure (Wang et al., 2004), which measures structural similarity between images.
  • $\mathcal { L } _ { \mathrm { LPIPS } }$: Learned Perceptual Image Patch Similarity (Zhang et al., 2018), which uses deep features to assess perceptual similarity.
  • $\mathcal { L } _ { \mathrm { DreamSim } }$: DreamSim loss (Fu et al., 2023), which measures semantic similarity.
  • To exclude images with potentially inaccurate pose estimations, losses corresponding to images with a similarity loss higher than 0.8 are discarded.
• Calculating Velocity Correction Term ($\Delta v_t$): This loss is used to compute a compensation term $\Delta v_t$ that corrects the predicted velocity: $$\Delta v _ { t } = \frac { \partial \mathcal { L } } { \partial \hat { x _ { 0 } } } \frac { \partial \hat { x _ { 0 } } } { \partial v _ { t } } = - t \frac { \partial \mathcal { L } } { \partial \hat { x _ { 0 } } }$$ Where:
  • $\Delta v_t$: The velocity compensation term at time step $t$.
  • $\mathcal { L }$: Represents the $\mathcal { L } _ { \mathrm { RVC } }$ for simplicity.
  • $\hat { x _ { 0 } }$: Is the predicted target SLAT (the clean data) at the current time step $t$, which is estimated as $\hat { x _ { 0 } } = x _ { t } - t \cdot v _ { t }$ (where $x_t$ is the noisy SLAT at time $t$, and $v_t$ is the velocity predicted by the DiT model).
  • This formula essentially backpropagates the image-level rendering loss to find how much the predicted underlying clean SLAT needs to change, and then converts that into a correction for the velocity.
• Updating Noisy SLAT: The noisy SLAT for the next time step, $x _ { t _ { \mathrm { p r e v } } }$, is then updated using the corrected velocity: $$x _ { t _ { \mathrm { p r e v } } } = x _ { t } - ( t - t _ { \mathrm { p r e v } } ) ( v + \alpha \cdot \Delta v )$$ Where:
  • $x _ { t _ { \mathrm { p r e v } } }$: The noisy SLAT for the previous (next) time step in the denoising process.
  • $x_t$: The current noisy SLAT.
  • $t$ and $t_{\mathrm{prev}}$: Current and previous time steps.
  • $v$: The velocity predicted by the DiT model.
  • $\alpha$: A predefined hyperparameter (set to 0.1 in practice) that controls the extent or strength of the compensation. By applying this RVC, the input images provide strong, explicit guidance, ensuring that the denoising trajectory for each local SLAT vector leads to highly accurate 3D results that are consistent with all input images in detail.

5. Experimental Setup

5.1. Datasets

The experiments used two primary datasets for training and evaluation:

• Objaverse Deitke et al. (2024): This large-scale 3D object dataset was used for fine-tuning the LoRA adapter of the VGGT aggregator and the TRELLIS sparse structure transformer.

  • Scale: Contains 390,000 3D data samples (likely referring to the Objaverse-XL version with over 10 million objects).
  • Characteristics: Provides a rich variety of shapes and textures.
  • Training Data Preparation: For each object mesh, 60 views were rendered from various camera positions to serve as input for fine-tuning. For evaluation, 150 view images were rendered per object at a resolution of $512 \times 512$ under uniform lighting conditions, following the setup of original TRELLIS work.

• Dora-Bench Chen et al. (2024): A benchmark dataset used for thorough evaluation of the model's performance.

  • Source: Combines 3D data selected from Objaverse (Deitke et al., 2023), ABO (Collins et al., 2022), and GSO (Downs et al., 2022) datasets.
  • Characteristics: Organized into 4 levels of complexity.
  • Evaluation Sampling: 300 objects were randomly sampled from Dora-Bench for evaluation.
  • View Settings: 40 views were rendered following TRELLIS's camera trajectory, and 4 specific views (No. 0, 9, 19, and 29) with a uniform interval were chosen as multi-view input to align with baseline methods like LGM and InstantMesh.

• OmniObject3D Wu et al. (2023b): Another large-vocabulary 3D object dataset used for evaluation.

  • Scale: Contains 6,000 high-quality textured meshes.

  • Characteristics: Scanned from real-world objects, covering 190 daily categories.

  • Evaluation Sampling: 200 objects covering 20 categories were randomly sampled.

  • View Settings: 24 views were rendered at different elevations, and 4 of these were randomly chosen as multi-view input for evaluation.

    These datasets were chosen because they provide diverse and challenging 3D object data, suitable for evaluating both generative capabilities (completeness, plausibility) and reconstruction accuracy (fidelity to input views). The specific rendering strategies ensure fair comparisons with various baseline methods.

5.2. Evaluation Metrics

The performance of ReconViaGen was evaluated using a combination of image-based metrics (for novel view synthesis quality) and 3D geometry-based metrics (for shape accuracy and completeness).

5.2.1. PSNR (Peak Signal-to-Noise Ratio)

• Conceptual Definition: PSNR is an engineering metric that quantifies the difference between two images. It is commonly used to measure the quality of reconstruction of lossy compression codecs or, in this context, the fidelity of synthesized novel views to ground-truth images. A higher PSNR value indicates higher similarity (lower distortion) between the reconstructed image and the original.
• Mathematical Formula: $ \mathrm{MSE} = \frac{1}{MN} \sum_{i=0}^{M-1}\sum_{j=0}^{N-1} [I(i,j) - K(i,j)]^2 $ $ \mathrm{PSNR} = 10 \cdot \log_{10} \left( \frac{\mathrm{MAX}_I^2}{\mathrm{MSE}} \right) $
• Symbol Explanation:
  • I(i,j): The pixel value at coordinates (i,j) in the original image.
  • K(i,j): The pixel value at coordinates (i,j) in the reconstructed (synthesized) image.
  • M, N: The dimensions (width and height) of the image.
  • $\mathrm{MSE}$: Mean Squared Error, the average of the squared differences between the pixels of the two images.
  • $\mathrm{MAX}_I$: The maximum possible pixel value of the image (e.g., 255 for an 8-bit grayscale image, or 1.0 for normalized floating-point images).
  • $\log_{10}$: The base-10 logarithm.

5.2.2. SSIM (Structural Similarity Index Measure)

• Conceptual Definition: SSIM is a perceptual metric that evaluates the similarity between two images based on three comparison components: luminance, contrast, and structure. Unlike PSNR which measures absolute error, SSIM aims to better reflect human visual perception of image quality. Values range from -1 to 1, where 1 indicates perfect similarity.
• Mathematical Formula: $ \mathrm{SSIM}(x,y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)} $
• Symbol Explanation:
  • x, y: Two image patches being compared.
  • $\mu_x$: The mean of image patch $x$.
  • $\mu_y$: The mean of image patch $y$.
  • $\sigma_x^2$: The variance of image patch $x$.
  • $\sigma_y^2$: The variance of image patch $y$.
  • $\sigma_{xy}$: The covariance of image patches $x$ and $y$.
  • $c_1 = (K_1L)^2$, $c_2 = (K_2L)^2$: Small constants to avoid division by zero or near-zero values. $L$ is the dynamic range of the pixel values (e.g., 255 for 8-bit grayscale), and $K_1, K_2 \ll 1$.

5.2.3. LPIPS (Learned Perceptual Image Patch Similarity)

• Conceptual Definition: LPIPS is a metric that measures the perceptual similarity between two images by comparing their features extracted from a pre-trained deep neural network (e.g., AlexNet, VGG). It correlates better with human judgment of similarity than traditional metrics like PSNR or SSIM, as it captures more semantic and stylistic differences. A lower LPIPS value indicates higher perceptual similarity.
• Mathematical Formula: $ \mathrm{LPIPS}(\mathbf{x}, \mathbf{x}0) = \sum_l \frac{1}{H_l W_l} \sum{h,w} | w_l \odot (\phi_l(\mathbf{x})_{h,w} - \phi_l(\mathbf{x}0){h,w}) |_2^2 $
• Symbol Explanation:
  • $\mathbf{x}$: The original image.
  • $\mathbf{x}_0$: The reconstructed (synthesized) image.
  • $l$: Index referring to a specific layer in the pre-trained feature extractor network.
  • $\phi_l(\cdot)$: The feature map extracted from layer $l$ of the pre-trained network.
  • $H_l, W_l$: Height and width of the feature map at layer $l$.
  • $w_l$: A learnable vector that scales the differences at each channel of the feature map at layer $l$.
  • $\odot$: Element-wise multiplication.
  • $\|\cdot\|_2^2$: The squared L2 norm, measuring the Euclidean distance between feature vectors.

5.2.4. Chamfer Distance (CD)

• Conceptual Definition: Chamfer Distance is a metric used to quantify the similarity between two point clouds (or, more generally, two sets of points). It measures the average closest-point distance between the points of one set to the other, and vice-versa. A lower CD indicates higher geometric similarity.
• Mathematical Formula: $ \mathrm{CD}(S_1, S_2) = \frac{1}{|S_1|} \sum_{x \in S_1} \min_{y \in S_2} |x-y|2^2 + \frac{1}{|S_2|} \sum{y \in S_2} \min_{x \in S_1} |x-y|_2^2 $
• Symbol Explanation:
  • $S_1, S_2$: The two point clouds being compared.
  • $|S_1|, |S_2|$: The number of points in point cloud $S_1$ and $S_2$, respectively.
  • $x$: A point in $S_1$.
  • $y$: A point in $S_2$.
  • $\min_{y \in S_2} \|x-y\|_2^2$: The squared Euclidean distance from point $x$ to its nearest neighbor in $S_2$.
  • $\min_{x \in S_1} \|x-y\|_2^2$: The squared Euclidean distance from point $y$ to its nearest neighbor in $S_1$.

5.2.5. F-score

• Conceptual Definition: The F-score (also known as F1-score or F-measure) is a metric that combines precision and recall, often used in information retrieval and classification tasks. In 3D geometry, it's adapted to measure the quality of a reconstructed 3D shape against a ground truth. It considers both how much of the ground truth is captured by the reconstruction (recall) and how much of the reconstruction actually belongs to the ground truth (precision). A higher F-score indicates better overall geometric accuracy and completeness. The paper defines it based on a given radius $r$.
• Mathematical Formula: (Adapted for 3D shapes, typically based on point-to-surface distances) Given two point clouds, $S_{pred}$ (predicted) and $S_{gt}$ (ground truth), and a threshold distance $r$: $ \mathrm{Precision}(S_{pred}, S_{gt}, r) = \frac{1}{|S_{pred}|} \sum_{p \in S_{pred}} \mathbb{I}(\min_{q \in S_{gt}} |p-q|2 \leq r) $ $ \mathrm{Recall}(S{pred}, S_{gt}, r) = \frac{1}{|S_{gt}|} \sum_{q \in S_{gt}} \mathbb{I}(\min_{p \in S_{pred}} |q-p|2 \leq r) $ $ \mathrm{F-score}(S{pred}, S_{gt}, r) = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}} $
• Symbol Explanation:
  • $S_{pred}$: The point cloud sampled from the predicted (reconstructed) 3D model.
  • $S_{gt}$: The point cloud sampled from the ground truth 3D model.
  • $r$: The radius (threshold distance) for considering points as "matched" or "correct."
  • $p \in S_{pred}$: A point in the predicted point cloud.
  • $q \in S_{gt}$: A point in the ground truth point cloud.
  • $\|p-q\|_2$: The Euclidean distance between points $p$ and $q$.
  • $\min_{q \in S_{gt}} \|p-q\|_2$: The minimum distance from point $p$ to any point in $S_{gt}$.
  • $\mathbb{I}(\cdot)$: The indicator function, which is 1 if the condition inside is true, and 0 otherwise.
  • $\mathrm{Precision}$: The proportion of points in $S_{pred}$ that are within distance $r$ of some point in $S_{gt}$.
  • $\mathrm{Recall}$: The proportion of points in $S_{gt}$ that are within distance $r$ of some point in $S_{pred}$.
  • The paper specifies sampling $100 \mathrm{k}$ points for CD and F-score, with all objects normalized to $[-1, 1]^3$, and the F-score radius $r$ set to 0.1.

5.2.6. Camera Pose Metrics (Appendix A.3)

• RRE (Relative Rotation Error): Measures the angular difference between predicted and ground-truth camera rotations. A lower RRE indicates more accurate rotation.
• Acc.@15°, Acc.@30°: These are accuracy percentages, representing the proportion of camera poses where the RRE is below $15^\circ$ and $30^\circ$ respectively. Higher percentages are better.
• TE (Translation Error): Measures the distance between predicted and ground-truth camera centers. A lower TE indicates more accurate translation. The paper addresses scale ambiguity by computing relative translations between views and normalizing them.

5.3. Baselines

ReconViaGen was compared against a wide range of existing SOTA baseline methods, categorized into three groups:

5.3.1. 3D Generation Models

These models are primarily designed for generating 3D content, often from limited inputs, but may struggle with consistency.

• TRELLIS-S (Xiang et al., 2024): The stochastic mode of TRELLIS, where it randomly chooses one input view to condition each denoising step.
• TRELLIS-M (Xiang et al., 2024): The multi-diffusion mode of TRELLIS, which computes the average denoised results conditioned on all input views.
• Hunyuan3D-2.0-mv (Zhao et al., 2025): A generative model that conditions on concatenated DINO features from input images (from fixed viewpoints) to generate meshes.

5.3.2. Large Reconstruction Models with Known Camera Poses

These methods require camera poses as input and aim for accurate reconstruction.

• InstantMesh (Xu et al., 2024b): Predicts Triplane representations for mesh outputs from multiple images with fixed viewpoints.
• LGM (Large Multi-view Gaussian Model, Tang et al., 2025): Predicts pixel-aligned 3D Gaussians from multiple images with fixed viewpoints.

5.3.3. Pose-Free Large Reconstruction Models with 3DGS or Point Cloud Outputs

These models do not require pre-calibrated camera poses and often output 3D Gaussians or point clouds.

• LucidFusion (He et al., 2024): Predicts relative coordinate maps for 3D Gaussian outputs.
• VGGT (Wang et al., 2025a): The base Visual Geometry Grounded Transformer model, which reconstructs a point cloud from multi-view inputs in a feed-forward manner. (For comparison with 3D generation models, VGGT's output was aligned to ground-truth models using the same camera pose estimation approach used for ReconViaGen).

5.3.4. Commercial 3D Generation Models

For qualitative "in-the-wild" testing, ReconViaGen was also compared against closed-source commercial models.

• Hunyuan3D-2.5

• Meshy-5

  This comprehensive set of baselines covers various paradigms in 3D generation and reconstruction, allowing for a thorough assessment of ReconViaGen's performance across different aspects.

5.4. Implementation Details

• LoRA Fine-tuning:
  • Rank: Set to 64.
  • Alpha Parameter for LoRA Scaling: Set to 128.
  • Dropout Probability for LoRA Layers: 0.
  • Application: Adapter applied only to the qkv (query, key, value) mapping layer and projectors of each attention layer.
  • VGGT Aggregator Fine-tuning: Randomly sampled $1 \sim 4$ views from 150 views during training. Learning rate of $1 \times 10^{-4}$.
• TRELLIS Sparse Structure Transformer Fine-tuning:
  • Base Model: Built upon TRELLIS (Xiang et al., 2024).
  • Classifier-Free Guidance (CFG): Incorporated with a drop rate of 0.3.
  • Optimizer: AdamW.
  • Learning Rate: Fixed at $1 \times 10^{-4}$.
  • Hardware: 8 NVIDIA A800 GPUs (80GB memory).
  • Training Steps: 40,000 steps with a batch size of 192.
• Inference Settings:
  • CFG Strengths: 7.5 for SS (Sparse Structure) generation, 3.0 for SLAT (Structured LATent) generation.
  • Sampling Steps: 30 for SS generation, 12 for SLAT generation to achieve optimal results.
  • Rendering-aware Velocity Compensation ($\alpha$): Set to 0.1.

6. Results & Analysis

6.1. Core Results Analysis

The experiments demonstrate that ReconViaGen achieves state-of-the-art performance across various metrics, validating its ability to produce complete and accurate 3D models consistent with input views.

The following are the results from Table 1 of the original paper:

6.1.1. Quantitative Results Analysis (Table 1)

• Overall Superiority: ReconViaGen consistently outperforms all baseline methods across both the Dora-bench and OmniObject3D datasets for all evaluation metrics: PSNR, SSIM, LPIPS, CD, and F-score. This strong performance validates its claims of achieving both high image reconstruction consistency and accurate 3D geometry.
• Image Reconstruction Consistency (PSNR↑, SSIM↑, LPIPS↓):
  • ReconViaGen achieves the highest PSNR and SSIM, and the lowest LPIPS on both datasets. For example, on Dora-bench, it achieves 22.632 PSNR (vs. 20.672 for LGM, the next best) and 0.911 SSIM (vs. 0.899 for LGM). This indicates that its synthesized novel views are highly consistent and perceptually similar to the ground truth, reflecting accurate geometry and texture.
• Geometry Accuracy and Completeness (CD↓, F-score↑):
  • The method also records the lowest CD and highest F-score, signifying superior geometric accuracy and completeness. On Dora-bench, CD is 0.089 (vs. 0.095 for LGM), and F-score is 0.953 (vs. 0.939 for VGGT + 3DGS). This highlights the effectiveness of integrating generative priors to complete missing regions while maintaining fidelity.
• Comparison with Base Models (TRELLIS and VGGT):
  • ReconViaGen significantly surpasses TRELLIS-S and TRELLIS-M in all metrics by a large margin (e.g., ~6 PSNR points higher). This demonstrates that the reconstruction-based conditioning and refinement strategies effectively address the inconsistency issues of pure generative models.
  • It also outperforms VGGT + 3DGS (which leverages VGGT's reconstruction capabilities directly) in most metrics, especially PSNR, CD, and F-score. This is crucial as VGGT provides the reconstruction prior, showing that ReconViaGen successfully integrates these priors to create a better overall solution.
• Performance on Different Datasets: The relative performance gaps are similar across both Dora-bench and OmniObject3D, suggesting good generalizability. The paper notes VGGT performs better on Dora-bench than OmniObject3D because Dora-bench's uniformly distributed views offer richer visual cues, aiding reconstruction. Despite this, ReconViaGen still maintains a lead, particularly benefiting from its generative prior on challenging datasets like OmniObject3D.

6.1.2. Qualitative Results Analysis (Figure 3, Figure 4, Figure 7)

该图像是图表，展示了图3中ReconViaGen与多种基线方法在Dora-bench和OminiObject3D数据集上的3D重建效果对比，包括定量指标和多视角3D模型渲染结果，突出展示了ReconViaGen在结构和细节一致性上的优势。

Figure 3: Reconstruction result comparisons between our ReconViaGen and other baseline methods on samples from the Dora-bench and OminiObject3D datasets. Zoom in for better visualization.

• Figure 3 (Dora-bench & OmniObject3D samples): Visual comparisons confirm the quantitative superiority. ReconViaGen produces 3D models with the most accurate geometry and textures. Baselines often show artifacts, missing details, or blurred textures, especially in challenging regions. For instance, TRELLIS variants might produce plausible but clearly inconsistent shapes, while reconstruction models might have holes. ReconViaGen manages to maintain sharp details and correct global structure.

  该图像是图4的示意图，展示了不同方法在真实场景样本上的多视角3D重建效果。左侧为多视角输入，右侧依次为TRELLIS-S、TRELLIS-M、Meshy-5-mv、Hunyuan3D-2.5-mv及本文提出方法的结果，体现了本方法对任意相机视角的鲁棒性和重建细节的准确性。

Figure 4: Reconstruction results on in-the-wild samples. Note that commercial 3D generators require input images from orthogonal viewpoints, while ours can accept views from arbitrary camera poses for robust outputs. Zoom in for better visualization in detail.

• Figure 4 (In-the-wild samples): This figure highlights the robustness of ReconViaGen even against commercial models like Hunyuan3D-2.5-mv and Meshy-5. A key advantage shown is ReconViaGen's ability to accept input views from arbitrary camera poses, whereas some commercial models might require orthogonal viewpoints. This practical flexibility, combined with high-fidelity outputs, makes it suitable for real-world scenarios. The results demonstrate that ReconViaGen can produce cleaner, more detailed, and more geometrically accurate outputs compared to its competitors in uncontrolled settings.

  该图像是图7，展示了TRELLIS-M、TRELLIS-S与ReconViaGen在多视角图像生成样本上的重建结果对比，包含输入图像、多视角视图及三种方法的法线渲染效果，直观体现了各方法在细节和一致性上的表现差异。

Figure 7: Reconstruction result comparisons between TRELLIS-M, TRELLIS-S, and our ReconViaGen on samples produced by the multi-view image generator.

• Figure 7 (Reconstruction on Generated Multi-view Images): This experiment tests the robustness of ReconViaGen on synthesized multi-view inputs (generated by Hunyuan3D-1.0). Such inputs often suffer from cross-view inconsistencies. ReconViaGen demonstrates strong robustness, producing higher-quality and more consistent reconstructions even from these challenging, imperfect inputs compared to TRELLIS-based baselines. This suggests that its internal mechanisms for handling multi-view information are resilient.

6.2. Data Presentation (Tables)

6.2.1. Quantitative Ablation Results on Dora-bench Dataset (Table 2)

The following are the results from Table 2 of the original paper:

6.2.2. Evaluation with More Input Images on the Dora-bench Dataset (Table 3)

The following are the results from Table 3 of the original paper:

6.2.3. Evaluation of Camera Pose Estimation on the Dora-bench Dataset (Table 4)

The following are the results from Table 4 of the original paper:

6.2.4. Quantitative Ablation Results of the Number of Input Images on the Dora-bench Dataset (Table 5)

The following are the results from Table 5 of the original paper:

6.2.5. Quantitative Ablation Results of Condition at SS Flow on the Dora-bench Dataset (Table 6)

The following are the results from Table 6 of the original paper:

6.2.6. Quantitative Ablation Results of Condition at SLAT Flow on the Dora-bench Dataset (Table 7)

The following are the results from Table 7 of the original paper:

6.3. Ablation Studies / Parameter Analysis

The paper conducts comprehensive ablation studies to justify the contribution of each proposed component and design choice.

6.3.1. Effectiveness of Core Components (GGC, PVC, RVC) (Table 2 & Figure 5)

• Baseline (a): ReconViaGen without any novel designs (TRELLIS-M equivalent): This variant (row (a) in Table 2) serves as the baseline, essentially using TRELLIS-M which computes average denoised results conditioned on all input views. It achieves moderate performance (e.g., PSNR 16.706, F-score 0.843).

• Impact of Global Geometry Condition (GGC) (b vs. a):

  • Integrating GGC (row (b)) into SS Flow significantly boosts performance across all metrics (e.g., PSNR jumps from 16.706 to 20.462, F-score from 0.843 to 0.941, CD drops from 0.144 to 0.093).
  • Analysis: This large gain confirms that GGC is crucial for providing accurate global shape information, which is then used to generate a much better coarse structure. The VGGT's strong reconstruction prior, effectively aggregated into GGC, guides the generator to learn more correct foundational geometry.
  • Qualitative (Figure 5): The "Global Geometry Condition" variant shows a vastly improved global shape compared to the "Baseline," which often has distorted or incorrect overall geometry.

• Impact of Local Per-View Condition (PVC) (c vs. b):

  • Adding PVC (row (c)) for SLAT Flow further improves image-level consistency (PSNR increases from 20.462 to 21.045, SSIM from 0.894 to 0.905, LPIPS drops from 0.102 to 0.093). While CD and F-score show minor changes here, the perceptual and image quality metrics indicate better local detail.
  • Analysis: This demonstrates that PVC effectively provides the fine-grained, view-specific information needed for generating accurate local geometry and texture. It helps in better aligning details with individual input views.
  • Qualitative (Figure 5): The "Per-View Condition" variant shows better local details and texture patterns compared to just GGC.

• Impact of Rendering-aware Velocity Compensation (RVC) (d vs. c):

  • Finally, applying RVC (row (d)) during inference brings additional significant improvements (e.g., PSNR rises from 21.045 to 22.632, SSIM from 0.905 to 0.911, F-score from 0.937 to 0.953, CD from 0.093 to 0.089).
  • Analysis: Even though RVC is only applied at inference time, its role in explicitly guiding the denoising trajectory towards pixel-level alignment is powerful. It fine-tunes the generated details to be highly consistent with the input images, leading to overall best performance in both geometry and appearance.
  • Qualitative (Figure 5): The "Rendering-aware Compensation" variant shows impressively refined fine-grained appearance and sharp details, closely matching the input views.

6.3.2. Evaluation with More Input Images (Table 3 & Figure 6)

• Quantitative Analysis (Table 3):

  • ReconViaGen consistently outperforms Object VGGT + 3DGS (a strong reconstruction baseline) at 6, 8, and 10 input views, under both uniform and limited-view sampling strategies.
  • This highlights the crucial role of the generative prior in ReconViaGen for completing invisible object regions, especially compared to pure reconstruction which struggles even with more views if coverage is still sparse.

• Scaling with Number of Images (Table 5):

  • Performance metrics (PSNR, SSIM, F-score) generally improve as the number of input images increases from 2 to 8. This is expected, as more views provide richer information.
  • However, the marginal gains diminish as the number of views increases (e.g., PSNR from 4 to 6 views (22.632 to 22.823) is smaller than from 2 to 4 views (19.568 to 22.632)).
  • Analysis: This indicates a saturation effect, meaning that beyond a certain point, additional uniformly distributed views provide diminishing returns. This is an important insight for practical applications, balancing data collection with reconstruction quality.

• Qualitative Analysis (Figure 6):

  该图像是图6，展示了ReconViaGen在不同数量输入图像下的三维重建结果对比。每组包含多视角输入图像和对应渲染的三维模型，体现了输入视图数目对重建质量的影响。

  Figure 6: Qualitative comparisons for different numbers of input images with ReconViaGen. Zoom in for better visualization in detail.

  • Visual results demonstrate that even with only 2 input images, ReconViaGen can produce plausible reconstructions. As more images are added, the reconstructed details become sharper and more accurate, confirming the quantitative trends. The ability to handle an arbitrary number of inputs from any viewpoint is a key flexibility.

6.3.3. Evaluation of Camera Pose Estimation (Table 4)

• Object VGGT vs. Original VGGT: Fine-tuning VGGT on object-specific data (Object VGGT) significantly improves pose estimation (RRE drops from 8.575 to 7.257, Acc.@15° from 90.67 to 93.44, TE from 0.066 to 0.055). This validates the benefits of domain-specific fine-tuning.
• ReconViaGen (Ours) vs. Object VGGT: ReconViaGen further improves Acc.@30° (96.11 vs. 94.11) and TE (0.046 vs. 0.055), achieving the best overall performance in pose estimation.
  • Analysis: The paper attributes this to the generative prior effectively "densifying" sparse views. By creating a more complete and coherent 3D representation, the image matching for pose refinement (PnP + RANSAC) becomes more robust. The slight increase in RRE for ReconViaGen (7.925 vs. 7.257 for Object VGGT) is acknowledged as potentially due to minor discrepancies between the generated 3D model (used for rendering during pose refinement) and the ground-truth geometry, suggesting a minor trade-off between absolute geometric fidelity and robust pose estimation in some cases.

6.3.4. Ablation Study on the Form of Condition for SS Flow (Table 6)

• The study compares four strategies for conditioning SS Flow:
  • (i) Feature Volume: Downsamples VGGT point cloud to an occupancy volume, projects and averages DINO features.
  • (ii) Concatenation: Fuses VGGT and DINO features, then concatenates all input-view tokens.
  • (iii) PVC (Local Per-View Condition): Uses the same local per-view condition as SLAT Flow.
  • (iv) GGC (Global Geometry Condition): The proposed method.
• Analysis:
  • GGC (row (iv)) achieves the best performance across all metrics (e.g., PSNR 20.462, F-score 0.941).
  • The "Feature Volume" method (i) performs poorly, likely due to inaccuracies in predicted poses or point clouds introducing noise during projection into the volume.
  • "Concatenation" (ii) and "PVC" (iii) perform better than "Feature Volume" but significantly worse than GGC. The paper suggests that these view-level features are not effectively aggregated, leading to redundancy and over-reliance on the raw VGGT outputs without proper holistic integration.
  • This ablation strongly validates that the proposed Condition Net for aggregating multi-view VGGT features into a unified Global Geometry Condition is the most effective way to provide reconstruction-aware conditioning for coarse structure generation.

6.3.5. Ablation Study on the Form of Condition for SLAT Flow (Table 7)

• This study compares two strategies for conditioning SLAT Flow:
  • (i) GGC (Global Geometry Condition): Uses the same global condition as SS Flow.
  • (ii) PVC (Local Per-View Condition): The proposed local per-view condition.
• Analysis:
  • PVC (row (ii)) substantially outperforms GGC (row (i)) in all metrics (e.g., PSNR 22.632 vs. 17.784, F-score 0.953 vs. 0.939).
  • Explanation: The paper attributes this to the information compression inherent in GGC. While GGC is excellent for coarse structure, it loses the fine-grained details necessary for refining geometry and texture at the SLAT Flow stage. PVC, by providing view-specific token lists, retains this detailed information, allowing for more accurate local generation. This explains the design choice of using GGC for coarse generation and PVC for fine-grained generation.

6.4. Reconstruction on Generated Multi-view Images or Videos (Figure 7)

• Robustness to Imperfect Inputs: ReconViaGen demonstrates strong robustness when reconstructing from multi-view images generated by another model (Hunyuan3D-1.0), which often suffer from cross-view inconsistencies.

• Comparison: Even with such challenging inputs, ReconViaGen produces visually superior results compared to TRELLIS-M and TRELLIS-S, exhibiting better consistency and detail. This suggests that its internal mechanisms (like cross-attention with reconstruction priors and RVC) are effective at harmonizing potentially inconsistent inputs and generating a coherent 3D model.

  In summary, the extensive experiments and detailed ablation studies rigorously demonstrate the efficacy of ReconViaGen's novel architecture and its individual components. The integration of reconstruction and generation priors, coupled with multi-view-aware conditioning and explicit pixel-level alignment, is key to its SOTA performance.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces ReconViaGen, a novel coarse-to-fine framework for accurate and complete multi-view 3D object reconstruction. The core innovation lies in its effective integration of strong reconstruction priors (derived from a fine-tuned VGGT model) with diffusion-based 3D generative priors (from TRELLIS). The authors meticulously analyze the shortcomings of existing methods, specifically identifying insufficient cross-view correlation modeling and poor controllability during the stochastic denoising process as major hurdles for achieving consistency.

To overcome these, ReconViaGen proposes three key mechanisms:

1. Global Geometry Condition (GGC): Aggregates multi-view VGGT features into a global token list to guide coarse 3D structure generation, significantly improving global shape accuracy.

2. Local Per-View Condition (PVC): Generates view-specific token lists from VGGT features to provide fine-grained conditions for detailed geometry and texture generation.

3. Rendering-aware Velocity Compensation (RVC): An inference-only mechanism that uses pixel-level alignment losses from rendered intermediate 3D models to explicitly correct the denoising trajectory, ensuring high consistency with input views in fine details.

   Extensive experiments on the Dora-bench and OmniObject3D datasets, alongside detailed ablation studies, confirm that ReconViaGen achieves state-of-the-art performance. It demonstrates superior results in both global shape accuracy and completeness, as well as in the fidelity of local geometric and textural details, effectively bridging the gap between complete generation and accurate reconstruction.

7.2. Limitations & Future Work

The paper implicitly points out the limitations of existing methods, which ReconViaGen aims to address.

• Limitations of Pure Reconstruction: These methods inherently struggle with incompleteness, holes, and artifacts in cases of sparse views, occlusions, or weak textures.

• Limitations of Pure 3D Generation: These methods suffer from stochasticity, leading to results that are plausible but often inconsistent and inaccurate at a pixel level with the input views.

• Limitations of Other Prior Integration: Previous works using 2D diffusion priors often have inconsistency issues, while regression-based 3D priors tend to produce smoother, less detailed results. 3D volume diffusion models may suffer from poor compactness and representation capability.

  As for future work, the authors suggest:

• Integrating Stronger Priors: With ongoing advancements in both 3D reconstruction and 3D generation, ReconViaGen's modular framework allows for the integration of even stronger reconstruction or generation priors to further enhance reconstruction quality. This implies that the current version, while SOTA, is not the ultimate limit of what can be achieved with this paradigm.

7.3. Personal Insights & Critique

• Elegant Integration of Paradigms: ReconViaGen presents a highly elegant solution to a long-standing problem in 3D vision. Instead of viewing reconstruction and generation as competing approaches, it judiciously combines their strengths. The use of a powerful reconstructor (VGGT) to condition and guide a generative model (TRELLIS) is a particularly insightful architectural choice. It allows the generative model to "dream" plausible completions while being anchored firmly to the observed reality.

• Effective Handling of Multi-view Information: The explicit design of Global Geometry Condition (GGC) and Local Per-View Condition (PVC) using a Condition Net is crucial. This layered approach to conditioning, tailoring the information aggregation for coarse vs. fine details, effectively addresses the challenge of building robust cross-view connections. This is a significant improvement over simpler conditioning schemes (e.g., just concatenating features).

• Novel Inference-time Refinement: The Rendering-aware Velocity Compensation (RVC) mechanism is a clever and practical innovation. By performing pixel-level consistency checks during inference and using these to adjust the diffusion model's denoising trajectory, the method sidesteps the need for complex end-to-end retraining for fine alignment. This makes the generated output highly faithful to the input images without sacrificing the generative model's ability to hallucinate. The dynamic weighting and discarding of unreliable pose estimations further add to its robustness.

• Potential Areas for Improvement/Critique:

  • Computational Cost of RVC: While effective, performing decoding to a mesh, rendering, and backpropagation for RVC at each relevant denoising step during inference could be computationally intensive. Although it's only in the later stages (t < 0.5), this might impact inference speed, especially for very high-resolution outputs or a large number of sampling steps. The paper does not provide specific inference time benchmarks, which would be valuable.
  • Dependency on Base Models: The performance of ReconViaGen is inherently tied to the capabilities of VGGT and TRELLIS. While they are SOTA, any limitations or biases in these foundational models could propagate. As future work suggests, integrating "stronger priors" is key to continued improvement, implying this dependency.
  • Generalizability to Complex Scenes: The focus of ReconViaGen is on "3D Object Reconstruction." While Figure A.7 mentions scene reconstruction by segmenting and stitching objects, the core framework is tailored for individual objects. Extending ReconViaGen to handle highly complex, cluttered scenes or environments directly, beyond individual objects, might require further architectural adaptations.
  • Robustness to Adversarial/Out-of-Distribution Inputs: The paper shows robustness against imperfect generated images, which is great. However, it would be interesting to see its performance with highly noisy, occluded, or adversarial real-world inputs that significantly deviate from the training distribution, especially for the VGGT component.

• Broader Impact and Transferability: The principle of using a robust reconstruction module to provide conditioned guidance for a generative model can be highly transferable. This approach could be adapted for:

  • Video-to-3D Reconstruction: Leveraging temporal consistency priors from video analysis to guide 3D generation of dynamic objects.

  • Medical Imaging: Reconstructing complete 3D organs from sparse 2D slices, where generative priors can fill in missing data while anatomical knowledge ensures accuracy.

  • Digital Content Creation: Providing artists with powerful tools to rapidly create complete and detailed 3D assets from limited concept art or photographs, ensuring creative intent is preserved while leveraging generative power.

    Overall, ReconViaGen represents a significant advancement in the quest for comprehensive and precise 3D reconstruction. Its meticulous design and strong empirical results make it a compelling solution and a foundational step for future research at the intersection of 3D vision and generative AI.