1. Bibliographic Information

1.1. Title

Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding

1.2. Authors

Zhihao Yuan, Jinke Ren, Chun-Mei Fen, Heghuan Zhao, Shuang Cui, Li Li. The authors are affiliated with The Future Network of Intelligence Institute and the School of Science and Engineering at The Chinese University of Hong Kong (Shenzhen), IHPC, A*STAR, Singapore, and The University of Hong Kong. Their research backgrounds appear to be in computer vision, robotics, and artificial intelligence, particularly focusing on 3D scene understanding and language models.

1.3. Journal/Conference

Published as a preprint on arXiv. While not a peer-reviewed journal or conference publication yet, arXiv is a highly influential platform for rapid dissemination of research in fields like AI and computer vision.

1.4. Publication Year

2023

1.5. Abstract

This paper introduces a novel visual programming approach for zero-shot open-vocabulary 3D visual grounding (3DVG). Traditional supervised methods for 3DVG are limited by extensive annotation requirements and predefined vocabularies. To overcome these, the proposed method leverages Large Language Models (LLMs) to establish a foundational understanding of zero-shot 3DVG through a unique dialog-based approach. Building on this, the authors design a visual program consisting of three types of modules: view-independent, view-dependent, and functional modules, specifically tailored for 3D scenarios to perform complex reasoning and inference. Furthermore, an innovative language-object correlation (LOC) module is developed to extend existing 3D object detectors into open-vocabulary scenarios. Extensive experiments demonstrate that this zero-shot approach can outperform some supervised baselines, marking a significant advancement towards effective 3DVG.

1.6. Original Source Link

https://arxiv.org/abs/2311.15383 Publication Status: Preprint on arXiv (v2 posted 2023-11-26).

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the limitation of 3D Visual Grounding (3DVG) methods that rely on extensive annotations and predefined vocabularies. 3DVG is the task of localizing specific objects within a 3D scene based on a textual description. This capability is crucial for burgeoning applications such as autonomous robotics, virtual reality (VR), and the metaverse.

Current supervised 3DVG approaches typically treat the problem as a matching task, where 3D detectors generate object proposals, and a best match is found by fusing visual and textual features. While these methods can achieve high precision, they face two significant challenges:

1. Resource-intensive annotation: Acquiring sufficient object-text pair annotations for 3D scenes is prohibitively expensive and time-consuming for real-world applications.

2. Closed-vocabulary limitation: These approaches are constrained to pre-defined vocabularies during training, making them ineffective in open-vocabulary scenarios where descriptions might include novel object categories or attributes not seen during training.

   The paper's entry point is to leverage the zero-shot learning capabilities of pre-trained models (like CLIP in the 3D domain) and the powerful planning and reasoning capabilities of Large Language Models (LLMs) to address these limitations.

2.2. Main Contributions / Findings

The paper makes several primary contributions:

• Novel Visual Programming Approach for Zero-shot 3DVG: The authors propose an innovative 3D visual programming approach that eliminates the need for extensive object-text pair annotations, moving towards a zero-shot paradigm.

• Dialog-based LLM for Foundational Understanding: A unique dialog-based method is introduced where LLMs are engaged to establish an initial understanding of zero-shot 3DVG. Although it has limitations, it lays the groundwork for programmatic reasoning.

• Design of Tailored 3D Visual Program Modules: The visual program is composed of three types of modules—view-independent, view-dependent, and functional—specifically designed for 3D contexts to facilitate complex reasoning and inference tasks that LLMs alone struggle with (e.g., spatial relations, mathematical calculations).

• Innovative Language-Object Correlation (LOC) Module: A LOC module is developed to extend the scope of existing 3D object detectors into open-vocabulary scenarios. This module merges the geometric discernment of 3D point clouds with the fine-grained appearance acumen of 2D images.

• Extensive Evaluation and Superior Performance: The approach is extensively evaluated on popular ScanRefer and Nr3D datasets, including for the first time on the full validation sets. The results demonstrate that the proposed zero-shot approach can outperform some supervised baselines, significantly advancing 3DVG.

• Robustness and Interpretability: The modular design of the visual program provides structured and accurate inference sequences, integrating 3D and 2D data for robust and interpretable results.

  These findings collectively address the annotation burden and vocabulary restrictions of prior 3DVG methods, paving the way for more flexible and scalable 3D scene understanding.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a beginner should be familiar with the following core concepts:

• 3D Visual Grounding (3DVG): This is the task of identifying and localizing a specific 3D object within a 3D scene based on a natural language description. For example, given a 3D scan of a room and the query "the blue chair near the window," a 3DVG system would output the bounding box of that specific chair.
• Zero-Shot Learning: A machine learning paradigm where a model can classify or recognize categories it has not seen during training. It achieves this by leveraging auxiliary information (e.g., semantic descriptions, attribute vectors) that links unseen classes to seen classes. In this context, it means the model can ground objects described by novel terms without requiring explicit object-text pair annotations for those terms.
• Open-Vocabulary: Refers to the ability of a model to handle an unconstrained set of categories or descriptions, rather than being limited to a fixed, predefined vocabulary that it was trained on. This is crucial for real-world applications where object types and descriptions can be infinitely varied.
• Large Language Models (LLMs): These are deep learning models with billions of parameters, trained on vast amounts of text data to understand, generate, and process human language. Examples include GPT-3, GPT-4, and Llama. They exhibit strong reasoning, planning, and commonsense knowledge capabilities, which the paper leverages for interpreting natural language descriptions and generating visual programs.
• Visual Programming: In this context, visual programming refers to generating a sequence of executable operations (a "program") from a natural language query, which then interacts with vision modules to solve a task. Instead of directly mapping language to a visual output, an LLM translates the query into a structured, modular program.
• CLIP (Contrastive Language-Image Pre-training): A neural network trained on a wide variety of image-text pairs from the internet. It learns to associate images with their textual descriptions. A key feature is its ability to compute cosine similarity between image embeddings and text embeddings, enabling zero-shot image classification and open-vocabulary object recognition by matching images to arbitrary text queries. The paper uses CLIP (or CLIP-like models) for its 2D open-vocabulary capabilities when processing 2D imagery of 3D object proposals.
• 3D Point Clouds: A set of data points in a three-dimensional coordinate system. These points represent the external surface of an object or environment. Each point typically has X, Y, Z coordinates, and may also include color (RGB) or intensity information. They are the primary representation of 3D scenes in this paper.
• Bounding Box (BBox): A rectangular or cuboid region that tightly encloses an object in an image or 3D space. In 3DVG, the goal is to predict the bounding box of the target object.

3.2. Previous Works

The paper contextualizes its contribution by discussing prior research in three main areas:

• Supervised 3DVG:

  • Most existing methods ([4, 6, 14, 18, 41, 58, 60]) treat 3DVG as a matching problem. They typically use pre-trained 3D object detectors ([21, 37]) to generate candidate objects (e.g., bounding boxes) within a 3D scene. Then, visual features from these candidates and textual features from the query are fused (e.g., through multimodal encoders) to rank the candidates and find the best match.
  • Examples include ScanRefer [4] and ReferIt3DNet [1] which learn to align 3D point clouds and language features. More advanced methods like TGNN [17] and InstanceRefer [60] explore instance-wise features and object attributes/relations. Recent advancements have also incorporated Transformer architectures ([65]) and DETR ([20]) for 3DVG.
  • NS3D [16] employs CodeX [5] to generate hierarchical programs but still requires extensive data annotations, lacking open-vocabulary and zero-shot capabilities.
  • Limitation: These methods require densely-annotated datasets (like ScanRefer [4] and Referit3D [1]) with well-aligned object-text pairs and are generally restricted to a closed set of semantic class labels seen during training.

• Indoor 3D Scene Understanding (Open-Vocabulary):

  • The emergence of RGB-D scan datasets ([9, 43, 51]) has propelled research in tasks like 3D object classification [35, 36], 3D object detection [30, 37], and 3D instance segmentation [21, 42, 50]. However, these are often closed-vocabulary.
  • Inspired by 2D open-vocabulary segmentation [13, 28], some 3D methods have emerged:
    • LERF [23] learns a language field within NeRF [31] by rendering CLIP features to generate 3D relevancy maps for arbitrary language queries.
    • OpenScene [34] extracts 2D open-vocabulary segmentation features ([13, 26]) and projects them into 3D to train a 3D network.
    • OpenMask3D [45] uses a closed-vocabulary network for instance masks, discarding the classification head.
  • Limitation: While these advance open-vocabulary 3D scene understanding, they often lack spatial and commonsense reasoning abilities that are critical for 3DVG.

• LLMs for Vision-Language Tasks:

  • Recent LLMs ([33, 46, 48]) offer impressive zero-shot planning and reasoning abilities, enhanced by prompting techniques like Chain-of-Thought [53]. They can interpret instructions, break down goals, and control robot agents [2, 19, 29].
  • When combined with vision models, LLMs significantly improve vision-language tasks:
    • Visual ChatGPT [55] uses ChatGPT as an orchestrator for various visual foundation models.
    • VISPROG [15] generates high-level modular programs for complex natural language reasoning and image editing.
    • ViperGPT [44] generates executable Python code for image grounding by feeding API descriptions to LLMs.
  • Limitation: Leveraging these LLM capabilities for zero-shot 3D language grounding remained an unexplored area, specifically addressing the challenges of 3D spatial reasoning and view-dependent relations.

3.3. Technological Evolution

The field of 3DVG has evolved from supervised methods that rely heavily on paired 3D-text annotations and are restricted to closed vocabularies, to open-vocabulary 3D scene understanding methods that leverage 2D vision-language models (like CLIP) to extend recognition to unseen categories. However, these open-vocabulary methods often lack sophisticated spatial and commonsense reasoning. Concurrently, Large Language Models (LLMs) have shown remarkable reasoning and planning abilities in zero-shot settings for vision-language tasks. This paper positions itself at the intersection of these advancements, integrating LLM-driven programmatic reasoning with open-vocabulary 3D perception to create a zero-shot 3DVG system that can handle complex spatial relations and arbitrary object descriptions without extensive 3D annotations.

3.4. Differentiation Analysis

Compared to the main methods in related work, the core differences and innovations of this paper's approach are:

• From Supervised to Zero-Shot: Unlike traditional supervised 3DVG methods that require large object-text pair annotations and predefined vocabularies, this approach is inherently zero-shot and open-vocabulary, dramatically reducing the annotation burden.
• Explicit Programmatic Reasoning: Instead of implicit learning of visual-textual alignment, the paper uses LLMs to generate explicit, executable visual programs. This allows for structured step-by-step reasoning, which is more robust and interpretable, especially for complex spatial relations.
• Addressing 3D-Specific Challenges with Modules:
  • It explicitly designs view-dependent and view-independent modules to handle the unique challenges of 3D spatial reasoning, particularly the dynamic nature of view-dependent relations which LLMs (in a naive dialog setting) and 2D-only methods struggle with. The 2D egocentric view approach for view-dependent relations is a key innovation here.
  • The functional modules (e.g., MIN, MAX) address the LLMs' weakness in precise mathematical calculations crucial for 3DVG.
• Language-Object Correlation (LOC) Module for Open-Vocabulary 3D Detection: While open-vocabulary 3D scene understanding exists (e.g., LERF, OpenScene), they often lack precise localization and reasoning. The LOC module specifically addresses open-vocabulary instance segmentation in 3D by combining closed-vocabulary 3D object detection with open-vocabulary 2D image-text models, allowing the identification of novel objects/attributes. This integration is more direct and effective for grounding.
• Bridging LLMs and 3D Visual Perception: Previous LLM-vision works often focused on 2D domains (e.g., Visual ChatGPT, ViperGPT, VISPROG). This paper extends LLM-driven visual programming to the complex 3D domain, tackling its unique challenges head-on.

4. Methodology

4.1. Principles

The core idea behind this method is to leverage the reasoning and language understanding capabilities of Large Language Models (LLMs) to interpret natural language descriptions for 3D Visual Grounding (3DVG). Instead of directly training an end-to-end neural network with extensive 3D object-text pair annotations, the approach proposes to translate the natural language query into an executable visual program. This visual program then orchestrates specialized vision modules (both 3D and 2D) to perform the necessary steps for object localization. This programmatic approach aims to provide explicit control over spatial reasoning, handle view-dependent relations, and enable open-vocabulary grounding in a zero-shot manner, overcoming the limitations of LLMs in direct spatial calculation and visual perception without fine-tuning.

4.2. Core Methodology In-depth (Layer by Layer)

The methodology evolves from a simple dialog-based approach with LLMs to a more structured visual programming framework, enhanced by a novel Language-Object Correlation (LOC) module.

4.2.1. Dialog with LLM (Vanilla Approach)

The paper first introduces a vanilla approach where an LLM directly engages in a dialogue to perform 3DVG.

• Input: The input consists of a real-world RGB-D scan (represented as a point cloud $\mathbf{P} \in \mathbb{R}^{N \times 6}$ where $N$ is the number of color-enriched 3D points) and a free-form text description $\tau$.
• Scene Transformation: To bridge the gap between LLM's text understanding and the spatial nature of 3DVG, the 3D scene is first transformed into a textual narrative. This narrative lists objects detected in the scene, their categories, positions, and dimensions. The format is: $Object <id> is a <category> located at (x, Y, z) with sizes (width, length, height).$
• Dialogue Process: The LLM is provided with this scene description and the text query $\tau$. It acts as an agent that needs to identify the specified object and explain its reasoning.
  • Example: As shown in Figure 2(a) (part of image 6), if the LLM receives object information (e.g., keyboard, door with their coordinates), it can extract relevant objects from the query and attempt to identify the target (e.g., the keyboard closest to the door) by calculating distances.
• Limitations:
  1. View-dependent issues: LLMs struggle with view-dependent queries (e.g., "the right window") because they typically reason based on x-y coordinates and lack the ability to adapt to dynamic viewpoints.
  2. Mathematical Calculation Weakness: LLMs are generally poor at precise mathematical calculations (e.g., exact distance computations), which are often critical for resolving spatial relations (e.g., "closest"). These limitations stem from the inherent stochasticity and control limitations of LLMs.

The following figure (Figure 2 from the original paper) shows a comparison between the dialog with LLM approach and the visual programming approach.

该图像是一本论文中的示意图，展示了基于大语言模型（LLM）的零样本开放词汇3D视觉定位的视觉编程流程。图中包括对话示例和由视觉程序驱动的推理过程，描述如何结合3D扫描和语言信息实现目标预测。

4.2.2. 3D Visual Programming

To address the limitations of the dialog-based approach, the paper proposes a visual programming approach.

• Program Generation:
  • The reasoning process for 3DVG is transformed into a scripted visual program.
  • LLMs are used to generate these programs by providing a set of in-context examples that demonstrate human-like problem-solving tactics in 3DVG.
  • Each program is a sequence of operations. An operation consists of a module name, input parameters, and an assigned output variable. The output of one step can be reused in subsequent steps, creating a logical flow.
• Example: For the description "The round cocktail table in the corner of the room with the blue and yellow poster," the process is:
  1. $BOX0 = LOC('round cocktail table')$: The LOC (Language-Object Correlation) operator identifies potential round cocktail tables.
  2. $BOX1 = LOC('blue and yellow poster')$: The LOC operator identifies the blue and yellow poster as an anchor reference.
  3. $RESULT = CLOSEST(BOX0, BOX1)$: The CLOSEST module computes proximity between BOX0 (potential tables) and BOX1 (poster) and selects the table closest to the poster.
• Module Types: The visual program leverages three types of modules tailored for 3D contexts (summarized in Table 1):

  1. View-independent modules: Operate on 3D spatial relations regardless of the observer's position. Examples: near, close, next to, far, above, below, under, top, on, opposite, middle, CLOSEST.

  2. View-dependent modules: Depend on the observer's vantage point. Examples: front, behind, back, right, left, facing, leftmost, rightmost, looking, across, between.

  3. Functional modules: Perform operations like MIN and MAX to select objects based on extremal criteria (e.g., smallest, tallest) or properties like size, length, width.

     The following are the results from Table 1 of the original paper:

     View-independent	near, close, next to, far, above, below,under, top, on, opposite, middle
     View-dependent	front, behind, back, right, left, facing,leftmost, rightmost, looking, across, be-tween
     Functional	min, max, size, length, width

These modules enable flexible composability, structured inference, and integrate 3D and 2D data.

4.2.3. Addressing View-Dependent Relations

This section details how the system handles view-dependent relations, a significant challenge in 3D space due to their dynamic nature based on the observer's viewpoint.

• Challenge: View-dependent relations (e.g., "left," "right," "front," "behind") are straightforward in 2D (e.g., X-axis for left/right) but complex in 3D where viewpoint changes everything.
• Solution: 2D Egocentric View: The approach adopts a 2D egocentric view to establish a consistent frame of reference.
  • A virtual camera is assumed to be at the center of the room ($P_{\mathrm{center}}$).
  • This camera rotates to align with the anchor objects ($P_{\mathbf{o}_{a}}$).
  • 3D objects are then projected onto a 2D plane from this vantage point.
• Projection Formula: The 2D projections are obtained using a standard view transformation and projection process: $ R, T = \mathrm{Lookat}(P_{\mathrm{center}}, P_{\mathbf{o}_{a}}, up) $ $ (u, v, w)^{\mathrm{T}} = I \cdot (R | t) \cdot P $ Where:
  • $R$: Rotation matrix derived from the Lookat function.
  • $T$ (or $t$ in the projection equation): Translation matrix (or vector) derived from the Lookat function.
  • $\mathrm{Lookat}(\cdot)$: A view transformation function that computes $R$ and $T$ given the camera's eye position ($P_{\mathrm{center}}$), target position ($P_{\mathbf{o}_{a}}$), and an up-vector (e.g., for camera orientation).
  • $P = (x, y, z, 1)^{\mathrm{T}}$: The homogeneous 3D coordinate vector of an object.
  • $I$: The intrinsic matrix of the orthogonal camera.
  • $(u, v, w)^{\mathrm{T}}$: The projected coordinates. $u$ is the X-axis on the 2D plane, $v$ is the Y-axis on the 2D plane, and $w$ is the depth value.
• Spatial Reasoning:

  • The $u$ value of an object's center determines its left or right position (a lower $u$ indicates left).

  • The $w$ value allows distinguishing front from behind.

  • These concepts are combined to define complex relations like between.

    The following figure (Figure 3 from the original paper) illustrates the shift to a 2D egocentric view for view-dependent relations.

    该图像是图表，展示了论文中ScanRefer和Nr3D数据集的错误来源分布对比，包括Prog、Exec、Rel、Cls、Loc及Correct各项占比情况，体现了不同错误类型在两数据集中的比例差异。

4.2.4. Language-Object Correlation (LOC) Module

The LOC module is crucial for enabling open-vocabulary segmentation and extending 3D object detectors beyond predefined classes.

• Motivation: While the visual programming handles reasoning, a basic vision model is still needed for object localization. Existing 3D detectors ([21, 37]) are typically closed-vocabulary.

• Two-Stage Approach: The LOC module combines 3D and 2D networks to achieve open-vocabulary capabilities.

  1. Closed-Vocabulary 3D Filtering: First, a pre-trained 3D instance segmentation network (e.g., Mask3D [42]) is used to generate object proposals and their predicted labels from a fixed vocabulary (e.g., "table"). This filters the scene to a subset of objects relevant to a broad category. For example, if the query is "round cocktail table", the 3D network first identifies all objects labeled "table".
  2. Open-Vocabulary 2D Refinement: For the filtered subset of 3D proposals, their corresponding 2D imagery (e.g., rendered images from multiple views) is used. 2D multi-modal models then analyze these images against the fine-grained query (e.g., "round cocktail table") to pinpoint the exact target.

• Types of 2D Multi-modal Models Utilized:

  1. Image classification models: A dynamic vocabulary is constructed (e.g., "round cocktail table" and "table"). Cosine similarity between these terms and the 2D imagery features (e.g., from CLIP [40]) is computed to find the best correlation.
  2. Visual Question Answering (VQA) models: A question like "Is there a [query]?" (e.g., "Is there a round cocktail table?") is posed to a VQA model (e.g., ViLT [24]), which provides a yes/no answer or a more detailed response.
  3. General Large Models: A similar inquiry is submitted to a general large model (e.g., BLIP-2 [27]) to verify alignment between the detected object and the query based on its generated text response.

• Flexibility: The approach is not limited to specific 3D or 2D models, allowing for versatile incorporation of various models and benefiting from advancements in foundation models.

  The following figure (Figure 4 from the original paper) illustrates the Language-Object Correlation module.

  该图像是一个室内场景的三维点云重建示意图，展示了基于视觉编程的零样本开放词汇3D视觉定位中对目标区域的识别与框选，重点突出镜子上的标注框以示范模块协同推理能力。

5. Experimental Setup

5.1. Datasets

The experiments are conducted on two popular 3D Visual Grounding (3DVG) datasets:

• ScanRefer [4]:

  • Source & Characteristics: Tailored for 3DVG, it contains 51,500 sentence descriptions for 800 ScanNet scenes [9]. ScanNet provides densely reconstructed 3D indoor scenes with semantic annotations.
  • Domain: Indoor scenes, rich in diverse objects and complex spatial arrangements.
  • Usage: Used for evaluating object proposal generation and grounding accuracy. The validation split is used for zero-shot evaluation.
  • Data Sample (Example from paper's abstract): Given a 3D scan and its description - "It is the keyboard closest to the door," the goal is to pinpoint the keyboard.

• Nr3D [1]:

  • Source & Characteristics: A human-written and free-form dataset for 3DVG, collected through a 2-player reference game in 3D scenes.

  • Subsets: Sentences are categorized into "easy" (target object has only one same-class distractor) and "hard" (multiple same-class distractors). It's also partitioned into "view-dependent" and "view-independent" subsets based on whether the sentence requires a specific viewpoint for grounding.

  • Domain: Similar to ScanRefer, focuses on indoor scenes but with an emphasis on referring expressions and spatial relations.

  • Usage: Used for evaluating grounding accuracy given ground-truth object masks (focuses on classification rather than localization error). The validation split is used for zero-shot evaluation.

    These datasets are well-suited for validating the method's performance as they cover diverse 3D indoor scenes, complex natural language descriptions, and allow for evaluation of both localization and classification aspects of 3DVG, including challenges like distractors and viewpoint dependencies.

5.2. Evaluation Metrics

The paper considers two settings for performance evaluation, each with specific metrics:

1. Setting 1: Object Proposal Generation (Default for ScanRefer)

   • This setting aligns with real-world applications where the system must first generate object proposals (e.g., bounding boxes) and then ground them.
   • Metrics: Acc@0.25 and Acc@0.5.
     • Conceptual Definition: These metrics measure the percentage of correctly predicted bounding boxes. A prediction is considered correct if its Intersection over Union (IoU) with the ground-truth bounding box exceeds a specified threshold (0.25 or 0.5). They quantify both the localization accuracy (how well the predicted box overlaps with the true box) and the classification accuracy (if the correct object was identified).
     • Mathematical Formula (IoU): $ \mathrm{IoU}(B_p, B_{gt}) = \frac{\mathrm{Area}(B_p \cap B_{gt})}{\mathrm{Area}(B_p \cup B_{gt})} $ Where:
       • $B_p$: The predicted bounding box.
       • $B_{gt}$: The ground-truth bounding box.
       • $\mathrm{Area}(B_p \cap B_{gt})$: The area of overlap between the predicted and ground-truth boxes.
       • $\mathrm{Area}(B_p \cup B_{gt})$: The area of the union of the predicted and ground-truth boxes.
     • Mathematical Formula (Accuracy@k): $ \mathrm{Acc@k} = \frac{\mathrm{Number of correct predictions with IoU > k}}{\mathrm{Total number of predictions}} \times 100% $ Where $k$ is the IoU threshold (0.25 or 0.5).

2. Setting 2: Ground-Truth Object Masks Furnished (Default for Nr3D)

   • This setting focuses purely on classification accuracy by providing ground-truth object masks, thereby eradicating localization error. The objective is to achieve high grounding accuracy by correctly identifying the target object among given candidates.
   • Metric: Top-1 Accuracy.
     • Conceptual Definition: This metric measures the percentage of instances where the model's top-ranked prediction for the target object is indeed the correct ground-truth object. It primarily assesses the model's ability to classify or identify the correct object among a set of candidates, assuming perfect localization (since ground-truth boxes/masks are provided).
     • Mathematical Formula (Accuracy): $ \mathrm{Accuracy} = \frac{\mathrm{Number of correctly identified objects}}{\mathrm{Total number of objects to be grounded}} \times 100% $

5.3. Baselines

The proposed method is compared against several supervised and open-vocabulary 3D scene understanding approaches:

• Supervised Approaches: These models are trained on large object-text pair annotated datasets and represent state-of-the-art in traditional 3DVG.

  • ScanRefer [4]: A foundational 3DVG model that encodes 3D point clouds and language features separately and fuses them to rank objects.
  • ReferIt3DNet [1]: Similar to ScanRefer, it processes 3D point clouds and language to identify objects.
  • TGNN [17]: Extends previous methods by learning instance-wise features within a graph neural network framework.
  • InstanceRefer [60]: Focuses on cooperative holistic understanding through instance multi-level contextual referring. It also learns instance-wise features.
  • 3DVGTransformer [65]: Utilizes the Transformer architecture [47] for relation modeling in 3D visual grounding.
  • BUTD-DETR [20]: Employs Bottom-Up Top-Down Detection Transformers [3] for language grounding in images and point clouds, representing a strong Transformer-based approach.

• Open-Vocabulary 3D Scene Understanding Approaches: These methods aim to learn 3D representations aligned with 2D vision-language features to enable free-form language grounding, typically without specific 3DVG training.

  • LERF [23]: Language Embedded Radiance Fields. It learns a language field within NeRF [31] by volume rendering CLIP [40] features, allowing it to generate 3D relevancy maps for arbitrary language queries. It processes the query with a CLIP text encoder and computes similarity against extracted 3D features.

  • OpenScene [34]: Extracts image features using 2D open-vocabulary segmentation models [13, 26] and trains a 3D network to produce point features aligned with multi-view fused pixel features. It then clusters points with the highest CLIP similarity to determine the target object.

    These baselines are representative as they cover both traditional supervised methods (which the paper aims to outperform with a zero-shot approach) and emerging open-vocabulary 3D methods (which the paper aims to improve upon in terms of reasoning and localization precision).

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate the strong performance of the proposed zero-shot visual programming approach for 3DVG, often outperforming supervised baselines and significantly exceeding other open-vocabulary methods.

• ScanRefer Dataset (Table 2):

  • The Ours (full model) achieves Acc@0.5 of 32.7% and Acc@0.25 of 36.4%. This is notable because it surpasses established supervised approaches like ScanRefer (24.3% Acc@0.5) and TGNN (29.7% Acc@0.5).
  • While BUTD-DETR (a state-of-the-art supervised method) still performs better (39.8% Acc@0.5), the fact that a zero-shot approach can achieve comparable or better results than some supervised methods is a significant finding.
  • Other open-vocabulary approaches like LERF (0.9% Acc@0.5) and OpenScene (6.5% Acc@0.5) perform poorly, highlighting their limitations in reasoning and localization precision when compared to the proposed method. This validates the need for a more structured reasoning framework like visual programming.
  • The superior performance of the full model over Ours (2D only) and Ours (3D only) (e.g., 32.7% Acc@0.5 vs. 17.6% and 29.3% respectively) demonstrates the effectiveness of integrating both 3D geometric information and 2D appearance acumen via the LOC module.

• Nr3D Dataset (Table 3):

  • On Nr3D (where ground-truth masks are provided, focusing on classification accuracy), Ours achieves an Overall Accuracy of 39.0%.
  • This result again excels a supervised approach like InstanceRefer (38.8% Overall).
  • Notably, on the "view-dependent" split, Ours achieves 36.8%, which is a 2% accuracy gain over 3DVG-Transformer (34.8%). This specific improvement highlights the efficacy of the relation modules and the 2D egocentric view approach in handling view-dependent spatial relations.

• Qualitative Results (Figure 5 - image 9):

  • The qualitative examples show that dialog with LLM and visual programming can accurately predict for view-independent relations (e.g., above, under).

  • Crucially, BUTD-DETR and dialog with LLM struggle with view-dependent relations (e.g., left, front), whereas the visual programming approach can make accurate predictions by leveraging 2D egocentric views.

  • A failure case (Figure 5e) shows dialog with LLM failing due to lack of open-vocabulary detection (e.g., "chair has wheels"). The visual programming initially makes a wrong prediction but can be corrected with a precise program (e.g., using CLOSEST). This indicates the importance of robust program generation and module design.

    Overall, the results strongly validate the effectiveness of the proposed zero-shot visual programming approach, particularly its ability to handle complex spatial reasoning and open-vocabulary scenarios while remaining competitive with, and often surpassing, supervised methods in specific aspects.

6.2. Data Presentation (Tables)

The following are the results from Table 2 of the original paper:

The following are the results from Table 3 of the original paper:

6.3. Ablation Studies / Parameter Analysis

6.3.1. Dialog with LLM vs. Visual Programming

This study compares the two proposed zero-shot approaches using different LLM versions on the ScanRefer validation set.

The following are the results from Table 4 of the original paper:

• Findings:
  • GPT-4 consistently yields higher accuracy than GPT-3.5 for both approaches, but at a significantly higher computational cost (due to more tokens and GPT-4's pricing).
  • The visual programming approach always outperforms the dialog with LLM approach in terms of accuracy (32.1% vs 25.4% for GPT-3.5; 35.4% vs 27.5% for GPT-4).
  • Crucially, visual programming is also far more cost-efficient (e.g., 0.19` vs 3.05forGPT-3.5), as it requires fewer tokensfrom theLLMfor program generation compared to extensive dialogue. This demonstrates the superior effectiveness and efficiency of thevisual programming` paradigm.

6.3.2. Relation Modules

Ablation studies analyze the impact of different view-dependent and view-independent relation modules on system performance.

The following are the results from Table 5 of the original paper:

• View-Dependent Modules (Table 5):

  • Starting from an accuracy of 26.5% without any view-dependent modules, adding LEFT or RIGHT alone significantly boosts accuracy (32.4% and 35.9% respectively).

  • Including LEFT, RIGHT, FRONT, BEHIND, and BETWEEN progressively improves performance, reaching 39.0% accuracy.

  • Finding: LEFT and RIGHT are identified as the most important view-dependent relations, which aligns with the motivation for the 2D egocentric view approach.

    The following are the results from Table 6 of the original paper:

    CLOSEST	FARTHEST	LOWER	HIGHER	Accuracy
    				18.8
    > >				30.7
    	✓			34.0
    ✓	✓	✓		36.8
    ✓	✓	✓	✓	39.0

• View-Independent Modules (Table 6):

  • Starting from 18.8% accuracy without any view-independent modules, adding CLOSEST alone dramatically increases accuracy to 30.7%.
  • The inclusion of FARTHEST, LOWER, and HIGHER further boosts performance to 39.0%.
  • Finding: CLOSEST is the most important view-independent relation, which also aligns with intuitive spatial reasoning in 3DVG.

6.3.3. LOC Module

The LOC module's effectiveness is analyzed by comparing the full model with 2D-only and 3D-only variants (as seen in Tables 2 and 3).

• Findings:
  • Ours (2D only) performs the worst (17.6% Acc@0.5 on ScanRefer, 23.6% Overall on Nr3D). This is attributed to the complexity of indoor scene images and potential domain gaps with 2D training samples.
  • Ours (3D only) performs better (29.3% Acc@0.5 on ScanRefer, 36.7% Overall on Nr3D), leveraging geometric information and closed-set labels.
  • The full model (Ours) consistently achieves the best performance (32.7% Acc@0.5 on ScanRefer, 39.0% Overall on Nr3D).
  • Conclusion: This validates that the LOC module's integration of 3D geometric distinctiveness (from 3D point clouds) and open-vocabulary appearance acumen (from 2D image models) is critical for optimal performance in open-vocabulary 3DVG.

6.3.4. Generalization

The framework's adaptability to different 3D backbones and 2D models is tested.

The following are the results from Table 7 of the original paper:

• Different 2D Models (Table 7):

  • Using CLIP, ViLT, and BLIP-2 as 2D assistance models, the Acc@0.25 scores are 35.7%, 35.1%, and 36.4% respectively.

  • Finding: The framework is compatible with various 2D multi-modal models, and advancements in these 2D foundation models (e.g., BLIP-2 performing slightly better) can directly improve the system's performance.

    The following are the results from Table 8 of the original paper:

    3D Backbone	View-dep.	View-indep.	Overall
    PointNet++	35.8	39.4	38.2
    PointBert	36.0	39.8	38.6
    PointNeXt	36.8	40.0	39.0

• Different 3D Backbones (Table 8):

  • $PointNet++$ [36], PointBERT [59], and PointNeXt [38] are used as 3D backbones. PointNeXt achieves the highest Overall Accuracy of 39.0%.
  • Finding: The framework demonstrates strong cross-model effectiveness and robustness, indicating it can leverage different 3D foundational models to enhance performance.

6.3.5. Effect of Prompt Size

The impact of the number of in-context examples provided to the LLM for program generation is analyzed.

The following figure (Figure 6 from the original paper) shows the ablation study on the number of in-context examples.

该图像是论文中展示的一个视觉程序示例，展示了以模块化代码形式表达的零样本三维视觉定位操作，包含对象定位及空间关系描述。

• Findings: Performance on both Nr3D and ScanRefer generally improves as the number of in-context examples increases, suggesting that more examples guide LLMs to generate better visual programs. However, this improvement follows the law of diminishing marginal utility.
• Voting Technique: Applying a voting technique (aggregating results from multiple runs) provides additional performance gains.

6.3.6. Error Analysis

To understand limitations, an error analysis was conducted on subsets of ScanRefer and Nr3D validation sets.

The following figure (Figure 7 from the original paper) illustrates the breakdown of error sources.

该图像是一张室内场景的三维点云示意图，展示了通过3D视觉定位技术对桌面区域的识别与分割，图中部分区域被用红色和蓝色边框标记，体现了视角相关模块在三维视觉编程中的应用。

• Findings:
  1. Program Generation (Prog): The primary source of error is the generation of accurate visual programs by the LLM. This suggests that improving LLM capabilities (e.g., with more powerful LLMs or better prompting) could significantly boost performance.
  2. Object Localization and Classification (Loc/Cls): These are the second most significant error sources, indicating that the underlying 3D object detection and classification components remain critical areas for improvement.
  3. Relation Handling (Rel): Errors in handling specific spatial relations also contribute, highlighting the need for developing additional modules to cover a wider array of relations (e.g., "opposite").

6.4. Qualitative Results

The following figure (Figure 5 from the original paper) shows qualitative examples comparing ground truth, BUTD-DETR, Dialog with LLM, and Visual Programming.

该图像是一个室内办公环境的三维点云示意图，展示了多个以不同颜色框选的三维空间区域，用于三维视觉定位任务中的目标对象标注。

• Figure 5(a) and Figure 5(b) demonstrate that both Dialog with LLM and Visual Programming can achieve accurate predictions for view-independent relations (e.g., "above," "under") without extensive training.
• Figure 5(c) and Figure 5(d) highlight the BUTD-DETR and Dialog with LLM approaches' inability to address view-dependent relations (e.g., "left," "front"). In contrast, the visual programming approach successfully leverages 2D egocentric views for accurate predictions in these scenarios.
• Figure 5(e) illustrates a failure case for Dialog with LLM due to its lack of open-vocabulary detection (e.g., cannot recognize "chair has wheels"). The visual programming approach also initially makes a wrong prediction due to LLM misinterpreting the relation "pushed," but can be corrected with a more appropriate module (e.g., CLOSEST).

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces a novel zero-shot visual programming approach for 3D Visual Grounding (3DVG), effectively addressing the limitations of extensive annotation requirements and predefined vocabularies in conventional supervised methods. The core of the approach involves engaging Large Language Models (LLMs) to generate explicit visual programs, which orchestrate specialized view-independent, view-dependent, and functional modules tailored for 3D scenarios. A key innovation is the Language-Object Correlation (LOC) module, which seamlessly integrates 3D geometric features with 2D open-vocabulary appearance information to enable robust object localization. Extensive experiments on ScanRefer and Nr3D datasets demonstrate that this zero-shot approach can not only outperform existing open-vocabulary methods but also achieve performance comparable to, and in some cases exceeding, supervised baselines. The modular design fosters both accuracy and interpretability, marking a significant stride towards more flexible and scalable 3D scene understanding.

7.2. Limitations & Future Work

The authors identify several limitations and suggest future research directions, primarily informed by their error analysis:

• Program Generation Accuracy: The primary error source is the generation of accurate visual programs by LLMs.
  • Future Work: This can be improved by using more powerful LLMs and more comprehensive in-context examples in the prompting process.
• Object Localization and Classification: The accuracy of underlying 3D object detection and classification remains a critical component and error source.
  • Future Work: Continued development in 3D perception models (e.g., for instance segmentation and open-vocabulary detection) will directly benefit the system.
• Limited Spatial Relations: The current set of relation modules may not cover the full spectrum of possible spatial relations in natural language descriptions.
  • Future Work: Developing additional modules to handle a wider array of spatial relations (e.g., "opposite," "inside," "next to each other in a line") is necessary for increased robustness.
• LLM Capabilities: Despite advances, LLMs still have inherent weaknesses in precise mathematical calculations and sometimes misinterpret complex relational phrases.
  • Future Work: Further research into improving LLM's numerical reasoning or designing more robust functional modules that externalize these computations will be beneficial.

7.3. Personal Insights & Critique

The paper presents a highly innovative and promising direction for 3DVG. The shift from implicit end-to-end learning to LLM-orchestrated visual programming is a powerful paradigm that brings interpretability and zero-shot generalization to a complex 3D task.

• Inspirations and Applications:

  • The modular nature of the visual program makes it highly adaptable. This approach could be transferred to other multi-modal tasks requiring complex reasoning and external tool use, such as robot manipulation planning based on natural language commands in dynamic 3D environments.
  • The LOC module's strategy of combining closed-vocabulary 3D detection with open-vocabulary 2D refinement is particularly clever. This hybrid approach could be adopted by other open-vocabulary 3D tasks where precise 3D localization is needed alongside semantic flexibility.
  • The explicit handling of view-dependent relations using a 2D egocentric projection is an elegant solution to a notoriously difficult problem in 3D scene understanding.

• Potential Issues and Areas for Improvement:

  • LLM Robustness: While the visual programming approach reduces LLM stochasticity compared to direct dialogue, the reliability of program generation itself (as identified in the error analysis) remains a bottleneck. LLM hallucinations or misinterpretations in program synthesis could lead to systematic errors. More robust prompt engineering, few-shot examples, or even LLM fine-tuning on program generation specifically for 3DVG might further enhance this.

  • Module Granularity and Coverage: While the three module types are a good start, the complexity of human language and 3D environments suggests that the set of modules might need to grow significantly. Managing this growing library of modules and ensuring LLMs can correctly select and compose them for arbitrary queries will be a challenge.

  • Computational Cost for 2D Refinement: The LOC module relies on rendering 2D images from 3D proposals and then processing them with 2D multi-modal models. This process can be computationally intensive, especially for scenes with many objects or complex queries requiring multiple 2D model inferences. Optimizing this pipeline for speed is crucial for real-time applications.

  • Geometric Precision of 3D Detectors: The performance is still fundamentally limited by the quality of the initial 3D instance segmentation (e.g., Mask3D). Errors in 3D object detection will propagate through the system. Improvements in 3D object detection (especially for small or partially occluded objects) are essential.

  • Dynamic Environments: The current approach implicitly assumes static 3D scenes. Extending this to dynamic environments (e.g., objects moving, changes in scene layout) would introduce new challenges for both 3D scene representation and programmatic reasoning.

    Overall, this paper represents a significant step towards practical zero-shot open-vocabulary 3DVG, showcasing the immense potential of combining LLMs with specialized vision modules for robust and interpretable AI systems.