Bibliographic Information

• Title: Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answering
• Authors: Kaixuan Jiang, Yang Liu, Weixing Chen, Jingzhou Luo, Ziliang Chen, Ling Pan, Guanbin Li, Liang Lin
• Affiliations: Sun Yat-sen University, Peng Cheng Laboratory, Hong Kong University of Science and Technology
• Journal/Conference: The paper is an arXiv preprint, a non-peer-reviewed manuscript submitted to a public repository. This allows for rapid dissemination of research but means it has not yet undergone formal academic peer review.
• Publication Year: 2025 (as per the preprint metadata)
• Abstract: The authors address critical limitations in Embodied Question Answering (EQA), including inefficient exploration, biased datasets that encourage non-exploratory reasoning, and inadequate evaluation metrics. To solve these issues, they introduce three main contributions:
  1. EXPRESS-Bench: The largest EQA dataset specifically designed to evaluate both exploration and reasoning, consisting of 777 trajectories and 2,044 question-trajectory pairs.
  2. Fine-EQA: A hybrid exploration model that combines frontier-based and goal-oriented navigation to guide agents more effectively to task-relevant areas.
  3. Exploration-Answer Consistency (EAC): A novel evaluation metric that assesses whether an agent's answer is grounded in the visual evidence it gathered during exploration. The paper concludes that extensive experiments demonstrate the effectiveness of their benchmark and model in advancing the field.
• Original Source Link:
  • Original Source: https://arxiv.org/abs/2503.11117v3
  • PDF: https://arxiv.org/pdf/2503.11117v3.pdf

Executive Summary

Background & Motivation (Why)

Embodied Question Answering (EQA) is a task where an AI agent must navigate a 3D environment to visually find the answer to a question (e.g., "What color is the blanket on the bed?"). Unlike traditional question answering that uses static images or text, EQA requires a combination of navigation, perception, and reasoning.

The paper identifies three major problems with current EQA research:

1. Inefficient Exploration: Most agents use a frontier-based exploration strategy, where they simply move to the edge of explored territory. This works in open spaces but is highly inefficient in cluttered, complex environments like a real house, leading to redundant or incomplete exploration.
2. Biased Datasets and "Disembodied Reasoning": Existing datasets often contain biases or clues that allow AI models to "cheat." For instance, if a question mentions the "living room," a model might guess the answer based on general knowledge about living rooms without ever navigating there. This is called unfaithful question answering, where the answer is not grounded in what the agent actually saw.
3. Inadequate Evaluation: Standard metrics typically just check if the generated answer is textually similar to the correct answer. They fail to verify if the agent actually saw the evidence needed to answer the question, and they cannot detect when a model "hallucinates" a plausible but unverified answer.

Main Contributions / Findings (What)

To tackle these challenges, the paper introduces a comprehensive, three-part solution:

1. A New Benchmark (EXPRESS-Bench): The authors created a large-scale dataset specifically designed to test exploration. It avoids common biases by ensuring questions require active navigation and that answers are unique within the scene, forcing the agent to find the correct location rather than guessing.

2. A Hybrid Exploration Model (Fine-EQA): They propose a more intelligent navigation model that combines two strategies:

   • Frontier-Based Exploration (FBE): For broadly mapping the environment.
   • Goal-Oriented Exploration (GOE): For finely investigating specific, task-relevant regions (e.g., focusing on the kitchen if the question is about the stove). This hybrid approach allows the agent to explore both efficiently and thoroughly.

3. A Grounding-Aware Metric (Exploration-Answer Consistency - EAC): This novel metric evaluates answers on two criteria: correctness (is the answer right?) and grounding (did the agent see the visual evidence to support its answer?). This provides a much more rigorous and faithful assessment of an agent's true capabilities.

   Experiments show that Fine-EQA outperforms existing methods on EXPRESS-Bench and other datasets, while the EAC metric successfully penalizes models that generate ungrounded, "hallucinated" answers.

Prerequisite Knowledge & Related Work

Foundational Concepts

• Embodied AI: A subfield of artificial intelligence where agents (e.g., robots or virtual avatars) learn to operate within an environment. Unlike disembodied AI that processes static data, embodied agents must perceive their surroundings, make decisions, and perform actions (like moving or interacting with objects) to achieve goals.
• Embodied Question Answering (EQA): A specific task in Embodied AI. An agent is placed in a 3D environment (like a simulated house) and given a question (e.g., "How many chairs are at the dining table?"). The agent must navigate the environment to find the relevant visual information and then provide an answer. This contrasts with Visual Question Answering (VQA), where the AI is given a single, static image and a question about it.
• Frontier-Based Exploration (FBE): A classic navigation algorithm. The agent maintains a map of the environment, dividing it into three types of space: explored, unexplored, and frontiers. Frontiers are the boundaries between explored and unexplored regions. The agent's strategy is to continually move to the nearest frontier to expand its map and discover new areas. While systematic, it can be inefficient in complex rooms or hallways.
• Goal-Oriented Exploration (GOE): A more directed exploration strategy. Instead of exploring everything, the agent uses the question's content to identify semantically relevant areas. For a question about a "microwave," a GOE strategy would prioritize navigating to the "kitchen" over the "bedroom."
• Vision-Language Models (VLMs): AI models, such as GPT-4o, that can process and understand information from both images (vision) and text (language) simultaneously. In EQA, they are crucial for understanding the question, interpreting the agent's visual observations, and generating a natural language answer.
• Habitat Simulator: A popular open-source simulation platform for Embodied AI research. It allows researchers to train and test agents in realistic 3D environments, such as the HM3D (Habitat-Matterport 3D) dataset, which consists of reconstructions of real-world buildings.

Previous Works

The paper situates its contributions by comparing against previous EQA datasets and methods.

• Early EQA Datasets (EQA-v1, MP3D-EQA): These datasets often used rule-based templates to generate questions (e.g., "What color is the [OBJECT]?"). This led to simple, repetitive questions and limited linguistic diversity. Many also included biases that allowed models to succeed without true exploration.
• Large Model-Generated Datasets (HM-EQA, S-EQA): With the rise of VLMs, newer datasets used these models to generate more natural and diverse questions. However, the paper argues that these datasets still overlook the active exploration component and suffer from issues like non-unique answers (e.g., multiple bathrooms in a house, leading to ambiguity). OpenEQA was a step forward with manually created open-ended questions but, according to the authors, still focused more on memory than active exploration.
• Exploration Methods: Most prior EQA agents relied heavily on frontier-based exploration. While some works integrated semantic information (e.g., using a VLM to assign importance scores to frontiers), they still struggled with inefficiencies in cluttered scenes and failed to perform fine-grained exploration of task-relevant areas.

Differentiation

EXPRESS-Bench, Fine-EQA, and EAC collectively offer a more complete solution to the EQA problem:

• EXPRESS-Bench vs. Other Datasets: It is the largest benchmark focused on exploration and is carefully filtered to ensure unique answers, forcing agents to find the correct object/location. It provides full ground-truth exploration trajectories, which is missing in many other datasets. (This is a manual transcription of the data in Table 1.)

  	Simulator	Dataset	Real Scenes	Exploration Track	Track Numbers	Target Point	Question Creation	Open Vocab
  EQA-v1 [7]	House3D	SUNCG	✗	✓		✓	Rule-Based	✗
  MP3D-EQA [37]	MINOS	MP3D	✓	✓		✓	Rule-Based	✗
  MT-EQA [43]	House3D	SUNCG	✗	✓		>xx×x*x	Rule-Based	✗
  IQA [11]	AI2THOR		✗	✗			Rule-Based	✗
  VideoNavQA [4]	House3D	SUNCG	✗	✗			Rule-Based
  K-EQA [35]	AI2Thor		✗	✗			Rule-Based	✗
  HM-EQA [31]	Habitat	HM3D	✓	✗			VLMs	✗
  S-EQA [8]	VirtualHome		✗	✗			LLMs	✗
  NoisyEQA [39]				✗			VLMs
  CityEQA [45]	EmbodiedCity		✗	✗		✓	Manual	✓
  OpenEQA [29]	Habitat	ScanNet/HM3D	√		152	✗	Manual	√
  EXPRESS-Bench (Ours)	Habitat	HM3D			777		VLMs

• Fine-EQA vs. Other Models: It proposes a hybrid exploration strategy that flexibly switches between broad and fine-grained search, making it more efficient and effective than pure FBE or GOE models.

• EAC vs. Other Metrics: It is the first metric to explicitly measure answer grounding, providing a more reliable assessment of a model's performance and penalizing unfaithful, "hallucinated" answers.

Methodology (Core Technology & Implementation Details)

The paper's methodology can be broken down into three core technical contributions: the benchmark (EXPRESS-Bench), the evaluation metric (EAC), and the exploration model (Fine-EQA).

3.1-3.3. EXPRESS-Bench: A New Benchmark

The construction of EXPRESS-Bench follows a three-stage pipeline, designed to create high-quality, exploration-centric EQA tasks.

该图像是图2的示意图，展示了EXPRESS-Bench构建流程，包括轨迹生成、问答对生成和数据筛选的步骤，强调了手动筛查确保数据真实性和唯一性。

1. Stage 1: Trajectory Generation.

   • In a simulated HM3D scene, two random navigable points are chosen: an initial position and a target position.
   • The shortest path (sequence of "move forward," "turn left," "turn right") between these points is calculated to serve as the ground truth trajectory.
   • The agent's state (coordinates, orientation, visual observation) is recorded at every step. This sequence of first-person views is also compiled into a trajectory video.

2. Stage 2: Question-Answer Pair Generation.

   • The final visual observation from the ground truth trajectory (i.e., the view at the target location) is fed into a VLM (GPT-4o-mini). This is a key design choice, as it ensures the question is answerable from a specific, reachable viewpoint.
   • The VLM is prompted to generate open-ended questions and answers that are natural for a home scenario, based on the provided image.

3. Stage 3: Data Filtering.

   • A crucial manual review process is performed to ensure data quality and eliminate biases.

   • Uniqueness of Answers: To prevent ambiguity (e.g., "the bedroom" when there are three bedrooms), questions are filtered. A question is kept only if the target region is unique or is the closest one of its type to the agent's starting point. This forces the agent to explore and locate the correct region.

   • Relevance and Clarity: Reviewers ensure questions are relevant to the scene and add contextual details if necessary to make the task clear.

     The final EXPRESS-Bench contains 777 trajectories and 2,044 question-trajectory pairs, covering seven question categories as shown below.

     该图像是图表，展示了EXPRESS-Bench数据集中不同类型问题的示例和统计分布。左侧列举了七类问题（object、knowledge、existence、counting、attribute、location、state）对应的问题-回答示例，右侧为2044个问题的分类比例环形图。

3.4. Exploration-Answer Consistency (EAC) Metric

To address unfaithful answering, the authors propose the EAC metric, which evaluates not just what the agent answered, but how it arrived at that answer. It combines a correctness score and a grounding score.

该图像是图4，展示了探索-答案一致性（EAC）评价指标的示意图。图中包括问题、答案、响应和对应图像输入，通过视觉语言模型（VLM）计算多个变量，最终得出路径探索一致性得分 $E_{path}$。

• Correctness Score ($\sigma_i$): A VLM evaluates the agent's generated answer ($A_i$) against the ground truth answer ($A_i^*$) and the question ($Q_i$). Importantly, it also considers the agent's final visual observation ($I_i$). This allows the VLM to give partial credit for answers that are plausible given what the agent saw, even if they don't perfectly match the ground truth. $$\sigma _ { i } = \varphi ( Q _ { i } , A _ { i } ^ { * } , A _ { i } , I _ { i } )$$

  • $\sigma_i$ is a score from 1 to 5.

• Grounding Score ($\delta_i$): This novel component assesses if the agent's final observation ($I_i$) is relevant to the question ($Q_i$) and supports its answer ($A_i$). $$\delta _ { i } = \psi ( Q _ { i } , A _ { i } , I _ { i } )$$

  • $\delta_i$ is assigned one of three values:

    • 1: The observation is relevant, and the answer is consistent with the observation.

    • 0.5: The observation is relevant, but the answer is incorrect or misdescribes what is seen.

    • 0: The observation is irrelevant to the question. The agent is likely hallucinating an answer without having found the necessary visual evidence.

      Using these scores, two final metrics are calculated:

1. Overall Correctness ($C$): The average score, weighted by both correctness and grounding. $$C = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \frac { \sigma _ { i } \times \delta _ { i } } { 5 } \times 1 0 0 \%$$

2. Path Efficiency ($E_{path}$): This metric balances correctness with navigation efficiency. An agent gets a high score if it gives a correct, grounded answer while taking a path similar in length to the optimal one. $$E _ { p a t h } = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \frac { \sigma _ { i } \times \delta _ { i } } { 5 } \times \frac { l _ { i } } { m a x ( p _ { i } , l _ { i } ) } \times 1 0 0 \%$$

   • $N$: Total number of questions.
   • $l_i$: Length of the ground-truth shortest path.
   • $p_i$: Length of the path actually taken by the agent.

4. Fine-EQA: A Hybrid Exploration Framework

Fine-EQA is a two-stage framework that intelligently switches between exploration strategies to efficiently find task-relevant information.

该图像是一个示意图，展示了论文中提出的结合主动探索和目标导向探索的混合导航模型及其在语义地图构建和问答任务中的应用流程。

4.2. Frontier-Based Exploration (FBE)

This stage is used for broad, initial exploration to map out the environment. It enhances traditional FBE by incorporating semantic information.

• Semantic Map ($M_{sem}$): The agent builds a 2D map of the environment. For different points on the map, a VLM assigns a semantic value based on task relevance. This value is a fusion of a local value $v_l$ (relevance of a specific point) and a global value $v_g$ (relevance of the overall scene).
• Frontier Selection: The agent identifies frontiers (boundaries of the known area). Each frontier is assigned a weight $w_i$ based on:
  • $v_{sem}^i$: Its semantic value.
  • $r_e^i, r_o^i$: The amount of unexplored and unoccupied space beyond it.
  • $dis(f_i, p_{cur})$: Its distance from the agent (closer frontiers are preferred). The agent then probabilistically selects a frontier to move toward, prioritizing those that are semantically relevant and lead to large, unexplored areas.

4.3. Goal-Oriented Exploration (GOE)

This stage is activated when a task-relevant region is identified, allowing for fine-grained investigation.

• Functional Region Semantic Mapping: During exploration, a VLM classifies the agent's view into functional regions (e.g., "kitchen," "bedroom"). This information is stored in a separate functional region semantic map ($M_{reg}$).

• Task-Relevant Region Prioritization: An LLM analyzes the question to determine a priority order for these functional regions. For "Where is the toaster?", the "kitchen" would be the highest priority.

• Masked Semantic Mapping: Once the agent enters a high-priority region, it "masks" its global semantic map ($M_{sem}$) to focus only on that region. It then navigates to the point with the highest semantic value within that masked region. This ensures a thorough search of the most important area. To avoid getting stuck, a limit is placed on exploration within any single region.

  By switching between FBE for general mapping and GOE for focused searching, Fine-EQA achieves both broad coverage and detailed investigation.

Experimental Setup

Datasets

• EXPRESS-Bench: The primary dataset used for evaluation, created by the authors. It contains 2,044 question-trajectory pairs in HM3D scenes, simulated in Habitat. Its key features are its large scale, focus on active exploration, and manually-verified unique answers.
  • Data Sample Example: As shown in Figure 13 of the paper, a trajectory might end in a bathroom. An example question could be:
    • Question Type: Location
    • Question: I forgot where I leave my mug in the dining room. Do you see it? (Note: this example text is from a prompt, the actual question would be about the bathroom scene). A more fitting question for the bathroom view would be: Where is the bathrobe?
    • Answer: It's hanging on the hook on the door.
• OpenEQA (A-EQA subset): A publicly available benchmark used for external validation. The authors test their model on the "Active EQA" subset, which requires exploration.
• HM-EQA: Another existing EQA dataset used for comparison. It features multiple-choice questions.

Evaluation Metrics

The primary metrics are those introduced by the paper, based on the EAC framework.

• $C$ (Overall Correctness): Measures answer accuracy, penalized if the answer is not grounded in visual evidence.
• $E_{path}$ (Path Efficiency): Measures grounded accuracy, penalized for inefficient navigation paths.
• $d_T$ (Target Distance): The final geodesic distance between the agent and the ground-truth target location. A smaller value indicates better navigation. $$d _ { T } = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } d i s _ { g } ( P _ { E } ^ { i } , P _ { T } ^ { i } )$$
  • $dis_g$: Geodesic distance (shortest path distance within the environment).
  • $P_E^i$: Agent's final position.
  • $P_T^i$: Ground-truth target position.
• $C*$: A variant of $C$ that ignores grounding ($\delta_i=1$ for all samples). It measures raw correctness, similar to metrics from prior work, and is used to evaluate non-exploring models. $$C ^ { * } = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \frac { \sigma _ { i } } { 5 } \times 1 0 0 \%$$
• For experiments on other datasets, their respective metrics are used ($C'$ and $E'$ for OpenEQA, Accuracy and Path Length for HM-EQA).

Baselines

The proposed Fine-EQA model is compared against a comprehensive set of baselines grouped into five categories:

1. Blind LLMs: Large Language Models (GPT-4, DeepSeek-V3) that answer the question with no visual input, relying solely on prior knowledge.
2. Socratic Models: An LLM is given textual descriptions of several frames from the ground-truth trajectory, generated by a VLM (GPT-4o-mini or LLaVA). The LLM must reason over these descriptions.
3. Multi-Frame VLMs: A VLM (GPT-4o-mini or LLaVA) is directly given multiple image frames from the trajectory and the question.
4. Exploring Agents: Agents that actively navigate the environment.
   • RE (Random Exploration): Agent explores randomly.
   • FBE (Frontier-Based Exploration): Agent uses a pure frontier-based strategy without semantics.
   • GOE (Goal-Oriented Exploration): Agent starts randomly and switches to goal-oriented search once a relevant region is found.
5. Human Agent: Human participants were given the ground-truth trajectory and question to provide an answer. This serves as an upper bound on performance.

Results & Analysis

The experiments rigorously evaluate the proposed benchmark, model, and metric.

Core Results

The main results on EXPRESS-Bench are presented in Table 2.

(This is a manual transcription of the data in Table 2.)

Key Observations:

• Exploration is Crucial: Blind LLMs and other non-exploratory agents perform poorly on metrics that matter ($C$ and Epath). Socratic Models achieve a decent $C*$ score, but this is based on being fed ground-truth frames, not active exploration.
• The Power of the EAC Metric: The Random Exploration (RE) agent achieves a high $C*$ score (62.75), suggesting good performance. However, its grounded correctness score $C$ plummets to 36.95. This large gap reveals that the agent is often "guessing" correctly without finding the right visual evidence—a hallucination problem that EAC successfully detects.
• Fine-EQA Excels: The proposed Fine-EQA model achieves the highest scores across all key metrics for exploring agents: $C$ (40.55), Epath (16.22), and dT (6.43, lowest distance to target). This demonstrates its superior ability to navigate efficiently, find the relevant information, and provide a grounded answer.
• Gap to Human Performance: All models are far behind the Human Agent's $C*$ score of 83.99, indicating that exploration-aware EQA remains a very challenging task.

Experiments on Other Datasets

Fine-EQA was also tested on existing benchmarks to prove its generalizability.

(This is a manual transcription of the data in Tables 3 and 4.) Table 3: Performance on OpenEQA (A-EQA subset)

Table 4: Performance on HM-EQA

Fine-EQA outperforms the state-of-the-art on both datasets. The most dramatic improvement is in $E'$ (efficiency) on OpenEQA, where Fine-EQA scores 29.16 compared to 7.5, showcasing its vastly superior navigation strategy.

Ablation Studies

Ablation studies were conducted to understand the contribution of each component of Fine-EQA.

(This is a manual transcription of the data in Table 5.) Table 5: Ablation of Model Modules

Removing either Frontier-Based Exploration (FBE) or Goal-Oriented Exploration (GOE) hurts performance. Removing FBE (leaving only GOE) results in a larger drop, highlighting the importance of the broad exploration phase for initially mapping the environment.

Effectiveness of Exploration and Answering

The paper provides qualitative and quantitative evidence for Fine-EQA's effectiveness.

• Exploration Effectiveness (Fig. 6): Visualizations of exploration paths show that RE is chaotic, while FBE and GOE are more structured but can still be inefficient. Fine-EQA's path is the most direct and purposeful, quickly identifying the relevant "bathroom" region and exploring it thoroughly.

  该图像是一个示意图，展示了四种不同的环境探索策略（随机探索RE、基于边界的探索FBE、目标导向探索GOE和Fine-EQA）的路径及其对应的视角照片。图中包含了提问“我把浴巾放在浴室哪里？”及其答案，突出Fine-EQA策略在任务相关区域的高效探索能力。

• Answer Faithfulness (Fig. 7 & Table 7): The authors use a VLM to rate the confidence that an image can be used to answer the question. The final frame observed by the Fine-EQA agent consistently receives the highest confidence score, confirming that the agent stops at a viewpoint that contains the necessary information. Quantitative results in Table 7 further support this, with Fine-EQA achieving the highest ACE (Average Confidence) and WCE (path-weighted confidence).

  该图像是一个示意图，展示了EXPloration-awaRe Embodied queStion anSwering Benchmark (EXPRESS-Bench)中的探索轨迹及问答过程。右侧显示连续视角帧及对应的环境探索置信度，体现了Fine-EQA模型在任务相关区域的导航和答案推理。

Conclusion & Personal Thoughts

Conclusion Summary

This paper makes a significant, multi-faceted contribution to the field of Embodied Question Answering. The authors identify and address systemic issues of unfaithful reasoning, inefficient exploration, and inadequate evaluation. Their three key contributions are:

1. EXPRESS-Bench: A large-scale, high-quality benchmark that pushes the community to develop agents with genuine exploration capabilities.

2. Fine-EQA: A novel and effective hybrid exploration model that balances broad discovery with focused, task-relevant investigation, setting a new state-of-the-art.

3. EAC Metric: An innovative evaluation protocol that measures the crucial link between exploration and answering, promoting the development of more "faithful" and less "hallucinating" agents.

   Together, these contributions provide a more rigorous framework for developing and evaluating the next generation of embodied AI agents.

Limitations & Future Work

The paper's conclusion is brief, but we can infer several limitations and directions for future work:

• Model Complexity and Cost: Fine-EQA is not a single end-to-end model but an engineered system that relies on multiple, separate calls to powerful (and often proprietary and expensive) models like GPT-4 and GPT-4o-mini. This makes it computationally intensive and may limit its applicability in real-time robotics.
• Scalability of Dataset Creation: The high quality of EXPRESS-Bench is partly due to a meticulous manual filtering process. This process is difficult to scale and may inadvertently introduce its own subtle human biases.
• Handling Ambiguity: The benchmark deliberately removes ambiguity by ensuring unique answers. While this is useful for evaluation, real-world scenarios are often ambiguous. Future work could explore how agents can handle such ambiguity, for example, by asking clarifying questions.
• From Simulation to Reality: The work is conducted entirely in simulation. Transferring these complex navigation and reasoning strategies to a physical robot, with noisy sensors and unpredictable dynamics, remains a major challenge.

Personal Insights & Critique

This is a strong and well-rounded paper that moves the EQA field in the right direction.

• Holistic Approach: The most impressive aspect is the holistic nature of the contribution. Instead of just proposing a new model, the authors tackled the entire research cycle: the data (EXPRESS-Bench), the model (Fine-EQA), and the evaluation (EAC). This is far more impactful than a piecemeal improvement.

• EAC is a Game-Changer: The Exploration-Answer Consistency metric is arguably the most important contribution. For too long, AI research has been plagued by models that find shortcuts and produce plausible-sounding nonsense. By explicitly measuring and rewarding the grounding of answers in evidence, EAC promotes the development of AI that is not only correct but also trustworthy and interpretable. This principle is transferable to many other domains beyond EQA.

• Critique of the Fine-EQA Model: While effective, Fine-EQA feels more like a carefully engineered pipeline than a learned, emergent behavior. It combines several handcrafted modules and heuristics (e.g., region-specific exploration limits, switching logic). The future of Embodied AI likely lies in developing more integrated, end-to-end learning systems that can discover such complex strategies on their own through interaction and reinforcement learning, rather than having them pre-programmed.

  Overall, this paper provides a valuable benchmark and a strong baseline model, but its most lasting impact may be the push for more faithful and rigorous evaluation in embodied intelligence.