1. Bibliographic Information

1.1. Title

EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments

1.2. Authors

Dongping Li (Zhejiang University, UIUC), Tielong Cai (Zhejiang University), Tianci Tang (Zhejiang University), Wenhao Chai (University of Washington), Katherine Rose Driggs-Campbell (UIUC), Gaoang Wang (Zhejiang University).

1.3. Journal/Conference

Published at (UTC): 2025-03-11. Status: Preprint (arXiv). Context: While currently a preprint, the work targets the intersection of robotics and large language models (LLMs), a rapidly evolving field often presented at top conferences like CVPR, ICRA, or IROS.

1.4. Publication Year

2025

1.5. Abstract

This paper addresses the challenge of developing autonomous home robots that can perform complex, long-horizon tasks via natural language instructions. The authors identify key gaps in current research: the lack of unified benchmarks for complex tasks, inadequate evaluation metrics, and data incompatibility between LLMs and robot trajectories. To solve this, they propose EMMOE, a comprehensive benchmark integrating high-level planning and low-level execution. They introduce EMMOE-100, a dataset of 100 diverse everyday tasks with detailed annotations, and HoMiEBoT, an agent system combining a Visual Language Model (VLM) planner with lightweight execution policies. They also propose three new metrics (Task Progress, Success End Rate, Success Re-plan Rate) to better evaluate diverse aspects of robot performance.

1.6. Original Source Link

https://arxiv.org/abs/2503.08604

2. Executive Summary

2.1. Background & Motivation

The dream of having a robot butler that can "tidy up the living room" or "prepare breakfast" just by asking is a long-standing goal in AI and robotics. This field is known as Embodied AI.

• The Problem: While Large Language Models (LLMs) like GPT-4 are great at reasoning, and robot control policies are getting better at specific movements (like picking up a cup), connecting the two is difficult.
• Existing Gaps:
  1. Fragmentation: Most benchmarks focus either on high-level planning (deciding what to do) or low-level control (deciding how to move motors), but rarely both in a unified, complex setting.
  2. Evaluation Limits: Current metrics usually just check if the final goal was reached (Success/Fail). They don't account for how the robot behaved, whether it realized it failed and tried again, or if it knew when to stop.
  3. Data Mismatch: LLMs are trained on internet text (conversations), while robots need trajectory data (sequences of states and actions). This makes "grounding" (connecting language to physical reality) hard.

2.2. Main Contributions / Findings

1. EMMOE Benchmark: A unified testing ground that requires agents to interpret instructions and execute long-horizon daily tasks in a continuous 3D simulated environment.

2. EMMOE-100 Dataset: A collection of 100 complex tasks (e.g., "Check if there are bananas in the fridge; if not, get one from the kitchen") with rich annotations, including reasoning chains and "re-plan" data where the robot fixes its own mistakes.

3. New Metrics: Three novel metrics designed to evaluate the process of execution, not just the result:

   • Task Progress (TP): How much of the task was completed?
   • Success End Rate (SER): Did the robot know when to stop?
   • Success Re-plan Rate (SRR): Could the robot recover from failure?

4. HoMiEBoT Agent: A hierarchical agent system that uses a fine-tuned Multi-modal LLM for planning and specialized lightweight models for navigation and manipulation.

5. Data Alignment Strategy: A method to convert robot trajectory data into conversation-style data suitable for training LLMs, including Direct Preference Optimization (DPO) to align the model with correct behaviors.

   The following figure (Figure 1 from the original paper) illustrates an example task from the EMMOE-100 dataset, highlighting the need for reasoning (checking the fridge first) and interleaved execution (moving, opening, picking).

   该图像是图示示例，展示了在EMMOE-100中的任务执行过程。图中代理人需要先前往冰箱并打开，然后在厨房拿取香蕉。这一过程强调了推理及交错执行的必要性，只有完成所需步骤才能成功。

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Embodied AI: Artificial Intelligence that controls a physical body (like a robot) or a simulated body within an environment. Unlike a chatbot that lives on a server, an embodied agent must perceive its surroundings (vision) and act upon them (motor control).
• Mobile Manipulation: A specific type of robot task where the robot must move around (navigate) to different locations and interact with objects (manipulate), such as opening doors or picking up items.
• Task and Motion Planning (TAMP): A hierarchical approach to robotics.
  • Task Planning (High-Level): Discrete logic, e.g., "Go to kitchen -> Open fridge -> Pick apple."
  • Motion Planning (Low-Level): Continuous control, e.g., "Move wheels at velocity $v$ for $t$ seconds," "Rotate arm joint $\theta$ by 30 degrees."
• Large Language Model (LLM) & Visual Language Model (VLM):
  • LLM: AI trained on text to generate human-like text (e.g., GPT-4).
  • VLM / LMM (Large Multi-modal Model): An LLM that can also "see" images or videos. In this paper, the planner sees what the robot sees to make decisions.
• Direct Preference Optimization (DPO): A training technique for LLMs. Instead of just teaching the model "say this next," DPO uses pairs of answers—one "chosen" (better) and one "rejected" (worse)—to mathematically force the model to prefer the better behavior. This is often more stable than traditional Reinforcement Learning from Human Feedback (RLHF).
• Grounding: The problem of mapping abstract concepts (the word "apple") to concrete physical data (pixels of a red round object, or the coordinates (x,y,z)).

3.2. Previous Works

• ALFRED: A famous benchmark for following instructions to complete daily tasks. However, it relies on discrete actions (teleporting or pre-defined moves) rather than continuous control, limiting its realism.
• Habitat: A high-performance 3D simulator for training embodied agents in photorealistic environments. EMMOE is built inside Habitat.
• Visual Language Navigation (VLN) & Action (VLA):
  • VLN: Focuses only on moving to a target (e.g., "Go to the bedroom").
  • VLA: End-to-end models (like RT-2) that take vision/language in and output robot actions directly. The authors argue VLA models can be heavy and hard to generalize to long tasks, preferring a hierarchical approach.

3.3. Differentiation Analysis

• vs. ALFRED/BEHAVIOR: EMMOE integrates continuous mobile manipulation with high-level planning. Unlike ALFRED (discrete) or BEHAVIOR (often assumes perfect high-level plans or focuses on one aspect), EMMOE forces the agent to handle both the "thinking" (logic) and the "doing" (continuous movement) simultaneously.
• vs. End-to-End VLA: Instead of training one giant model to do everything, EMMOE promotes a modular system (HoMiEBoT) where a VLM handles logic and small, specialized models handle specific skills (like grasping). This allows for easier debugging and re-planning.

4. Methodology

4.1. Principles

The core principle of EMMOE is hierarchical modularity with feedback.

1. High-Level Planning (HLP): A "brain" (VLM) that looks at the environment and decides the next immediate subtask (e.g., "Open the drawer").

2. Low-Level Execution (LLE): "Muscle" models (specialized neural networks) that execute that subtask (e.g., move the arm to the drawer handle).

3. Closed-Loop Feedback: If the "muscle" fails (e.g., "drawer is locked"), it reports this error back to the "brain," which then reasons out a new plan (e.g., "Find the key").

   The following figure (Figure 2 from the original paper) provides an overview of this hierarchical framework, showing the flow from High-Level Planning to Low-Level Execution.

   该图像是图示，展示了 HomieBot 的层次框架。此框架通过高层规划将任务分解为易于管理的动作，并通过低层执行实现这些动作，同时提供实时反馈。图中展示了 LLM、动作感知及环境交互的过程。相关公式可表示为 $O = \{A, S, m\}$。

4.2. Core Methodology In-depth

4.2.1. High-Level Planning (HLP)

The HLP module is responsible for embodied decision-making. It uses a Multi-modal LLM (specifically Video-LLaVA) fine-tuned on the EMMOE dataset.

Step 1: Multi-modal Instruction Construction To make a decision, the agent constructs a rich input prompt $I$. The formula for this input is: $$I = \{ o _ { 1 \sim 4 } , s , T , i n v , h , f \}$$

• $o_{1 \sim 4}$: Visual observation. Four images representing the First-Person View (FPV) in four directions (front, left, back, right). This gives the agent a panoramic understanding.
• $s$: System information. Constant instructions defining the agent's role (e.g., "You are a home robot...").
• $T$: User Task. The natural language command (e.g., "Put a banana in the fridge").
• inv: Inventory. What the robot is currently holding (e.g., "banana" or "None").
• $h$: Execution history. A log of past actions and whether they succeeded or failed.
• $f$: Feedback. The result of the immediately preceding action (e.g., "Success" or "Failed: Target too far").

Step 2: Output Generation The model processes $I$ and generates a structured output $O$: $$O = M ( I ) = \{ A , S , m \}$$ Where the subtask $S$ is defined as: $$S = \{ \mathsf { a c t i o n , t a r g e t } \}$$

• $A$ (Analysis): A Chain-of-Thought (CoT) reasoning text where the agent explains why it is choosing the next step (e.g., "I see the fridge is closed, so I must open it before putting the banana inside.").
• $S$ (Subtask): The specific discrete command.
  • $\mathsf{action}$: Chosen from a pre-defined list (Go to, Pick, Put, Open, Close, End).
  • $\mathsf{target}$: The object or location to interact with (e.g., "fridge").
• $m$ (Model Selection): The agent selects which low-level "muscle" model to use (e.g., "Use PixNav for navigation" or "Use RT-1-X for picking"). This is a unique feature; the "brain" chooses the best tool for the job.

4.2.2. Low-Level Execution (LLE)

Once the HLP generates a subtask (e.g., [Pick, banana]), the LLE module takes over.

• Model Execution: The system calls the specific model selected by $m$. The authors use lightweight models like NoMaD (for navigation), RT-1-X (for picking/placing), and Octo (for opening/closing).
• Error Detection: This is critical for the "feedback" loop. The system categorizes errors into four types to give specific feedback to the planner:
  • Logical Error ($L$): e.g., Trying to pick something when hands are full (L1), or opening a non-openable object (L4).
  • Distance Error ($D$): e.g., Too far to reach (D1), or too close to move arm (D2).
  • Format Error ($F$): e.g., Hallucinating an object that doesn't exist (F2).
  • Execution Error ($E$): e.g., The low-level model tried but dropped the object (E1).

4.2.3. Data Augmentation & Alignment (SFT & DPO)

To make the LLM good at this specific robotic reasoning, the authors augment their data.

1. SFT (Supervised Fine-Tuning) Dataset Construction:

• They collect robot trajectories.
• They use GPT-4o to rewrite the raw trajectory logs into the conversation format described above ($I \rightarrow O$).
• They generate "Analysis" (reasoning chains) for each step using GPT-4o, creating a dataset where the model learns how to think.

2. DPO (Direct Preference Optimization) Dataset Construction: DPO requires triplets of {Prompt, Chosen Answer, Rejected Answer} to teach the model what not to do.

• Re-plan Data: If the robot failed at step $i$ (Output $O_i$) but succeeded after re-planning with Output $O_{i+1}$, then:
  • Prompt: Input $I_i$
  • Rejected: $O_i$ (The failed attempt)
  • Chosen: $O_{i+1}$ (The successful correction)
• Synthetic Negatives: Since natural failures are rare, they artificially create "Rejected" examples:
  • Order Change: Shuffle the correct sequence of steps (teaching logic).
  • Action Change: Swap valid actions with invalid ones (e.g., "Pick" -> "Fetch" if "Fetch" isn't in the allowed dictionary).
  • Model Change: Swap the correct model with a wrong one (e.g., using a navigation model to pick up an object).

5. Experimental Setup

5.1. Datasets

• Name: EMMOE-100.
• Source: Collected using Fetch Robots in the Habitat-Lab 2.0 simulator, based on scenes from the Replica Challenge.
• Scale: 100 tasks in total (90 for training, 10 for testing). This seems small, but each task is a long-horizon activity containing many sub-steps (total 966 subtasks).
• Characteristics: Tasks are categorized into 5 attributes:
  1. Short-horizon: Simple pick-and-place.
  2. Long-horizon: >10 steps.
  3. Open-ended: Multiple solutions possible (e.g., "Clean the table" - order doesn't matter).
  4. Logical: Vague descriptions requiring inference (e.g., "I am hungry" -> Find food).
  5. Human-style: Natural conversational commands.

5.2. Evaluation Metrics

The paper argues that standard "Success Rate" is not enough for long tasks. They propose three new metrics.

5.2.1. Task Progress (TP)

Concept: Measures how much of the task was completed, even if it wasn't fully finished. This rewards partial success. Formula: $$T P = \operatorname* { m a x } _ { k _ { i } \in K _ { T } } \left( \frac { \mathrm { l e n } ( k _ { i } ^ { \mathrm { c h e c k } } ) } { \mathrm { l e n } ( k _ { i } ) } \right)$$ Symbol Explanation:

• $K_T$: The set of all valid "keypaths" (sequences of essential subtasks) that can complete task $T$. A task might have multiple valid ways to solve it.
• $k_i$: The $i$-th keypath in that set.
• $\mathrm{len}(k_i)$: The total number of steps in that keypath.
• $k_i^{\mathrm{check}}$: The subset of steps in $k_i$ that were successfully matched and executed by the robot in the correct order.
• The metric takes the maximum ratio across all possible valid paths.

5.2.2. Success End Rate (SER)

Concept: Evaluates the robot's self-awareness. Did the robot realize it finished the task and issue the [End] command? A robot that finishes the task but keeps spinning in circles is not fully successful. Formula: $$S E R = { \frac { \ \operatorname { l e n } ( S ) } { \sum _ { t \in M } \operatorname { c o u n t } _ { t } ( \operatorname { e n d } ) } }$$ Symbol Explanation:

• $S$: The set of truly successful trajectories (where TP = 100%).
• $M$: The set of all trajectories attempted.
• $\operatorname{count}_t(\operatorname{end})$: A binary function (1 or 0). It is 1 if the robot outputted the End action at the conclusion of trajectory $t$.
• Interpretation: The formula essentially calculates: Of all the times the robot said "I'm done" (outputted End), what fraction were actually successful? (Note: The text description says "ratio of successful trajectories to the number of trajectories that the agent deemed successful." The formula matches this interpretation).

5.2.3. Success Re-plan Rate (SRR)

Concept: Measures resilience. If the robot fails a step, can it recover? Formula: $$S R R = \frac { \sum _ { t \in S } \mathrm { c o u n t } _ { t } ( \mathrm { r e p l a n } ) } { \sum _ { t \in M } \mathrm { c o u n t } _ { t } ( \mathrm { r e p l a n } ) }$$ Symbol Explanation:

• $\mathrm{count}_t(\mathrm{replan})$: The number of times a "re-plan" event occurred in trajectory $t$. A re-plan happens when an error is detected and the agent tries a new action.
• Numerator: Total re-plans that occurred in successful trajectories.
• Denominator: Total re-plans that occurred in all trajectories.
• Interpretation: This ratio tells us: When the robot tries to fix a mistake, how often does that effort contribute to an eventual victory?

5.3. Baselines

• Closed-Source LLMs: GPT-4o, Gemini-1.5-Pro, OpenAI o1 (reasoning model).
• Open-Source VLMs: Qwen2-VL-7B, MiniCPM-V 2.6.
• These are compared against the authors' HoMiEBoT (both SFT and SFT+DPO versions).

6. Results & Analysis

6.1. Core Results Analysis

The experimental results highlight the difficulty of the EMMOE benchmark and the effectiveness of the proposed alignment methods (SFT and DPO).

The following are the results from Table 2 of the original paper, comparing different models on the EMMOE-100 tasks:

Analysis:

1. DPO Superiority: HoMiEBoT (SFT+DPO) achieves the highest Success Rate (30.30%) and Task Progress (51.39%), validating that the DPO training (specifically teaching the model what not to do) significantly improves adherence to the complex task logic.

2. Open Source vs. Closed Source: Standard open-source models (Qwen2, MiniCPM) fail almost completely (SR ~1%) without fine-tuning. This proves that the EMMOE tasks require specific domain grounding that general VLMs lack.

3. The Power of Reasoning (o1): The OpenAI o1 model performs remarkably well (28.67% SR), nearly matching the specifically fine-tuned HoMiEBoT. Notably, it has the highest SRR (13.80%), suggesting that stronger inherent reasoning capabilities allow for better recovery from failures (re-planning).

4. SER Metric: HoMiEBoT performs best on SER (60.81%), meaning it is the most reliable at knowing when the task is truly finished, likely due to the specific [End] token training in the DPO phase.

   The following are the results from Table 3 of the original paper, showing the train/test split performance to analyze generalization:

   MODEL	TRaiN SPLIT					TeST SPLIT
   	SR	PLWSR	TP	SRR	SER	SR	PLWSR	TP	SRR	SER
   HomieBoT (SFT)	28.52	21.49	50.16	9.59	53.85	20.00	15.36	51.19	6.55	54.55
   HoMIEBoT (SFT+DPO)	31.84	25.82	52.29	9.69	60.71	16.67	14.36	43.39	3.08	62.50

Analysis:

• Generalization Gap: The DPO model performs better on the training set but drops below the SFT model on the test set (SR 16.67% vs 20.00%). This indicates that DPO might be "overfitting" to the specific preferences in the training tasks, while SFT maintains slightly better generalization to unseen tasks. This highlights a trade-off between peak performance on known tasks and robustness on new ones.

6.2. Error Analysis

The paper conducts a detailed breakdown of why failures happen using the error codes (L, D, F, E).

The following figure (Figure 3 from the original paper) visualizes the distribution of error types across different models:

Key Findings:

• F2 Error (Physical Grounding/Hallucination): This is the most common error for baseline models (e.g., GPT-4o). The model asks to interact with an object that simply isn't there or cannot be seen (F2). This confirms that "grounding"—connecting language to the visible scene—is the biggest hurdle for general LLMs.
• D1 Error (Distance): This is common in successful trajectories. It means the robot was too far, received feedback ("Target too far"), and successfully corrected itself by moving closer. This shows the feedback loop is working.
• E1/E2 (Execution): Errors from the low-level models (dropping items) are persistent, limiting the overall ceiling of success regardless of how smart the planner is.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper makes a significant step towards practical embodied AI by introducing EMMOE, a benchmark that rigorously tests both the "brain" (planning) and "body" (execution) of robots.

• They proved that general-purpose VLMs struggle with robotic grounding without specific fine-tuning.
• They demonstrated that DPO is a powerful tool for aligning VLMs with robotic constraints (like knowing when to stop).
• The HoMiEBoT system shows that a hierarchical approach—separating logic from skill execution—is a viable path for long-horizon tasks.

7.2. Limitations & Future Work

Limitations identified by authors:

1. Simulation Only: All experiments were in Habitat. Real-world physics and noise are much harsher.
2. Limited Action Space: The robot has a fixed set of high-level actions (Pick, Place, etc.), which limits creativity.
3. DPO Generalization: As seen in the results, DPO improved training performance but hurt generalization to unseen test tasks compared to SFT.

Future Work:

• expanding the dataset size.
• moving to real-world robot evaluation.
• improving the efficiency of the system (reducing inference time).

7.3. Personal Insights & Critique

• Critique of Metrics: The SER (Success End Rate) is a brilliant addition. Many robot demos fail because the robot doesn't know it's done. Formalizing this as a metric is valuable. SRR (Re-plan Rate) is also insightful; it differentiates "lucky" successes from "robust" successes where the agent fought back from failure.
• The "Brain-Body" Gap: The error analysis shows that even with a smart planner, simple execution errors (E1 - dropping things) cause failure. This suggests that future work shouldn't just make LLMs smarter, but must make low-level controllers more robust or "aware" of the task context.
• DPO for Robotics: The application of DPO here is fascinating. Usually, DPO is for "chat safety" or "style." Using it to enforce "robotic logic" (e.g., don't pick if hands are full) is a novel and effective transfer of the technique. However, the overfitting issue observed in Table 3 is a critical warning for the community: preference optimization might narrow the model's flexibility if not careful.