1. Bibliographic Information

1.1. Title

Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning

1.2. Authors

Guillermo Garcia-Hernando, Edward Johns, and Tae-Kyun Kim. Their affiliations typically indicate a strong background in computer vision, robotics, and machine learning, particularly in areas involving human-computer interaction, hand pose estimation, and reinforcement learning for manipulation tasks.

1.3. Journal/Conference

Published at arXiv on 2020-08-07T00:00:00.000Z. arXiv is a popular open-access preprint server for research in physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. While it is not a peer-reviewed journal or conference in itself, papers posted on arXiv are often submitted to and later published in prestigious venues. The publication year indicates it is a relatively recent work in the field as of its publication.

1.4. Publication Year

2020

1.5. Abstract

This paper addresses the challenge of performing dexterous manipulation of virtual objects using only a depth sensor and a 3D hand pose estimator (HPE). The core problem lies in the physics-based nature of virtual environments and the lack of force feedback, which makes interactions prone to failure due to HPE inaccuracies. Traditional methods of generating contact forces are insufficient for complex dexterous manipulations like in-hand manipulation. Existing reinforcement learning (RL) and imitation learning (IL) approaches typically train agents without incorporating online user input.

The authors propose a novel approach where a model learns to map noisy input hand poses to target virtual poses that facilitate the necessary contacts in a physics simulator. This model is trained as a residual agent using a model-free hybrid RL+IL approach. A 3D hand pose estimation reward is introduced to improve HPE accuracy when the corrected poses are remapped to the input space, ensuring the generated motion remains visually consistent with the user's input. To address the lack of HPE sequences for successful virtual interactions, a data generation scheme is proposed for training and evaluation. The framework is validated in two applications: hand-object interactions in VR and hand-object motion reconstruction in-the-wild.

1.6. Original Source Link

https://arxiv.org/abs/2008.03285 This is the arXiv preprint link.

1.7. PDF Link

https://arxiv.org/pdf/2008.03285.pdf

2. Executive Summary

2.1. Background & Motivation

The paper tackles the problem of enabling dexterous manipulation of virtual objects using bare hands in virtual reality (VR) or augmented reality (AR) environments, relying solely on depth sensors and 3D hand pose estimators (HPEs). This is a crucial step towards creating more intuitive and immersive interactive experiences.

The core challenges are multifaceted:

1. Physics Realism: Virtual environments are governed by physics laws (e.g., object weights, surface frictions). Realistic interactions require the virtual hand to respect these laws.
2. Lack of Force Feedback: Unlike physical interactions, users manipulating virtual objects do not receive haptic or force feedback, making precise control difficult.
3. HPE Inaccuracies: 3D hand pose estimators provide noisy and imperfect estimates of human hand poses. Even slight errors in finger tips or contact points can cause physical simulations to fail, leading to unnatural or unsuccessful interactions.
4. Limitations of Prior Art:

   • Simple Contact Models: Previous methods often generate contact forces based on finger penetration into virtual objects. While useful for simple grasping, these approaches are inadequate for dexterous manipulations (e.g., in-hand manipulation where an object is repositioned within the hand) because they don't produce physically realistic motion and are sensitive to HPE noise.

   • Commercial Solutions: Some commercial products simplify interaction by ignoring physics and "attracting" objects to the hand, resulting in artificial motion.

   • RL/IL without User Input: Existing Reinforcement Learning (RL) and Imitation Learning (IL) approaches train agents to learn skills autonomously, often from expert demonstrations, but typically do not incorporate online user input to assist in real-time.

     The paper's innovative idea is to address these issues by introducing a residual agent that refines noisy user input (estimated hand poses) in real-time, making it physically plausible and task-effective within a physics simulator, while still keeping the generated motion visually similar to the user's intention.

2.2. Main Contributions / Findings

The paper makes several significant contributions:

1. Residual Learning Framework: Proposes a novel residual learning framework where an agent learns to apply small but crucial corrections (residual actions) to noisy input hand poses from an HPE. This allows for dexterous manipulation in a physics simulator while maintaining visual resemblance to the user's input.

2. Hybrid RL+IL Approach: The residual agent is trained using a model-free hybrid Reinforcement Learning (RL) and Imitation Learning (IL) approach. This combination helps the agent learn task-specific skills while also encouraging natural, human-like motion by mimicking expert demonstrations.

3. 3D Hand Pose Estimation Reward: Introduces a novel 3D hand pose estimation reward (r^{\mathsf{pose}}) during training. This reward term encourages the virtual hand's pose to stay close to the ground-truth human hand pose, improving HPE accuracy when the corrected virtual poses are re-mapped back to the input space. This helps maintain visual fidelity.

4. Data Generation Scheme: Develops a novel data generation scheme to create training data. Since HPE sequences of successful, physically accurate interactions do not naturally exist, the scheme leverages mocap datasets of successful manipulations and large-scale 3D hand pose datasets to synthesize noisy input hand poses corresponding to expert actions.

5. Validation in Diverse Applications: Demonstrates the framework's effectiveness in two distinct applications:

   • Hand-object interactions in VR: Simulating dexterous manipulations (e.g., door opening, in-hand pen manipulation, tool use, object relocation) using mid-air hand pose estimates.
   • Hand-object motion reconstruction in-the-wild: Reconstructing physics-based hand-object sequences from RGBD videos captured in uncontrolled environments, using estimated hand poses and initial object pose estimates.

6. Superior Performance: Experiments show that the proposed method consistently outperforms various RL/IL baselines and a simple prior art of enforcing hand closure in terms of both task success and hand pose accuracy.

   The key conclusion is that a residual learning approach, integrating RL, IL, and pose estimation rewards, can effectively bridge the gap between noisy vision-based hand pose estimates and physically realistic dexterous manipulation in virtual environments, offering a more intuitive and successful user experience without expensive haptic devices.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a foundational grasp of several concepts from computer vision, robotics, and machine learning is essential:

• 3D Hand Pose Estimation (HPE):

  • Conceptual Definition: 3D Hand Pose Estimation is the process of determining the 3D spatial coordinates (positions) of key anatomical points (called joints or keypoints) of a human hand from an input image (e.g., RGB, depth, or RGB-D). These keypoints typically correspond to the knuckles, fingertips, and wrist.
  • Relevance: HPE provides the user input for the proposed system. However, HPEs are inherently noisy and imperfect, especially in complex scenarios or with occlusions, which is the core problem this paper aims to address.

• Inverse Kinematics (IK):

  • Conceptual Definition: In robotics and computer graphics, kinematics describes the motion of a system of interconnected bodies (like a robot arm or a hand) without considering the forces that cause the motion. Forward Kinematics calculates the position of the end-effector (e.g., fingertip) given the joint angles. Inverse Kinematics is the reverse: it calculates the required joint angles (or actuator parameters) of a kinematic chain to achieve a desired end-effector position and orientation.
  • Relevance: The paper uses IK to map the estimated 3D keypoints of a human hand ($x_t$) to the actuator parameters ($a_t$) (joint angles) of a virtual hand model ($z_t$). This mapping ($κ$) forms the initial, potentially imperfect, user input for the residual agent.

• Reinforcement Learning (RL):

  • Conceptual Definition: Reinforcement Learning is a paradigm of machine learning where an agent learns to make optimal decisions by interacting with an environment. The agent performs actions, observes the state of the environment, and receives a reward or penalty. The goal is to learn a policy (a mapping from states to actions) that maximizes the cumulative reward over time.
  • Relevance: The residual agent in this paper is trained using RL to learn the residual actions ($f_t$) that correct the user input. RL allows the agent to discover complex behaviors that lead to successful physical interactions.

• Imitation Learning (IL):

  • Conceptual Definition: Imitation Learning (also known as learning from demonstration) is a type of machine learning where an agent learns a policy by observing demonstrations from an expert (e.g., human demonstrations). Instead of discovering behaviors through trial and error as in RL, the agent tries to mimic the expert's actions.
  • Relevance: RL alone can sometimes lead to unnatural or sub-optimal human-like motions. IL is combined with RL in this work (a hybrid RL+IL approach) to ensure the residual agent produces motions that are not only task-effective but also visually similar to expert human demonstrations. Generative Adversarial Imitation Learning (GAIL) is a specific IL technique mentioned.

• Physics Simulator (MuJoCo):

  • Conceptual Definition: A physics simulator is a software system that models the laws of physics (e.g., gravity, friction, collisions, joint limits) to predict how objects will move and interact in a virtual environment. MuJoCo (Multi-Joint dynamics with Contact) is a popular physics engine often used in robotics and RL research due to its speed and accuracy.
  • Relevance: The entire interaction and learning process takes place within a physics simulator. This is crucial for evaluating the physical plausibility and success of dexterous manipulations. The simulator provides the state information ($s_t$) to the RL agent.

• Proximal Policy Optimization (PPO):

  • Conceptual Definition: PPO is a model-free, on-policy algorithm for Reinforcement Learning. It's an actor-critic method that tries to take the largest possible improvement step on a policy without collapsing its performance. It achieves this by using a clipping mechanism or penalty to limit how much the policy can change in one update step, making it more stable and robust than previous policy gradient methods.
  • Relevance: PPO is the specific RL algorithm chosen to optimize the residual agent's policy due to its reputation for stability and success in dexterous manipulation tasks.

• Generative Adversarial Imitation Learning (GAIL):

  • Conceptual Definition: GAIL is an Imitation Learning algorithm that leverages the Generative Adversarial Network (GAN) framework. It consists of a generator (the policy trying to mimic expert behavior) and a discriminator. The discriminator tries to distinguish between expert trajectories and trajectories generated by the policy. The policy then learns to produce trajectories that can fool the discriminator, thereby mimicking expert behavior.
  • Relevance: The adversarial IL reward (r_t^{\mathsf{il}}) in this paper is based on GAIL, encouraging the residual agent to generate actions similar to those in expert demonstration trajectories.

• Residual Learning / Residual Policy Learning:

  • Conceptual Definition: Residual learning is a concept where a model learns a "residual function" that represents the difference or correction needed relative to some initial estimate or baseline. Instead of learning the entire output from scratch, it learns how to improve upon a given starting point.
  • Relevance: The core of the paper's method is a residual agent that learns residual actions ($f_t$) to correct the imperfect user input ($κ(x_t(φ_t))$), rather than generating the entire action from zero. This helps RL exploration and keeps the output visually close to the user's intent.

• Degrees of Freedom (DoF):

  • Conceptual Definition: In robotics or mechanics, Degrees of Freedom refer to the number of independent parameters (e.g., joint angles, translational movements) that define the configuration or state of a system. For a robot arm, each joint typically contributes one DoF (e.g., rotation around an axis). A rigid body in 3D space has 6 DoF (3 for position, 3 for orientation).
  • Relevance: The virtual hand models used (e.g., ADROIT, MPL) have many DoF (24 to 29 DoF), representing the complexity of dexterous hand manipulation. The action space of the RL agent directly corresponds to these DoF.

• PID controllers:

  • Conceptual Definition: A Proportional-Integral-Derivative (PID) controller is a control loop feedback mechanism widely used in industrial control systems. It calculates an "error" value as the difference between a measured process variable and a desired setpoint. The controller attempts to minimize the error by adjusting the process control inputs using proportional, integral, and derivative terms.
  • Relevance: In robotics simulation, PD (Proportional-Derivative) or PID controllers are often used to translate desired joint angles (the actions output by the policy) into torques that are applied to the joints of the virtual hand model to reach the target angles.

• Mocap (Motion Capture):

  • Conceptual Definition: Motion capture is the process of recording the movement of objects or people. In the context of hands, it often involves data gloves or optical tracking systems that precisely measure joint angles or 3D positions of markers. Mocap data is typically high-fidelity and noise-free.
  • Relevance: Mocap datasets (like the one from Rajeswaran et al. [1]) provide expert demonstrations that are used in the Imitation Learning component and as a source for the data generation scheme.

3.2. Previous Works

The paper contextualizes its work by reviewing several areas of related research:

1. 3D Hand Pose Estimation (HPE):

   • Early success came from depth sensors ([6], [22], [23]) and deep learning ([24], [25], [16]). More recent work uses single RGB images ([26], [27]) or aims to estimate 3D hand meshes ([28], [29]) which could simplify the mapping to robot models.
   • Background: Modern HPEs often output 3D joint locations. The challenge of mapping these locations to specific joint angles of a kinematic model is a key motivation for the Inverse Kinematics component of this paper.

2. Vision-based Teleoperation:

   • Historically, teleoperation used contact devices ([30], [31], [32]).
   • Vision-based approaches ([33], [34], [35], [5], [8], [36]) exist but are often limited to simple grasping.
   • Example: Antotsiou et al. [8]: This work is explicitly mentioned and used as a baseline. It combines inverse kinematics with a PSO (Particle Swarm Optimization) function to encourage contact between object and hand surfaces. While it aims for realistic interactions, the paper argues that simply forcing contact is insufficient for complex dexterous actions like in-hand manipulation.
   • Commercial Solutions (e.g., Leap Motion [4], Hololens [9]): These approaches often recognize gestures and trigger pre-recorded outputs or "attract" objects artificially, ignoring physics and producing unnatural motion. This paper explicitly aims to correct user input slightly while respecting physics laws.
   • Physics-based Contact Modeling ([45], [46], [47], [48], [49], [10]): These methods infer contact forces from mesh penetration between the user's hand and the object. The paper points out that these rely on high-precision HPE and can apply forces that don't necessarily transfer realistically.

3. Physics-based Pose Estimation:

   • Tzionas et al. [11] uses a physics simulator within an optimization framework to refine hand poses, building on generative and discriminative model fitting work ([50], [51], [52], [53]).
   • Hasson et al. [12] proposes an end-to-end deep learning model with contact loss and mesh penetration penalty for plausible hand-object mesh reconstruction.
   • These works often deal with single-shot images and simple physical constraints.
   • Yuan and Kitani [14] use RL to estimate and forecast physically valid body poses from egocentric videos.

4. Motion Retargeting and Reinforcement Learning:

   • This area is particularly relevant, sharing similarities with full body motion retargeting ([56]) and methods using RL for control policies in physics-accurate target spaces ([57], [58], [59], [60], [13]).
   • Peng et al. [13] (SFV): This work uses RL for physical skills from videos. It aims to teach an agent to autonomously perform skills by observing reconstructed and filtered videos. The key difference from the current paper is that SFV focuses on offline skill learning to mimic motions, while this paper aims to correct noisy user hand poses online and assist the user, similar to shared autonomy ([15]).

5. Robot Dexterous Manipulation and Reinforcement Learning:

   • The paper highlights Rajeswaran et al. [1], Zhu et al. [21], and OpenAI et al. [18] for learning robotic manipulation skills with RL and IL.
   • Rajeswaran et al. [1]: This work is directly built upon by the current paper, particularly its simulation framework and glove demonstration dataset. The current paper extends these environments to vision-based hand pose estimation.
   • Zhu et al. [21]: Shares a similar adversarial hybrid loss with this paper, though the current work deals with significantly more degrees of freedom.

6. Residual Policy Learning:

   • Johannink et al. [61] and Silver et al. [62] propose similar residual policy ideas.
   • The core commonality is that improving an action (learning a residual) instead of learning from scratch significantly aids RL exploration and produces more robust policies.
   • Differentiation: The main difference is that Johannink et al. and Silver et al. apply residual actions on top of a pre-trained policy, whereas this paper's residual action operates directly on online user input. This means the policy observes both the user's action and the world state, rather than just the world state, which helps to align the agent with the user's intention.

3.3. Technological Evolution

The field has evolved from expensive and intrusive motion capture (mocap) systems (like gloves [1] or exoskeletons [2]) towards vision-based solutions that use depth sensors or RGB cameras to estimate hand poses. Early vision-based methods often struggled with dexterous manipulation due to HPE inaccuracies and the complexity of physics-based interactions without force feedback.

Initial attempts to integrate physics involved simple contact force generation based on mesh penetration ([10]), which proved insufficient for complex tasks. Concurrently, Reinforcement Learning (RL) emerged as a powerful paradigm for learning complex control policies, particularly in physics simulators ([1], [18]). However, RL policies often generated unnatural motions or required extensive expert demonstrations (Imitation Learning).

This paper sits at the intersection of these advancements, specifically addressing the gap where noisy, vision-based user input meets physics-realistic dexterous manipulation. It innovates by combining residual learning (to refine noisy input) with hybrid RL+IL (to learn robust and natural corrections), all within a physics simulator, and introduces a novel data generation scheme to make such training feasible. This represents a step towards truly intuitive and physically plausible bare-hand interaction in VR/AR without the need for specialized hardware beyond a depth sensor.

3.4. Differentiation Analysis

The paper distinguishes its approach from prior work primarily through:

1. Online User Input Correction vs. Offline Skill Learning:

   • Unlike RL approaches that learn to mimic a skill offline from pre-processed reference motion (e.g., [13], [58]), this work corrects noisy user hand poses "as they come" in an online fashion. The agent acts as an assistant to the user, a setting similar to shared autonomy ([15]).

2. Residual Policy on User Input vs. Pre-trained Policy:

   • While other residual policy learning methods (e.g., [61], [62]) improve upon a pre-trained policy, this paper's residual action operates directly on imperfect user input from a hand pose estimator. This means the policy observes both the user's intended action and the world state, aligning the agent's corrections with the user's intention.

3. Physics-Guided Correction with Visual Fidelity:

   • Commercial solutions (e.g., Leap Motion [4]) often ignore physics or "attract" objects, leading to artificial motion.
   • Methods modeling contact physics based on penetration ([10]) depend heavily on high-precision HPE and often fail for dexterous tasks due to HPE noise.
   • This paper explicitly learns minor but crucial joint displacements for contacts through RL, ensuring physical accuracy while simultaneously using a 3D hand pose estimation reward to keep the generated motion visually close to the user input. This balance of physics realism and visual fidelity is a key differentiator.

4. Novel Data Generation Scheme:

   • Recognizing the absence of HPE sequences performing successful virtual interactions, the paper proposes a unique data generation scheme. This scheme uses mocap datasets of successful actions and large-scale 3D hand pose datasets to synthesize noisy input sequences for training, effectively bridging the data gap.

     In essence, the paper provides a complete system that can take noisy, vision-based user hand poses and transform them into physically accurate, visually plausible, and task-successful dexterous manipulations in VR/AR, overcoming the limitations of both HPE noise and simplified physics models.

4. Methodology

4.1. Principles

The core idea behind this method is to enable physics-based dexterous manipulation in virtual environments using noisy, vision-based hand pose estimates as input. Instead of trying to directly map the imperfect estimated hand pose to a physically accurate virtual hand posture (which is prone to failure), the system employs a residual learning approach. This means an agent learns to apply small, corrective adjustments, called residual actions, to the initial, imperfect user input. These residual actions are designed to introduce the necessary contacts and subtle movements to accomplish a manipulation task in a physics simulator, while simultaneously ensuring that the corrected motion remains visually similar to the user's original input.

The theoretical basis and intuition are rooted in addressing the domain gap and noise inherent in hand pose estimation (HPE):

1. Tackling Noise and Imperfect Mapping: HPE outputs are noisy and may not perfectly align with the kinematics of a virtual hand model. A direct Inverse Kinematics (IK) mapping from these noisy estimates will likely produce a virtual hand pose that either penetrates virtual objects or fails to make stable contact, leading to physical simulation failures. The residual agent learns to "fix" these small but critical errors.

2. Leveraging User Intent: The assumption is that the user input (even if noisy) broadly represents the user's intention for manipulation. Instead of ignoring this intent and learning an entirely new action (which can be hard for Reinforcement Learning to explore), the residual agent builds upon this intent, making only necessary adjustments. This makes the RL problem easier and helps maintain visual fidelity.

3. Physics Realism and Natural Motion: The agent is trained within a physics simulator, ensuring that the learned residual actions result in physically plausible interactions. To prevent unnatural motion often associated with RL alone, Imitation Learning (IL) is incorporated to guide the agent towards motions that resemble expert human demonstrations.

4. Closing the Loop with Pose Reward: A 3D hand pose estimation reward is introduced to encourage the virtual hand's corrected pose to remain close to the ground-truth human pose. This helps to improve the overall HPE accuracy by providing physics-guided feedback that aligns the virtual motion with the visual input.

   In essence, the system acts as an intelligent co-pilot, taking the user's noisy steering input, subtly correcting it for the rough terrain of physics simulation, and ensuring the journey looks and feels natural.

4.2. Core Methodology In-depth (Layer by Layer)

The proposed framework consists of several integrated components: Inverse Kinematics for initial mapping, a Residual Hand Agent trained with hybrid RL+IL, and a Data Generation Scheme to facilitate training.

4.2.1. Inverse Kinematics: From Human Hand Pose to Virtual Pose

The first step is to translate the user's observed hand motion into the control parameters of the virtual hand model. Given a user's estimated hand pose at time step $t$, denoted as $x_t$, which typically consists of the 3D locations of 21 joints of a human hand (e.g., from a dataset like BigHand2.2M [3]), captured from a visual representation $\phi_t$ (e.g., a depth image). The goal is to obtain a visually similar hand posture $z_t$ in the virtual model. This requires estimating the parameters $a_t$, which are the actuators or actions of the virtual hand model. These actuators usually define the target angles between hand joints, which are then controlled by PID controllers to reach the desired configuration.

The mapping from the estimated 3D pose $x_t$ to the virtual hand's actuator rotations $a_t$ is performed by an Inverse Kinematics (IK) function, denoted as $\kappa$. This mapping can be either manually designed or learned (e.g., with a supervised neural network if input-output pairs are available). The relationship is formally expressed as:

$a_t = \kappa(x_t(\phi_t))$

Where:

• $a_t$: The actuator parameters (e.g., joint angles) of the virtual hand model at time $t$. This is the "action" that will be applied to the virtual hand.

• $\kappa$: The Inverse Kinematics function, which maps 3D joint locations to virtual hand model actuator parameters.

• $x_t$: The estimated 3D hand pose of the human user at time $t$, represented as 3D locations of 21 joints.

• $\phi_t$: The visual representation (e.g., depth image) from which $x_t$ was estimated.

  The paper notes that for simplicity, $\kappa(x_t(\phi_t))$ in the action space is often referred to as the user input, distinguishing it from the user's estimated hand pose $x_t$ in the pose space.

Challenges with IK: IK is an ill-posed problem. Different human and virtual hand models can lead to multiple possible $a_t$ for a target $z_t$, or no solution at all. This is exacerbated by the noisy nature of $x_t$ from HPE. This imperfect user input necessitates a residual approach.

4.2.2. Residual Hand Agent

Since the IK function's output, $\kappa(x_t(\phi_t))$, is often imperfect due to kinematic mismatches and HPE noise, it might not be sufficient to succeed in dexterous manipulation tasks. A small error early in a sequence can lead to catastrophic failures later. To address this, a residual controller is introduced. This controller produces a residual action $f_t$, which is a correction applied to the user input.

The residual action $f_t$ is a function of the current simulation state $s_t$, the user input $\kappa(x_t(\phi_t))$, and optionally the visual representation $\phi_t$ (or extracted visual features). The final action $a_t$ applied to the environment is the user input adjusted by this residual action:

$$a_t = \kappa(x_t(\phi_t)) - f_t(s_t, \kappa(x_t(\phi_t)), \phi_t)$$

Where:

• $a_t$: The final action applied to the virtual hand model at time $t$.

• $\kappa(x_t(\phi_t))$: The initial user input derived from the estimated hand pose via IK.

• $f_t(\cdot)$: The residual action function, representing the learned correction.

• $s_t$: The current simulation state at time $t$. This includes information like relative positions between the target object and the virtual hand, hand velocity, etc.

• $\phi_t$: The visual representation (e.g., image or visual features) at time $t$.

  To prevent the residual action from deviating too much from the user input, $f_t$ is typically constrained to be within a certain zero-centered interval. The learning of this residual policy is formulated as a Reinforcement Learning (RL) problem. An agent interacts with a simulated environment by following a policy $\pi_\theta(f | s, \kappa, \phi)$, which is parameterized by $\theta$ (e.g., a neural network).

At each time step $t$, the agent observes the state $s_t$, the user input $\kappa(x_t(\phi_t))$, and visual information $\phi_t$. It then samples a residual action $f_t$ from its policy $\pi_\theta$. The final action $a_t$ is computed and applied to the environment, moving it to the next state $s_{t+1}$. A scalar reward $r_t$ is received, quantifying the quality of this transition.

The goal of RL is to find an optimal policy $\pi_\theta$ that maximizes the expected return $J(\theta)$, defined as:

$$J(\theta) = \mathbb{E}_{\tau \sim p_\theta(\tau)}\left[\sum_{t=0}^{T} \gamma^t r_t\right]$$

Where:

• $J(\theta)$: The objective function to be maximized, representing the expected cumulative discounted reward.

• $\mathbb{E}_{\tau \sim p_\theta(\tau)}[\cdot]$: The expected value over all possible trajectories $\tau$ generated by following the policy $\pi_\theta$.

• $\tau = (s_0, \kappa(x_0), \phi_0, f_0, s_1, \dots)$: A trajectory representing a sequence of states, user inputs, visual observations, and residual actions.

• $T$: The horizon or length of the trajectory, which can be variable depending on the hand pose input sequence.

• $\gamma \in [0, 1]$: The discount factor, which determines the present value of future rewards.

  The policy parameters $\theta$ are optimized using Proximal Policy Optimization (PPO) [17], a policy gradient approach. PPO learns both a policy network (which outputs residual actions) and a value function (which estimates the expected return from a given state).

4.2.2.1. Reward function

The total reward function $r_t$ is a crucial component that guides the learning process of the residual agent. It combines three different objectives, each with a weighting factor:

$$r_t = \omega^{\mathsf{task}} r_t^{\mathsf{task}} + \omega^{\mathsf{il}} r_t^{\mathsf{il}} + \omega^{\mathsf{pose}} r_t^{\mathsf{pose}}$$

Where:

• $r_t$: The total reward received at time $t$.

• $\omega^{\mathsf{task}}$, $\omega^{\mathsf{il}}$, $\omega^{\mathsf{pose}}$: Weighting factors for each reward component, controlling their relative importance.

  a) Task-oriented reward ($r_t^{\mathsf{task}}$): This reward component is specific to each environment and directly encourages the agent to accomplish the designated manipulation task. It can include short-term rewards (e.g., getting closer to the object of interest) and long-term rewards (e.g., successfully opening a door). The exact definition varies per task (details are in the Appendix).

b) Imitation learning reward ($r_t^{\mathsf{il}}$): RL policies, when trained solely on task rewards, can sometimes discover effective but unnatural behaviors (i.e., not human-like). To address this, an adversarial Imitation Learning (IL) reward is incorporated, similar to Generative Adversarial Imitation Learning (GAIL) [20] or Zhu et al. [21]. This reward encourages the agent's actions to resemble expert demonstration data.

The IL reward is defined as:

$$r_t^{\mathsf{il}} = (1 - \lambda) \log(1 - D_\psi(s_t, a_t))$$

Where:

• $D_\psi(s_t, a_t)$: A score given by a discriminator network (with parameters $\psi$) that quantifies how likely the state-action pair $(s_t, a_t)$ is to have come from an expert policy versus the agent's policy. A higher score from $D_\psi$ means it looks more like expert data.

• $\lambda$: A hyperparameter (set to 0.5 in experiments) that balances the influence of the discriminator in the agent's reward.

  To incorporate this objective, a min-max optimization scheme is used, characteristic of GANs:

$$\min_\theta \max_\psi \mathbb{E}_{\pi_E}[\log D_\psi(s, a)] + \mathbb{E}_{\pi_\theta}[\log(1 - D_\psi(s, a))]$$

Where:

• $\min_\theta$: The policy network (generator) aims to minimize this objective, meaning it wants to generate state-action pairs that fool the discriminator (i.e., make $D_\psi(s, a)$ close to 0.5 for its own actions).

• $\max_\psi$: The discriminator network aims to maximize this objective, meaning it wants to correctly distinguish expert state-action pairs from those generated by the policy.

• $\mathbb{E}_{\pi_E}[\cdot]$: Expected value over expert trajectories generated by expert policy $\pi_E$.

• $\mathbb{E}_{\pi_\theta}[\cdot]$: Expected value over trajectories generated by the agent's policy $\pi_\theta$.

  This objective encourages the policy $\pi_\theta$ to produce residual actions $f_t$ that, when combined with the user input $\kappa(x_t(\phi_t))$, generate state-action pairs $(s_t, a_t)$ that are indistinguishable from those of an expert. The expert data $\mathcal{D} = (s_i, a_i)_{i=1 \dots N}$ is obtained from Rajeswaran et al. [1], which used a data glove to capture noise-free sequences.

c) 3D hand pose estimation reward ($r^{\mathsf{pose}}$): The task-oriented and IL rewards might sometimes lead to virtual poses ($z_t$) that diverge significantly from the pose depicted in the user input image ($\phi_t$), especially if the hand pose estimator performs poorly due to object occlusion. To maintain visual resemblance between the virtual hand and the user's actual hand pose, an additional reward is introduced. This reward requires access to annotated ground-truth hand poses ($\bar{x}_t$) during training.

The 3D hand pose estimation reward is defined as:

$$r_t^{\mathsf{pose}} = - \sum_{j}^{21} ||z_t^j - \bar{x}_t^j||_2$$

Where:

• $z_t^j$: The 3D position of the $j$-th joint of the virtual hand model at time $t$.

• $\bar{x}_t^j$: The 3D position of the $j$-th joint of the ground-truth human hand at time $t$.

• $||\cdot||_2$: The Euclidean distance (L2 norm).

  This reward term penalizes large Euclidean distances between the virtual hand's joints and the ground-truth human hand's joints, thus encouraging the policy network to produce actions that keep the virtual hand posture visually consistent with the user's actual pose.

4.2.3. Data Generation Scheme

Training the residual policy requires a dataset of estimated hand poses $\{x_t\}$ that represent natural hand motions, ideally those that would lead to successful interactions if the system were perfect. Collecting such HPE sequences in an online fashion is difficult because users often fail and stop. Directly adding random noise to ground-truth mocap data might not accurately reflect the structured noise from a real HPE.

To overcome this, the paper proposes a novel data generation scheme. The idea is to:

1. Start with Mocap Data: Utilize a mocap dataset (e.g., Rajeswaran et al. [1]) which provides successful sequences of state-action pairs $(s_t, a_t)$ from expert demonstrations. These actions, when executed, generate a sequence of virtual poses $\{z_t\}$.

2. Generate Virtual Poses and Viewpoints: For each action $a_t$ in the mocap dataset, execute it in the physics simulator and record the resulting virtual pose $z_t$ (by placing virtual sensors) and the relative camera viewpoint $v_t$ (e.g., elevation and azimuth angles) from a virtual camera observing the hand.

3. Query a 3D Hand Pose Dataset: Given the sequences of virtual poses $z_t$ and their associated viewpoints $v_t$, the scheme retrieves corresponding real hand images from a large, public 3D hand pose dataset (e.g., BigHand2.2M [3]). This dataset is dense in articulations and viewpoints.

   • Retrieval Process: First, virtual poses $z_t$ are normalized (joint links to unit vectors) and aligned (palm to a certain plane) to $\hat{z}_t$. The BigHand2.2M dataset is queried to find ground-truth poses ($\bar{x}_t$) that match the viewpoint $v_t$ and then the nearest neighbor based on Euclidean distance between the normalized and aligned poses ($\hat{z}_t$ and the candidate ground-truth poses).
   • Image and HPE Generation: Once a match is found in BigHand2.2M, its associated depth image is retrieved. This image is then passed through an off-the-shelf 3D hand pose estimator (e.g., [63]) to compute the estimated hand pose $x_t$.

4. Synthesize Noisy Inputs: These estimated hand poses $x_t$ (generated from real images, thus containing structured noise similar to actual HPEs) are then used as the noisy input for training the residual agent. Additionally, noise can be added to the ground-truth translations to generate diverse realistic sequences.

   This pipeline effectively generates pairs of noisy estimated hand poses ($x_t$) and expert actions ($a_t$) that, when combined with simulation states ($s_t$) and visual features ($φ_t$), can be used to train the residual policy (see Algorithm 2 in the Appendix).

The overall training algorithm, incorporating the data generation scheme, PPO, and GAIL, is described in Algorithm 1 in the Appendix. It iteratively generates user input sequences, collects policy rollouts in the simulator, computes rewards, and updates the policy, value function, and discriminator networks.

The following figure (Figure 2 from the original paper) shows the overall framework:

该图像是一个示意图，展示了基于残差强化学习的手势估计（HPE）和手部动作生成的框架。图中包括用户输入、残差手势代理与物理模拟器之间的交互，以及数据生成的流程。视觉特征与手势估计的整合，旨在实现更精确的虚拟操作。

This diagram illustrates the flow:

1. Input: A Depth Image is fed into a 3D Hand Pose Estimator (HPE).
2. User Input: The estimated hand pose ($x_t$) is then processed by an Inverse Kinematics (IK) function to generate an initial user input (actions $\kappa(x_t(\phi_t))$) for the virtual hand model.
3. Residual Hand Agent: This agent observes the current simulation state ($s_t$), the user input ($\kappa(x_t(\phi_t))$), and potentially visual features ($\phi_t$). It then predicts a residual action ($f_t$).
4. Combined Action: The residual action is added to the user input to produce the final action ($a_t$) that is applied to the Virtual Hand in the Physics Simulator.
5. Physics Simulator: The simulator executes the action, updates the environment state ($s_{t+1}$), and generates virtual poses ($z_t$) and rewards ($r_t$).
6. Reward Function: The reward integrates task-oriented rewards ($r^{\mathsf{task}}$), imitation learning rewards ($r^{\mathsf{il}}$), and 3D hand pose estimation rewards ($r^{\mathsf{pose}}$).
7. Data Generation (Offline): Separately, a Data Generation Scheme creates synthetic HPE sequences by combining mocap expert data with a 3D hand pose dataset and an HPE to produce noisy estimated hand poses that are used for training.

5. Experimental Setup

5.1. Datasets

The paper uses several datasets, depending on the specific experiment:

• Rajeswaran et al. [1] (MoCap Dataset):

  • Source: This dataset provides successful expert trajectories of dexterous manipulation recorded using a mocap glove and a tracking system [32].
  • Scale & Characteristics: It contains noise-free sequences of state-action pairs for various dexterous manipulation tasks (e.g., door opening, in-hand pen manipulation, tool use, object relocation). There are approximately 24 mocap trajectories per task.
  • Domain: Robotic hand manipulation (specifically, the ADROIT anthropomorphic hand).
  • Usage: This dataset serves as the source of expert demonstrations for Imitation Learning and is the foundation for the data generation scheme in Experiment A, where noise is either synthetically added or simulated from HPE.

• BigHand2.2M [3]:

  • Source: A large public 3D hand pose dataset.
  • Scale & Characteristics: Designed to densely capture articulation and viewpoint spaces of human hands in mid-air and in an object-free setup. It contains pairs of depth images and ground-truth 3D hand pose annotations.
  • Domain: Human hand poses from depth images.
  • Usage: Used in the data generation scheme (Section III-C) in Experiment A.2. The virtual poses generated from mocap actions are used to query BigHand2.2M to find corresponding depth images, which are then fed into an HPE to generate structured noisy input poses.

• First-Person Hand Action Benchmark (F-PHAB) [64]:

  • Source: A benchmark specifically for hand-object interactions.
  • Scale & Characteristics: Provides hand-object interaction sequences with hand and object pose annotations from RGB-D videos. It includes different manipulation tasks covering power and precision grasps (e.g., pour juice from a carton to a glass, give coin to other person). Each task has 24 annotated video sequences from 6 different users.
  • Domain: Real-world hand-object interactions captured in-the-wild with RGB-D sensors.
  • Usage: Used as a testbed in Experiment B for physics-based hand-object sequence reconstruction. This dataset provides estimated hand poses and initial object pose estimates from real RGB-D inputs.

Effectiveness for Validation: These datasets are well-chosen to validate different aspects of the proposed method:

• Rajeswaran et al. [1] provides a controlled environment with expert mocap data, ideal for studying the residual agent's ability to handle noise and learn from demonstrations.
• BigHand2.2M [3] enables the generation of realistic HPE noise by mapping expert virtual poses to real depth images and then back through an HPE, bridging the gap between synthetic and real noise.
• F-PHAB [64] presents a challenging in-the-wild scenario, directly testing the framework's robustness against real-world HPE errors and object occlusions in complex hand-object interactions.

5.2. Evaluation Metrics

The paper employs several metrics to evaluate the performance of the proposed method, focusing on task success, pose accuracy, and stability.

1. Task Success:

   • Conceptual Definition: This metric measures the percentage of times the agent successfully completes the designated manipulation task within a given tolerance or criteria (e.g., door opened past a certain angle, object in target location). It directly quantifies the effectiveness of the policy in achieving its goal.
   • Mathematical Formula: Not explicitly given as a standard formula but generally calculated as: $ \text{Task Success} = \frac{\text{Number of successful episodes}}{\text{Total number of episodes}} \times 100% $
   • Symbol Explanation:
     • Number of successful episodes: Count of trials where the task completion criteria were met.
     • Total number of episodes: Total number of trials or test sequences run.

2. $E_{\mathsf{pose}}$ (3D Hand Pose Error):

   • Conceptual Definition: This metric quantifies how similar the virtual hand posture ($z_t$) is to the actual visual pose of the human hand ($\bar{x}_t$). It is measured by reprojecting the virtual hand's joints into the input image space and comparing them to ground-truth annotations. A lower $E_{\mathsf{pose}}$ indicates better visual fidelity and pose accuracy.
   • Mathematical Formula: The paper defines the pose reward as the negative Euclidean distance between virtual and ground-truth joints. The error itself is typically the average of this distance. $ E_{\mathsf{pose}} = \frac{1}{N \cdot J} \sum_{t=1}^{N} \sum_{j=1}^{J} ||z_t^j - \bar{x}_t^j||_2 $
   • Symbol Explanation:
     • $N$: Total number of time steps or frames in a sequence.
     • $J$: Number of joints in the hand model (e.g., 21).
     • $z_t^j$: The 3D position of the $j$-th joint of the virtual hand model at time $t$.
     • $\bar{x}_t^j$: The 3D position of the $j$-th joint of the ground-truth human hand at time $t$.
     • $||\cdot||_2$: The Euclidean distance (L2 norm) between two 3D points.
     • $\frac{1}{N \cdot J} \sum \sum$: Represents the average over all joints and all time steps.

3. $\bar{T}$ (Average Length of Sequence Before Instability):

   • Conceptual Definition: This metric measures the stability and robustness of the simulation. It represents the average percentage of the total sequence length that is successfully completed before the simulation becomes unstable and the task is not completed (e.g., object falls, hand breaks contact). A higher $\bar{T}$ indicates a more stable and robust interaction.
   • Mathematical Formula: Not explicitly given, but conceptually: $ \bar{T} = \frac{1}{\text{NumEpisodes}} \sum_{k=1}^{\text{NumEpisodes}} \left( \frac{\text{Actual sequence length}_k}{\text{Total possible sequence length}_k} \times 100% \right) $
   • Symbol Explanation:
     • NumEpisodes: Total number of interaction sequences or trials.
     • Actual sequence length}_k: The number of frames or time steps completed successfully in episode $k$ before instability or failure.
     • Total possible sequence length}_k: The maximum possible length of episode $k$.

5.3. Baselines

The paper compares its proposed method against several baselines, covering different approaches to RL and IL, and simple IK-based or contact-enforcing methods:

1. Inverse Kinematics (IK):

   • Description: This baseline applies actions based solely on the user's estimated hand pose mapped directly to the virtual hand model via the IK function $\kappa$. No RL or IL corrections are applied.
   • Representativeness: Represents the most straightforward approach of pose retargeting without considering physics or HPE noise. It establishes a lower bound for performance in many scenarios.

2. Reinforcement Learning (RL) - no user:

   • Description: An RL agent (trained with PPO) observes only the simulation state $s_t$ (and potentially $\phi_t$) but does not explicitly observe or use the user input $\kappa(x_t(\phi_t))$. It learns to perform the task autonomously.
   • Representativeness: This baseline evaluates the capability of RL to learn the task from scratch without guidance from user intent. The paper states this can be similar to triggering a prerecorded sequence.

3. Reinforcement Learning (RL) + user reward:

   • Description: Similar to RL - no user, but an additional reward term $r_{\text{user}} = -0.1 ||a - \kappa(x)||_2$ is added to encourage the RL agent to follow the user input $\kappa(x_t(\phi_t))$. The agent still operates in a non-residual way (i.e., it learns the full action $a_t$, not just a residual $f_t$).
   • Representativeness: Tests if simply penalizing deviation from user input within a standard RL framework can improve performance, without the residual learning architecture.

4. Imitation Learning (IL) - no user:

   • Description: An IL agent (based on GAIL [20]) learns from expert demonstrations without observing or using the user input $\kappa(x_t(\phi_t))$. It tries to reproduce expert trajectories.
   • Representativeness: Assesses IL's ability to learn the task from expert data independently of user input.

5. Hybrid learning (Hybrid) - no residual / Hybrid + user reward:

   • Description: This combines RL and IL (similar to the proposed method's reward structure), but the agent operates in a non-residual way (i.e., it learns the full action $a_t$ from scratch, not a residual correction). There's a version with and without the user reward term.
   • Representativeness: Evaluates the benefit of combining RL and IL without the residual learning architecture.

6. Closing hand baseline (Experiment B specific):

   • Description: This baseline, implemented for Experiment B, acts on top of an IK function (specifically from [8]). It attempts to tighten the grasp or generate more contact forces by moving actuators towards the object, similar to methods by Höll et al. [10].
   • Representativeness: Represents a common heuristic-based approach for grasping and contact generation, which is a simplified form of physics-based interaction.

5.4. Hand Models, Simulator and Tasks

• Hand Models:
  • ADROIT anthropomorphic platform [1]: Used in Experiment A. It has 24 degrees-of-freedom (DoF) for joint angle rotations of the Shadow dexterous hand, plus 6 DoF for the 3D position and orientation of the hand.
  • MPL hand model [32]: Used in Experiment B. It consists of 23 DoF for joint angles + 6 DoF for global position and orientation. An extended version by Antotsiou et al. [8] is specifically mentioned.
• Simulator: MuJoCo physics simulator [19] is used for all experiments. It's known for its accuracy and speed in model-based control and robotics.
• Tasks (Experiment A): Four dexterous manipulation scenarios defined in Rajeswaran et al. [1] are used:
  • Door Opening: Undo a latch and swing a door open.
  • In-Hand Manipulation: Reposition a blue pen to match a target orientation (green pen).
  • Tool Use (Hammer): Pick up a hammer and drive a nail into a board.
  • Object Relocation: Move a blue ball to a green target location. Each task has specific success criteria and reward functions (detailed in the Appendix).
• Tasks (Experiment B): Two hand-object interaction tasks from F-PHAB [64] are used:
  • Pour Juice: Hold a juice carton and perform a pouring action into a glass.
  • Give Coin: Pick up a coin and place it into another person's hand (represented by a target box).
• Policy Network: The policy network ($\pi$), value function network, and discriminator network (for IL) are all Multi-Layer Perceptrons (MLPs) with two hidden layers, each having 64 neurons (a $(64, 64) MLP$).
• Residual Policy Limit: The residual action $f_t$ is constrained to be within 20% of the total action space in Experiment A, and 25% in Experiment B, ensuring it remains a "small correction."
• Action Representation: Actions are modeled as a Gaussian distribution with a state-dependent mean and a fixed diagonal covariance matrix.
• PPO Parameters: Policy updates are performed after collecting a batch of $m = 4096$ samples, with minibatches of size 256. Adam optimizer is used with learning rates of $3 \cdot 10^{-4}$ for the policy and value function, and $10^{-4}$ for the discriminator. 15 parameter updates for policy/value and 5 for discriminator per batch. Temporal discount ($\gamma$) of 0.995, GAE parameter of 0.97, clipping threshold of 0.2. Initial policy standard deviation of 0.01.
• Hybrid Reward Weight ($\lambda$): Set to 0.5 for all experiments involving hybrid reward.
• Pose Reward Weight ($\omega^{\mathsf{pose}}$): Set to 0.01 in Experiment B. Not used in Experiment A.1 (due to object-free BigHand2.2M) nor A.2 (due to absence of visual features to policy network).
• HPE for Experiment A.2: Yuan et al. [63] (DeepPSO) trained on the remaining subjects of BigHand2.2M.
• HPE for Experiment B: Oberweger and Lepetit [65] ($DeepPrior++$) trained on F-PHAB [64] dataset, yielding an average test joint error of 14.54 mm. Visual features $\phi_t \in \mathbb{R}^{1024}$ are extracted from its FC2 layer.

6. Results & Analysis

The experiments are divided into two main parts: Experiment A focuses on dexterous manipulations in a virtual space with estimated hand poses in mid-air, while Experiment B tackles physics-based hand-object sequence reconstruction from in-the-wild RGBD sequences.

6.1. Core Results Analysis

6.1.1. Experiment A: Performing dexterous manipulations in a virtual space with estimated hand poses in mid-air

This experiment evaluates the framework's ability to handle noisy HPE inputs in a controlled virtual environment.

Experiment A.1: Overcoming random noise on demonstrations This sub-experiment tested the framework's resilience to synthetic Gaussian noise added to expert successful actions (from mocap data [1]). This allowed for controlled analysis of noise impact.

The following are the results from Table I of the original paper:

• Analysis: The table shows task success rates for the door opening task when training and testing with different levels of Gaussian noise ($\sigma$ in radians) added to each actuator.

  • The User input row indicates the success rate if only the noisy IK output (without any residual correction) is applied. As test noise ($\sigma_{test}$) increases, the user input success rate drastically drops (e.g., from 80% at $\sigma_{test}=0.00$ to 2.7% at $\sigma_{test}=0.20$).

  • Our residual agent (trained at various $\sigma_{train}$) is able to recover meaningful motion and maintain much higher success rates, especially when the training noise magnitude is similar to or higher than the test noise. For example, when trained with $\sigma_{train}=0.10$, it achieves 87% success at $\sigma_{test}=0.10$, while the raw user input only achieves 33.8%.

  • The agent can generalize to higher test noise if trained on higher noise levels. It can recover up to $\sigma_{test}$ of 0.20 rad. This demonstrates the residual agent's robustness and ability to correct significant noise.

    The following are the results from Table II of the original paper:

    Method	Door opening		Tool use		In-hand man.		Object rel.
    	Train	Test	Train	Test	Train	Test	Train	Test
    IK	64.00	74.00	50.00	56.00	67.67	69.92	77.00	83.00
    RL-no user	75.00	59.00	51.00	44.00	43.61	38.34	0.00	0.00
    IL-no user	0.00	0.00	0.00	0.00	4.00	6.77	0.00	0.00
    Hybrid-no res.	0.00	0.00	0.00	0.00	4.00	0.00	0.00	0.00
    RL+user reward	69.92	62.40	6.01	9.02	48.12	27.81	0.00	0.00
    Hybrid+user rew.	0.00	0.00	56.39	33.08	9.02	7.51	0.00	0.00
    Ours	81.33	83.00	61.00	58.00	90.97	87.21	49.62	16.54

• Analysis: This table compares various baselines with the proposed Ours method for a fixed Gaussian noise of 0.05 rad across four dexterous manipulation tasks.

  • Our Method (Ours): Consistently achieves the highest task success rates in Door opening (83.00% Test), Tool use (58.00% Test), and In-hand man. (87.21% Test). It significantly outperforms all baselines.

  • IK: While performing reasonably well in some tasks (e.g., Object rel. 83.00% Test), it often struggles in more complex dexterous tasks like Tool use (56.00% Test) and In-hand man. (69.92% Test) when noise is present.

  • RL-no user: Shows some capability, especially in Door opening (59.00% Test), but completely fails in Object rel.. This baseline doesn't follow user input, effectively acting as a pre-recorded sequence.

  • IL-no user & Hybrid-no res.: These baselines generally perform very poorly or fail entirely (0% success in many cases), suggesting that IL alone or hybrid learning without residual correction struggles when starting from a noisy input and having to learn the full action.

  • RL+user reward & Hybrid+user rew.: Trying to force RL to follow user input with an explicit reward helps in some cases (e.g., RL+user reward in Door opening 62.40% Test, Hybrid+user rew. in Tool use 33.08% Test), but it's not as effective or robust as the residual approach.

  • Object Relocation: All methods, including Ours, show relatively low success rates on Object relocation (16.54% Test for Ours). The authors hypothesize that the low result of PPO propagates to their algorithm, suggesting that this task might be inherently more challenging for the chosen RL optimizer or task setup.

  • Convergence Speed: The paper notes that Our residual policy converges significantly faster (3.8M and 5.2M samples for door opening and in-hand manipulation respectively) compared to RL baseline (7.9M and 13.8M samples). This faster convergence is attributed to the user input providing better guidance for RL exploration.

    The following figure (Figure 3 (a) from the original paper) shows an example of task success rate curves:

    该图像是多个示意图，包括任务成功率曲线图（a）、手部操作演示（b）及3D手部运动重构（c）。图a展示了不同方法的任务成功率变化，图b则展示了手部与目标物体的接触情况，图c呈现了手部运动的3D重构过程。

• Analysis of Figure 3 (a): This plot illustrates the task success rate during training for door opening (top) and in-hand manipulation (bottom). Our method (blue line) consistently achieves higher success rates and converges faster than the RL baseline (red line). This quantitatively supports the claim that residual learning with user input significantly aids exploration and leads to more effective policies.

Ablation Study on RL and IL components:

• The paper states that combining both RL and IL components (as in Ours) leads to accomplishing the task while keeping a motion that resembles the human experts more closely.
• In terms of task success alone, RL alone achieves 75.9% while IL alone achieves 36.5%. This indicates that while RL is strong for task achievement, IL helps shape the motion quality. The combination leverages the strengths of both.

Experiment A.2: Overcoming structured hand pose estimation and mapping errors This experiment moves from synthetic random noise to structured noise generated by a real HPE and mapping function using the proposed data generation scheme (Section III-C).

The following are the results from Table III of the original paper:

• Analysis: This table (expanded in Table V in the Appendix) presents results for task success on structured HPE errors for Door opening and In-hand manipulation. GT refers to ground-truth poses (noise-free), while Est. refers to estimated hand poses from the HPE (with structured noise).

  • Supervised IK Baseline: An IK network trained with supervised learning on $(x_t, a_t)$ pairs is generally poor. It achieves 49.62% with GT poses in Door opening but drops to 27.81% with Est. poses. For In-hand man., it completely fails with GT poses (0.00%). This highlights the IK's limitation when dealing with real HPE errors.

  • Our Method (Ours (Est. poses)): When trained and tested with estimated poses (the most realistic setting), Ours achieves high success rates: 85.95% (Train) and 70.67% (Test) for Door opening, and 20.33% (Train) and 57.14% (Test) for In-hand man.. Notably, for In-hand man., Ours (Est. poses) significantly improves over IK (20.30% Test) and RL + user reward (Est.) (12.03% Test).

  • RL - no user: This baseline performs well for Door opening with GT poses (98.49% Train, 76.69% Test) and Est. poses (66.16% Train, 71.42% Test), indicating it can learn the task autonomously. However, as discussed, it does not follow user input and resembles a triggered sequence. For In-hand man., its performance is poor (0.00% Test with Est. poses).

  • RL + user reward: This baseline struggles, often showing 0% success for Door opening with or without GT/Est. poses, suggesting that simply adding a user following reward to a non-residual RL agent is not robust enough for structured HPE noise.

  • Comparison with A.1: Applying models trained on random noise (Experiment A.1) to structured HPE noise (Experiment A.2) performs poorly (Ours (Experiment A.1) row). This emphasizes the necessity of the data generation scheme to simulate realistic HPE errors.

  • Conclusion: The residual approach (Ours (Est. poses)) is generally robust to structured noise and significantly outperforms baselines, especially in complex dexterous tasks like In-hand manipulation where IK and simple RL struggle.

    The following figure (Figure 4 from the original paper) shows qualitative results on 'in-hand manipulation task':

    该图像是一个插图，展示了在手中操作任务中的定性结果。图中显示了估计的手势（中）与逆向运动学结果（上）和我们的方法结果（下）。深度图像是通过数据生成方案获取的。

• Analysis of Figure 4: This figure visually demonstrates the effectiveness of the proposed method for in-hand manipulation.

  • Middle (estimated hand pose): Shows the noisy input hand pose from the HPE.
  • Top (IK result): Shows the direct Inverse Kinematics mapping of the estimated pose. It's evident that the virtual hand is not correctly grasping or manipulating the pen, likely due to HPE inaccuracies leading to non-physical contacts or penetrations.
  • Bottom (Our result): Shows the output of Our method. The virtual hand is successfully grasping and manipulating the pen, with physically plausible contacts. The pose still visually resembles the user's input, but minor corrections have been applied to achieve the task. This image qualitatively supports the quantitative results, showing that Our method can correct HPE errors to enable successful dexterous manipulation.

6.1.2. Experiment B: Physics-based hand-object sequence reconstruction

This experiment tests the framework on real-world RGBD sequences (in-the-wild) from the F-PHAB dataset [64]. Here, expert demonstrations are not available, so $r^il$ and the data generation scheme are not used. A 3D hand pose estimation reward ($r^{\mathsf{pose}}$) is used.

The following are the results from Table IV of the original paper:

• Metrics:

  • T↑: Average length of sequence before instability (in percentage over total length). Higher is better.
  • Epose↓: 3D hand pose error (in mm). Lower is better.
  • Success ↑: Task success rate (percentage). Higher is better.

• Analysis for 'Pour Juice':

  • Our method (Ours): Achieves the best performance across all metrics on both training and test sets. It has the highest Success (93.0% Train, 65.0% Test) and $T$ (98.2% Train, 59.6% Test), indicating both high task completion and stability. Its Epose (25.43 mm Train, 33.15 mm Test) is also the best or competitive.
  • Ours w/o pose reward: This ablation shows the impact of the pose reward. Without it, Success drops from 93.0% to 84.0% (Train) and 65.0% to 47.0% (Test), and Epose slightly worsens. This confirms the pose reward's importance in maintaining visual fidelity and contributing to task success.
  • Closing hand: This baseline performs better than IK but significantly worse than Ours. While it can achieve 55.0% Success (Train), its Epose is higher, and $T$ is lower than Ours. Its reliance on simple contact enforcement is not as robust.
  • IK [8]: Performs poorly, especially on the test set (5.0% Success). This highlights the challenge of in-the-wild HPE noise and the inadequacy of IK alone.

• Analysis for 'Give Coin':

  • Our method (Ours): Dominates this task, achieving 80.0% Success (Train) and 83.3% (Test), with high $T$ (95.5% Train, 92.1% Test) and the lowest Epose (24.3 mm Train, 25.30 mm Test). This task is considered very challenging due to the small, light coin and the need for precision.
  • Closing hand: Shows some limited success (25.0% Train, 28.57% Test) but is far from Ours. Its Epose is worse.
  • IK [8]: Completely fails (0.0% Success on both Train and Test), underscoring the difficulty of this task for simple IK.

• Training-Test Gap: A significant gap between training and test results is observed for all methods, especially in Pour Juice. This is more severe in Give coin, where slight inaccuracies can make the light and thin coin fall. The authors attribute this to more severe HPE errors in this experiment and the small number of training sequences leading to overfitting. They suggest more training data or data/trajectory augmentation as potential solutions.

• Conclusion: Our approach successfully accomplishes dexterous manipulation tasks in-the-wild while maintaining hand posture similar to the visual input, outperforming all baselines.

  The following figure (Figure 5 from the original paper) shows qualitative results from Experiment B:

  该图像是一个示意图，展示了使用深度传感器和3D手势估计进行物体的灵巧操作。图中显示了不同阶段的手部动作，包括抓取和移动物体的过程，反映了物理模拟中的手-物相互作用和姿态估计的校正。

• Analysis of Figure 5: This figure illustrates the physics-based hand-object sequence reconstruction for the pour juice task.

  • (a) RGB/depth image and estimated 3D hand pose: Shows the raw input from the F-PHAB dataset and the HPE result.
  • (b) IK function $\kappa$ [8] on HPE: Displays the output of the Inverse Kinematics baseline. The hand pose might be visually similar but fails to make proper contact or interact physically with the juice carton, often leading to penetration.
  • (c) Closing hand baseline on top of $\kappa$: Shows the effect of the closing hand baseline. While it attempts to create contact, the interaction often looks unnatural or unstable.
  • (d) Our result: Demonstrates the virtual hand successfully grasping and manipulating the juice carton. The pose is physically accurate (making proper contact) and visually consistent with the user's input, enabling the pouring action. This qualitatively supports the method's superior performance in in-the-wild scenarios.

6.2. Data Presentation (Tables)

The following are the results from Table V of the original paper:

• Comprehensive Analysis of Table V (Experiment A.2): This table expands on Table III, providing task success rates for all baselines and tasks when facing structured HPE errors (generated via the data generation scheme). GT indicates using ground-truth hand poses as input to IK, while Est. indicates using estimated hand poses from an HPE for IK.
  • Our Method (Ours (Est. poses)): Consistently demonstrates the strongest performance in the most challenging and realistic setting (using estimated poses).
    • Door opening: 85.95% (Train) / 70.67% (Test)
    • In-hand man.: 20.33% (Train) / 57.14% (Test) - Note: The higher test success than train success for 'In-hand man.' (Est.) for 'Ours' might indicate some robustness or perhaps variability in test sets, but generally, it's a very challenging task where other baselines perform worse.
    • Tool use (hammer): 78.94% (Train) / 71.42% (Test)
    • Object relocation: 34.00% (Train) / 35.00% (Test)
  • IK: Performs poorly for In-hand man. (0.00% for GT, 20.30% for Est.) and Door opening (27.81% for Est.), confirming its inadequacy for dexterous tasks with HPE noise.
  • RL - no user (Est.): Shows reasonable performance for Door opening (71.42% Test) and Tool use (54.89% Test), but fails for In-hand man. (0.00%) and Object relocation (0.00%), confirming its limitations when not guided by user input or for complex precision tasks.
  • IL - no user (GT/Est.) & Hybrid - no user (GT/Est.): These Imitation Learning and Hybrid (RL+IL without residual) baselines generally fail to achieve significant success across most tasks, especially when presented with estimated hand poses. This indicates that IL needs clean, expert input and that learning a full action from scratch with noisy input is very difficult.
  • RL + user reward (GT/Est.) & Hybrid + user reward (GT/Est.): These baselines, which try to follow user input with an explicit reward, also largely struggle or fail, especially for estimated poses. This further emphasizes that the residual learning architecture is critical for effectively incorporating noisy user guidance.
  • Object Relocation: Remains challenging for all methods. Our method achieves 35.00% Test success with Est. poses, which is the highest among all compared baselines, but still low compared to other tasks. This supports the authors' hypothesis about PPO's potential limitations for this specific task.
  • Ours (Experiment A.1): The row "Ours (Experiment A.1)" shows the performance of the model trained only with random Gaussian noise (from Experiment A.1) when applied to the structured noise scenarios of Experiment A.2. Its performance is significantly worse than Ours (Est. poses), highlighting that random noise is not an adequate proxy for structured HPE errors, and thus validating the data generation scheme's importance.

6.3. Ablation Studies / Parameter Analysis

• Impact of RL and IL Components:

  • The paper mentions an ablation study in Experiment A.1, concluding that combining both RL and IL leads to accomplishing the task while keeping a motion that resembles the human experts more closely.
  • Quantitatively, for task success (in the random noise setting), RL alone achieved 75.9% while IL alone achieved 36.5%. This indicates that RL is primarily responsible for task accomplishment, while IL helps ensure the naturalness and human-likeness of the motion. The hybrid approach (Ours) leverages both.

• Impact of Pose Reward ($r^{\mathsf{pose}}$):

  • In Experiment B, the comparison between Ours and Ours w/o pose reward explicitly demonstrates the effect of the 3D hand pose estimation reward.
  • For Pour Juice, removing $r^{\mathsf{pose}}$ leads to a decrease in Task Success (from 65.0% to 47.0% on Test) and an increase in Epose (from 33.15 mm to 37.88 mm on Test). This clearly shows that the pose reward helps keep the virtual pose closer to the visual input and contributes to overall task success.

• Noise Level Generalization (Table I): This can be viewed as an ablation on the training noise level. It showed that the model generalizes best when the training noise is similar to or greater than the test noise, emphasizing the importance of training with realistic noise distributions.

• Convergence Speed: The paper highlights that Our method converges significantly faster than other RL baselines (e.g., 3.8M vs. 7.9M samples for door opening). This is attributed to the user input providing a strong exploration signal, thereby reducing the exploration problem in RL.

  The following figure (Figure 7 from the original paper) displays training curves for the door task (Experiment A.2):

  该图像是图表，展示了实验 A.2 中不同方法的训练曲线（门的任务成功率）。横轴为时间步（百万），纵轴为任务成功率。在不同方法中，'Ours (GT)' 和 'Ours (Est.)' 表现较好，'RL (GT)' 和 'IK (GT)' 的表现亦相对较高。

• Analysis of Figure 7: This graph shows the training curves (task success rate vs. time steps) for the door opening task in Experiment A.2 (structured HPE error).

  • Ours (Est. poses) (green) and Ours (GT poses) (blue) achieve the highest task success rates and show faster convergence.
  • RL - no user (GT) (red) also reaches a high success rate but is slower to converge.
  • The IK baselines (IK (GT) and IK (Est.)) plateau at lower success rates, especially IK (Est.), reinforcing their limitations.
  • Other baselines (e.g., RL + user reward (GT)) either converge very slowly or fail to reach high success, demonstrating the superiority of the residual learning approach.

6.4. Visualizations of Contact Forces

Figure 3 (c) (part of the larger Figure 3 image) shows generated contact forces for hand-object interaction during 3D hand motion reconstruction.

• The image in Figure 3 (c) shows a virtual hand interacting with an object (e.g., a cup or a bottle), with arrows emanating from the fingertips towards the object, representing contact forces. This visualization provides qualitative evidence that the residual agent successfully learns to generate physically plausible contacts that enable stable manipulation within the physics simulator. These forces are crucial for successful interaction, especially in tasks like in-hand manipulation or pouring juice.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper presents a robust and innovative framework for enabling physics-based dexterous manipulation of virtual objects using only estimated hand poses from a depth sensor, eliminating the need for expensive or intrusive haptic hardware. The core contribution is a residual agent that learns to apply minor yet crucial joint displacements to correct noisy user input (estimated hand poses), facilitating successful task accomplishment within a physics simulator. This agent is trained using a model-free hybrid Reinforcement Learning (RL) and Imitation Learning (IL) approach, guided by task-oriented, imitation learning, and 3D hand pose estimation rewards. A novel data generation scheme addresses the challenge of acquiring suitable training data by synthesizing realistic noisy HPE sequences from mocap demonstrations and a large-scale hand pose dataset. Experiments across various dexterous manipulation tasks in VR and in-the-wild RGBD reconstruction demonstrate the proposed method's superior performance in task success and hand pose accuracy compared to RL/IL baselines and simple contact-enforcing techniques.

7.2. Limitations & Future Work

The authors identify several limitations and propose promising directions for future research:

1. Training-Test Gap: A significant performance gap was observed between training and test results, particularly in in-the-wild scenarios (Experiment B). This suggests that the synthetic HPE noise generated for training might not fully capture the complexity and diversity of real-world noise and occlusions.

   • Future Work:
     • End-to-end Framework: Making the entire framework end-to-end, allowing gradients to propagate from the simulator back to the hand pose estimator, could lead to physics-based pose estimation that is jointly optimized for manipulation.
     • Improved Data Generation: Generating more realistic synthetic data (e.g., by fitting realistic hand models [67] on mocap data or trained policies) could help narrow the training-test gap.
     • Data/Trajectory Augmentation: Employing techniques to augment training data or trajectories could improve generalization.

2. Lack of 6D Object Pose Estimation: Experiment B relies on ground-truth 6D object pose initialization.

   • Future Work: Integrating a 6D object pose estimator [66] into the loop would enable fully vision-based hand-object interaction without relying on ground-truth object poses.

3. Scalability to More Tasks: The current framework's scalability to a much larger number of diverse tasks is not immediately clear.

   • Future Work: Research into RL generalization for in-the-wild scenarios and to a wider range of dexterous manipulation tasks is needed. Developing methods to scale up the number of tasks within this framework would be valuable.

4. Online Deployment Challenges: While the framework shows promise for VR systems, deployment to receive poses in a stream for real-time interaction may introduce additional challenges (e.g., latency, robustness to varying real-world conditions).

7.3. Personal Insights & Critique

This paper offers a compelling solution to a critical problem in VR/AR and robotics: enabling natural, physics-based dexterous manipulation without specialized haptic devices. The residual learning paradigm is particularly elegant, as it cleverly leverages existing, albeit noisy, user intent and focuses on learning targeted corrections rather than re-learning entire actions. This approach not only makes the RL exploration problem tractable but also ensures the virtual motion remains visually aligned with the user's input, which is crucial for immersive experiences.

Transferability and Applications: The methodology has high transferability potential. Beyond VR/AR, it could be applied to:

• Teleoperation of Robotic Hands: Enhancing the control of dexterous robotic hands by human operators, where noisy vision-based input could be refined to execute precise physical tasks.
• Assisted Manipulation for Humans with Motor Impairments: Providing intelligent assistance to individuals who may have difficulty with fine motor control, allowing their intended (but imperfect) motions to be corrected for successful interaction with virtual or even real objects (via robotic proxies).
• Virtual Prototyping and Training: Creating highly realistic virtual environments for design, prototyping, or training where physical interaction fidelity is paramount.

Potential Issues and Areas for Improvement:

1. HPE Robustness to Occlusion: While the pose reward helps with visual fidelity, the initial HPE itself can significantly degrade under self-occlusion or object occlusion (especially for objects like coins). If the initial HPE is severely inaccurate, the residual agent might struggle to find a feasible correction within its limited action space. An end-to-end system or HPE specifically trained for occluded hand-object interactions could mitigate this.

2. Generalization to Novel Objects and Tasks: The RL component is task-specific. Scaling to novel objects or tasks not seen during training would likely require meta-learning or zero-shot generalization techniques, which are active areas of RL research. The current framework may need to be re-trained for each new task, limiting its broad applicability.

3. Real-time Performance: The paper mentions offline motion reconstruction and online prediction. While PPO and MLP architectures can be fast, the computational demands of the physics simulator, HPE, and policy inference in a real-time VR system could be substantial, requiring further optimization.

4. User Experience and Latency: The residual corrections are applied by an agent. While designed to be minor, there's a delicate balance between task success and user agency. If the corrections feel too intrusive or introduce noticeable latency, it could detract from the VR experience. More research into human-agent collaboration and adaptive shared autonomy could be beneficial.

   Overall, this paper provides a robust foundation for bare-hand interaction in physics-rich virtual environments, intelligently addressing the long-standing challenges of HPE noise and physical realism through a well-designed hybrid learning framework. Its emphasis on residual correction and visual fidelity makes it a significant step towards truly immersive and intuitive human-computer interaction.