1. Bibliographic Information

• Title: Transformer-Based Imitative Reinforcement Learning for Multirobot Path Planning
• Authors: Lin Chen, Yaonan Wang, Zhiqiang Miao, Yang Mo, Mingtao Feng, Zhen Zhou, and Hesheng Wang. The authors are affiliated with several prominent Chinese universities, including Hunan University, Xidian University, and Shanghai Jiao Tong University, with expertise in robotics, visual perception, control engineering, and machine learning.
• Journal/Conference: IEEE Transactions on Industrial Informatics (TII). This is a highly reputable, top-tier journal in the field of industrial electronics and informatics, known for its rigorous peer-review process and focus on practical applications of advanced technologies.
• Publication Year: 2023 (Published January 30, 2023).
• Abstract: The paper addresses the Multirobot Path Planning (MRPP) problem, focusing on decentralized solutions for large-scale systems. The authors identify a key limitation in existing state-of-the-art methods: their inability to perform efficiently in dense environments without inter-robot communication. To solve this, they make two main contributions. First, they introduce a Transformer-based architecture for the policy neural network to better extract features for robot collaboration. Second, they propose a novel imitative reinforcement learning framework that combines contrastive learning with Double Deep Q-Network (DDQN) to effectively train this complex network. Simulation results demonstrate that their method achieves state-of-the-art performance in non-communicating scenarios. The policy is also validated on a physical system with three robots.
• Original Source Link: /files/papers/68ee03c058c9cb7bcb2c7e9a/paper.pdf (Formally published paper).

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: The paper tackles the Multirobot Path Planning (MRPP) problem, which involves finding collision-free paths for multiple robots from their starting points to their goals. The focus is on decentralized policies, where each robot makes decisions independently based on its local observations.
  • Key Challenge: Existing decentralized methods, even those using advanced techniques like reinforcement learning (RL) and imitation learning (IL), struggle in dense environments (many robots and/or obstacles) when inter-robot communication is unavailable. Without communication, robots cannot explicitly coordinate their actions, leading to deadlocks and collisions. Previous policy networks (often based on CNNs) have difficulty capturing the long-range dependencies and global context needed for implicit coordination.
  • Innovation: The paper introduces the Transformer architecture to the policy network for the first time in MRPP. Transformers are known for their ability to model long-range relationships in data, which the authors hypothesize will allow robots to learn implicit coordination strategies by better understanding the spatial relationships of other robots and obstacles from local observations. They also design a novel training framework to handle the challenges of training such a powerful model.

• Main Contributions / Findings (What):

  1. A Novel Transformer-Based Policy Network: They propose a new deep neural network architecture that uses a Transformer encoder to process a robot's local observations. This network is designed to extract richer feature representations that encode implied collaborative relationships among robots, improving navigation in dense settings.
  2. A Hybrid Imitation Reinforcement Learning Framework: Training a Transformer with sparse RL rewards is difficult. To address this, they develop a training method that combines:
     • Double Deep Q-Network (DDQN): An advanced RL algorithm to learn an optimal action-value function.
     • Supervised Contrastive Learning: An IL technique that uses expert demonstrations to guide the training. It helps the model learn from expert data more effectively, especially when the data is imbalanced (some actions are more frequent than others), and provides stronger supervisory signals than standard imitation methods.
  3. State-of-the-Art Performance without Communication: Through extensive simulations, the proposed method, named Transformer-based Imitative Reinforcement Learning (TIRL), is shown to outperform existing communication-free methods in terms of success rate. It also remains competitive with methods that rely on communication, showcasing a significant improvement in system robustness. The method's feasibility is confirmed with a real-world experiment.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Multirobot Path Planning (MRPP): Also known as Multi-Agent Path Finding (MAPF), this is the problem of finding non-conflicting paths for a group of agents from their start to goal locations.
  • Centralized vs. Decentralized Planning:
    • Centralized: A single central controller has full knowledge of all robots and the environment. It calculates all paths simultaneously. This is computationally expensive and not scalable to large numbers of robots.
    • Decentralized: Each robot plans its own path based on local information. This is scalable but makes coordination difficult, especially without communication. This paper focuses on a fully decentralized approach.
  • Reinforcement Learning (RL): A machine learning paradigm where an agent learns to make decisions by performing actions in an environment to maximize a cumulative reward. Key concepts include states, actions, rewards, and a policy (a strategy for choosing actions).
  • Imitation Learning (IL): A learning approach where an agent learns to perform a task by mimicking demonstrations from an expert. This is useful for complex problems where designing a good reward function for RL is difficult.
  • Deep Q-Network (DQN): A type of RL algorithm that uses a deep neural network to approximate the optimal action-value function (Q-function), which estimates the expected reward for taking a certain action in a given state.
  • Transformer: A neural network architecture originally developed for natural language processing. Its core component is the self-attention mechanism, which allows it to weigh the importance of different parts of the input data when processing it. This makes it exceptionally good at capturing long-range dependencies and contextual relationships, which is why the authors apply it to understand the spatial context in MRPP.

• Previous Works:

  • Classical Approaches:
    • ORCA (Optimal Reciprocal Collision Avoidance): A popular decentralized method, but it is reactive and prone to deadlocks in complex environments.
    • CBS (Conflict-Based Search): A centralized, optimal or sub-optimal planner. It is powerful but suffers from high computational complexity as the number of robots increases.
    • ODrM*: A centralized optimal planner used in this paper to generate expert data for imitation learning.
  • Learning-Based Approaches:
    • PRIMAL: A pioneering work that combined IL and RL for decentralized MRPP without communication. However, its CNN-based policy network struggles to capture complex interactions in dense scenarios.
    • GNN-based methods (e.g., MAGAT, DHC, DCC): These methods use Graph Neural Networks (GNNs) to model interactions between robots. They achieve high performance but critically rely on explicit communication, making the system vulnerable to communication failures.

• Technological Evolution: The field has moved from classical, search-based algorithms (CBS) to decentralized, learning-based policies. Initial learning models used simpler architectures like CNNs (PRIMAL). The next step was to incorporate communication using GNNs (DHC, DCC). This paper proposes the next logical step: using a more powerful architecture (Transformer) to achieve strong performance without the dependency on communication.

• Differentiation: The proposed TIRL method is distinct from prior work in two key ways:

  1. It is the first to apply a Transformer architecture to the MRPP problem, aiming for superior feature extraction from local observations to enable implicit coordination.
  2. It uses a unique combination of DDQN and supervised contrastive learning as its training framework. This addresses the difficulty of training a deep Transformer model and better leverages expert data compared to the simpler IL approach in PRIMAL.

4. Methodology (Core Technology & Implementation)

The core of the paper is a novel policy network and a specialized training framework.

Problem Formulation

The MRPP problem is modeled as a partially observable Markov game, described by the tuple $( \mathbb { S } , A, Pr , R , O , Z , \gamma )$.

• $\mathbb{S}$: The global state space of all robots.

• $A$: The joint action space for all robots.

• Pr: The state transition probability function.

• $R$: The reward function.

• $O$: The set of local observations for each robot.

• $Z$: The observation function. Each robot only gets a partial observation $o \in O$ from the global state $s \in \mathbb{S}$.

• $\gamma$: The discount factor for future rewards.

  The goal is to learn a policy $\pi$ that maps observations to actions to maximize the expected future discounted reward: $\mathbb { E } [ \sum _ { t = 0 } ^ { \infty } \gamma ^ { t } R ( s _ { t } , a _ { t } ) ]$. The paper uses a value-based RL method, specifically DQN and its variant DDQN, which learns an action-value function $Q(s, a)$ that predicts this expected return.

Learning Environment

• Observation Space: Each robot has a partially observable view of the world. The observation is composed of:

  1. A $6 \times 9 \times 9$ tensor representing a $9 \times 9$ grid centered on the robot. The 6 channels are binary matrices indicating:
     • Channel 1: Locations of other robots.
     • Channel 2: Locations of static obstacles.
     • Channels 3-6: Four heuristic channels (as in DHC [2]) guiding the robot towards its goal.
  2. A $5 \times 1$ vector representing the action taken in the previous step (one-hot encoded).
  3. A $2 \times 1$ vector representing the robot's current coordinates on the map.

• Action Space: Each robot can choose from five discrete actions at each time step: move North, East, South, West, or stay idle.

• Reward Design: The reward structure encourages efficient, collision-free navigation. This is a manual transcription of the paper's Table I.

  Action	Reward
  Move [N/E/S/W]	-0.075
  Collision (obstacle/robots)	-0.5
  No movement (ON/OFF goal)	0, -0.075
  Finish episode	3

  A small negative reward for moving encourages shorter paths. A large negative reward penalizes collisions. A large positive reward is given only when all robots reach their goals.

A. Transformer-Based Policy Network

This network maps a robot's observation to Q-values for each of the five possible actions. Its architecture is shown in the figure below.

该图像是图2，展示了基于Transformer的策略神经网络结构示意图。该网络输入包含观察数据、机器人位置和前一步动作，通过嵌入层和Transformer编码器进行特征提取，最后通过全连接层输出动作价值函数Q(s,a)。

• Input Processing: The main input is the $9 \times 9 \times 6$ observation data. This grid is divided into nine non-overlapping $3 \times 3 \times 6$ patches. This segmentation captures local spatial information corresponding to different directions around the robot (e.g., front, back, left, right, and diagonals).

• Patch and Action Embeddings:

  • Each of the nine $3 \times 3 \times 6$ patches is flattened into a vector of size 54.
  • A learnable [action] token is created, similar to the [class] token in Vision Transformer (ViT). This token aggregates global information from all patches to eventually produce the final Q-values.
  • The flattened patches and the action token are passed through a trainable linear projection (MLP) to map them to a uniform dimension $D$.

• Positional Embeddings: To retain spatial information, learnable positional embeddings are added to the patch embeddings. The final input to the Transformer encoder is a sequence of 10 vectors (9 for patches + 1 for the action token).

  $$\mathbf { z } _ { 0 } = \left[ \mathbf { o } _ { \mathrm { a c t i o n } } ; \mathbf { o } _ { p } ^ { 1 } \mathbf { W } ; \mathbf { o } _ { p } ^ { 2 } \mathbf { W } ; \cdot \cdot \cdot ; \mathbf { o } _ { p } ^ { 9 } \mathbf { W } \right] + \mathbf { W } _ { \mathrm { p o s } }$$

  • $\mathbf{o}_{\mathrm{action}}$: The learnable action token embedding.
  • $\mathbf{o}_{p}^{i}$: The $i$-th flattened patch.
  • $\mathbf{W}$: The weight matrix of the linear projection layer.
  • $\mathbf{W}_{\mathrm{pos}}$: The learnable positional embedding matrix.

• Transformer Encoder: The sequence of embeddings $\mathbf{z}_0$ is passed through a standard Transformer encoder composed of $L$ identical blocks.

  该图像是图3，Transformer编码器的示意图，展示了输入嵌入补丁通过多头注意力层、标准化层和多层感知器的处理流程，并细化了多头注意力机制中的关键计算步骤，包括查询Q、键K、值V矩阵及其softmax操作。

  Each block consists of:

  1. Layer Normalization (LN): Normalizes the inputs to stabilize training.
  2. Multi-Head Self-Attention (MSA): This is the core of the Transformer. It allows each patch embedding to attend to all other patch embeddings, calculating attention weights to determine how much focus to place on other parts of the observation. This is how the model captures long-range dependencies and implicit relationships between robots and obstacles. The MSA mechanism computes Query ($\mathbf{Q}$), Key ($\mathbf{K}$), and Value ($\mathbf{V}$) matrices and calculates attention scores as: $$\mathbf { A } _ { \ell } = \mathrm { s o f t m a x } \left( \mathbf { Q } _ { \ell } \mathbf { K } _ { \ell } ^ { \top } / \sqrt { D _ { h } } \right)$$
  3. Residual Connection: The output of the MSA block is added to its input, preventing vanishing gradients.
  4. Layer Normalization (LN).
  5. Multilayer Perceptron (MLP): A standard feed-forward network applied to each token independently.
  6. Residual Connection.

• Output Head: The output embedding corresponding to the [action] token is passed through a final MLP to produce the Q-values for the five possible actions.

B. Imitation Reinforcement Learning

The paper proposes a hybrid training framework to effectively train the deep Transformer network. The total loss is a weighted sum of an RL loss and an IL loss.

该图像是图4，展示了本文提出的模仿强化学习训练流程示意图，包含状态、动作、奖励等采样过程及双深度Q网络框架。损失函数为 $Loss(\theta) = L_1(\theta) + L_2(\theta)$，描述了策略网络的更新和误差反馈。

• Reinforcement Learning with DDQN:
  • The RL component uses Double Deep Q-Network (DDQN). Standard DQN is known to sometimes overestimate Q-values, leading to suboptimal policies. DDQN mitigates this by decoupling action selection from action evaluation.
  • The loss function for the RL part ($L_1$) is the mean squared error between the predicted Q-value and the target Q-value, where the target is calculated using the DDQN update rule: $$y _ { \mathrm { d o u b l e } } = r _ { t } + \gamma Q \left( s _ { t + 1 } , \underset { a } { \arg \operatorname* { m a x } } Q \left( s _ { t + 1 } , a _ { t } ; \theta _ { i } \right) ; \theta _ { i - 1 } \right)$$ $$L _ { 1 } \left( \theta _ { i } \right) = \mathbb { E } \left[ \left( y _ { \mathrm { d o u b l e } } - Q \left( s _ { t } , a _ { t } ; \theta _ { i } \right) \right) ^ { 2 } \right]$$
    • $\theta_i$: Parameters of the main Q-network at iteration $i$.
    • $\theta_{i-1}$: Parameters of the target network (a delayed copy of the main network).
• Imitation Learning with Supervised Contrastive Learning:

  • The IL component uses expert demonstrations (generated by the ODrM* planner) to provide strong supervisory signals. Instead of simple behavioral cloning, it uses a supervised contrastive loss.

  • The goal is to make the network's output Q-vector for a given state similar to a target vector representing the expert's action, and dissimilar to vectors representing other actions.

  • To achieve this, the expert action $a^*$ is converted into a "hot vector" $h(a^*)$. This is not a standard one-hot vector. Instead, it has a high value for the expert action and small negative values for all other actions. This is shown in the manually transcribed Table II below (where $0 < \tau < 0.2$).

    Expert action (a*)	One-hot vector (oh(a*))	Hot vector (h(a*))
    forward	(1, 0, 0, 0, 0)	(1-$\tau$, -$\tau$, -$\tau$, -$\tau$, -$\tau$)
    backward	(0, 1, 0, 0, 0)	(-$\tau$, 1-$\tau$, -$\tau$, -$\tau$, -$\tau$)
    left	(0, 0, 1, 0, 0)	(-$\tau$, -$\tau$, 1-$\tau$, -$\tau$, -$\tau$)
    right	(0, 0, 0, 1, 0)	(-$\tau$, -$\tau$, -$\tau$, 1-$\tau$, -$\tau$)
    stop	(0, 0, 0, 0, 1)	(-$\tau$, -$\tau$, -$\tau$, -$\tau$, 1-$\tau$)

  • The contrastive loss function ($L_2$) encourages the dot product (similarity) between the network's predicted Q-vector $Q(s; \theta)$ and the hot vector $h(a^*_p)$ to be high for samples with the same expert action, and low for samples with different expert actions. $$\mathcal { L } _ { 2 } \left( \theta \right) = \sum _ { i \in I } \frac { - 1 } { | P ( i ) | } \sum _ { p \in P ( i ) } \log \frac { \exp \left( \sin \left( h \left( a _ { p } ^ { * } \right) , Q ( s _ { p } ; \theta ) \right) / \tau \right) } { \sum _ { i \in I } \exp \left( \sin \left( h \left( a _ { p } ^ { * } \right) , Q ( s _ { i } ; \theta ) \right) / \tau \right) }$$

    • $I$: The set of samples in a training batch.
    • $P(i)$: The set of "positive" samples in the batch that have the same expert action as sample $i$.
    • $\sin(\cdot, \cdot)$: Cosine similarity, measured via dot product.
    • $\tau$: A temperature parameter.
• Total Loss: The final loss function is a weighted sum of the DDQN loss and the contrastive IL loss. $$\mathcal { L } o s s ( \theta _ { i } ) = \alpha \mathcal { L } _ { 1 } ( \theta _ { i } ) + \beta \mathcal { L } _ { 2 } ( \theta _ { i } )$$
  • $\alpha$ and $\beta$ are weights that balance the influence of reinforcement learning and imitation learning.

5. Experimental Setup

• Datasets: There are no fixed datasets. Instead, a simulation environment is used for training and testing.
  • Training: Map sizes are randomly chosen between $40 \times 40$ and $80 \times 80$. The number of robots (team size) is randomly chosen between 1 and 64.
  • Testing: Performed on $40 \times 40$ and $80 \times 80$ maps with a fixed obstacle density of 0.3. The number of agents is varied (4, 8, 16, 32, 64). Each test scenario consists of 200 different cases.
  • Expert Data: The centralized planner ODrM* is used to generate expert trajectories for imitation learning.
• Evaluation Metrics:
  1. Success Rate:
     • Conceptual Definition: This metric measures the percentage of test cases where all robots successfully reach their designated goals within a predefined maximum number of time steps (256 for $40 \times 40$ maps, 386 for $80 \times 80$ maps). It is the primary metric for evaluating the policy's effectiveness and robustness.
     • Mathematical Formula: $$\text{Success Rate} = \frac{\text{Number of tasks completed successfully}}{\text{Total number of tasks attempted}} \times 100\%$$
  2. Average Steps:
     • Conceptual Definition: This metric measures the average number of time steps taken by the robots to complete a task, calculated only over the successful episodes. It reflects the efficiency of the paths generated by the policy. A lower value is better.
     • Mathematical Formula: $\text{Average Steps} = \frac{\sum_{i=1}^{N_s} \text{Steps}_i}{N_s}$
     • Symbol Explanation: $N_s$ is the number of successful episodes, and $\text{Steps}_i$ is the number of time steps for the $i$-th successful episode.
• Baselines: The proposed TIRL method is compared against a wide range of baselines:
  • Traditional Methods: ORCA (decentralized) and CBS (centralized).
  • Learning-Based (No Communication): PRIMAL, DHC-Baseline (DHC without communication and heuristics), DHC/Comm (DHC without communication).
  • Learning-Based (With Communication): DHC, RR-N2, DCC.
  • Ablation Baselines: TIRL/PE (TIRL without positional embedding) and Res-2 (TIRL with the Transformer module replaced by a ResNet-like CNN).

6. Results & Analysis

• Core Results:

  • Training Convergence: Figure 5 shows the loss curve during training, indicating that the proposed hybrid loss function allows the complex network to converge successfully.

    该图像是图表，展示了论文中提出的TIRL训练过程中损失函数随训练轮次（Episode）变化的曲线。曲线表明损失值随着训练进行呈下降趋势，最终趋于稳定，反映了训练过程中的收敛情况。

  • Comparison with Traditional Methods (Fig. 6): TIRL significantly outperforms both ORCA and CBS. ORCA fails in dense scenarios due to deadlocks. CBS, being centralized, struggles with computational complexity for a high number of agents (32 and 64), leading to very low success rates.

    该图像是图表，展示了图6中TIRL与传统方法在多机器人路径规划中成功率的比较。左图(a)比较了TIRL与集中式方法的成功率，右图(b)展示了TIRL与去中心化方法ORCA的成功率，均显示TIRL在不同机器人数量下表现更优。

  • Comparison with Non-Communicating Methods (Fig. 7): TIRL achieves a state-of-the-art success rate across all tested agent counts and map sizes when compared to other methods that do not use communication (PRIMAL, DHC-Baseline, DHC/Comm). This confirms the hypothesis that the Transformer architecture is more effective at learning implicit coordination than the CNNs used in previous methods.

    该图像是图表，展示了图7中TIRL方法与三种无通信多机器人路径规划方法在不同代理数量下的成功率比较。图中分别显示了地图尺寸40×40和80×80、障碍物密度0.3条件下的性能表现，TIRL取得了最高的成功率。

  • Comparison with Communicating Methods (Fig. 8): TIRL is highly competitive with methods that rely on communication. It outperforms them with a smaller number of agents. For a very large number of agents (e.g., 64), communication-based methods like DCC and RR-N2 eventually achieve a higher success rate. This is expected, as explicit communication provides more direct information for coordination. However, TIRL's strong performance highlights its ability to learn powerful coordination strategies without the fragility of a communication link, thus offering greater system robustness.

    该图像是图表，展示了在障碍物密度为0.3的环境中，不同地图大小（40×40和80×80）和不同数量智能体情况下，TIRL、DHC、RR-N2及DCC方法的成功率对比。图中显示随着智能体数量增加，TIRL表现优于其他方法。

  • Path Efficiency (Table III): The following is a manual transcription of Table III from the paper. TIRL generates highly efficient paths, often having the lowest Average Steps among the learning-based methods, particularly in the $40 \times 40$ map. This shows that the learned policy not only avoids collisions but also finds near-optimal paths. It only starts to become less efficient than communication-based methods in the largest, most crowded scenarios.

    Average Steps in 40×40 Map size

    Agents	ODrM*(=10)	TIRL	DCC	DHC	PRIMAL
    4	47.86	47.86	48.575	52.33	79.08
    8	55.47	56.42	59.60	63.9	76.53
    16	61.89	62.85	71.34	79.63	107.14
    32	66.94	73.46	93.54	100.1	155.21
    64	85.27	103.70	135.55	147.26	170.48

    Average Steps in 80×80 Map size

    Agents	ODrM*(=10)	TIRL	DCC	DHC	PRIMAL
    4	91.87	91.87	93.89	96.72	134.86
    8	105.72	105.72	109.89	109.24	153.20
    16	112.64	114.06	122.24	122.54	180.74
    32	122.28	136.54	132.99	138.32	250.07
    64	131.19	176.25	159.67	163.50	321.63

• Ablations / Parameter Sensitivity:

  • Ablation Study (Fig. 9): This study is crucial.

    • TIRL vs. TIRL/PE: Removing positional embeddings (PE) causes a significant drop in success rate, proving that encoding the spatial location of observed patches is vital for the Transformer to make sense of the input.

    • TIRL vs. Res-2: Replacing the Transformer with a standard CNN (Res-2) also leads to much lower performance. This directly supports the central claim that the Transformer's ability to capture long-range dependencies is superior to CNNs for this task.

      该图像是图表，展示了图9中TIRL与TIRL/PE及Res-2算法在不同地图尺寸40×40和80×80、障碍物密度为0.3时，随着机器人数量增加的成功率对比。

• Real-World Experiment:

  • The policy was deployed on three physical Autonomous Ground Vehicles (AGVs) in a $6 \times 4$ grid with obstacles. The experiment, shown below, demonstrates that the policy trained in simulation can be successfully transferred to the real world (sim-to-real). The robots successfully navigated to their goals without collisions, proving the practical viability of the approach. The paper also notes the model size and inference time are practical for deployment on embedded hardware like a Raspberry Pi.

    该图像是由多机器人路径规划实验的实景照片和对应的二维网格示意图组成，展示了三机器人从起点到终点在障碍环境中路径规划的过程与轨迹变化，图(a)-(f)为机器人移动实景，图(g)-(l)为对应路径规划示意。

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces a Transformer-based policy network for multirobot path planning, establishing a new state-of-the-art for decentralized, communication-free scenarios. The proposed imitative reinforcement learning framework, combining DDQN with supervised contrastive learning, is shown to be highly effective for training the complex Transformer architecture. The results demonstrate that by leveraging the Transformer's ability to capture long-range dependencies, robots can learn to coordinate implicitly and efficiently, leading to high success rates and robust performance in dense environments.

• Limitations & Future Work:

  • Success Rate: The policy does not achieve a 100% success rate, indicating that some complex scenarios still lead to failures or deadlocks.
  • A Priori Knowledge: The system requires each robot to know the complete map layout (static obstacles) and all start/goal positions at the beginning of the task. This is a strong assumption.
  • Future Work: The authors suggest integrating onboard sensors (like cameras or LiDAR) to allow robots to perceive their environment dynamically, removing the need for pre-supplied maps. They also plan to further investigate methods to establish even stronger correlations between observations and effective cooperative planning signals.

• Personal Insights & Critique:

  • Strengths:
    • The application of Transformers to MRPP is a significant and logical step forward for the field, moving beyond CNNs and GNNs. The results strongly validate this choice.
    • The hybrid training framework is innovative. Using supervised contrastive learning with a custom "hot vector" target is a clever way to apply this technique to a value-based RL problem and provides a much richer training signal than standard imitation learning losses.
    • The emphasis on a communication-free solution is a major practical advantage. In many real-world applications (e.g., disaster response, exploration in remote areas), reliable communication cannot be guaranteed. A robust, communication-free policy is therefore highly valuable.
  • Potential Weaknesses & Open Questions:
    • Scalability of Observation: The fixed $9 \times 9$ observation window might be a limitation. In very large, sparse environments, this window may not capture distant agents that could become relevant later. A dynamic or multi-scale observation approach could be an interesting extension.
    • Expert Data Dependency: The method still relies on a centralized optimal planner (ODrM*) to generate expert data. This could be a bottleneck for training in extremely large or complex environments where generating optimal solutions is intractable. Future work could explore learning without any expert data or using sub-optimal but easily generated demonstrations.
    • Generalization: While tested on random maps, the policy's ability to generalize to drastically different obstacle configurations or map types (e.g., non-grid worlds, environments with narrow corridors) would be interesting to explore further.