1. Bibliographic Information

1.1. Title

ATLAS: Actor-Critic Task-Completion with Look-ahead Action Simulation

1.2. Authors

The paper is listed under "Anonymous authors," indicating it is currently under double-blind review. Therefore, specific authors, their research backgrounds, and affiliations are not disclosed at this stage.

1.3. Journal/Conference

The paper is published on OpenReview.net, indicated by the original source link. OpenReview is a platform widely used for managing peer-review for major conferences, particularly in machine learning (e.g., ICLR, NeurIPS). Given that the paper is "under double-blind review" and its content aligns with cutting-edge research in AI agents and large language models, it is likely submitted to a highly reputable conference in artificial intelligence or machine learning.

1.4. Publication Year

2025 (indicated by the Published at (UTC) timestamp: 2025-10-08T00:00:00.000Z).

1.5. Abstract

The paper introduces ATLAS (Actor-Critic Task-completion with Look-ahead Action Simulation), a novel memory-augmented web agent designed to overcome the limitations of current state-of-the-art web agents, which struggle to adapt to new environments without neural network fine-tuning and often produce inefficient execution plans due to a lack of environmental awareness. ATLAS addresses this by grounding its plans in an explicit model of the environment, simulating action consequences in cognitive space. The agent first builds a cognitive map through lightweight, curiosity-driven exploration. Its modular architecture comprises a planner for proposing actions, a simulator for predicting consequences, a critic for selecting the best rollout and updating plans, and a browser executor. ATLAS achieves a 63% success rate on the WebArena-Lite Benchmark, outperforming the previous state-of-the-art's 53.9% without requiring website-specific LLM fine-tuning. Ablation studies confirm the crucial and complementary roles of its world-model, hierarchical planner, and look-ahead-based replanner.

1.6. Original Source Link

Official Source: https://openreview.net/forum?id=hwwn9hAAo5 PDF Link: https://openreview.net/pdf?id=hwwn9hAAo5 Publication Status: The paper is currently under double-blind review.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper addresses is the inability of current state-of-the-art web agents to effectively adapt to new web environments without extensive neural network fine-tuning. These agents often produce inefficient execution plans because they lack an understanding of the new environment's structure and dynamics.

This problem is critical in the current field because autonomous agents that can reliably navigate and act on the web have immense potential for performing complex tasks on behalf of users, such as information gathering, transactions, and site configurations. However, current web agents fall short of human-level reliability, especially on long-horizon tasks (tasks requiring many sequential steps and potentially spanning multiple pages or sessions). The challenges stem from partial observability (not all information is immediately visible), vast action spaces (many possible actions at any given time), and the need for multi-step planning and memory in dynamic web environments. For example, tasks on benchmarks like WebArena require understanding site structures, remembering states (like login or applied filters), and avoiding irreversible mistakes. Existing LLM (Large Language Model) agents, while adept at semantic understanding, are typically reactive (only respond to immediate observations) and lack structured memory and explicit planning capabilities. Systems like Plan-and-Act often require website-specific fine-tuning for their planner and executor modules, which limits their adaptability to new websites or use-cases.

The paper's entry point or innovative idea is to introduce ATLAS, an inference-time, actor-critic web agent that mitigates these issues by planning before acting, employing look-ahead simulation, and retrieving structured memories. This allows ATLAS to build an internal model of the environment, enabling more grounded and efficient planning without site-specific fine-tuning.

2.2. Main Contributions / Findings

The paper makes several primary contributions:

• An actor-critic planner with LLM-based look-ahead that evaluates actions via simulated outcomes: ATLAS integrates an actor-critic framework with LLM-powered look-ahead simulation, allowing the agent to predict the consequences of candidate actions in a cognitive space before executing them in the real environment. This significantly enhances the agent's ability to choose safer and more goal-aligned actions.

• A multi-layer memory with a cognitive map built through exploration and agentic summarization, used online for retrieval and replanning: ATLAS features a sophisticated memory system comprising Working Memory, a Cognitive Map (a graph of state transitions and action outcomes), and Semantic Memory (world knowledge). The Cognitive Map is constructed through curiosity-driven exploration and agentic summarization, which distills action-outcome deltas in natural language, making it efficient and interpretable. This memory is dynamically queried for planning and updated during execution.

• A practical modular architecture that integrates planning, memory, and simulation to transform high-level instructions into safe, executable action sequences for long-horizon web tasks: The modular design of ATLAS (Planner, Actor, Critic, Memory) allows for effective integration of these components. Crucially, this architecture enables strong performance without requiring website-specific LLM fine-tuning, making it easily portable to new web domains and adaptable to different underlying LLMs.

  The key findings demonstrate that ATLAS achieves a 63% success rate on the WebArena-Lite Benchmark, significantly outperforming the previously published state-of-the-art (53.9%). Ablation studies confirm that the world-model (cognitive map), hierarchical planner, and look-ahead-based replanner are all crucial and complementary components, as removing any of them leads to a substantial drop in performance. These findings highlight ATLAS's ability to perform complex web tasks reliably and adaptively without extensive retraining.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand ATLAS, a few foundational concepts from AI, reinforcement learning, and LLM agents are essential:

• Partially Observable Markov Decision Process (POMDP): This is a mathematical framework for modeling sequential decision-making in environments where the agent does not have direct access to the true state of the environment. Instead, it receives observations that are probabilistically related to the underlying state. In web navigation, an agent rarely sees the full HTML or backend state, only a rendered view or specific DOM elements, making it a POMDP.
  • A POMDP is formally defined by a tuple $(\mathcal{S}, \mathcal{A}, \mathcal{O}, T, R)$, where:
    • $\mathcal{S}$ is the set of possible hidden states of the environment.
    • $\mathcal{A}$ is the set of actions the agent can take.
    • $\mathcal{O}$ is the set of observations the agent can perceive.
    • $T$ is the state transition function $P(s' | s, a)$, which specifies the probability of transitioning to state $s'$ given the current state $s$ and action $a$.
    • $R$ is the reward function R(s, a), which provides a scalar reward for taking action $a$ in state $s$.
• Large Language Models (LLMs): These are advanced neural networks trained on massive amounts of text data, enabling them to understand, generate, and reason with human language. LLMs are at the core of ATLAS's Planner, Actor, Critic, and Memory modules, allowing them to process natural language instructions, generate coherent plans, propose actions, and summarize observations.
• Actor-Critic Methods: In reinforcement learning, actor-critic methods combine two components:
  • An Actor: This component is responsible for selecting actions. It learns a policy (a strategy for choosing actions) that maps states to actions.
  • A Critic: This component evaluates the actions taken by the actor. It learns a value function that estimates the expected future reward from a given state or state-action pair. The critic's evaluation helps the actor improve its policy. ATLAS adapts this concept: the Actor proposes candidate actions, and the Critic evaluates them using LLM-based assessment and simulation.
• World Model: A world model is an internal representation that an agent learns about its environment. It allows the agent to predict future states or consequences of its actions without actually performing them in the real world. This is crucial for planning, as an agent can simulate different action sequences to determine the best path to a goal. ATLAS's Cognitive Map serves as its world model, enabling look-ahead simulation.
• Look-ahead / Tree Search: These are planning techniques where an agent explores possible future sequences of actions and states to make better current decisions. Tree search algorithms (like Monte Carlo Tree Search or beam search) build a tree of possible future states and evaluate paths to find the optimal one. Look-ahead specifically refers to simulating consequences for a few steps into the future. ATLAS uses Look-ahead Action Simulation to evaluate candidate actions.
• Curiosity-Driven Exploration: In reinforcement learning, curiosity can be used as an intrinsic reward signal to encourage an agent to explore novel or uncertain states in its environment. This is particularly useful in environments with sparse or delayed external rewards. ATLAS employs curiosity-driven exploration to build its initial Cognitive Map and Semantic Memory.
• Agentic Summarization: This refers to using an LLM agent to process raw information (like HTML changes or trajectories) and distill it into concise, human-readable, and relevant summaries. Instead of storing raw data, ATLAS's memory agent curates what is stored, emphasizing deltas (changes) and affordances (new interaction possibilities).

3.2. Previous Works

The paper grounds its contributions by referencing significant prior research:

• LLM-Based Autonomous Agents:
  • ReAct (Yao et al., 2023): This framework was pioneering in demonstrating how LLMs can interleave reasoning (generating thoughts) and acting (performing actions) in interactive environments. It structured LLM agents to generate a thought, then an action, then observe, and repeat.
  • Reflexion (Shinn et al., 2024): Extended ReAct by incorporating self-reflection mechanisms, allowing agents to learn from past mistakes and improve decision-making over longer horizons. The agent could critique its own past trajectories and generate verbal feedback to refine its policy.
• Web Navigation Agents:
  • Early Systems (Liu et al., 2018): Relied on rule-based approaches and predefined scripts, which were interpretable but lacked adaptability to dynamic web environments.
  • WebArena (Zhou et al., 2024): A comprehensive benchmark designed to evaluate web agents across realistic multi-step tasks, providing a standardized environment for research.
  • WebArena-Lite (Liu et al., 2024a,b): A curated, quality-controlled subset of WebArena addressing scalability and quality concerns, used as the primary benchmark in ATLAS.
  • WebRL (Qi et al., 2024): Applied reinforcement learning principles to web navigation, focusing on policy optimization for improved performance.
  • WebPilot (Zhang et al., 2025): Emphasized multimodal understanding of web content for navigation.
  • Plan-and-Act (Erdogan et al., 2025): Focused on hierarchical task decomposition for long-horizon tasks. ATLAS specifically highlights Plan-and-Act's limitation of requiring website-specific fine-tuning.
  • Agent Workflow Memory (AWM) (Wang et al., 2024a,b): Demonstrated the importance of persistent memory for multi-step web tasks.
  • AgentOccam (Yang et al., 2024): Showed the effectiveness of simplifying the action space to natural language commands for LLM agents. ATLAS builds upon AgentOccam as its base agent for experimentation.
• Memory-Augmented Agents:
  • Memory Networks (Weston et al., 2014): Established foundational work on external memory modules for neural networks, adapted for sequential decision-making.
  • MemoryBank (Zhong et al., 2023): A comprehensive framework for managing episodic and semantic memory in LLM-based agents.
  • Cognitive Maps (Tolman, 1948): The original concept from cognitive science, describing how animals and humans build internal mental representations of their environment.
  • Neural Cognitive Mapping (Wayne et al., 2018; Park et al., 2023): Showed neural implementations of cognitive mapping in RL and how LLMs can maintain spatial-temporal memory for behavioral simulation.
• Planning and Simulation in AI Agents:
  • Monte Carlo Tree Search (MCTS) (Browne et al., 2012): A widely used tree search algorithm, proven effective in discrete domains, that explores game states by simulating random playouts.
  • Tree of Thoughts (Yao et al., 2024): Enabled LLMs to explore multiple reasoning paths through structured search, akin to MCTS but in conceptual space.
  • World Models (Ha & Schmidhuber, 2018; Micheli et al., 2022): Foundational work on learned world models and how transformer architectures can serve as effective models for sequential decision-making.
• Actor-Critic Methods and Look-ahead Planning:
  • Traditional Actor-Critic (Sutton & Barto, 2018): Core RL training methods.
  • AlphaGo (Silver et al., 2016): Demonstrated the power of combining tree search with learned value functions in an actor-critic framework.
• Curiosity-Driven Exploration:
  • Intrinsic Curiosity Modules (Pathak et al., 2017): Introduced intrinsic curiosity as a reward signal based on prediction error to drive self-supervised exploration.
  • Language-Based Curiosity (Mu et al., 2024): Extended curiosity-driven exploration to language-based and embodied AI settings.

3.3. Technological Evolution

Web navigation agents have evolved significantly. Initially, they were predominantly rule-based, relying on predefined scripts for specific tasks on known websites. This approach was brittle and lacked adaptability to dynamic web environments or changes in UI. The advent of machine learning led to learning-based methods, where agents could learn policies from data, but often struggled with generalization and long-horizon tasks.

The rise of Large Language Models (LLMs) marked a paradigm shift. LLMs brought unprecedented capabilities in semantic understanding and natural language generation, enabling agents to interpret open-ended instructions, reason about web content, and generate natural language actions. Frameworks like ReAct showed how LLMs could bridge reasoning and acting. However, even LLM-based agents faced challenges: they could be reactive rather than proactive, lacked robust memory structures, and often hallucinated outcomes, especially when dealing with unfamiliar environments or complex multi-step tasks. Many required fine-tuning for specific websites, hindering portability.

ATLAS fits into this evolution by addressing the limitations of reactive LLM agents and fine-tuning dependencies. It represents a step towards more robust, adaptable, and interpretable web agents by explicitly incorporating memory (especially a learned cognitive map), look-ahead simulation, and a hierarchical planning mechanism. By grounding plans in an explicit model of the environment rather than relying solely on LLM imagination, and by using curiosity-driven exploration for initial environment modeling, ATLAS advances the field towards agents that can truly adapt to new web environments inference-time.

3.4. Differentiation Analysis

Compared to the main methods in related work, ATLAS presents several core differences and innovations:

• Explicit World Model vs. Implicit Learning/Hallucination:
  • Previous Work (e.g., Plan-and-Act): Often attempts to learn an implicit world-model through neural network fine-tuning or relies on the LLM to envision action outcomes. This can be prone to hallucination and is not robust, especially in novel environments.
  • ATLAS: Leverages an explicit Cognitive Map built through curiosity-driven exploration and agentic summarization. This cognitive map serves as a trustworthy world model derived from real observations, allowing for more reliable look-ahead simulation.
• Adaptability without Fine-tuning:
  • Previous Work (e.g., Plan-and-Act): Many state-of-the-art web agents require website-specific LLM fine-tuning for their planner and executor modules to adapt to new environments. This is costly and limits scalability.
  • ATLAS: Its modular architecture is designed for inference-time planning and memory retrieval. It requires no website-specific LLM fine-tuning, making it readily adaptable to new domains and underlying LLMs by building its cognitive map through exploration.
• Comprehensive Look-ahead Planning vs. Greedy Search:
  • Previous Work (e.g., Tree of Thoughts): While some methods incorporate tree search, they often rely on LLMs as a reward function or world model and perform essentially greedy searches (one-step evaluation), pruning low-scoring branches immediately. This might overlook actions that are not immediately optimal but lead to better outcomes in subsequent steps.
  • ATLAS: Performs Look-ahead Action Simulation (LAS) akin to beam search over multiple steps. It considers the joint outcome of a sequence of actions by simulating $D$ steps into the future, providing a more comprehensive evaluation of action candidates.
• Efficiency of Simulation:
  • Previous Work: Actual execution of actions for exploration can be time-consuming and irreversible.
  • ATLAS: Its LAS is a simulation in conceptual space using the cognitive map. This is significantly more efficient than real-world execution and avoids stateful actions that cannot be recovered.
• Structured, Multi-layered Memory:
  • Previous Work (e.g., AWM, MemoryBank): While memory-augmented agents exist, ATLAS proposes a specific multi-layered memory system (Working Memory, Cognitive Map, Semantic Memory) with agentic summarization for efficient storage and retrieval of environment dynamics, constraints, and causal relationships, tailored for web navigation.
• Dynamic Replanning Grounded in Simulation:

  • ATLAS: Features Look-ahead Action Simulation-Backed Dynamic Replanning, where replanning is triggered when observations diverge from expectations and the results of the simulated tree search update the planner, preventing catastrophic forgetting and integrating a basic causal learning module.

    In essence, ATLAS moves beyond reactive or fine-tuned LLM agents by building an explicit, robust world model of the web environment through exploration and leveraging this model for proactive, look-ahead planning and dynamic replanning without the need for domain-specific LLM retraining.

4. Methodology

4.1. Principles

The core idea behind ATLAS is to enable an autonomous web agent to perform complex, long-horizon tasks in new web environments by actively constructing and leveraging an internal model of the environment. This is achieved through an inference-time actor-critic loop that integrates look-ahead action simulation in a cognitive space. The theoretical basis is that by understanding the dynamics and structure of the environment, the agent can make more informed, efficient, and safer decisions, moving beyond reactive behaviors or reliance on costly fine-tuning. The intuition is similar to how humans explore a new interface, learn its functionalities and constraints, and then plan their actions based on that learned mental model.

4.2. Core Methodology In-depth (Layer by Layer)

ATLAS frames web navigation as a Partially Observable Markov Decision Process (POMDP) and comprises four main modules: Planner, Actor, Critic, and Multi-layered Memory.

4.2.1. Problem Formulation

The problem of web navigation is defined as a POMDP by the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{O}, T, R)$:

• $\mathcal{S}$: The set of all possible underlying states of the web environment (e.g., full DOM structure, backend data, user session status). The agent typically does not observe this directly.

• $\mathcal{A}$: The set of actions the agent can take (e.g., click on an element, type text into a field, go_back, stop).

• $\mathcal{O}$: The set of observations the agent receives at each step (e.g., rendered HTML content, current URL, screenshots). These are partial views of the true state.

• $T$: The state transition function, which describes how the environment changes from one state to another given an action. $P(s' | s, a)$ is the probability of reaching state $s'$ from state $s$ by taking action $a$.

• $R$: The reward function, which gives a scalar value for achieving subgoals or the final task.

  Given a natural-language goal $q$, the agent's objective is to synthesize a plan and execute a sequence of actions $(a_0, \ldots, a_T)$ that leads to a goal-consistent terminal state, maximizing the reward (task fulfillment). At each time step $t$, the agent receives an observation $o_t \in \mathcal{O}$ and chooses an action $a_t \in \mathcal{A}$.

4.2.2. Architecture Overview

The overall architecture of ATLAS is illustrated in Figure 1 (a). It operates in an inference-time actor-critic loop with action simulation in conceptual space.

该图像是图1，展示了ATLAS系统的整体架构与流程示意，包括(a)系统流程，(b)基于好奇心驱动的记忆构建，以及(c)前瞻动作模拟（Look-ahead Action Simulation）的关键模块和信息流。

Figure 1 (a) Overall flow of ATLAS: The raw observation $o_t$ is summarized to lower cognitive load. Then the planner makes a plan $P_t$ based on the summarized observation o'_t. The actor proposes $N$ possible candidate actions for the next step. The critic provides judgment of action candidates and finalizes the best action $a_t$ to take by considering action outcomes obtained from the cognitive map.

The four main modules are:

1. Planner: Decomposes the task into subgoals and can dynamically replan.

2. Actor: Proposes a small set of $N$ diverse candidate actions for the next step.

3. Critic: Evaluates each candidate action by simulating its outcome using the Cognitive Map and selects the safest, most goal-advancing action.

4. Multi-layered Memory: Provides context, stores the Cognitive Map (state transitions), and World Knowledge (environment constraints), queried online and updated as needed.

   These modules work together to perform a simulated look-ahead tree search in conceptual space, enabling adaptive environment-grounded planning and efficient action selection.

4.2.2.1. Planner

The Planner module analyzes and decomposes the natural language task $q$ into a structured plan consisting of subtasks. Given the initial observation $o_0$, it produces an initial plan $P_0$. At each subsequent step $t$, it dynamically decides whether the plan needs to be updated (replanning) based on new evidence.

The planner's operation is formalized as: $$P_{0}=\operatorname{Planner}\left(q, o_{0}\right), \quad P_{t}=\operatorname{Planner}\left(q, o_{t}, s_{t}, M\right)$$ Here:

• $P_0$: The initial plan generated at the start of the task.

• $q$: The natural language task goal.

• $o_0$: The initial observation of the environment.

• $P_t$: The plan at time step $t$.

• $o_t$: The current observation.

• $s_t$: The current state of the agent (e.g., internal variables, current step in a subtask).

• $M$: The Multi-layered Memory, specifically the Cognitive Map and Semantic Memory, which provides context and knowledge for planning.

  The plans are concise lists of sub-goals with success predicates (e.g., "Reports $\rightarrow$ Sales $\rightarrow$ Set dates $\rightarrow$ Read table"). The outputs of the Planner are included in the context provided to the Actor and Critic. The Planner is implemented in the style of Chae et al. (2025) and extended as described in Section 3.5 of the paper.

4.2.2.2. Actor-Critic Interplay with Look-ahead

At each time step $t$, the Actor proposes a set of $N$ executable candidate actions, along with their reasoning. The Critic then evaluates these candidates.

The Actor's function is defined as: $$C_{t}=\operatorname{Actor}\left(q, P_{t}, o_{t}, s_{t}, M\right), \quad\left|C_{t}\right|=N$$ Here:

• $C_t$: The set of $N$ candidate actions proposed by the Actor at time $t$.

• $q$: The natural language task goal.

• $P_t$: The current plan from the Planner.

• $o_t$: The current observation.

• $s_t$: The current agent state.

• $M$: The Multi-layered Memory.

• $|C_t|=N$: The number of candidate actions proposed is $N$.

  The Critic evaluates each candidate action $a_t^i \in C_t$ and selects the best next action $a_t$. This evaluation is based on a value function V(a).

The Critic's selection is defined as: $$a_{t}=\arg \max _{a \in \mathrm{C}_{t}} V\left(a \mid q, P_{t}, o_{t}, s_{t}, M\right)$$ Here:

• $a_t$: The best action selected by the Critic to be executed.

• $a \in C_t$: Any candidate action from the set $C_t$.

• V(a): The utility estimate or value assigned to action $a$.

  The utility estimate V(a) is derived from an LLM-based assessment that considers several factors: goal alignment (how well the action aligns with the task goal), state viability (whether the resulting state is recoverable), action coherence (logical consistency of the action), plan consistency (how well it fits the current plan), and outcome risk (e.g., destructive or dead-end transitions). Crucially, unlike systems that attempt to learn an implicit world model via neural network fine-tuning, ATLAS leverages its Cognitive Map (part of $M$) to retrieve the predicted outcomes of each candidate action. This provides the agent with the ability to look ahead into the future consequences of its actions. The paper later extends this with simulated tree search in Section 3.4 for enhanced exploration.

4.2.2.3. Multi-layered Memory

ATLAS employs three complementary types of memory, designed to provide comprehensive contextual information to the agent:

• Working Memory: This is a task-specific memory that stores facts and observations relevant to the current episode. It is optionally stored within the LLM context for immediate use during a particular task execution. This acts as a short-term buffer for recent and crucial information.

• Cognitive Map: This memory layer is a graph of transitions $M = \{(o, a, o')\}$ that encodes structured knowledge about the environment's dynamics. Instead of storing raw HTML, it uses agentic summaries that capture deltas (differences between observations) and new affordances (e.g., "clicking Reports reveals {Sales, Products,...}"). This map supports retrieval for simulation and planning, specifically $\hat{o}_{t+1}=M(o_t, a)$, which predicts the next observation $\hat{o}_{t+1}$ given the current observation $o_t$ and action $a$. This is the core of ATLAS's world model.

• Semantic Memory (World Knowledge): This layer captures environment-specific constraints and learned dynamics (e.g., specific date formats, search rules, or non-recoverable states on a particular website). It is used to penalize risky actions and inform simulation by providing crucial context about how the environment behaves. For example, it might contain knowledge like "the date picker only accepts input in MM/DD/YYYY format".

  These memory layers are updated online as the agent interacts with the environment and are queried on demand by the Planner, Actor, and Critic.

4.2.3. MEMORY CONSTRUCTION VIA CURIOSITY-DRIVEN EXPLORATION

4.2.3.1. Motivation

Existing agents often fail because they lack awareness of potential action outcomes (e.g., difficulty canceling an order after placing it) or familiarity with environment-specific requirements (e.g., date formats). Humans can easily predict action outcomes using world knowledge. To bridge this gap, ATLAS encourages agents to explore the environment and store findings in memory to avoid undesirable outcomes.

4.2.3.2. Memory Construction Process

Inspired by artificial curiosity (Pathak et al., 2017), ATLAS augments its agent with a curiosity module to initialize memory. Before evaluation, a curiosity-driven exploration of the web environment is performed to seed the Cognitive Map and World Knowledge.

The process involves:

• Exploration policies: A series of lightweight explorer subagents are launched. These agents have diverse LLM generation temperatures and exploration policies. Coverage incentives are embedded into their prompts to encourage broad exploration (balancing breadth, depth, and entropy) within a fixed memory budget. Crucially, no task-completion reward is used during this phase to prevent information leakage from the test set.
• LLM based trajectory-mining: The collected exploration trajectories are processed by an LLM to convert them into agentic summaries of environmental transitions, which are then stored in the Cognitive Map. Additionally, the LLM produces agentic summaries of site-specific rules, constraints, and hazards for the Semantic Memory.

4.2.3.3. Memory layer 1: Cognitive Map

The Cognitive Map stores structured knowledge about the environment's dynamics, specifically state transitions and causal relationships. It functions as a learned world model, capturing how actions change observations. For example, "clicking Add to Cart on a product page" leads to a "cart update notification".

Formally, it's represented by tuples $(o_t, a_t, o_{t+1})$, where:

• $o_t$: Observation at step $t$ (e.g., HTML content, URL).

• $a_t$: Action executed at step $t$.

• $o_{t+1}$: Subsequent observation at step $t+1$.

  During exploration, for each step, $o_t$, $a_t$, and $o_{t+1}$ are documented. To enhance interpretability and reduce the cognitive load for the LLM agent, an agentic memory strategy is adopted: an LLM agent curates what is written into memory. It produces concise summaries emphasizing:

• Differences between successive observations $(o_t, o_{t+1})$.

• Newly available actions in $o_{t+1}$ after executing $a_t$.

  For retrieval, the Cognitive Map is queried with $(o_t, a_t)$, returning the predicted next raw observation $o_{t+1}$ and its LLM summaries. This design balances fidelity (retaining raw states) with abstraction (summarized transitions). If a query hits an unexplored node, a generic-placeholder observation is returned.

This process is depicted in Figure 1 (b). Figure 1 (b) Memory construction with curiosity-driven exploration: We build cognitive map by employing exploratory lightweight agents to interact with the environment.

4.2.3.4. Memory layer 2: Semantic Memory (World Knowledge)

This memory layer captures environment-specific knowledge that is crucial for robust web interaction. This includes:

• Constraints: E.g., "the date picker only accepts input in MM/DD/YYYY format".

• Formats: E.g., specific search query formats.

• Idiosyncratic behaviors: E.g., "the admin portal does not support exporting tables into CSV files".

  By recording these particulars from prior explorations, Semantic Memory bridges specific past experiences and working memory, which maintains immediate environmental awareness. This allows agents to adapt to recurring patterns and site-specific limitations. Both the Cognitive Map and Semantic Memory are optionally updated online if execution encounters unseen transitions or world dynamics.

4.2.4. Look-ahead Action Simulation (LAS)

The standard actor-critic interplay provides a good baseline but can suffer from insufficient exploration and lack of foresight. Look-ahead Action Simulation (LAS) addresses this.

At step $t$:

1. The Actor generates a set of candidate actions $C_t$ (as described in Section 4.2.2.2).
2. For each candidate action $a_t^i \in C_t$, the Critic hypothetically selects it for execution.
3. The predicted next observation $\hat{o}_{t+1}^i$ for each candidate action is retrieved from the Cognitive Map: $$\hat{o}_{t+1}^{i}=M\left(o_t, a_t^{i}\right)$$ Here:

   • $\hat{o}_{t+1}^i$: The predicted next observation if action $a_t^i$ is taken from current observation $o_t$.

   • $M$: The Cognitive Map function that predicts the next observation.

   • $o_t$: Current observation.

   • $a_t^i$: The $i$-th candidate action at time $t$.

     This process is repeated $D$ times, generating a set of rolled-out trajectories of length $D$. Each simulated trajectory $\hat{\tau}$ is assigned a value $V(\hat{\tau})$. This value is then confidence-weighted based on the transition uncertainty U(s, a): $$\hat{V}(\hat{\tau})=V(\hat{\tau}) \cdot \prod_{(s, a) \in \hat{\tau}}(1-U(s, a))$$ Here:

• $\hat{V}(\hat{\tau})$: The confidence-weighted value of the simulated trajectory $\hat{\tau}$.

• $V(\hat{\tau})$: The base value of the trajectory $\hat{\tau}$ (e.g., how well it leads to the goal).

• $\prod_{(s, a) \in \hat{\tau}}(1-U(s, a))$: A product term that penalizes trajectories passing through uncertain state-action transitions.

• U(s, a): The transition uncertainty for taking action $a$ in state $s$. A higher uncertainty (e.g., for unexplored transitions in the Cognitive Map) reduces the confidence in the trajectory's value.

  The trajectory with the highest $\hat{V}(\hat{\tau})$ determines the real action $a_t$ to be executed.

This process is shown in Figure 1 (c). Figure 1 (c) Look-ahead Action Simulation (LAS): At each step, ATLAS simulates all candidate actions with the observation from the cognitive map, providing ability to look-ahead. We employ the memory agent to learn from LAS trajectories to make a better plan and update memory if necessary.

Comparison to Prior Work (Advantages of LAS):

1. Trustworthiness: Unlike prior works that rely on LLMs to imagine outcomes (prone to hallucination), ATLAS's method leverages real observations stored in the Cognitive Map, making its predictions much more trustworthy.
2. Comprehensiveness: Previous methods often perform greedy search (one-step evaluation), pruning low-scoring branches immediately. ATLAS's LAS is akin to beam search over multiple steps, considering the joint outcome of a sequence of actions, which can identify paths that are not immediately optimal but lead to better long-term results.
3. Efficiency: The exploration conducted by LAS is a simulation in conceptual space using the internal Cognitive Map. This is significantly more efficient than actually executing actions in the real environment and avoids potentially irrecoverable stateful actions.

4.2.5. Look-ahead Action Simulation-Backed Dynamic Replanning and Memory Update

4.2.5.1. Replanning

ATLAS dynamically triggers replanning when the observed environment state diverges significantly from what was expected based on the Cognitive Map's predictions.

The condition for replanning is: $$\text{replan} = 1\left[\left\|o_t^{\text{obs}} - \hat{o}_t^{\text{exp}}\right\| > \varepsilon\right]$$ Here:

• $\text{replan}$: A binary indicator that is 1 if replanning is needed, and 0 otherwise.

• $o_t^{\text{obs}}$: The actual observation received at time $t$.

• $\hat{o}_t^{\text{exp}}$: The expected observation at time $t$, as predicted by the Cognitive Map based on the previous state and action.

• $\left\| \cdot \right\|$: A distance or dissimilarity metric between observations.

• $\varepsilon$: A threshold value; if the discrepancy exceeds this, replanning is triggered.

  When replanning is triggered, the Planner integrates a brief exploration digest—what worked/failed, newly exposed affordances, uncovered prerequisites—distilled by the memory writer. This information updates the current plan $P_t$. This mechanism can be seen as a simplified implementation of a causal learning module, updating the agent's internal causal model of the world. This approach also helps prevent catastrophic forgetting by ensuring that important context is retained during replanning.

4.2.5.2. Memory Update

In addition to replanning, the agent continuously updates its memory during action simulation and real execution. This applies to both the Cognitive Map and Semantic Memory. New patterns, constraints, or dynamics encountered are incorporated into the agent's long-term knowledge.

Crucially, decisions about what information to retain, update, or forget are delegated to the memory agent. This agent curates information based on task relevance and environmental novelty, which is particularly important during curiosity-driven exploration to refine the environment representation without memory overload from redundant or irrelevant details.

4.2.6. Agent Prompts (Appendix A)

The paper includes several LLM prompts in Appendix A, which define the behavior of the different modules:

• Planner Prompt (A.1): Guides the LLM to generate structured checklists of subgoals from a user instruction, initial URL, and initial observation, emphasizing high-level interactions and concise analysis.

• Replanning Prompt (A.2): (Content not fully provided in the excerpt, but implies guidance for LLM on how to adjust plans).

• Actor Prompt (A.3): Instructs the LLM to generate candidate actions (click, type, go_back, go_home, note, stop, branch, prune) given interaction history, current state, and action. branch and prune are specific planning actions.

• Critic Prompt (A.4): Guides the LLM to assess the value and risk of proposed web actions based on interaction history, current state, and action.

• Cognitive Map Prompt (A.5): (Content not fully provided in the excerpt, but relates to how the LLM processes data for the Cognitive Map).

• Episodic Memory Prompt (A.6): This is a detailed prompt for the LLM acting as an "expert in summarizing agent exploration." It guides the LLM to update environment dynamics by incorporating new evidence from exploration trajectories. It asks the LLM to identify Allowed Actions, Prohibited/Invalid Actions, Environment specific formats (date, search, URL), Newly Exposed Options, Environment Reliability (inconsistencies, errors), and Coverage & Unknowns. The prompt specifies a concise output format (less than 5 bullets per section, max 500 tokens) and explicitly states not to discard prior information unless contradicted.

  These prompts highlight the central role of LLMs in interpreting complex information, generating structured outputs, and adapting to dynamic situations within the ATLAS framework.

5. Experimental Setup

5.1. Datasets

The experiments for ATLAS were conducted on the WebArena-Lite benchmark.

• WebArena (Zhou et al., 2024): This is a realistic simulation environment designed for evaluating web agents. It comprises a broad array of web navigation tasks, including content retrieval, task execution, and form completion. Tasks vary in complexity and simulate real-world scenarios, such as purchasing items on e-commerce sites or updating code repositories on GitLab. The original WebArena dataset consists of 811 tasks. However, the paper notes that many of these tasks were unperformable, with humans only able to complete 78% of them.

• WebArena-Lite (Liu et al., 2024b): This is a quality-controlled and smaller subset of WebArena, consisting of 165 tasks. It was introduced to address quality and scalability concerns in the original benchmark. WebArena-Lite has been adopted by prior work in the web agent space (e.g., WebRL, Plan-and-Act) as a higher-quality and more scalable benchmark. It incorporates realistic scenarios such as unexpected environment failures, making it a robust testbed for web agents.

  • The WebArena-Lite benchmark categories include: Gitlab, Reddit, Shopping, Shopping Admin, Maps, and Multi-Site tasks. Each category tests different facets of web interaction and reasoning. For example, Shopping tasks involve navigating e-commerce sites to purchase items, while Gitlab tasks might involve code repository interactions. Multi-Site tasks require interactions across multiple distinct websites.

    The choice of WebArena-Lite is effective for validating the method's performance because it provides a diverse, realistic, and quality-controlled set of long-horizon tasks that challenge agents in partially observable and dynamic web environments. Its realistic nature, including potential environmental failures, makes it a suitable benchmark for assessing the adaptability and robustness of ATLAS.

5.2. Evaluation Metrics

The primary evaluation metric used in the paper is success rate.

1. Conceptual Definition: In the context of web agent task completion, success rate quantifies the percentage of tasks that an agent successfully completes according to the predefined task objectives. It measures the agent's ability to navigate the web environment, understand instructions, make correct decisions, and execute actions to reach a goal-consistent terminal state. A higher success rate indicates better performance and reliability.

2. Mathematical Formula: The success rate is typically calculated as: $ \text{Success Rate} = \frac{\text{Number of Successfully Completed Tasks}}{\text{Total Number of Tasks}} \times 100% $

3. Symbol Explanation:

   • Number of Successfully Completed Tasks: The count of individual tasks within the benchmark (e.g., WebArena-Lite) for which the agent achieved the specified goal.

   • Total Number of Tasks: The total number of tasks in the evaluated benchmark (e.g., 165 tasks in WebArena-Lite).

   • $100\%$: A scaling factor to express the result as a percentage.

     The paper reports both an Avg w/ Multi-site success rate and an Avg w/o Multi-site success rate, along with individual success rates for each task category (e.g., Gitlab, Reddit, Shopping).

5.3. Baselines

The paper compares ATLAS against several established and state-of-the-art web agent systems, all evaluated on the WebArena-Lite dataset:

• WebPilot + GPT-4o (Zhang et al., 2025): WebPilot is described as a versatile and autonomous multi-agent system for web task execution with strategic exploration. Its use of GPT-4o (a powerful LLM) suggests a focus on advanced multimodal understanding and reasoning capabilities.

• AWM + GPT-4-0613 (Wang et al., 2024a): AWM (Agent Workflow Memory) highlights the importance of persistent memory in multi-step web tasks. Its combination with GPT-4-0613 (another strong LLM) indicates an approach that emphasizes structured memory for long-horizon interactions.

• WebRL (Qi et al., 2024): This system applies reinforcement learning principles to web navigation, focusing on policy optimization through self-evolving online curriculum reinforcement learning. It represents a learning-based approach to agent control.

• Plan-and-Act (Erdogan et al., 2025): As discussed in the paper's introduction and related work, Plan-and-Act emphasizes hierarchical task decomposition. The authors state that it requires website-specific model fine-tuning for its planner and executor modules, which ATLAS aims to avoid.

• AgentOccam (Yang et al., 2024): AgentOccam is described as a simple yet strong baseline for LLM-based web agents, demonstrating the effectiveness of simplifying the action space to natural language. ATLAS explicitly states that its own work builds on top of AgentOccam, using it as the base agent for its ablation studies and re-running AgentOccam with Claude-4-Sonnet.

  These baselines are representative because they cover a range of current approaches to LLM-based web agents, including those focused on multimodal understanding, persistent memory, reinforcement learning, hierarchical planning, and simplified action spaces. Comparing against them allows ATLAS to demonstrate its advantages, particularly its adaptability without fine-tuning and its improved performance through look-ahead simulation and cognitive mapping.

6. Results & Analysis

6.1. Core Results Analysis

The main experimental results show that ATLAS significantly outperforms previous state-of-the-art methods on the WebArena-Lite Benchmark. Its strength lies in its ability to adapt to new environments without requiring website-specific LLM fine-tuning, a common limitation of prior systems.

The following are the results from Table 1 of the original paper:

Table 1: Evaluation Results for ATLAS versus other methods reported on WebArena-Lite. Best performance is in bold.

Key Observations:

• Overall Superiority: ATLAS achieves the highest average success rate, both with (63.0%) and without (67.1%) Multi-site tasks. This represents a substantial improvement over the previous state-of-the-art (Plan-and-Act at 53.9% with Multi-site).

• Strong Performance Across Categories: ATLAS demonstrates leading performance in Gitlab (73.3%), Shopping Admin (77.1%), and Reddit (84.2% - tied with Plan-and-Act). While it doesn't lead in Shopping (53.3% vs. Plan-and-Act's 55.6%) or Maps (42.3% vs. Plan-and-Act's 46.6%), its overall strength is clear.

• Multi-Site Tasks: ATLAS ties with AgentOccam for the best performance on Multi-site tasks (40.0%), suggesting its robust memory and planning capabilities generalize well across different web domains.

• Adaptability without Fine-tuning: A crucial advantage highlighted is that ATLAS achieves these results without requiring website-specific LLM fine-tuning, unlike some prior systems. This indicates its strong generalization capabilities and inference-time adaptability, which is a major breakthrough for practical web agents.

  These results strongly validate the effectiveness of the proposed ATLAS method. The significant jump in success rates, particularly the overall average, demonstrates that ATLAS's core innovations—the memory-augmented agent, look-ahead action simulation, and cognitive map—provide a more robust and efficient way for LLM agents to interact with complex and unfamiliar web environments. Its modular design enables better planning and decision-making by grounding actions in an explicit understanding of environment dynamics, rather than relying solely on LLM general knowledge or domain-specific fine-tuning.

6.2. Ablation Studies / Parameter Analysis

The paper includes an ablation study to verify the individual and complementary contributions of ATLAS's key components: the cognitive map, hierarchical planner, and look-ahead replanner. The study starts with AgentOccam as the base agent.

The following are the results from Table 2 of the original paper:

Table 2: Ablation Study Results for Individual Components of ATLAS.

Analysis of Ablation Results:

1. Base Agent (AgentOccam) Performance:

   • AgentOccam (Base) achieves 47.9% (Avg w/ Multi-site) and 46.7% (Avg w/o Multi-site). This serves as the starting point for comparing the improvements from ATLAS's components.

2. Impact of Cognitive Map:

   • Base + CM-Raw (Cognitive Map with Raw HTML): This configuration shows a reduction in overall performance (44.8% Avg w/ Multi-site, 47.1% Avg w/o Multi-site) compared to the base AgentOccam. This suggests that simply storing raw HTML as a cognitive map is not effective and can even degrade performance, likely due to the high cognitive load and noise from raw HTML content for the LLM. The Multi-site performance drops to 0%, which is a severe degradation.
   • Base + CM (Cognitive Map with Agentic Summarization): When the Cognitive Map is augmented with agentic summarization (processing raw HTML into concise, relevant summaries), performance dramatically improves to 57.4% (Avg w/ Multi-site) and 55.8% (Avg w/o Multi-site). This is a significant gain of nearly 10 percentage points over the base AgentOccam, and even surpasses Plan-and-Act's overall average. This highlights the crucial role of abstracted, interpretable memory in enabling effective use of the world model by LLM agents.

3. Impact of Hierarchical Planner:

   • Base + HL (Base Agent + Hierarchical Planner): Integrating a hierarchical planner (in the style of Chae et al., 2025) on top of the base agent improves performance to 50.9% (Avg w/ Multi-site) and 54.2% (Avg w/o Multi-site). While not as dramatic as the Cognitive Map improvement, it still shows a positive contribution from structured, multi-level planning. This confirms that breaking down tasks into subgoals helps navigate complex tasks.

4. Full ATLAS System (Base + CM + HL + LA):

   • Combining all components—the Cognitive Map with agentic summarization (CM), the Hierarchical Planner (HL), and the Look-ahead Action Simulation (LA) for replanning and conditioning the planner on the cognitive map—yields the full ATLAS agent.

   • This integrated system achieves the best performance: 63.0% (Avg w/ Multi-site) and 67.1% (Avg w/o Multi-site). This confirms that the components are not only individually beneficial but also have complementary roles. The look-ahead ability, powered by the cognitive map and guiding the hierarchical planner, allows for superior decision-making and adaptation. The sizable drops observed when components are removed validate their necessity within the ATLAS design.

     In summary, the ablation study rigorously demonstrates that the Cognitive Map (especially with agentic summarization), the Hierarchical Planner, and the Look-ahead Action Simulation are all essential and work synergistically to enable ATLAS's state-of-the-art performance. The failure of CM-Raw underlines the importance of intelligent memory curation over raw data storage for LLM-based agents.

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper introduced ATLAS, an innovative web navigation agent designed to enhance LLM-based agents' performance on long-horizon web tasks. ATLAS couples explicit, structured memory with hierarchical planning and look-ahead action simulation to convert open-ended browsing into a series of verifiable, low-entropy decisions. Its modular architecture, composed of a Planner, Actor, Critic, and Multi-layered Memory, allows it to maintain situational awareness across web pages, decompose complex goals into intermediate subgoals, and adapt its strategies as interface or task constraints evolve. A key strength of ATLAS is its ability to build an internal cognitive map of the environment through curiosity-driven exploration and agentic summarization, which then grounds its look-ahead simulation and dynamic replanning. This approach enables ATLAS to achieve state-of-the-art results on the WebArena-Lite Benchmark (63% success rate) without requiring website-specific LLM fine-tuning, making it highly adaptable and sample-efficient. The comprehensive ablation studies confirmed the crucial and complementary contributions of its world-model, hierarchical planner, and look-ahead-based replanner.

7.2. Limitations & Future Work

The authors acknowledge several limitations and propose a future research agenda focusing on principled generalization rather than just benchmark tuning:

• World Model Representation: The current world-model (Cognitive Map) is still in its early stages. Future work should develop more sophisticated web-native world models that can abstract repeated patterns (e.g., filters, tables, forms) into sub-programs and support counterfactual "what-if" reasoning, going beyond simple retrieval of observed transitions.

• Budget and Safety-Aware Planning: Next-generation planning should inherently be budget-aware (e.g., computational cost, time) and safety-aware. This involves trading off success, latency, and risk through calibrated uncertainty and constraint handling mechanisms.

• System Robustness Measurement: Current evaluations often assume robustness. Future research needs to measure robustness through rigorous stress tests. These tests should include scenarios like UI drift (changes in web interface), authentication flows, stochastic failures, and long-horizon, multi-session tasks.

• Advanced Evaluation Metrics: As agents approach human performance, evaluation must evolve beyond simple pass/fail rates. New metrics should incorporate cost of computation, side-effect penalties, reproducibility across seeds, and transparency of the intermediate state to provide a more holistic assessment.

  The authors envision agents that learn enduring abstractions of the web, plan under explicit budgets and constraints, and offer interpretable interfaces for verification and collaboration, viewing the separation of concerns (memory, planning, control) as a crucial scaffold for future reliable and adaptable web agents.

7.3. Personal Insights & Critique

ATLAS presents a compelling step forward for LLM-based web agents, particularly in its emphasis on an explicit, learned world model (the Cognitive Map) and look-ahead simulation. My personal insights draw several inspirations:

• The Power of Explicit Models: The paper's most significant contribution, in my view, is demonstrating that an explicit Cognitive Map with agentic summarization is vastly superior to raw HTML memory or relying solely on LLM "imagination." This highlights a fundamental principle: even for highly capable LLMs, providing structured, curated representations of the environment (a world model) leads to more trustworthy and efficient planning. This approach reduces the burden on the LLM to implicitly learn and hallucinate environment dynamics, allowing it to focus on higher-level reasoning.
• Modularity for Adaptability: The modular Actor-Critic architecture, combined with a distinct Memory module, is highly pragmatic. This separation of concerns is likely why ATLAS can adapt without fine-tuning. It suggests that robust AI systems, especially those interacting with dynamic external environments, benefit greatly from specialized, interoperable components rather than monolithic end-to-end models. This modularity also inherently enhances interpretability, as the rationale behind actions can be traced through the planner, actor, and critic evaluations.
• Curiosity as an Enabler: The use of curiosity-driven exploration for building the initial Cognitive Map is an elegant solution to the cold-start problem in new environments. It allows the agent to proactively learn the environment's affordances and dynamics without task-specific rewards, making it truly adaptable.

Potential Issues and Areas for Improvement:

• Scalability of Cognitive Map: While agentic summarization helps reduce cognitive load, the Cognitive Map can still grow large in highly complex web environments with many states and transitions. The paper mentions a "fixed memory budget" for explorers; scaling this to arbitrary web sites might still be a challenge. The LLM summarization process itself might introduce its own hallucinations or biases if not carefully controlled, requiring robust mechanisms for truthfulness and consistency.
• Definition of Uncertainty: The transition uncertainty U(s,a) is a critical component of the confidence weighting for simulated trajectories. The paper doesn't detail how this uncertainty is quantified. A precise and robust method for estimating U(s,a) (e.g., based on exploration frequency, consistency of past outcomes, or LLM confidence scores) would be crucial for the system's reliability.
• Generalization of Prompts: The system heavily relies on various LLM prompts for its modules. While powerful, prompt engineering can be sensitive, and the generalization of these prompts across vastly different types of websites (beyond WebArena-Lite) could be an area for further investigation. The LLM's underlying world knowledge and common sense still play a significant role, even with the Cognitive Map.
• The "What-If" Reasoning Gap: The authors themselves point out that their world model is "still in its infancy" and needs to support counterfactual "what-if" reasoning. While LAS provides a form of look-ahead, true counterfactuals (e.g., "What if I had clicked X instead of Y 3 steps ago?") are more complex and would require a more sophisticated causal model within the cognitive map.

Transferability to Other Domains:

The principles of ATLAS are highly transferable beyond web navigation. Any domain requiring an autonomous agent to operate in a partially observable, dynamic environment where planning and adaptation are crucial could benefit from this architecture:

• Software Robotics/Robotics: Navigating physical environments, interacting with novel objects, and learning the affordances of tools could greatly benefit from a cognitive map-like structure.

• UI Automation for Desktop Applications: Similar to web agents, automating tasks on desktop applications with diverse UI elements and workflows could leverage look-ahead planning and memory to adapt to new software versions or layouts.

• Code Generation and Debugging: An agent that understands the causal relationships between code changes and program behavior (a cognitive map of a codebase) could use look-ahead simulation to propose and evaluate code modifications or debugging steps.

  Ultimately, ATLAS reinforces the idea that true intelligence in AI agents involves not just powerful LLMs, but also structured memory, proactive planning, and an explicit understanding of the world they operate in.