1. Bibliographic Information

1.1. Title

The central topic of the paper is the development and evaluation of RecUserSim, a realistic and diverse user simulator designed for evaluating Conversational Recommender Systems (CRS).

1.2. Authors

The authors and their affiliations are:

• Luyu Chen, Zeyu Zhang, Xueyang Feng, Xu Chen: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
• Quanyu Dai, Zhenhua Dong: Huawei Noah's Ark Lab, Shenzhen, China.
• Mingyu Zhang, Pengcheng Tang, Yue Zhu: Huawei Technologies Ltd., Shenzhen, China.

1.3. Journal/Conference

The paper is published in the Companion Proceedings of the ACM Web Conference 2025 (WWW Companion '25). The ACM Web Conference (WWW) is a highly reputable and influential conference in the field of web technologies, including web search, data mining, and recommendation systems. Its companion proceedings typically feature shorter papers, posters, and demos, but are still part of a prestigious academic venue.

1.4. Publication Year

The paper was published on 2025-06-25.

1.5. Abstract

The paper addresses the challenge of evaluating Conversational Recommender Systems (CRS) by proposing RecUserSim, an LLM agent-based user simulator. While existing Large Language Model (LLM)-based simulators fall short in emulating individual real users across diverse scenarios and lack explicit rating mechanisms, RecUserSim aims to enhance simulation realism and diversity while providing quantitative scores. It features a profile module for diverse user personas, a memory module for tracking history and discovering unknown preferences, and a core action module inspired by Bounded Rationality theory for nuanced decision-making and fine-grained actions. A refinement module further fine-tunes responses for output control. Experiments show that RecUserSim generates diverse, controllable, and high-quality dialogues, even with smaller base LLMs, and its generated ratings demonstrate high consistency across different LLMs, confirming its effectiveness for CRS evaluation.

1.6. Original Source Link

The official source link for this paper is https://arxiv.org/abs/2507.22897v1. It is currently a preprint on arXiv. The PDF link is https://arxiv.org/pdf/2507.22897v1.pdf.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the challenging evaluation of Conversational Recommender Systems (CRS). CRS interact with users through natural language over multiple turns to provide personalized recommendations. This dynamic and interactive nature makes their evaluation significantly more complex than traditional recommender systems.

This problem is important because CRS have vast potential applications in areas like e-commerce, streaming, and travel, offering a user-centered experience by actively engaging users and gathering real-time feedback. However, current evaluation methods fall short:

• Traditional metric-based evaluations (e.g., accuracy, BLEU) focus on isolated turns and fixed benchmarks, failing to capture the dynamic interaction process.
• Online user testing, while ideal, is prohibitively expensive and time-consuming for large-scale application.
• Traditional rule-based user simulators lack the flexibility and adaptability to realistically mimic user behavior in dynamic conversations.
• Recent Large Language Model (LLM)-based user simulators, while promising, still have limitations:

  • They often fail to perform fine-grained simulation of individual user behavior (language, actions, decision-making).

  • They struggle to capture the diversity of a real user population, often generating uniform language styles and limited action types due to reliance on fixed benchmarks and weak role-playing capabilities.

  • They lack explicit rating mechanisms for quantitative evaluation of CRS performance.

    The paper's entry point or innovative idea is to leverage the advanced capabilities of LLMs within an agent-based architecture to create a user simulator, RecUserSim, that specifically addresses the aforementioned gaps by focusing on:

1. Enhanced simulation realism: Emulating individual user behavior with fine-grained control over language, actions, and decision-making.
2. Diverse user population representation: Generating varied language styles and action types, moving beyond uniform outputs.
3. Explicit rating mechanisms: Providing quantitative scores for comprehensive CRS evaluation.

2.2. Main Contributions / Findings

The paper makes several primary contributions to address the challenges in CRS evaluation:

• Proposed RecUserSim, a novel LLM agent-based user simulator: This simulator is designed to enable both realistic individual role-playing and diverse user population representation, allowing for accurate and comprehensive evaluation of Conversational Recommender Systems.

• Introduction of key mechanisms for enhanced role-playing:

  • Three-tier 'Rating-Action-Response' mechanism: Inspired by Bounded Rationality theory, this mechanism models user decision-making processes more realistically, generating multi-dimensional ratings, fine-grained actions, and personalized responses.
  • Tool-augmented refinement method: This module provides fine-grained control over output language, ensuring responses adhere to specific user linguistic patterns (e.g., information richness, formality, sentence length), which enhances persona consistency.
  • Profile and Memory Modules: A profile module constructs fine-grained and diverse user personas with conflict resolution, and a memory module tracks history and dynamically uncovers unknown preferences through an excitation mechanism.

• Validation of RecUserSim's superiority: Through extensive comparative analyses, the paper demonstrates that RecUserSim outperforms existing simulators in generating more realistic, diverse, and high-quality dialogues. This superiority is maintained even when using smaller base LLMs, highlighting its robustness.

• Demonstration of rating mechanism reliability: The ratings generated by RecUserSim show high consistency across different LLM backbones, confirming its effectiveness and reliability in quantitatively assessing CRS performance.

• Successful industrial deployment: The paper highlights RecUserSim's practical applicability by deploying it in the development and evaluation of Huawei's Celia Food Assistant, where its results align well with human evaluations.

  These findings collectively solve the problem of comprehensive and realistic CRS evaluation by providing a robust, controllable, and scalable user simulator that can mimic diverse user behaviors and provide actionable quantitative feedback.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand RecUserSim, a reader should be familiar with the following fundamental concepts:

• Recommender Systems (RS): At its core, a recommender system is a type of information filtering system that aims to predict the "rating" or "preference" a user would give to an item. They are widely used in e-commerce, content platforms, and more, to suggest items (e.g., products, movies, articles) that users might like. Traditional RS usually operate based on historical data (past purchases, ratings, browsing history) and static user profiles.

• Conversational Recommender Systems (CRS): CRS are an evolution of traditional RS. Instead of static interactions, CRS engage users in multi-turn natural language dialogues to understand their preferences, provide recommendations, and refine suggestions based on real-time feedback. This interactive nature allows them to dynamically adapt to evolving user needs, making the recommendation process more personalized and user-centric. For example, a CRS might ask "Do you prefer spicy food?" or "Are you looking for a restaurant with outdoor seating?" to narrow down options.

• Large Language Models (LLMs): LLMs are advanced artificial intelligence models trained on vast amounts of text data to understand, generate, and process human language. They are capable of various natural language processing (NLP) tasks, including text generation, summarization, translation, question answering, and role-playing. Key to this paper, LLMs excel at generating coherent and contextually relevant dialogue, which makes them suitable for simulating user interactions. Examples include GPT-3.5, GPT-4, and GLM models.

• LLM Agents: An LLM agent is an LLM augmented with additional components (like memory, tools, and planning modules) that enable it to perform more complex tasks beyond simple text generation. Agents can maintain state, learn from interactions, reason, and make decisions, mimicking intelligent behavior. In the context of user simulation, an LLM agent can represent a virtual user with a persona, memory of past interactions, and the ability to choose actions and generate responses based on that persona and history.

• Bounded Rationality Theory: This theory, proposed by Herbert A. Simon, suggests that human decision-making is not perfectly rational but is limited by the available information, cognitive capabilities, and time. Instead of optimizing for the absolute best outcome, individuals tend to satisfice—i.e., choose an option that is "good enough" rather than exhaustively searching for the optimal one. In the context of RecUserSim, this theory inspires the Three-Tier Action Mechanism by modeling user behavior as a sequence of information processing, option evaluation, and decision-making, acknowledging that users don't always act perfectly logically but within their cognitive limits and available information.

3.2. Previous Works

The paper contextualizes RecUserSim by discussing prior approaches to CRS evaluation and user simulation:

• Metric-based Evaluation Methods: These are traditional approaches that evaluate CRS based on isolated metrics:

  • Recommendation Accuracy Metrics: Hit rate, precision, recall, F1 score, and NDCG (Normalized Discounted Cumulative Gain) [16] are commonly used. These measure how well the system recommends relevant items. For example, Precision measures the fraction of recommended items that are actually relevant.
  • Dialogue Quality Metrics: BLEU (Bilingual Evaluation Understudy) [27] and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [23] are used to compare generated text (e.g., system responses) against human-written reference texts. They quantify the similarity in n-grams or word sequences.
  • Limitations: These methods rely on predefined "ground-truth" conversations from fixed benchmark datasets. This is insufficient for CRS because real user feedback is dynamic and often deviates from these static references, failing to capture the true performance in interactive settings.

• Simulator-based Evaluation Methods: To address the dynamic nature of CRS, simulators aim to mimic complete user-system dialogues.

  • Human Evaluation: Considered the "gold standard" as it captures real user interactions. However, it is costly, time-consuming, and not scalable for large-scale evaluation [12, 13, 35].
  • Traditional User Simulators: These are rule-based or template-designed systems [1, 8, 19, 31, 37, 45]. They define a set of rules or templates to generate user responses and actions.
    • Limitations: They lack flexibility and adaptability. Their pre-defined rules cannot realistically capture the nuanced and diverse behaviors of real users in complex, dynamic conversations.

• LLM-based User Simulators: Leveraging the powerful capabilities of LLMs in dialogue understanding and generation, these are a more recent development for CRS evaluation.

  • Single-prompt LLM Simulators: Models like iEvalLM [33], MACRS [7], and PEARL [17] use a single, overarching prompt to guide the LLM in simulating user interactions.
    • Limitations: They suffer from limited dialogue diversity, repetitive conversation flows, and insufficient control over generated outputs. The single prompt struggles to enforce fine-grained persona adherence and varied interaction patterns.
  • Agent-based LLM Simulators: CSHI [43, 44] represents an advancement by introducing LLM-based agents with basic user profiles and a limited action space.
    • Limitations: While an improvement, CSHI still struggles to capture truly diverse user behaviors and lacks precise control over the generated outputs. This makes it difficult to produce realistic, persona-consistent interactions and limits its effectiveness in simulating a broad range of users.

3.3. Technological Evolution

The evaluation of recommender systems has evolved significantly:

1. Early Recommender Systems (Pre-2010s): Focused primarily on accuracy metrics (e.g., RMSE, MAE, Precision, Recall) using offline datasets of explicit ratings (e.g., MovieLens). Evaluation was mostly static.
2. Introduction of Interaction and Dialogue (Mid-2010s): With the rise of dialogue systems, the need for interactive evaluation became apparent. This led to the development of Conversational Recommender Systems (CRS). Initial evaluations involved human studies or simple rule-based simulators, which were limited.
3. Rise of Deep Learning in CRS (Late 2010s): Deep learning models improved the performance of CRS. Evaluation still struggled with the dynamic nature, relying on turn-level metrics (like BLEU, ROUGE) against fixed ground truth.
4. Emergence of Large Language Models (Early 2020s): LLMs revolutionized NLP, offering unprecedented capabilities in dialogue and role-playing. This led to LLM-based user simulators, first with single-prompt methods, then evolving to more structured agent-based architectures.
5. Current State (This paper's contribution): RecUserSim represents the cutting edge by addressing the limitations of prior LLM-based simulators. It aims for deeper realism (fine-grained persona, Bounded Rationality), greater diversity (expanded action space, profile module), and robust quantitative evaluation (explicit rating mechanism, refinement module), pushing the field towards more comprehensive and reliable CRS evaluation.

3.4. Differentiation Analysis

RecUserSim distinguishes itself from prior LLM-based user simulators (specifically iEvaLM and CSHI) through several core innovations:

• Enhanced Realism with 'Rating-Action-Response' Mechanism:

  • Previous: iEvaLM and other single-prompt methods lack explicit modeling of user decision-making, leading to less realistic interactions. CSHI introduced a basic action selection but with rigid, predefined actions.
  • RecUserSim: Inspired by Bounded Rationality theory, it employs a three-tier Rating-Action-Response mechanism. This involves Multi-Dimensional Rating (Language, Action, Recommendation quality with justifications), Fine-Grained Action Selection (expanded action space with five actions, allowing multi-action selection, and personalized action patterns), and Personalized Response Generation (integrating persona, history, ratings, and linguistic patterns). This comprehensive approach better mimics real human cognitive processes.

• Diverse User Population Representation through Richer Profile and Action Space:

  • Previous: iEvaLM and similar models often generate uniform language styles dictated by the base LLM. CSHI has a basic profile but a limited action space (e.g., only three predefined actions), resulting in rigid and less diverse user behaviors.
  • RecUserSim: Features a fine-grained profile module that constructs diverse user personas across basic information, environment, preferences, and behavior traits. It includes a conflict resolution mechanism for realistic profiles. Its expanded action space with five distinct actions (and the ability to combine them) significantly enhances user simulation diversity and flexibility.

• Fine-Grained Output Control with Tool-Augmented Refinement:

  • Previous: LLMs often struggle to balance multiple output constraints simultaneously [24], leading to generated responses that may not fully align with a specific persona's linguistic patterns.
  • RecUserSim: Introduces a dedicated refinement module with specialized refinement tools for information richness, formality, and sentence length. These tools apply constraint-specific adjustments sequentially, ensuring that the final output closely matches the predefined user persona's linguistic patterns, providing a level of control absent in previous simulators.

• Explicit Rating Mechanism for Quantitative Evaluation:

  • Previous: Most LLM-based user simulators primarily focus on dialogue generation and lack an explicit, quantitative rating mechanism for evaluating CRS performance across multiple dimensions.
  • RecUserSim: The Multi-Dimensional Rating submodule provides explicit numerical scores for Language quality, Action quality, and Recommendation quality, offering a more precise and actionable assessment of CRS performance, which is crucial for development and comparison.

• Dynamic Discovery of Unknown Preferences:

  • Previous: Some approaches attempt to simulate unknown preferences by simply hiding a subset of known preferences and revealing them later. This assumes all "unknown" preferences are predefined.

  • RecUserSim: Implements an LLM-driven unknown preference excitation mechanism within the memory module that dynamically uncovers truly latent interests. If a recommended item aligns with a user's tastes but wasn't explicitly stated, the LLM recognizes it as a new preference, enhancing realism.

    In summary, RecUserSim takes a more holistic and granular approach to user simulation by integrating advanced LLM agent capabilities with behavioral economic theories and fine-grained control mechanisms, leading to a simulator that is not only more realistic and diverse but also provides a robust quantitative evaluation framework.

4. Methodology

4.1. Overview and Principles

The core idea behind RecUserSim is to create an LLM-based autonomous agent that can realistically and diversely simulate user behavior in interactions with Conversational Recommender Systems (CRS). The simulator is built upon the principle that accurate user simulation requires:

1. A strong foundation of fine-grained and diverse user personas.

2. Consistent behavioral modeling through memory of past interactions and evolving preferences.

3. Realistic decision-making processes that mimic human cognition.

4. Precise control over output language to adhere to specific linguistic patterns of the persona.

   Inspired by the Bounded Rationality Model [30] from economics, which conceptualizes decision-making as a three-step process (receiving information, evaluating options, making decisions), RecUserSim formalizes this into a Tree-Tier Action Mechanism (Rating-Action-Response).

The RecUserSim framework comprises four main modules, as illustrated in Figure 1, which work collaboratively to achieve these objectives:

• Profile Module: Establishes realistic and diverse user personas.

• Memory Module: Tracks interaction history and discovers unknown preferences.

• Action Module: The core decision-making unit, generating ratings, actions, and responses.

• Refinement Module: Fine-tunes generated responses for persona adherence.

  The following figure (Figure 1 from the original paper) shows an overview of the RecUserSim architecture:

  Figure 1: An overview of RecUserSim. It is an LLM agent-based user simulator which comprises four modules: profile, memory, action, and refinement. The profile module creates diverse user personas, which are stored and tracked by the memory module. The action module, inspired by Bounded Rationality, generates multi-dimensional ratings, fine-grained actions, and personalized responses. Finally, the refinement module fine-tunes these responses to align with user's linguistic patterns.

4.2. Profile and Memory Management

4.2.1. Profile Module

The profile module is foundational for ensuring simulation realism and diversity by constructing detailed user personas.

• Diverse Profile Construction:

  • A user profile in RecUserSim is comprehensive, covering four key aspects:
    1. Basic information: e.g., age, gender, occupation.
    2. Environment information: e.g., current location, time constraints.
    3. Preferences: e.g., dietary restrictions, preferred cuisine types, price range.
    4. Behavior traits: These define the user's communication style and decision-making tendencies, such as linguistic patterns (e.g., verbose, concise, formal, informal) and action patterns (e.g., decisive, indecisive, exploratory).
  • To ensure diversity and avoid manual bias, user profiles are constructed by randomly sampling from predefined dictionaries for each attribute, following prior probability distributions. This allows for a wide range of user types.

• Profile Conflict Resolution:

  • Random sampling can inadvertently create contradictory user profiles (e.g., a user who dislikes spicy food but prefers Sichuan cuisine, known for its spiciness).
  • To prevent such illogical combinations and ensure realism, an LLM is employed to assess the sampled profile attributes for consistency. If conflicts are detected, the LLM adjusts the conflicting attributes to ensure a coherent and realistic user representation.

4.2.2. Memory Module

The memory module acts as a bridge between the profile module and the action module. It is responsible for maintaining the user's current state and historical context, crucial for consistent and adaptive behavior simulation.

• Storage and Tracking: It stores the detailed user profile generated by the profile module. It also continuously tracks the entire interaction history with the CRS, including past recommendations, user feedback, and actions taken. This ensures that the simulator's behavior remains consistent with previous turns in the conversation.

• Unknown Preference Excitation:

  • Real users often discover new interests or realize latent preferences during interactions. To mimic this, RecUserSim introduces an LLM-driven unknown preference excitation mechanism.
  • Limitation of previous methods: Existing approaches might predefine a set of "hidden" preferences and reveal them only when a matching item is recommended. This doesn't account for genuinely latent interests that users themselves might not be consciously aware of.
  • RecUserSim's approach: If the CRS recommends an item that is not explicitly in the user's known preferences but the LLM determines it to be highly agreeable (e.g., it aligns with their broader taste profile or other implicit cues), the LLM recognizes this as a new preference. This preference is then dynamically added to the user's memory. This adaptive update mechanism allows the simulator to simulate evolving user interests, enhancing the realism of long-term interactions.

4.3. Three-Tier Action Mechanism

This mechanism is the core of RecUserSim, designed to accurately model individual user behavior by formalizing the decision-making process. It is inspired by the Bounded Rationality Model [30], which breaks down decision-making into three steps:

1. Receiving and processing information: The user receives a CRS response.

2. Evaluating options: The user evaluates the recommendation or information provided.

3. Making decisions: The user decides on their next action (e.g., accept, reject, clarify).

   RecUserSim adapts this into a Tree-Tier Action Mechanism: Rating-Action-Response, which unfolds sequentially upon receiving a CRS response.

4.3.1. Multi-Dimensional Rating

This submodule simulates the user's evaluation process, providing quantitative scores that serve as crucial feedback for CRS performance and guide the simulator's subsequent actions and response generation.

• Generative Verifier Approach: Inspired by Generative Verifier [36], RecUserSim enhances rating reliability by first prompting the LLM to generate explicit justifications for its rating before assigning a score. This process makes the LLM's reasoning transparent and the resulting scores more robust.
• Rating Dimensions: Ratings are structured across three key dimensions, each scored from 1 to 5:
  • Language quality:
    • Definition: Reflects how natural, fluent, and clear the CRS's generated dialogue is.
    • Purpose: Assesses the CRS's ability to communicate effectively and user-friendly.
  • Action quality:
    • Definition: Evaluates whether the CRS selected the correct action and accurately understood the user's request.
    • Example: If a user asks for a spicy restaurant, the CRS should take a recommendation action and provide a list of spicy restaurants. A score of 5 indicates perfect alignment in both action selection and intent comprehension.
    • Purpose: Measures the CRS's ability to interpret user intent and respond appropriately.
  • Recommendation quality:
    • Definition: This applies only when the CRS performs a recommendation action. The final score is a sum of an objective score (1-5) and subjective modifiers (each ranging from -1 to +1).
      • Objective score: Measures how well the recommendation objectively matches the user's known and discovered preferences.
      • Subjective modifiers: Adjust the score based on the user's specific behavior traits. For instance, a user who enjoys exploring new cuisines might give a higher score to a novel but moderately aligned recommendation, whereas a cautious user might score it lower.
    • Example: A restaurant that moderately aligns with preferences (base score 3) might receive a +1 modifier if the user has an exploratory behavior trait and the restaurant is novel, resulting in a final score of 4.
    • Purpose: Assesses the relevance and personalization of the CRS's item suggestions, accounting for individual user nuances.

4.3.2. Fine-Grained Action Selection

This module determines the simulator's next actions based on the generated multi-dimensional ratings and the user's behavior traits, providing more flexible and diverse behaviors than previous simulators.

• Expanded Action Space: Unlike CSHI [44], which uses a limited set of three actions, RecUserSim significantly expands the action space and allows for simultaneous selection of multiple actions. This better reflects real user behavior (e.g., providing negative feedback while also clarifying preferences).
• The five user action types are:
  1. Request for recommendations: User actively seeks item suggestions (e.g., "What should I eat tonight?"). Typically occurs at the start or after previous recommendations are exhausted.
  2. Preference clarification: User provides more details about their preferences or refines existing ones (e.g., "I prefer Italian food, and it should be vegetarian"). Often used after an unsuitable recommendation.
  3. Feedback on recommendation: User explicitly expresses satisfaction or dissatisfaction with a recommended item (e.g., "I like that!", "No, I don't want a Chinese restaurant"). This guides the CRS in refining future suggestions.
  4. Item attribute inquiry: User asks for more specific details about a recommended item (e.g., "What's the price range?", "Where is it located?").
  5. End conversation: User decides to terminate the interaction, either because they are satisfied with a recommendation or frustrated by repeated unsuitable results.
• Personalized Action Patterns: The simulator incorporates user-specific action patterns from the profile module. This means that even under similar preference and recommendation scenarios, different user personas can choose different actions. For example, a casual user might end a conversation immediately after a satisfactory recommendation, while an indecisive user might ask for more details or alternative options.
• Extensibility: The design is highly adaptable, allowing new actions to be easily added or existing ones removed based on specific research goals (e.g., adding a "chit-chat" action).

4.3.3. Personalized Response Generation

This submodule is responsible for transforming the selected actions and internal states into natural language responses that reflect the individual user's persona and attitudes.

• Integration of Multiple Inputs: To generate personalized responses, the LLM takes into account:
  • The user profile (especially linguistic patterns and behavior traits).
  • The dialogue history.
  • The satisfaction ratings from the Multi-Dimensional Rating module.
  • The selected actions from the Fine-Grained Action Selection module.
• Satisfaction Score Conversion: Since LLMs often struggle to directly interpret numerical satisfaction scores, these scores are first converted into descriptive text (e.g., "very satisfied," "mildly dissatisfied"). This allows the LLM to better interpret the user's emotional state and attitude towards the CRS, leading to more contextually appropriate responses.
• Linguistic Pattern Embedding: Users' specific linguistic patterns (e.g., formality, verbosity, conciseness) from their profile are embedded directly into the LLM prompts. This ensures that the generated responses align with the individual user's speaking style.
• Challenge and Solution: Acknowledging that LLMs might struggle to satisfy all multiple constraints (e.g., concise, informal, information-rich) simultaneously, RecUserSim introduces a refinement module as a subsequent step to fine-tune the generated responses.

4.4. Tool-augmented Refinement

The tool-augmented refinement module addresses the challenge that Large Language Models often struggle to simultaneously satisfy multiple, sometimes conflicting, output constraints (e.g., be concise and formal and information-rich) [24]. This module applies constraint-specific adjustments sequentially to ensure the final output strictly adheres to the predefined user persona's linguistic patterns.

• Modular Design: The refinement module consists of several specialized refinement tools, each designed to fine-tune outputs based on a specific linguistic pattern. These patterns include information richness, formality, and sentence length.

• Tool Structure (Judger + Refiner): Each refinement tool includes two main components:

  1. Judger: Assesses whether the current LLM-generated response aligns with the target linguistic pattern specified in the user's persona.
  2. Refiner: If the judger identifies a misalignment, the refiner modifies the output to bring it into compliance.

• Rule-based vs. LLM-based Components:

  • For straightforward patterns like sentence length, the judger can be rule-based (e.g., simply counting words). The refiner, however, typically uses an LLM to intelligently adjust the text without losing meaning.
  • For complex patterns like formality, both the judger and the refiner rely on LLMs due to the nuanced nature of such adjustments.

• Conditional Activation: In certain scenarios, strict adherence to linguistic patterns might not be necessary (e.g., when a user ends a conversation). The refinement tools can be deactivated for such specific actions to allow for more natural termination flows.

• Extensibility: The module is highly adaptable. New refinement tools can be easily added by providing in-context examples to the LLM that demonstrate how to judge and refine responses based on new user persona attributes.

  The three designed refinement tools are:

1. Information richness:

   • Mechanism: The tool leverages an LLM to identify key points (e.g., time, location, specific preferences) within a generated sentence and counts their occurrences.
   • Adjustment: If the number of key points deviates from a predefined threshold for the user's persona (e.g., a "low-information" user should have no more than 2 key points), the refiner (an LLM) adjusts the information density of the response accordingly (e.g., by adding or removing details).

2. Formality:

   • Mechanism: This tool determines and adjusts the formality level of the response.
   • Judger: An LLM-based judger classifies the formality level of the generated text (e.g., informal, neutral, formal) based on examples provided in its prompt.
   • Refiner: If the classified formality does not match the user's formality trait in their profile, an LLM-based refiner modifies the response to align with the specified degree of formality (e.g., changing "wanna" to "want to," or "hello" to "greetings").

3. Sentence length:

   • Mechanism: This tool controls the overall length of the response.
   • Judger: A rule-based judger compares the word count of the generated response against a predefined length threshold for the user's persona (e.g., "no more than 20 words" for a concise user).
   • Refiner: If the length deviates, an LLM-powered refiner adjusts the response length (e.g., by summarizing or elaborating) while simultaneously attempting to preserve formality and information richness.

5. Experimental Setup

5.1. Datasets

The experiments are conducted in an unconstrained food recommendation scenario. This means that the recommended restaurants are not limited by real-world availability or specific geographical constraints for the evaluation of the simulators themselves. While the paper mentions that RecUserSim was deployed for Huawei's Celia Food Assistant, which does account for geographical locations and restaurant availability, the core simulator evaluation is done in this unconstrained setting.

The paper does not explicitly name a publicly available dataset of food recommendations used to train or evaluate the baselines in the main simulation quality comparison. Instead, the LLMs generate the content of the recommendations and conversations based on the prompts and the context of the food domain. User profiles are constructed by random sampling from predefined dictionaries, rather than derived from a specific dataset.

Example of data context: A user profile might include preferences for "spicy food," "vegetarian options," and a "casual dining environment." The CRS would then recommend a restaurant matching these criteria, and the simulator would generate responses like "I like spicy food, but I'm not feeling Indian tonight. Do you have any Thai options?"

5.2. Evaluation Metrics

The evaluation of RecUserSim is multi-faceted, employing both subjective and objective metrics for simulation quality, and quantitative metrics for rating reliability.

5.2.1. Subjective Evaluation Metrics (for User Simulator Performance)

These metrics assess the human-like quality of the simulated dialogues and are evaluated via pairwise comparison by a gpt-4o-based judge [41].

• Conceptual Definition: These metrics evaluate how natural, coherent, and realistic the user simulator's outputs are from a human perspective, covering different granularities of interaction.
• Metrics:
  • Single-turn output quality:
    • Naturalness: How fluid, idiomatic, and human-like a single response from the simulator sounds.
    • Clarity: How easily understandable and unambiguous the information conveyed in a single response is.
  • Single-round interaction quality:
    • Adaptability: The simulator's ability to adjust its response and behavior appropriately based on the CRS's action and previous turn.
    • Relevance: How pertinent the simulator's feedback or request is to the ongoing conversation and the CRS's output.
  • Overall dialogue quality:
    • Role-play ability: How well the simulator consistently maintains its assigned user role throughout the entire dialogue, without contradicting its persona or trying to act as a CRS.
    • Realism: The overall impression of how closely the entire conversation resembles a genuine human-to-system interaction.

5.2.2. Objective Evaluation Metrics (for User Simulator Diversity and Controllability)

These metrics objectively quantify the variability and persona adherence of the simulator's outputs.

• Conceptual Definition: These metrics measure how varied the generated responses are across different linguistic dimensions, and how effectively the simulator can control these dimensions to match a specific user persona.
• Metrics:
  • Sentence length:
    • Conceptual Definition: The number of words in a generated response. It quantifies the verbosity or conciseness of the simulator's communication.
    • Mathematical Formula: $ L = \sum_{i=1}^{N} \mathbb{I}(w_i \in \text{words}) $
    • Symbol Explanation:
      • $L$: Total word count (sentence length).
      • $N$: Total number of tokens in the response.
      • $w_i$: The $i$-th token in the response.
      • $\mathbb{I}(\cdot)$: Indicator function, which is 1 if the condition is true, and 0 otherwise. Here, it counts tokens that are identified as words (ignoring punctuation, etc.).
  • Information richness:
    • Conceptual Definition: The density of key factual or preference-related points conveyed in a response. It quantifies how much meaningful information a user provides.
    • Measurement: A gpt-4o-based evaluator identifies and counts key points (e.g., time, location, preferences) in each sentence.
  • Formality:
    • Conceptual Definition: The level of politeness and adherence to standard linguistic conventions in a response. It quantifies the tone of communication.
    • Measurement: A gpt-4o-based evaluator assesses formality based on linguistic features and categorizes responses (e.g., informal, neutral, formal).

5.2.3. Quantitative Rating Metrics (for Rating Reliability)

These are the three dimensions defined in the Multi-Dimensional Rating submodule (Section 4.3.1), used to evaluate CRS performance quantitatively. Each is rated from 1 to 5.

• Action quality: Assesses correctness of CRS action selection and intent comprehension.
• Language quality: Reflects naturalness, fluency, and clarity of CRS dialogue.
• Recommendation quality: Assesses relevance and personalization of CRS item suggestions.

5.2.4. Consistency Metrics (for Rating Reliability)

These statistical measures quantify the agreement between RecUserSim's ratings when using different LLM backbones.

• Conceptual Definition: These metrics determine how consistently two different sets of ratings (e.g., ratings from RecUserSim with gpt-4o vs. gpt-4o-mini as its backbone) rank the same items or models.
• Metrics:
  • Pearson correlation coefficient ($r$):
    • Conceptual Definition: Measures the linear relationship between two sets of data. A value of +1 indicates a perfect positive linear correlation, -1 a perfect negative linear correlation, and 0 no linear correlation. It is sensitive to the magnitude of values.
    • Mathematical Formula: $ r_{xy} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2 \sum_{i=1}^{n} (y_i - \bar{y})^2}} $
    • Symbol Explanation:
      • $r_{xy}$: Pearson correlation coefficient between variables $x$ and $y$.
      • $n$: Number of data points.
      • $x_i$: Value of the $i$-th data point for variable $x$.
      • $y_i$: Value of the $i$-th data point for variable $y$.
      • $\bar{x}$: Mean of variable $x$.
      • $\bar{y}$: Mean of variable $y$.
  • Spearman correlation coefficient ($\rho$):
    • Conceptual Definition: Measures the monotonic relationship between two sets of data. It assesses how well the relationship between two variables can be described using a monotonic function. It is based on the ranks of the data points, making it less sensitive to outliers and non-linear relationships than Pearson. A value of +1 indicates a perfect positive monotonic correlation, -1 a perfect negative monotonic correlation, and 0 no monotonic correlation.
    • Mathematical Formula: $ \rho = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)} $
    • Symbol Explanation:
      • $\rho$: Spearman correlation coefficient.
      • $d_i$: Difference between the ranks of corresponding values for each pair of data points $(x_i, y_i)$.
      • $n$: Number of data points.

5.3. Baselines

The paper uses different baselines for evaluating simulation quality versus rating reliability.

5.3.1. Baselines for Simulation Quality (Section 4.1)

RecUserSim is compared against two main categories of LLM-based user simulators in an unconstrained food recommendation scenario:

• iEvaLM [33]: A single-prompt simulator that interacts with the CRS using a target item as input. It represents the simpler, less structured LLM-based approach.

• CSHI [44]: An agent-based LLM user simulator that incorporates profile, memory, and action modules. It represents a more advanced agent-based approach, but with limitations RecUserSim aims to overcome.

  To ensure fair comparison, prompts and plugin parameters for iEvaLM and CSHI were adapted for the food recommendation task. A custom gpt-4o-mini-based single-prompt CRS was constructed for these simulators to interact with.

5.3.2. Baselines for Rating Reliability (Section 4.2)

To assess the reliability of RecUserSim's rating mechanism, it interacts with different CRS models implemented using various LLM backbones. Two CRS frameworks are used:

• BaseCRS: A single-prompt CRS that generates recommendations based solely on the dialogue history. This represents a simpler CRS architecture.
• AgentCRS: A more sophisticated CRS agent equipped with:

  • A planning module to select actions.

  • A memory module to store interaction history and extract user preferences.

  • An action module to perform "ask," "recommend," and "answer" actions, each guided by specific prompts.

    Each of these CRS frameworks (BaseCRS and AgentCRS) is implemented using three different LLM backbones: gpt-3.5-turbo, gpt-4o-mini, and gpt-4o. This creates a total of six distinct CRS models for RecUserSim to evaluate.

5.3.3. LLM Bases for RecUserSim Itself

To demonstrate RecUserSim's robustness across different underlying LLMs, it is deployed using the following LLM backbones:

• gpt-3.5-turbo
• gpt-4o-mini
• gpt-4o
• glm-4-9b-chat (a locally hosted open-source model)

6. Results & Analysis

6.1. Core Results Analysis

6.1.1. Subjective Evaluation of User Simulator Performance

The subjective evaluation assesses RecUserSim's ability to emulate a single user based on six metrics across three dimensions: single-turn output quality (naturalness, clarity), single-round interaction quality (adaptability, relevance), and overall dialogue quality (role-play ability, realism). The evaluation was conducted via a pairwise comparison with gpt-4o-based judge, generating 500 dialogues for each simulator.

The following figure (Figure 2 from the original paper) presents the subjective comparison results:

Figure 2: Subjective comparison of RecUserSim and baseline simulators. Higher win rates indicate better performance.

Analysis:

• As shown in Figure 2, RecUserSim consistently outperforms both iEvaLM and CSHI across all six subjective metrics when using the same base LLM (gpt-4o-mini). This demonstrates RecUserSim's superior capability in producing higher-quality and more human-like dialogue outputs.
• Notably, even when RecUserSim uses a smaller, open-source model (glm-4-9b-chat) as its base LLM, it still achieves higher win rates than iEvaLM (which uses gpt-4o) in most metrics, with the exception of naturalness. This finding highlights RecUserSim's robustness and strong adaptability, indicating that its architectural design and mechanisms are effective enough to maintain high-quality simulation even with less powerful underlying LLMs. This is a significant advantage, as it suggests RecUserSim can be effectively deployed in resource-constrained environments or with open-source models while still delivering competitive performance.

6.1.2. Objective Evaluation of User Simulator Diversity

This evaluation objectively assesses the diversity of user simulator outputs by analyzing the distribution of sentence length, information richness, and formality across 500 generated dialogues. A more uniform distribution across categories indicates greater diversity, reflecting better capability in modeling a diverse user population.

The following figure (Figure 3 from the original paper) compares the output diversity of RecUserSim and baseline simulators:

Figure 3: Objective comparison of simulators' output diversity. More uniform distributions indicate greater diversity.

Analysis:

• Figure 3 clearly illustrates that RecUserSim exhibits significantly greater diversity across all three metrics: sentence length, information richness, and formality. Its distributions are well-balanced, meaning it generates a broad mix of short, medium, and long sentences; varying levels of information density; and different degrees of formality.
• In contrast, both iEvaLM and CSHI show skewed distributions.
  • iEvaLM tends to favor highly informative and formal responses, suggesting a limited range of output styles.
  • CSHI also leans towards formal and less informative replies, indicating a lack of variability in its generated dialogues.
• These results objectively confirm RecUserSim's superior capability in simulating a broad spectrum of user populations by producing outputs that reflect diverse linguistic patterns and communication styles, which is crucial for comprehensive CRS evaluation.

6.1.3. Controllability of RecUserSim over Outputs

This evaluation assesses RecUserSim's ability to control its outputs based on user-specific linguistic patterns, demonstrating its strength in individual user simulation. A well-controlled simulator should generate outputs that align with a user's predefined preferences for sentence length, information richness, and formality.

The following figure (Figure 4 from the original paper) illustrates the controllability of RecUserSim on outputs:

Figure 4: Controllability of RecUserSim on outputs based on users' linguistic patterns.

Analysis:

• Figure 4 provides strong evidence of RecUserSim's high controllability over its outputs, consistent across different base LLMs (gpt-4o-mini, gpt-3.5-turbo, glm-4-9b-chat).
• Panel (a) - Sentence Length Control: Shows that users defined as preferring short messages indeed generate responses skewed towards shorter lengths, while those preferring long messages produce outputs concentrated at longer lengths. This demonstrates effective control over response verbosity.
• Panel (b) - Information Richness Control: Illustrates that informative users generate responses with richer information (more key points) compared to uninformative users. This confirms the ability to modulate the density of meaningful content.
• Panel (c) - Formality Control: Demonstrates control over formality levels. Users inclined towards formal speech yield more formal responses, whereas those preferring informal speech generate more informal replies.
• These findings collectively validate RecUserSim's robust controllability based on distinct user linguistic patterns, underscoring its significant advantage in simulating diverse individual user behaviors with high fidelity. This precise control is largely attributed to its tool-augmented refinement module.

6.2. Evaluation on Rating Reliability

This section assesses the consistency and accuracy of RecUserSim's explicit multi-dimensional rating mechanism in evaluating CRS performance across different LLM backbones. RecUserSim (built on gpt-4o, gpt-4o-mini, and glm-4-9b-chat) interacted with six CRS models (BaseCRS and AgentCRS, each with gpt-3.5-turbo, gpt-4o-mini, and gpt-4o backbones), generating 500 dialogues per pair. Scores were computed for action quality, language quality, and recommendation quality, and Pearson and Spearman correlation coefficients were used to measure consistency.

The following are the results from Table 1 of the original paper:

Table 1: Evaluation results of RecUserSim in assessing CRS models with different base LLMs.

Analysis of Table 1:

• Accuracy of Evaluation: The scores assigned by RecUserSim to the CRS models generally align with the expected performance of their respective base LLMs for the CRS itself. Across both BaseCRS and AgentCRS, models powered by gpt-4o typically receive higher scores than gpt-4o-mini, which in turn receive higher scores than gpt-3.5-turbo. This trend ($gpt-4o > gpt-4o-mini > gpt-3.5-turbo$) demonstrates the accuracy of RecUserSim's evaluations in differentiating CRS models based on the capability of their underlying LLMs.

• AgentCRS vs. BaseCRS in Action Quality: Despite AgentCRS having a more sophisticated framework (planning, memory, action modules), its action ratings are often lower than BaseCRS. This is particularly noticeable when RecUserSim with gpt-4o evaluates the gpt-3.5-turbo versions (3.63 for AgentCRS vs. 4.48 for BaseCRS). The paper attributes this to AgentCRS's constrained action space, which limits its behavioral adaptability compared to the potentially more flexible single-prompt BaseCRS. This suggests that a complex architecture doesn't guarantee better performance if its components are too restrictive.

• Language Quality: Scores for language quality are consistently very high (close to 5.00) across all CRS models and RecUserSim backbones, indicating that LLM-based CRS generally produce high-quality language.

  The following are the results from Table 2 of the original paper:

  Evaluation Models	Pearson	Spearman
  gpt-4o vs gpt-4o-mini	0.99*	0.88*
  gpt-4o vs glm-4-9b	0.98*	0.82*
  gpt-4o-mini vs glm-4-9b	0.99*	0.89*

Table 2: Correlation values for Action ratings. The * symbol indicates statistical significance at $p < 0.05$.

Analysis of Table 2 (Action ratings correlation):

• The Pearson and Spearman correlation coefficients for Action ratings are exceptionally high (all $> 0.82$), with Pearson correlations consistently near 0.99. This indicates a very strong and statistically significant ($p < 0.05$) consistency in how RecUserSim assigns Action quality scores across its different LLM backbones. Regardless of whether RecUserSim uses gpt-4o, gpt-4o-mini, or glm-4-9b-chat, it largely agrees on the relative Action quality of the CRS models. This highlights the robustness of the Action quality evaluation.

  The following are the results from Table 3 of the original paper:

  Evaluation Models	Pearson	Spearman
  gpt-4o vs gpt-4o-mini	0.62	0.51
  gpt-4o vs glm-4-9b	0.53	0.81*
  gpt-4o-mini vs glm-4-9b	0.87*	0.81*

Table 3: Correlation value for Recommendation ratings. The * symbol indicates statistical significance at $p < 0.05$.

Analysis of Table 3 (Recommendation ratings correlation):

• The correlation coefficients for Recommendation ratings show moderate to high alignment. While some Pearson values are lower (e.g., 0.53 for gpt-4o vs. glm-4-9b), the Spearman correlations are generally higher (e.g., 0.81 for gpt-4o vs. glm-4-9b, and gpt-4o-mini vs. glm-4-9b), and often statistically significant ($p < 0.05$).

• The paper explains that the slightly lower Pearson correlations (compared to Action ratings) stem from minor score variations among similarly performing models. These minor differences can be amplified by the sensitivity of Pearson correlation to the exact numerical values, whereas Spearman (being rank-based) is more robust to such small magnitude differences.

• Nevertheless, the relatively high Spearman correlations (especially those statistically significant) still indicate a strong agreement between RecUserSim's Recommendation quality ratings across different LLM backbones regarding the ranking of CRS models. This supports the overall robustness of RecUserSim's evaluation mechanism even for the more nuanced Recommendation quality dimension.

  Overall, the findings validate both the accuracy (scores reflect CRS LLM capabilities) and reliability (consistency across different RecUserSim backbones) of RecUserSim in quantitatively assessing CRS performance.

6.3. Industrial Deployment

RecUserSim was deployed in the development and evaluation of Huawei's Celia Food Assistant, a real-world conversational food recommendation system that accounts for geographical locations and restaurant availability. It was used to evaluate both the demo and online versions of the assistant. To benchmark RecUserSim's performance, human evaluations were also conducted on 100 randomly selected conversations from each version, with six evaluators annotating them based on the same criteria as RecUserSim.

The following are the results from Table 4 of the original paper:

Table 4: Evaluation results of RecUserSim in developing Celia Food Assistant.

Analysis of Table 4:

• Consistency with Human Evaluation: Table 4 shows a general alignment between RecUserSim's evaluation results and human evaluators. While the absolute scores may differ slightly (e.g., RecUserSim's Action scores are higher than human scores for the Demo version), the relative trend between the Celia (Demo) and Celia (Online) versions is consistent. Both RecUserSim and human evaluators indicate that the Online version of Celia performs better across all metrics (Action, Language, Recommendation) compared to the Demo version.
• Demonstrated Effectiveness: This consistency in ranking and relative performance improvement between system versions highlights RecUserSim's effectiveness and practical applicability in evaluating real-world CRS in an industrial setting. It can reliably identify improvements or regressions in CRS performance, serving as a valuable tool during development cycles.
• Dual RecUserSim Backbones: The RecUserSim results are presented using both gpt-4o-mini and gpt-4o as backbones, further demonstrating that the simulator's findings are stable and not overly dependent on a single LLM base. The scores from both RecUserSim backbones generally follow the same trend as human evaluations.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper successfully introduced RecUserSim, a novel LLM agent-based user simulator designed to overcome the limitations of existing CRS evaluation methods. RecUserSim achieves realistic user role-playing and diverse user population representation through a meticulously crafted agent framework. Its profile module enables the creation of fine-grained and diverse user personas, which are consistently tracked and evolved by the memory module (including unknown preference excitation). The core action module, inspired by Bounded Rationality theory, implements a three-tier Rating-Action-Response mechanism for nuanced decision-making, multi-dimensional ratings, fine-grained actions, and personalized responses. Furthermore, the tool-augmented refinement module ensures that generated outputs strictly adhere to specific user linguistic patterns. Comprehensive experiments demonstrated RecUserSim's superiority over baselines in generating diverse, controllable, and high-quality dialogues, even with less powerful LLMs. Crucially, its explicit rating mechanism proved highly consistent across different LLM backbones, validating its reliability for quantitative CRS evaluation. The successful deployment in Huawei's Celia Food Assistant further underscores its practical utility and adaptability in real-world industrial applications.

7.2. Limitations & Future Work

While the paper does not explicitly dedicate a section to "Limitations," several implicit challenges and potential areas for improvement can be inferred from the methodology and discussion:

• Reliance on LLM for Judgment and Refinement: Both the profile conflict resolution, unknown preference excitation, multi-dimensional rating justifications, and aspects of the refinement module (especially for information richness and formality) rely on LLMs (specifically gpt-4o for evaluation). While powerful, LLMs can exhibit biases, lack true common sense, or occasionally generate hallucinations. This dependence on LLM judgments might introduce a circular dependency or hidden biases into the evaluation process.

• Cost of Powerful LLMs: Running advanced LLMs like gpt-4o (or even gpt-4o-mini) for extensive simulations and evaluations can be computationally expensive and may incur significant API costs, limiting scalability for extremely large-scale evaluations, especially with the multiple LLM calls involved in the multi-module architecture.

• Generalizability Beyond Food Recommendation: While RecUserSim demonstrated effectiveness in the food recommendation domain and was deployed for a food assistant, its generalizability to other CRS domains (e.g., movie, travel, fashion) might require specific adaptations of its profile module dictionaries, action space, and refinement tools examples. The core framework is adaptable, but domain-specific knowledge injection is still necessary.

• Complexity of Persona Definition: While fine-grained user profiles are an advantage, defining and maintaining a truly exhaustive and diverse set of behavior traits and linguistic patterns for a vast user population can still be a complex and labor-intensive task, even with random sampling and conflict resolution.

  Potential future research directions implied by the work or general challenges in the field:

• More Robust and Interpretable LLM-as-Judge: Further research into making LLM-based judges more transparent, auditable, and less prone to bias would enhance the reliability of simulator-based evaluations.

• Self-Improving User Simulators: Developing mechanisms for RecUserSim to learn and adapt its persona and behavior patterns dynamically from real user interactions (if limited human data becomes available) could further enhance realism.

• Expanding Action Space and Dialogue Complexity: Exploring even more complex and nuanced user actions or dialogue phenomena (e.g., humor, sarcasm, emotional states) could push the boundaries of user simulation.

• Cross-Domain Generalization: Investigating methods for RecUserSim to more easily adapt its persona and domain knowledge to new CRS scenarios with minimal manual intervention.

7.3. Personal Insights & Critique

RecUserSim offers significant inspiration by demonstrating how LLM agents can be meticulously engineered to address complex challenges like CRS evaluation. The paper's strength lies in its modular design and the thoughtful integration of theoretical concepts (like Bounded Rationality) into practical mechanisms.

• Innovation in Behavioral Modeling: The Three-Tier Action Mechanism (Rating-Action-Response) is a particularly insightful contribution. Breaking down user behavior into distinct cognitive steps provides a more structured and realistic approach than monolithic LLM prompts. The Multi-Dimensional Rating with explicit justifications is crucial for providing actionable feedback to CRS developers, moving beyond subjective impressions.

• Emphasis on Controllability and Diversity: The profile module and tool-augmented refinement module are excellent examples of how to impose fine-grained control over LLM outputs, a common challenge in LLM applications. This focus on ensuring both individual persona adherence and overall population diversity is vital for a robust evaluation tool. The objective metrics used to validate diversity are also well-chosen.

• Practical Applicability: The successful industrial deployment with Huawei's Celia Food Assistant is strong evidence of RecUserSim's practical value. It suggests that such simulators can significantly reduce the cost and time of CRS development and iteration.

  Critiques and potential areas for improvement:

• The "Black Box" of LLM Judgments: While LLM-as-a-judge is a growing paradigm, its internal decision-making for subjective metrics (like naturalness, realism, information richness, formality) remains opaque. While the paper uses gpt-4o for judging, further research into how to make these LLM judgments more transparent, explainable, and less prone to specific LLM biases would be beneficial. For example, explicitly defining rubric details for the LLM judge beyond a simple rating scale.

• True Latent Preference Discovery: The unknown preference excitation mechanism is a step forward, but the extent to which an LLM can truly "discover" latent preferences that are not implicitly encoded in its training data or the user's known preferences remains an open question. It might be more accurately described as inferring agreeable but unstated preferences rather than discovering entirely novel ones.

• Computational Overhead: The multi-module, sequential processing, and repeated LLM calls (for rating, action, response, then refinement with judgers and refiners) can be computationally intensive. While the paper shows robustness with smaller LLMs, optimizing the efficiency of these calls would be important for large-scale, cost-effective deployments.

• Scalability of Profile Construction: While random sampling helps, ensuring comprehensive and consistent profile dictionaries across various domains can still be a bottleneck. Exploring automated or semi-automated methods for generating or validating these base profile attributes would be valuable.

  Overall, RecUserSim represents a significant advancement in CRS evaluation, providing a principled and practical framework for creating realistic, diverse, and controllable user simulations. Its innovations can potentially be transferred to other interactive AI systems that require robust and nuanced user-side evaluation.