1. Bibliographic Information

1.1. Title

ChatCRS: Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems

1.2. Authors

• Chuang Li (National University of Singapore, NUS Graduate School for Integrative Sciences and Engineering)
• Yang Deng (National University of Singapore, Singapore Management University, Singapore)
• Hengchang Hu (National University of Singapore)
• Min-Yen Kan (National University of Singapore)
• Haizhou Li (National University of Singapore, Chinese University of Hong Kong, Shenzhen)

1.3. Journal/Conference

This paper was published at Findings of the North American Chapter of the Association for Computational Linguistics (NAACL) in 2025. NAACL is a highly reputable conference in the field of Natural Language Processing (NLP), indicating a strong peer-review process and significant impact within the research community.

1.4. Publication Year

2025

1.5. Abstract

The paper addresses the limitations of advanced Large Language Models (LLMs) like ChatGPT in domain-specific Conversational Recommender System (CRS) tasks. These limitations manifest in two key areas: 1) generating grounded responses that leverage recommendation-oriented knowledge, and 2) proactively guiding conversations towards different dialogue goals. Through a comprehensive evaluation, the authors demonstrate the critical need for external knowledge and goal guidance to improve both recommendation accuracy and language quality. In response, they propose ChatCRS, a novel framework that decomposes the complex CRS task into sub-tasks. ChatCRS integrates a knowledge retrieval agent, which uses a tool-augmented approach to reason over external Knowledge Bases (KBs), and a goal-planning agent for predicting dialogue goals. Experiments conducted on two multi-goal CRS datasets show that ChatCRS achieves state-of-the-art performance, enhancing the informativeness of language by 17%, proactivity by 27%, and recommendation accuracy by a factor of ten.

1.6. Original Source Link

https://aclanthology.org/2025.findings-naacl.17.pdf

2. Executive Summary

2.1. Background & Motivation (Why)

The paper tackles the challenge of efficiently enabling Large Language Models (LLMs) to perform effectively in domain-specific Conversational Recommender System (CRS) tasks. While LLMs excel in general natural language generation, they exhibit significant limitations when applied directly to CRS, particularly in:

1. Generating grounded responses with recommendation-oriented knowledge: LLMs often struggle to provide factual, domain-specific information crucial for recommendations, especially in domains with sparse internal knowledge or when requiring external, up-to-date facts.

2. Proactively leading conversations through different dialogue goals: LLMs may fail to steer the conversation effectively towards recommendation goals, leading to unhelpful or repetitive interactions.

   This problem is important because CRS aims to integrate conversational capabilities with recommendation systems, allowing for multi-round interactions and dynamic understanding of user needs. The existing gap lies in efficiently adapting powerful, general-purpose LLMs to this specialized task without requiring prohibitively expensive fine-tuning or relying solely on their often insufficient internal knowledge. Prior works incorporating external knowledge and goal guidance in CRS primarily used smaller, training-based language models, which are not scalable to LLMs. Existing retrieval-augmented LLM methods also face challenges in CRS due such as ambiguous query formulation and the need for future knowledge planning.

The paper's novel approach is the ChatCRS framework, which addresses these gaps by decomposing the complex CRS task into manageable sub-tasks handled by specialized agents that interact with the LLM.

2.2. Main Contributions / Findings (What)

The primary contributions of this work are:

1. Comprehensive Evaluation of LLMs in CRS: The paper provides a thorough evaluation of LLMs' capabilities and limitations in both recommendation and response generation tasks within CRS, highlighting the critical necessity of external knowledge and goal guidance.
2. Introduction of ChatCRS Framework: The authors propose ChatCRS, the first knowledge-grounded and goal-directed LLM-based CRS that employs a multi-agent architecture. This framework effectively decomposes the CRS problem into sub-tasks:
   • A knowledge retrieval agent uses a tool-augmented approach to interface and reason over external Knowledge Bases.
   • A goal planning agent predicts dialogue goals to proactively guide conversations. These agents operate atop any LLM backbone, providing external inputs without requiring costly fine-tuning.
3. Experimental Validation and Performance Enhancement: Experiments on two multi-goal CRS datasets (DuRecDial and TG-ReDial) validate the efficacy and efficiency of ChatCRS. The framework sets new state-of-the-art benchmarks, achieving:
   • 17% improvement in the informativeness aspect of language quality.
   • 27% improvement in the proactivity aspect of language quality.
   • A tenfold enhancement in recommendation accuracy. The analysis further elucidates how these external inputs contribute to the model's superior performance.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Conversational Recommender System (CRS): An interactive system that combines natural language conversation with item recommendation. Unlike traditional recommender systems that might just provide a list of items, CRS engages in a dialogue with the user to understand their preferences, ask clarifying questions, and provide recommendations within the conversational flow. It involves both a recommendation task (suggesting items) and a response generation task (producing natural language utterances).
• Large Language Models (LLMs): Advanced artificial intelligence models (e.g., ChatGPT, LLaMA) trained on vast amounts of text data, capable of understanding, generating, and processing human-like text. They exhibit strong natural language generation capabilities and often possess a wide range of implicit "world knowledge" learned during pre-training.
• Knowledge Bases (KBs): Structured repositories of factual information, often represented as a graph of entity-relation-entity triples (e.g., (Jiong He, Zodiac sign, Taurus)). KBs provide explicit, grounded knowledge that LLMs might lack or struggle to access reliably.
• In-Context Learning (ICL): A technique where LLMs learn a task by being provided with a few examples (demonstrations) in the input prompt, without any weight updates or fine-tuning. The model then generates an output for a new, unseen input based on these examples. This is also referred to as few-shot learning in the context of LLMs.
• Chain-of-Thought (CoT) Prompting: A prompting technique designed to elicit reasoning steps from LLMs. Instead of directly asking for an answer, the prompt encourages the LLM to think step-by-step, showing its "thought process" before arriving at the final answer. This can improve the model's ability to solve complex problems.
• Low-Rank Adaptation (LoRA): A parameter-efficient fine-tuning technique for large pre-trained models. Instead of fine-tuning all parameters of a large model, LoRA injects small, trainable low-rank matrices into the model's existing layers. This significantly reduces the number of trainable parameters and computational cost, making fine-tuning LLMs more feasible.
• Multi-Agent Systems: A system composed of multiple autonomous agents that interact with each other and their environment to achieve individual or collective goals. In the context of LLMs, this often means having a central LLM orchestrate specialized "agents" (which might be other LLMs or external tools) for specific sub-tasks.
• Tool-Augmented LLMs: LLMs that are enhanced by giving them access to external tools (e.g., search engines, calculators, knowledge base query tools). The LLM can decide when and how to use these tools to perform tasks that are beyond its intrinsic capabilities or to access up-to-date, grounded information.

3.2. Previous Works

The paper contextualizes its work by discussing prior research in CRS and LLMs.

Attribute-based or Conversational Approaches in CRS

• Attribute-based approaches: These systems (e.g., Zhang et al. 2018; Lei et al., 2020; Deng et al., 2021) focus on exchanging item attributes in an entity space, often without natural language conversations. The interaction is more structured, e.g., "Do you prefer movies with action or comedy?"
• Conversational approaches: These methods (e.g., Li et al., 2018b; Deng et al., 2023c; Wang et al., 2023a) use natural language generation to interact with users. They typically employ language models as backbones for generating responses. Early works like Li et al. (2018a) and Hayati et al. (2020) used general language models (e.g., DialoGPT) and incorporated external knowledge or guidance (goals, topics) to improve performance in domain-specific CRS tasks (Wang et al., 2022, 2021). The current paper builds on the conversational approach.

LLM-based CRS

Recent research explores LLMs in CRS in several ways:

• Zero-shot or few-shot recommenders: LLMs are prompted with item-based (Palma et al., 2023; Dai et al., 2023) or conversational inputs (He et al., 2023; Sanner et al., 2023; Wang et al., 2023b; Qin et al., 2024) to generate recommendations. This paper conducts an empirical analysis in this area.

• AI agents controlling pre-trained CRS or LMs: LLMs act as orchestrators, distributing CRS subtasks to other models and optimizing the ensemble (Feng et al., 2023; Liu et al. 2023a; Huang et al., 2023). This aligns with ChatCRS's multi-agent design.

• User simulators: LLMs are used to generate CRS datasets or evaluate interactive CRS systems (Wang et al., 2023c; Zhang and Balog, 2020; Huang et al., 2024).

  The paper points out a critical gap: a lack of prior work integrating external inputs specifically to improve LLM-based CRS models, which ChatCRS aims to fill.

Multi-agent and Tool-augmented LLMs

• LLMs, when designed as conversational agents, can achieve specific goals through multi-agent task decomposition and tool augmentation (Wang et al., 2023d).
• This involves delegating subtasks to specialized agents and invoking external tools like knowledge retrieval or function calls (Yao et al., 2023; Wei et al., 2023; Yang et al., 2023; Jiang et al., 2023; Zhang et al., 2024). This body of work provides the foundational ideas for ChatCRS's architecture.

3.3. Technological Evolution

The evolution leading to this paper can be traced as follows:

1. Traditional Recommender Systems (RS): Focused purely on item recommendation based on user historical data or item features.

2. Conversational Recommender Systems (CRS): Integrated natural language interaction to dynamically understand user preferences and provide recommendations, moving beyond static profiles. Early CRS models often used smaller, task-specific language models.

3. Knowledge-Enhanced CRS: Recognized the need for external knowledge and goal guidance to improve CRS performance, especially in domain-specific scenarios. These often involved training-based methods on smaller LMs (e.g., MGCG, UniMIND).

4. Rise of Large Language Models (LLMs): LLMs demonstrated unprecedented natural language generation capabilities, sparking interest in their application to CRS. Initial LLM-based CRS research focused on their inherent recommendation abilities, often in zero-shot or few-shot settings.

5. LLM Limitations in Domain-Specific CRS: Despite their power, LLMs revealed limitations in providing grounded, domain-specific knowledge and proactively guiding conversations in CRS tasks without external scaffolding.

6. Tool-Augmented and Multi-Agent LLMs: The broader NLP community started exploring how to augment LLMs with external tools and orchestrate them within multi-agent frameworks to overcome their inherent limitations (e.g., hallucination, lack of up-to-date knowledge).

   ChatCRS emerges at this juncture, leveraging the strengths of LLMs while addressing their domain-specific CRS weaknesses by integrating the proven benefits of external knowledge and goal guidance through a multi-agent, tool-augmented framework.

3.4. Differentiation

ChatCRS differentiates itself from previous approaches primarily by:

• Unified Framework for LLM-based CRS: Unlike earlier CRS works that primarily used smaller language models or focused on single CRS tasks, ChatCRS is explicitly designed for LLMs, jointly addressing both recommendation and response generation.
• Integration of External Inputs for LLMs: While previous CRS models integrated external knowledge/goals, applying these effectively to LLMs without prohibitive fine-tuning has been a challenge. ChatCRS provides an efficient solution via a multi-agent architecture.
• Multi-Agent, Tool-Augmented Design: It uniquely combines a goal planning agent (for dialogue flow) and a tool-augmented knowledge retrieval agent (for grounded information) within a single framework controlled by an LLM. This is a novel combination for CRS.
• Efficiency and Adaptability: It avoids costly full fine-tuning of LLMs by using in-context learning (ICL) for the main LLM and parameter-efficient fine-tuning (LoRA) for the goal planning agent. The modular design also makes it backbone-agnostic, meaning it can be applied to different LLMs.
• Addressing Retrieval Limitations: It tackles the limitations of generic retrieval-augmented generation (RAG) methods in CRS by employing a path-based, structured knowledge retrieval that allows for planning and reasoning over the Knowledge Base, rather than just simple keyword-based retrieval.

4. Methodology

4.1. Principles

The core idea behind ChatCRS is to overcome the inherent limitations of Large Language Models (LLMs) in domain-specific Conversational Recommender System (CRS) tasks by providing them with structured external knowledge and explicit goal guidance. The intuition is that while LLMs excel at language generation and general reasoning, they may lack specific, up-to-date, or domain-grounded knowledge, and might not proactively drive conversations towards specific recommendation goals. By decomposing the complex CRS task into sub-tasks and using specialized agents to procure these external inputs, ChatCRS allows a central LLM to leverage its powerful generative capabilities while remaining grounded and goal-oriented. This modular, multi-agent approach harnesses the strengths of LLMs without requiring computationally expensive full fine-tuning.

The paper formulates the CRS process as: $$( i , s _ { j + 1 } ^ { s y s } ) = C R S \left( C o v , K , G \right)$$ Where:

• $i$: The item recommended.

• $s_{j+1}^{sys}$: The next-turn system response.

• Conv: The dialogue history (conversation context).

• $K$: External knowledge, which can be factual knowledge (single triples like [Jiong— Star sign—Taurus]) or item-based knowledge (multiple triples describing an item/entity, like $[Cecilia—Star in—<i1> <i2> ... <in>]$).

• $G$: Dialogue goals, which guide the conversation's direction (e.g., "greeting," "ask question," "movie recommendation").

  This formulation highlights that the CRS outputs (recommendation and response) are a function of the conversation history, external knowledge, and dialogue goals.

The overall architecture of ChatCRS is shown in Figure 3.

该图像是论文中图3的示意图，展示了ChatCRS系统设计，包括a) 知识检索代理通过外部知识库推理，b) 目标规划代理预测对话目标，c) 对话代理结合对话历史和外部输入生成响应和推荐结果。

Figure 3:Overall ChatCRS system design including a) Knowledge retrieval agent that interfaces and reasons over external KB; b) Goal planning agent and c) Conversational agent generate final results for both CRS tasks.

4.2. Steps & Procedures

ChatCRS operates through a multi-agent framework comprising three main components: a Knowledge Retrieval Agent, a Goal Planning Agent, and an LLM-based Conversational Agent.

1. Input: A new dialogue history $C_j$ is received.
2. Task Decomposition: The LLM-based conversational agent acts as a controller, conceptually decomposing the complex CRS task into sub-tasks requiring external knowledge and goal prediction.
3. Knowledge Retrieval (via Knowledge Retrieval Agent):
   • Entity Extraction: Entities mentioned in the current user utterance are extracted. (The paper states this is directly provided by extracting entities in the KB from the dialogue utterance, as in Zou et al., 2022).
   • Candidate Relation Extraction (F1): For each extracted entity $E$, the agent uses a function F1 to identify all neighboring relations in the external Knowledge Base (KB). This provides a set of candidate relations.
   • Relation Planning by LLM: The core LLM, guided by N-shot In-Context Learning (ICL) (see example in Table 13), analyzes the dialogue history $C_j$ and the candidate relations to select the most pertinent and potential relation $R^*$. This step involves the LLM's reasoning capabilities to infer what knowledge might be relevant.
   • Knowledge Triple Retrieval (F2): Once $R^*$ is selected, the agent uses a function F2 along with the entity $E$ and selected relation $R^*$ to fetch the complete knowledge triples $K^*$ from the KB (e.g., $[Jimmy—Stars in—<movie 1, movie 2, ..., movie n>]$).
   • Handling Multiple Entities/Triples: If multiple entities are present in an utterance, knowledge retrieval is performed individually for each. If multiple item-based knowledge triples are returned, a fixed number (K) are randomly selected due to input token length limitations for the LLM.
4. Goal Planning (via Goal Planning Agent):
   • Instruction Fine-tuning: This agent uses a Low-Rank Adapter (LoRA) to parameter-efficiently fine-tune a smaller LLM (e.g., LLaMA 2-7b) for goal prediction. The fine-tuning process optimizes the model to generate the dialogue goal $G^*$ for the next utterance given the current dialogue history $C_j$.
   • Goal Prediction: The fine-tuned LoRA model takes the dialogue history $C_j$ as input and predicts the most appropriate dialogue goal $G^*$ for the upcoming turn. This goal helps in proactive conversation management.
5. Response and Recommendation Generation (via LLM-based Conversational Agent):

   • ICL Prompt Construction: The main LLM receives the dialogue history $C_j$, the retrieved knowledge $K^*$, and the predicted goal $G^*$. These are combined into an In-Context Learning (ICL) prompt, following a structure similar to the "Oracular Generation" examples shown in Figure 2c and Table 12.

   • Output Generation: Based on this comprehensive prompt, the LLM generates the final system response $s_{j+1}^{system}$ and/or item recommendation $i$.

     The modular design allows each agent to function independently, enabling easy integration of new LLMs or personalized agents.

4.3. Mathematical Formulas & Key Details

4.3.1. Overall CRS Formulation

The target function for CRS is expressed in two parts: given the dialogue history Conv, it generates 1) the recommendation of item $i$ and 2) a next-turn system response $s_{j+1}^{sys}$. $$( i , s _ { j + 1 } ^ { s y s } ) = C R S \left( C o v , K , G \right)$$

• $i$: The item recommended by the system.
• $s_{j+1}^{sys}$: The natural language response generated by the system for the next turn.
• Conv: The current dialogue history, typically represented as a sequence of user and system utterances up to the current turn $j$, denoted as $\{s_k^{sys}, s_k^u\}_{k=1}^j$.
• $K$: External knowledge, either factual or item-based, provided to the CRS to enrich its responses and recommendations.
• $G$: Dialogue goals, which guide the conversation flow and intent for the current or next turn.

4.3.2. Goal Planning Agent Loss Function

The LoRA model used for the goal planning agent is instruction fine-tuned to generate the dialogue goal $G^*$ for the next utterance. The optimization aims to minimize the negative log-likelihood of predicting the correct goal, given the dialogue history. $$L _ { g } = - \sum _ { k } ^ { N } \sum _ { j } ^ { T } \log P _ { \theta } \left( G ^ { * } | \mathbf { \it { C } } _ { j } ^ { k } \right)$$

• $L_g$: The loss function for goal prediction.
• $N$: The total number of dialogues in the training dataset.
• $T$: The total number of turns within a given dialogue.
• $P_{\theta}(G^* | \mathbf{\it{C}}_j^k)$: The probability of predicting the ground-truth dialogue goal $G^*$ at turn $j$ in dialogue $k$, given the dialogue history $\mathbf{\it{C}}_j^k$, parameterized by $\theta$.
• $\theta$: Represents the trainable parameters of the LoRA adapter, which are optimized during fine-tuning.

4.3.3. LLM-based Conversational Agent Output Generation

The LLM-based conversational agent takes the processed inputs from the knowledge retrieval and goal planning agents to generate its final outputs. $$i , s _ { j + 1 } ^ { s y s t e m } = L L M ( C _ { j } , K ^ { * } , G ^ { * } )$$

• $i$: The recommended item(s).
• $s_{j+1}^{system}$: The generated natural language response for the next turn.
• LLM: The underlying Large Language Model (e.g., ChatGPT, LLaMA).
• $C_j$: The current dialogue history provided to the LLM.
• $K^*$: The relevant knowledge triples retrieved by the knowledge retrieval agent.
• $G^*$: The dialogue goal predicted by the goal planning agent.

4.3.4. Knowledge Ratio Calculation

To measure the necessity of relevant knowledge for different goal types, a "Knowledge Ratio" is calculated. $${ \mathrm { K n o w l e d g e ~ R a t i o ~ } } ( \mathbf { K R } ) _ { G } = { \frac { N _ { K , G } } { N _ { G } } }$$

• (KR)_G: The Knowledge Ratio for a specific goal type $G$.
• $N_{K,G}$: The number of utterances with annotated knowledge that are associated with goal type $G$.
• $N_G$: The total number of utterances associated with goal type $G$.

5. Experimental Setup

5.1. Datasets

The experiments are conducted on two multi-goal, human-annotated CRS benchmark datasets:

• DuRecDial (Liu et al., 2021):

  • Origin: Collects knowledge and goal-guided CRS dialogues.
  • Language: Available in both English and Chinese.
  • Statistics: 10,000 dialogues, 11,000 items.
  • Annotations: Fully annotated for both knowledge (✓) and goal guidance (21 distinct goals) for each dialogue turn.
  • Justification: Provides rich, granular annotations for both knowledge and goals, making it ideal for evaluating models that leverage these external inputs in a bilingual context.

• TG-ReDial (Zhou et al., 2020):

  • Origin: Collects topic-guided dialogues.

  • Language: Primarily Chinese.

  • Statistics: 10,000 dialogues, 33,000 items.

  • Annotations: Fully annotated for goal guidance (8 distinct goals) for each dialogue turn.

  • Knowledge: Does not contain internal knowledge annotation. Instead, an external Knowledge Base, KBCN_DBpedia (Zhou et al., 2022), is used to provide knowledge.

  • Justification: Represents a different scenario where knowledge needs to be sourced externally, testing the robustness of the knowledge retrieval agent, and provides insights into domain transferability for LLMs trained on Chinese text.

    The following table shows the dataset statistics:

    Dataset	Statistics		External K&G
    	Dialogues	Items	Knowledge	Goal
    DuRecDial	10k	11k	✓	21
    TG-Redial	10k	33k	X	8

5.2. Evaluation Metrics

The paper employs a comprehensive set of automatic and human evaluation metrics for both response generation and recommendation tasks, as well as for the knowledge and goal agents.

5.2.1. Response Generation Metrics

• BLEU (Bilingual Evaluation Understudy) scores (bleu-n):

  • Conceptual Definition: BLEU measures the $n$-gram overlap between the generated response and one or more reference (ground-truth) responses. It assesses the fluency and grammatical correctness, and to some extent, the content preservation of the generated text. A higher BLEU score indicates better quality.
  • Mathematical Formula: The general formula for BLEU is: $$\mathrm{BLEU} = \mathrm{BP} \cdot \exp \left( \sum_{n=1}^{N} w_n \log p_n \right)$$ Where:
    • $\mathrm{BP}$ (Brevity Penalty): A penalty factor to discourage overly short generated sentences.
    • $N$: The maximum $n$-gram order (typically 4, giving BLEU-1, BLEU-2, BLEU-3, BLEU-4).
    • $w_n$: Weight for each $n$-gram precision (often uniform, e.g., $1/N$).
    • $p_n$: The $n$-gram precision, calculated as: $$p_n = \frac{\sum_{\text{sentence} \in \text{Candidates}} \sum_{n\text{-gram} \in \text{sentence}} \min(\text{Count}(n\text{-gram}), \text{Max\_Ref\_Count}(n\text{-gram}))}{\sum_{\text{sentence} \in \text{Candidates}} \sum_{n\text{-gram} \in \text{sentence}} \text{Count}(n\text{-gram})}$$ Here, Count(n-gram) is the count of $n$-grams in the candidate (generated) sentence, and Max_Ref_Count(n-gram) is the maximum count of that $n$-gram in any single reference sentence.
  • Symbol Explanation:
    • $\mathrm{BP}$: Brevity Penalty.
    • $p_n$: Modified $n$-gram precision.
    • $w_n$: Weights for $n$-gram precisions.
    • $\text{Count}(n\text{-gram})$: Frequency of an $n$-gram in a generated sentence.
    • $\text{Max\_Ref\_Count}(n\text{-gram})$: Maximum frequency of an $n$-gram in any reference sentence.

• F1 Score (for content preservation):

  • Conceptual Definition: F1 score is the harmonic mean of precision and recall. In response generation, it is often used to evaluate how well specific content keywords or entities from the ground truth are preserved in the generated response. It balances the model's ability to generate relevant content (precision) with its ability to cover all relevant content (recall). A higher F1 score indicates better content preservation.
  • Mathematical Formula: $$\mathrm{F1} = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$ Where:
    • $\mathrm{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$ (e.g., how many generated keywords are actually in the ground truth)
    • $\mathrm{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$ (e.g., how many ground-truth keywords are covered in the generated response)
  • Symbol Explanation:
    • $\mathrm{Precision}$: Proportion of correctly predicted positive instances among all predicted positive instances.
    • $\mathrm{Recall}$: Proportion of correctly predicted positive instances among all actual positive instances.
    • $\text{True Positives}$: Correctly identified content units (e.g., keywords, entities).
    • $\text{False Positives}$: Incorrectly identified content units.
    • $\text{False Negatives}$: Content units that were missed.

• Distinct (Dist-n) scores (for diversity):

  • Conceptual Definition: Distinct-n (Dist-1, Dist-2) measures the proportion of unique $n$-grams (1-grams for Dist-1, 2-grams for Dist-2) in the generated responses. It assesses the diversity and non-repetitiveness of the generated text. A higher Distinct score indicates more diverse and less generic responses.
  • Mathematical Formula: $$\mathrm{Dist}\text{-}n = \frac{\text{Number of unique } n\text{-grams}}{\text{Total number of } n\text{-grams}}$$
  • Symbol Explanation:
    • $\text{Number of unique } n\text{-grams}$: The count of distinct $n$-grams found across all generated responses.
    • $\text{Total number of } n\text{-grams}$: The total count of all $n$-grams across all generated responses.

5.2.2. Recommendation Metrics

• NDCG@k (Normalized Discounted Cumulative Gain at k):

  • Conceptual Definition: NDCG@k is a measure of ranking quality that considers the position of relevant items. It assigns higher scores to relevant items that appear higher in the recommendation list. It is normalized to be between 0 and 1, where 1 represents a perfect ranking.
  • Mathematical Formula: $$\mathrm{NDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}}$$ Where $\mathrm{DCG@k}$ (Discounted Cumulative Gain at k) is: $$\mathrm{DCG@k} = \sum_{j=1}^{k} \frac{2^{\mathrm{rel}_j} - 1}{\log_2(j+1)}$$ And $\mathrm{IDCG@k}$ (Ideal Discounted Cumulative Gain at k) is the maximum possible DCG@k, obtained by sorting all relevant items by their relevance: $$\mathrm{IDCG@k} = \sum_{j=1}^{k} \frac{2^{\mathrm{rel}_{(j)}} - 1}{\log_2(j+1)}$$
  • Symbol Explanation:
    • $\mathrm{rel}_j$: The relevance score of the item at position $j$ in the recommended list.
    • $\mathrm{rel}_{(j)}$: The relevance score of the item at position $j$ in the ideal (sorted) list.
    • $k$: The cut-off position (e.g., 10 or 50).

• MRR@k (Mean Reciprocal Rank at k):

  • Conceptual Definition: MRR@k measures the average of the reciprocal ranks of the first relevant item in a list of recommendations. If the first relevant item appears at rank $r$, its reciprocal rank is $1/r$. If no relevant item is found within the top $k$, the score is 0. A higher MRR@k indicates that relevant items are ranked higher.
  • Mathematical Formula: $$\mathrm{MRR@k} = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{1}{\mathrm{rank}_q}$$
  • Symbol Explanation:
    • $|Q|$: The total number of queries (or recommendation instances).
    • $\mathrm{rank}_q$: The rank position of the first relevant item for query $q$ within the top $k$ recommendations. If no relevant item is found within $k$, $\mathrm{rank}_q$ is considered $\infty$, making $1/\mathrm{rank}_q = 0$.

5.2.3. Knowledge and Goal Agent Metrics

• Accuracy (Acc):
  • Conceptual Definition: The proportion of correctly predicted instances (knowledge relations or goals) out of the total number of instances.
  • Mathematical Formula: $$\mathrm{Acc} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$
• Precision (P):
  • Conceptual Definition: The proportion of true positive predictions among all positive predictions made by the model.
  • Mathematical Formula: $$\mathrm{P} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$$
• Recall (R):
  • Conceptual Definition: The proportion of true positive predictions among all actual positive instances.
  • Mathematical Formula: $$\mathrm{R} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$$
• F1 Score (for knowledge/goal prediction):
  • Conceptual Definition: The harmonic mean of Precision and Recall, providing a balanced measure of a model's performance, especially useful when class distribution is imbalanced.
  • Mathematical Formula: $$\mathrm{F1} = 2 \cdot \frac{\mathrm{P} \cdot \mathrm{R}}{\mathrm{P} + \mathrm{R}}$$
  • Symbol Explanation (for Acc, P, R, F1):
    • $\text{True Positives}$: Instances correctly identified as positive.
    • $\text{False Positives}$: Instances incorrectly identified as positive.
    • $\text{False Negatives}$: Instances incorrectly identified as negative (missed positives).

5.2.4. Human Evaluation Metrics

For human evaluation, 100 dialogues from DuRecDial were randomly sampled, and responses were scored by three annotators on a scale of 0 (bad) to 2 (good) across four metrics:

• General Language Quality:
  • Fluency (Flu): Assesses if responses are grammatically correct and flow naturally.
  • Coherence (Coh): Evaluates the relevance and logical consistency of responses within the dialogue context.
• CRS-specific Language Quality:
  • Informativeness (Info): Quantifies the depth and breadth of knowledge or information conveyed in the responses.
  • Proactivity (Pro): Assesses how effectively responses anticipate and address the underlying goals or requirements of the conversation, proactively steering the dialogue.

5.3. Baselines

To validate ChatCRS's efficacy, both LLM-based and traditional training-based baselines are selected.

LLM Baselines (Few-shot settings):

• ChatGPT: A prominent closed-source Large Language Model.
• LLaMA 2-13b (Touvron et al., 2023): An open-source Large Language Model.
  • Representativeness: These represent the state-of-the-art in general-purpose LLMs, allowing for a direct comparison of their inherent capabilities with and without the ChatCRS framework's external inputs. They are used in N-shot In-Context Learning (ICL) settings, mimicking how ChatCRS uses LLMs.

Training-based Baselines (Fully-finetuned):

• UniMIND (Deng et al., 2023c): A multi-task learning framework specifically designed for multi-goal conversational recommender systems.
  • Representativeness: This is a crucial baseline as it is one of the few prior models capable of performing both response generation and recommendation tasks, making it a holistic CRS baseline similar to ChatCRS's aim.
• MGCG (Multi-type GRUs for encoding/generation) (Liu et al., 2020):
  • Focus: Primarily on the response generation task.
  • Representativeness: A GRU-based approach that encodes dialogue context, goals/topics for generating responses.
• MGCG-G (GRU-based for graph-grounded goal planning) (Liu et al., 2023b):
  • Focus: Primarily on the response generation task, with a focus on graph-grounded goal planning.
  • Representativeness: Extends MGCG by incorporating goal planning into response generation.
• TPNet (Transformer-based dialogue encoder and graph-based dialogue planner) (Wang et al., 2023a):
  • Focus: Primarily on the response generation task and goal-planning.
  • Representativeness: A Transformer-based model reflecting more advanced sequence-to-sequence approaches for dialogue generation.
• GRU4Rec (Gated Recurrent Unit for Recommendation) (Liu et al., 2016):
  • Focus: Recommendation task.
  • Representativeness: A strong sequential recommender system that leverages GRUs to model user preferences.
• SASRec (Self-Attentive Sequential Recommendation) (Kang and McAuley, 2018):
  • Focus: Recommendation task.
  • Representativeness: A Transformer-based sequential recommender system, representing a more advanced approach to capturing long-range dependencies in user behavior.
• BERT (Devlin et al., 2019):
  • Focus: Goal planning task (as a text-classification task).
  • Representativeness: A foundational Transformer-based pre-trained language model, used here for classifying dialogue goals.
• BERT+CNN:
  • Focus: Goal planning task.
  • Representativeness: A deep learning approach combining BERT representations with a Convolutional Neural Network (CNN) for next goal predictions.

5.4. Implementation Details

• LLMs for CRS tasks: ChatGPT and LLaMA 2-13b were used in few-shot settings with N-shot In-Context Learning (ICL) prompts (Dong et al., 2022; Sanner et al., 2023).
  • Prompting: $N$ training data examples were integrated into the ICL prompts in a consistent format for each task.
  • Recommendation task: LLMs were prompted to produce a top-$K$ item ranking list, focusing only on knowledge-guided generation (due to fixed dialogue goal of "Recommendations").
  • ChatGPT temperature: Set to 0 to ensure replicable output given the same input, minimizing randomness.
• Goal Planning Agent:
  • Model: QLoRA was used to fine-tune a smaller LLaMA 2-7b model, enhancing parameter efficiency (Dettmers et al., 2023; Deng et al., 2023c).
  • LoRA parameters: Attention dimension and scaling alpha were set to 16.
  • Training: The base language model was kept frozen; only the LoRA layers were optimized using the Adam optimizer.
  • Hyperparameters: Fine-tuned over 5 epochs, batch size of 8, learning rate of $1 \times 10^{-4}$.
• Knowledge Retrieval Agent and LLM-based Generation Unit: Employ the same N-shot ICL approach as in CRS tasks, using ChatGPT and LLaMA-13b (Jiang et al., 2023).
• Language Specifics: For TG-ReDial, which contains only Chinese conversations, a pre-trained Chinese LLaMA model was used for inference.
• Computational Resources: Experiments ran on a single A100 GPU or via the OpenAI API.
• Inference Duration: One-time ICL inference on DuRecDial test data ranged from 5.5 to 13 hours for LLaMA and ChatGPT, respectively.
• OpenAI API Cost: Approximately US$20 for the DuRecDial dataset inference.

5.5. Human Evaluation Details

• Dataset: 100 randomly sampled dialogues from the DuRecDial dataset.
• Models Evaluated: UniMIND, ChatGPT, LLaMA-13b, and ChatCRS.
• Annotators: Three research assistants, fluent in both English and Mandarin, and well-educated.
• Scoring: Each response was scored on a scale of 0 (bad), 1 (ok), and 2 (good) for four metrics.
• IRB Exemption: The human evaluation process received an IRB exemption.
• Dataset Access: The dataset used is publicly accessible.
• Compensation: Annotators were compensated at a rate of $15 per hour.
• Evaluation Criteria (reiterated from 5.2.4):
  • Fluency (Flu): Grammatical correctness and natural flow.
  • Coherence (Coh): Relevance and logical consistency within the dialogue context.
  • Informativeness (Info): Depth and breadth of knowledge or information conveyed.
  • Proactivity (Pro): Effectiveness in anticipating and addressing conversational goals.

6. Results & Analysis

6.1. Core Results

6.1.1. Empirical Analysis: Necessity of External Inputs

The preliminary empirical analysis evaluates LLMs' inherent capabilities with and without external knowledge and goal guidance.

The following table shows the results from Table 1, demonstrating the recommendation task performance:

The following table shows the results from Table 2, demonstrating the response generation task performance in DuRecDial:

Key Findings from Empirical Analysis:

• Finding 1: The Necessity of External Inputs: For both recommendation (Table 1) and response generation (Table 2), the Oracle approach (with gold-standard external knowledge and dialogue goals) consistently and significantly outperforms Direct Generation (DG) and Chain-of-Thought (COT) across all LLM baselines. This underscores that LLMs alone are insufficient for CRS tasks and highlights the indispensable role of external inputs. Notably, for the recommendation task, Oracle yields over a tenfold improvement compared to DG and COT.
• Finding 2: Improved Internal Knowledge/Goal Planning in Advanced LLMs: Table 2 indicates that the performance of Chain-of-Thought (COT) in a larger LLM (LLaMA-13b) can be comparable to or even slightly surpass the Oracle performance of a smaller LLM (LLaMA-7b) in some metrics. This suggests that more sophisticated LLMs possess better internal knowledge and goal-setting capabilities. However, these internal capabilities are still insufficient for domain-specific CRS, as the integration of more accurate external knowledge and goal guidance (the Oracle approach) continues to yield state-of-the-art (SOTA) performance.

6.1.2. ChatCRS Performance for Recommendation Task

The following table shows the results from Table 5, demonstrating ChatCRS's recommendation performance:

Analysis: ChatCRS significantly improves recommendation accuracy, especially compared to raw LLM baselines (ChatGPT, LLaMA-13b) in few-shot settings, achieving a tenfold enhancement. On DuRecDial, ChatCRS reaches comparable performance to UniMIND, a fully-finetuned baseline, despite using only few-shot methods. This highlights the effectiveness of external knowledge in grounding LLMs for recommendation tasks, especially when LLMs' internal knowledge might be limited or misaligned with specific domains. For TG-ReDial, ChatCRS shows substantial improvement over LLM baselines in MRR, aligning with UniMIND in NDCG@10 but surpassing it in MRR@10/50.

6.1.3. ChatCRS Performance for Response Generation Task

The following table shows the results from Table 6, demonstrating ChatCRS's response generation performance:

Analysis: ChatCRS outperforms existing fully-finetuned baselines in fluency (higher BLEU scores) and language diversity (high dist2 scores). In terms of F1 score (content preservation), some baseline models fine-tuned on training data perform better due to familiarity with the dataset's language style. However, automatic metrics may not fully capture language quality, necessitating human evaluation. ChatCRS shows significant improvements over raw LLMs and competitive performance against fully-finetuned models, especially in diversity (dist2), indicating its ability to generate varied and engaging responses.

6.1.4. Human Evaluation Results

The following table shows the results from Table 7, demonstrating human evaluation and ChatCRS ablations for language qualities on DuRecDial:

Analysis: Human evaluation confirms that LLMs generally excel in Fluency and Coherence compared to smaller LMs (like UniMIND). ChatCRS demonstrates superior performance across all human evaluation metrics. It notably enhances Coherence through its goal guidance. Critically, for CRS-specific language quality, Informativeness (quantifying depth of knowledge) and Proactivity (assessing effectiveness in anticipating goals), ChatCRS shows significant improvement over all baselines. This highlights the importance of incorporating both external knowledge and goals for high-quality CRS interactions.

6.1.5. Knowledge Retrieval Agent Results

The following table shows the results from Table 8, demonstrating the knowledge retrieval agent performance:

Analysis: ChatCRS shows a dramatic improvement in knowledge retrieval accuracy (e.g., F1 of 0.553) compared to raw LLMs (ChatGPT F1: 0.015, LLaMA-13b F1: 0.001). This confirms the severe limitation of LLMs' internal knowledge for domain-specific tasks and the effectiveness of ChatCRS's tool-augmented knowledge retrieval agent in interfacing with external KBs. ChatCRS also outperforms training-based approaches like TPNet and MGCG-G, indicating its ability to reason over and select pertinent knowledge.

6.1.6. Goal Planning Agent Results

The following table shows the results from Table 9, demonstrating the goal planning agent performance:

Analysis: The ChatCRS goal planning agent achieves state-of-the-art performance in goal prediction, particularly on the DuRecDial dataset (F1 of 0.97). This is a substantial improvement over raw LLMs, which perform poorly (e.g., ChatGPT F1 of 0.04). While the performance on TG-ReDial is slightly lower (F1 of 0.81), it still significantly surpasses LLM baselines. This indicates that the LoRA-fine-tuned goal planning agent effectively guides the dialogue flow. The slight drop in TG-ReDial performance is attributed to its higher proportion of recommendation-related goals and more multi-goal utterances, making prediction more challenging.

6.2. Ablations / Parameter Sensitivity

6.2.1. Ablation over Knowledge Types

The following table shows the results from Table 3, demonstrating the ablation over knowledge types utilising ChatGPT as the LLM backbone:

Analysis (Finding 3): This ablation study demonstrates that both factual and item-based knowledge jointly improve LLM performance on domain-specific CRS tasks. When either factual knowledge or item-based knowledge is removed, there's a noticeable decline in performance for both response generation and recommendation tasks. For example, in the recommendation task, removing factual knowledge drastically reduces NDCG@10 from 0.617 to 0.220. The paper suggests that even if a type of knowledge doesn't directly contribute to a specific task (e.g., factual knowledge for item recommendation), it can still benefit LLMs by helping them associate unknown entities with their internal knowledge, thus adapting more effectively to the target domain. This finding justifies ChatCRS's approach of leveraging both types of knowledge.

6.2.2. Ablation over Knowledge Retrieval and Goal Planning Agents

The human evaluation results in Table 7 also serve as an ablation study for the ChatCRS framework's components.

• w/o K* (without Knowledge retrieval agent): When the knowledge retrieval agent is removed, the Informativeness score drops from 1.76 to 1.49 (a decrease of 15.4%). This confirms that the knowledge retrieval agent is crucial for generating informative responses, as it provides the necessary external facts and details.

• w/o G* (without Goal planning agent): When the goal planning agent is removed, the Proactivity score drops from 1.69 to 1.55 (a decrease of 8.3%). This indicates that goal guidance is essential for the system to proactively steer the conversation and anticipate user needs. While there is a slight drop in Informativeness as well (from 1.76 to 1.72), the impact is more pronounced on Proactivity.

  These ablations empirically confirm the efficacy and necessity of both the knowledge retrieval and goal planning agents for enhancing CRS-specific language qualities within the ChatCRS framework.

6.3. Detailed Analysis (RQ3)

6.3.1. Knowledge Ratio for Goal Types

The paper investigates the Knowledge Ratio for different goal types on the DuRecDial dataset to understand the necessity of external knowledge.

The following figure shows the knowledge ratio for each goal type on DuRecDial dataset:

该图像是一个条形图，展示了DuRecDial数据集中不同对话目标类型对应的知识比例，反映了在问答及推荐场景中知识的使用频率和分布情况。

Figure 4: Knowledge ratio for each goal type on DuRecDial dataset.

Analysis: The Knowledge Ratio (defined by Equation 4) measures the proportion of utterances within a specific goal type that require annotated knowledge. The analysis in Figure 4 reveals:

• Goals like "Asking questions" have a very high knowledge ratio (98%), indicating that nearly all utterances under this goal type necessitate external knowledge for accurate responses.
• All recommendation-related goals consistently appear in the top 10 for knowledge necessity.
• "POI recommendation" (Point of Interest recommendation) ranks highest among recommendation goals, requiring pertinent knowledge in 75% of cases. This detailed analysis reinforces the empirical finding that domain-specific CRS tasks, especially recommendation and question-answering, heavily rely on external knowledge, a need that LLMs' internal knowledge alone cannot sufficiently fulfill.

6.3.2. Goal Prediction Across Datasets

The paper provides a detailed breakdown of goal prediction results for the top 5 goal types in each dataset.

The following figure shows the results of ChatCRS goal predictions with different goal types on DuRecDial (left) and TG-Redial (right) datasets:

该图像是图表，展示了ChatCRS在DuRecDial和TG-Redial数据集上不同对话目标类型的预测结果，左侧为DuRecDial数据集中五个主要目标的总数与正确预测数，右侧为TG-Redial数据集中五个目标类型组合的预测统计。

Figure 5: Results of ChatCRS goal predictions with different goal types on DuRecDial (left) and TG-Redial (right) datasets.

Analysis:

• DuRecDial: ChatCRS demonstrates SOTA performance in goal prediction for DuRecDial. This dataset has a better-balanced distribution between recommendation and non-recommendation goals, which aligns more closely with real-world conversational scenarios. The high accuracy across diverse goals (e.g., "Ask for opinion," "Ask questions," "Chat about celebrity," "Movie recommendation") indicates the goal planning agent's effectiveness.
• TG-ReDial: While ChatCRS still performs strongly on TG-ReDial, the accuracy is slightly lower than DuRecDial. This is attributed to two factors:
  1. TG-ReDial contains a higher proportion of recommendation-related goals compared to "Chit-Chat" goals.
  2. It features more multi-goal utterances (e.g., combining "Chit-Chat and Asking for Recommendation"), making goal predictions inherently more challenging due to increased complexity and ambiguity. This detailed breakdown confirms that the goal planning agent is highly effective in guiding dialogue flow, especially in well-balanced dialogue distributions.

6.3.3. Case Studies

The paper presents two case studies (Table 10) to illustrate the practical impact of ChatCRS compared to baselines.

The following table shows the case study for ChatCRS with baselines:

Analysis:

• Case Study 1 (Factual Knowledge Answer): This scenario, related to the "Asking questions" goal type, highlights ChatCRS's ability to leverage factual knowledge. While UniMIND correctly answers "Taurus", ChatGPT and LLaMA hallucinate incorrect zodiac signs ("Cancer" and "Aries," respectively). ChatCRS accurately retrieves and integrates the correct external knowledge, providing a grounded and factual response ("Taurus"). This demonstrates how external knowledge guidance mitigates the risks of generating implausible or inconsistent information by LLMs.
• Case Study 2 (Proactive Response Generation): Here, the target dialogue goal is "Movie Recommendation." ChatCRS not only recommends a relevant movie ($<Left Right...Destiny>$) but also proactively engages the user by adding descriptive details ("new movie this year," "taxi driver, very cool!") and posing a follow-up question ("Have you seen it?"). This contrasts with UniMIND and ChatGPT which offer recommendations but lack the same level of conversational proactivity. LLaMA even provides an unhelpful, generic response. This case study underscores how accurate goal direction, powered by the goal planning agent, enables ChatCRS to lead dialogues effectively, maintaining engagement and gathering information for refined recommendations, rather than devolving into unproductive turns seen in raw LLMs.

7. Conclusion & Reflections

7.1. Conclusion Summary

This work introduces ChatCRS, an innovative framework that significantly enhances Large Language Model-based Conversational Recommender Systems (CRS). The core contribution lies in its ability to effectively integrate external knowledge retrieval and goal-guided planning into LLM-driven CRS. Through comprehensive empirical analysis, the paper convincingly demonstrates that LLMs, despite their advanced capabilities, are inherently limited in domain-specific CRS tasks without such external scaffolding. ChatCRS addresses these limitations by adopting a multi-agent architecture: a knowledge retrieval agent leverages tool-augmented approaches to reason over external Knowledge Bases, and a goal planning agent predicts dialogue goals to proactively steer conversations.

The experimental results on DuRecDial and TG-ReDial datasets validate ChatCRS's efficacy, showcasing substantial improvements: a tenfold enhancement in recommendation accuracy, a 17% increase in informativeness, and a 27% boost in proactivity of generated responses. This establishes ChatCRS as a new state-of-the-art solution, offering a scalable and model-agnostic approach that reduces the reliance on expensive fine-tuning while maintaining high adaptability across diverse domains.

7.2. Limitations & Future Work

The authors acknowledge several limitations:

• Resource Constraints: The study was limited to few-shot learning and parameter-efficient techniques due to budget and computational constraints. It primarily used economically viable, smaller-scale closed-source LLMs (ChatGPT) and open-source models (LLaMA-7b and -13b).

• Data Scarcity: A significant challenge was the scarcity of datasets with adequate knowledge and goal-oriented annotations for each dialogue turn. This limitation hinders the development of more sophisticated conversational models capable of effectively understanding and navigating complex dialogues. The authors hope future datasets will address this.

  Potential future work suggested by the authors includes:

• Advanced Planning Mechanisms: Exploring more sophisticated planning mechanisms for the agents to further enhance CRS performance.

• Self-Improving Retrieval Strategies: Investigating self-improving retrieval strategies for the knowledge retrieval agent, allowing it to adapt and refine its knowledge access over time.

7.3. Personal Insights & Critique

The ChatCRS framework offers a compelling solution to a critical problem in LLM-based CRS: how to leverage the immense generative power of LLMs while ensuring their responses are grounded in factual knowledge and their conversations are purposefully directed. The multi-agent, tool-augmented approach is a practical and efficient way to achieve this, side-stepping the prohibitive costs of full fine-tuning for domain-specific tasks.

Novelty: The explicit decomposition of the CRS task into distinct knowledge retrieval and goal planning agents orchestrated by a central LLM, particularly with the tool-augmented knowledge retrieval and LoRA-based goal planning, is a significant contribution. It moves beyond generic Retrieval-Augmented Generation (RAG) by integrating planning into knowledge acquisition and proactivity into conversation management.

Strengths:

• Modular Design: The modularity is a major strength, allowing for easy integration of new LLMs or specialized tools without overhauling the entire system. This future-proofs the framework to some extent.
• Efficiency: The reliance on ICL and LoRA makes the solution economically viable and scalable, especially important for enterprises or researchers with limited computational resources.
• Strong Empirical Results: The tenfold improvement in recommendation accuracy and significant gains in language quality metrics (especially informativeness and proactivity) are highly impressive and demonstrate the practical utility of the framework.
• Addressing LLM Weaknesses: It directly tackles known LLM weaknesses like hallucination (through grounded knowledge) and lack of proactivity (through goal guidance).

Potential Areas for Improvement/Critique:

• Dynamic Knowledge Base Updates: While the knowledge retrieval agent interfaces with external KBs, the paper doesn't explicitly discuss how ChatCRS handles dynamic, real-time updates to these KBs. For highly volatile domains (e.g., trending news, stock prices), ensuring the KB is always current is crucial.

• User Feedback Integration: The framework currently focuses on system-side improvements. Exploring how explicit user feedback (e.g., "I don't like that recommendation," "tell me more") can iteratively refine the knowledge retrieval or goal planning strategies could lead to even more adaptive systems.

• Ambiguity Resolution: User queries can be ambiguous. The paper could delve deeper into how the knowledge retrieval agent handles ambiguous entity mentions or relations, or how the goal planning agent adapts when a user's true intent is unclear.

• Generalizability to Other Domains: While tested on two CRS datasets, the adaptability of the framework to vastly different domains (e.g., medical CRS, legal CRS) could be further explored. These domains might require more complex reasoning or different types of knowledge structures.

• Ablation of LLM Types: It would be interesting to see how the performance gains scale with different base LLMs (e.g., very small LLMs vs. much larger ones). The paper shows some comparison between ChatGPT and LLaMA, but a more systematic ablation of the LLM backbone itself could be insightful.

  Ethical Considerations: The paper thoughtfully includes an "Ethical Considerations" section, detailing their human evaluation protocol, public datasets, methodology for annotation, and fair compensation for annotators. This commitment to ethical research practices is commendable and crucial in the age of large language models and human data annotation.

Overall, ChatCRS represents a robust and well-validated step forward in making LLMs more effective and reliable for complex, domain-specific conversational tasks. Its architectural design is both elegant and practical, likely inspiring future research in multi-agent and tool-augmented LLM systems.