1. Bibliographic Information

1.1. Title

Asking Clarifying Questions for Preference Elicitation With Large Language Models

1.2. Authors

• Ali Montazeralghaem (alimontazer@google.com) - Google Mountain View, CA, USA

• Guy Tennenholtz (guytenn@google.com) - Google Mountain View, CA, USA

• Craig Boutilier (cboutilier@google.com) - Google Mountain View, CA, USA

• Ofer Meshi (meshi@google.com) - Google Mountain View, CA, USA

  All authors are affiliated with Google in Mountain View, CA, USA, indicating a strong industry research background, likely focusing on practical applications of AI and machine learning, particularly in areas relevant to Google's products (e.g., search, recommendations, conversational AI).

1.3. Journal/Conference

Published at GENNEXT@SIGIR'25. SIGIR (Special Interest Group on Information Retrieval) is a highly reputable and leading international conference in the field of information retrieval. GENNEXT@SIGIR'25 likely refers to a workshop or a co-located event, indicating that the work was presented at a venue associated with a top-tier conference, suggesting relevance and quality within the information retrieval and recommender systems community.

1.4. Publication Year

2025

1.5. Abstract

The paper addresses the challenge of eliciting user preferences in Conversational Recommendation Systems (CRS) augmented by Large Language Models (LLMs), especially when user history is limited. It proposes a novel two-stage method, inspired by diffusion models, to train LLMs to generate effective sequential clarifying questions. In the forward process, a complete user profile is 'corrupted' by generating clarifying questions and then removing the answers step by step, creating noisy user profiles. The reverse process trains an LLM to denoise these profiles by learning to ask effective clarifying questions to reconstruct the original profile. The results demonstrate that this method significantly improves the LLM's ability to ask funnel questions (moving from general to specific) and effectively elicit user preferences.

1.6. Original Source Link

https://arxiv.org/abs/2510.12015 (Preprint on arXiv)

1.7. PDF Link

https://arxiv.org/pdf/2510.12015v1.pdf (Preprint on arXiv)

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the effective elicitation of user preferences in Conversational Recommendation Systems (CRS) that leverage Large Language Models (LLMs). In recommendation systems (RSs), personalizing recommendations is crucial, but often challenging due to limited user interaction history (the cold-start problem) or privacy constraints that restrict access to past data. Additionally, current user preferences can be influenced by transient factors like mood or context, which passive observation cannot capture.

Traditional preference elicitation (PE) techniques exist, but with the advent of LLMs and conversational interfaces, there's an opportunity to perform PE through multi-turn dialogues. While simple prompting can make LLMs ask questions, the challenge lies in optimizing them to generate effective sequential clarifying questions across various domains. The existing gap is how to systematically train LLMs to ask high-quality, structured PE questions that efficiently reveal user preferences, especially in a funnel-like manner (starting general and becoming more specific).

The paper's entry point is an innovative idea: drawing inspiration from diffusion models to train LLMs for this task. By framing the problem as denoising a user profile through question-asking, they aim to generate a training methodology that naturally leads to structured and effective PE dialogues.

2.2. Main Contributions / Findings

The primary contributions of this paper are:

1. Novel Diffusion-Inspired Training Approach: Introduction of a two-stage process, inspired by diffusion models, for training LLMs to ask sequential clarifying questions for user preference elicitation. This involves a forward process of corrupting a user profile by removing answers and generating questions, and a reverse process of training an LLM to denoise the profile by asking effective questions.

2. Generation of Funnel Questions: The method is designed to produce funnel questions, meaning the LLM learns to start with more general inquiries and gradually progress to more specific ones, mimicking a natural and efficient conversational flow for PE.

3. Improved LLM Proficiency: Experimental results demonstrate that the proposed method significantly enhances the LLM's ability to ask effective clarifying questions, leading to improved reconstruction of the true user profile.

4. Effective User Preference Elicitation: The fine-tuned LLM shows superior performance in gathering relevant user information and building a comprehensive user profile through sequential questioning, outperforming non-fine-tuned LLMs and less effective user simulators.

5. User Simulator Fine-tuning: The paper also shows that fine-tuning a user simulator (another LLM) to provide accurate answers from a given profile significantly improves the overall interaction and Questioner performance.

   The key conclusions are that a diffusion model-inspired training paradigm can effectively optimize LLMs for the complex task of sequential preference elicitation, resulting in more human-like and efficient conversational RSs. These findings address the cold-start problem and enhance personalization by enabling LLMs to actively and strategically gather user preferences.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a beginner should be familiar with several core concepts:

• Large Language Models (LLMs): These are advanced artificial intelligence models, like GPT-3, Gemma, or Gemini, trained on vast amounts of text data. They can understand, generate, and process human language. Their strength lies in their ability to perform various Natural Language Processing (NLP) tasks, such as answering questions, summarizing text, and holding conversations, often in an open-ended manner. In this paper, LLMs are used both to generate questions and simulate user responses.

• Recommendation Systems (RSs): These are software tools that provide suggestions for items (e.g., movies, products, articles) to users. They typically work by analyzing user behavior, preferences, and item characteristics. The goal is to help users discover items they might like but haven't explicitly searched for.

• Conversational Recommendation Systems (CRS): This is a type of RS where the interaction with the user occurs through a dialogue interface. Instead of just presenting a list of recommendations, a CRS can ask questions, understand user feedback in natural language, and refine recommendations iteratively. LLMs are increasingly used to power the conversational aspects of CRS.

• Preference Elicitation (PE): This is the process of actively asking users questions to determine their tastes, needs, or priorities. In RSs, PE helps overcome the cold-start problem (when there's little or no historical data for a new user) and allows for dynamic adaptation to changing user preferences or contexts. The paper focuses on generating effective PE questions.

• Diffusion Models: Originating primarily in computer vision for tasks like image generation, diffusion models are a class of generative models that learn to reverse a diffusion process.

  • Forward (Corruption) Process: This process gradually adds noise to data (e.g., an image) until it becomes pure random noise. For discrete data (like text), it involves operations like inserting, deleting, or replacing tokens.
  • Reverse (Denoising) Process: This process learns to iteratively denoise the corrupted data, starting from noise and gradually reconstructing the original clean data. This paper adapts this concept to user profiles, where "noise" means missing information, and "denoising" means reconstructing the full profile by asking questions.

• Fine-tuning: This is a technique used to adapt a pre-trained LLM (a model already trained on a large, general dataset) to a specific task or domain. By training the model on a smaller, task-specific dataset, it learns to perform that particular task more accurately and efficiently.

• Parameter-Efficient Fine-Tuning (PEFT): PEFT methods are a family of techniques that allow for fine-tuning LLMs with significantly fewer trainable parameters than full fine-tuning. This reduces computational cost and memory requirements.

• Low-Rank Adaptation (LoRA): A specific PEFT technique. LoRA works by injecting trainable low-rank matrices into the transformer architecture of LLMs. During fine-tuning, only these newly added matrices are trained, while the original pre-trained model weights remain frozen. This makes fine-tuning much more efficient.

• JSON Format: JavaScript Object Notation is a lightweight data-interchange format. It's a human-readable way to represent structured data, consisting of key-value pairs and ordered lists. The paper uses JSON to represent user profiles in a structured manner, making it easier for LLMs to process and manipulate.

3.2. Previous Works

The paper builds upon existing research in recommendation systems, preference elicitation, and the application of LLMs in conversational contexts.

• Autoregressive Text Generation (Background 2.1): The foundation for how LLMs generate text. The probability of an entire sequence $S = [s_0, s_1, ..., s_N]$ (e.g., a sentence or a question) being generated is modeled as the product of conditional probabilities of each token, given the preceding tokens. $ P ( S ) = \prod _ { i = 0 } ^ { N } p ( s _ { t } | s _ { 0 } , s _ { 1 } , . . . , s _ { t - 1 } ) $ Here, P(S) is the probability of the entire sequence $S$. $s_t$ represents the token at position $t$ in the sequence. $p(s_t | s_0, s_1, ..., s_{t-1})$ is the conditional probability of generating the token $s_t$ given all the tokens that came before it ($s_0$ to $s_{t-1}$). This autoregressive property means each token is predicted based on the tokens already generated, moving from left to right.

• Diffusion Models (Background 2.3): The paper explicitly draws inspiration from diffusion models, particularly discrete diffusion models used in text generation. In computer vision, continuous diffusion models work by adding Gaussian noise progressively to an original image $x_0$ to get $x_T$ (pure noise) in a forward (corruption) process. A model is then trained to reverse this, denoising $x_{t+1}$ back to $x_t$. The paper adapts this to user preferences, where a null state (empty profile) is iteratively refined by asking questions, similar to how a noisy image is refined into a clear one.

• Preference Elicitation (PE) in RSs: References [16, 18, 23, 24, 26] discuss traditional PE techniques, highlighting its importance in clarifying user preferences and improving recommendations. These works typically focus on various strategies for asking questions to infer user preferences, often using explicit feedback.

• Conversational Recommendation Systems (CRS) with LLMs: References [6, 11, 19-21, 31] explore the integration of LLMs into RSs to create conversational interfaces. These works demonstrate the potential of LLMs to augment RSs with multi-turn dialogue capabilities, enabling more natural PE. Simple prompting of LLMs to ask questions is a common starting point, but this paper aims to optimize this process.

• LLM-based Preference Elicitation and Clarifying Questions:

  • Li et al. [17] introduced Generative Active Task Elicitation (GATE), where LLMs interact with users via free-form language to infer behavior.
  • Andukuri et al. [1] proposed STaR-GATE, focusing on teaching LLMs to ask clarifying questions to handle ambiguity.
  • Austin et al. [2] used Bayesian optimization with LLM-based acquisition functions for natural language PE, employing NLI and BO strategies.
  • Piriyakulkij et al. [22] presented an algorithm for active preference inference using LMs and probabilistic reasoning to generate informative questions.
  • Montazeralghaem et al. [19-21] also contributed to conversational search and recommendation, including using actor-critic frameworks for interactive CRS.

3.3. Technological Evolution

The field has evolved from traditional recommendation systems relying on implicit feedback and collaborative filtering to more interactive systems. Early PE involved structured questionnaires or simple feedback mechanisms. The rise of LLMs marks a significant shift, enabling RSs to engage users in natural language conversations. This allows for more nuanced preference elicitation than ever before. However, simply using LLMs out-of-the-box for PE can be suboptimal; they need to be guided to ask effective, sequential, and non-redundant questions. This paper represents a step in this evolution by proposing a systematic, diffusion-inspired training method to optimize LLMs for this specific conversational PE challenge. It moves beyond generic prompting to structured LLM fine-tuning for question generation.

3.4. Differentiation Analysis

Compared to other LLM-based PE approaches, the core innovation of this paper lies in its diffusion model-inspired training methodology for generating sequential clarifying questions.

• Structured Training for Sequentiality and Funneling: While other works explore LLMs asking clarifying questions (e.g., STaR-GATE [1]) or using Bayesian optimization for query generation (PEBOL [2]), this paper specifically designs a forward-reverse process to explicitly train the LLM to ask questions in a funnel-like manner (general to specific). This structured approach for sequential question generation is a key differentiator.

• Profile Corruption and Denoising Analogy: The direct analogy to diffusion models—corrupting a complete user profile by removing information (answers to questions) and then training an LLM to denoise (reconstruct) it by asking those questions—is a novel application of this paradigm to preference elicitation. This contrasts with frameworks that might infer preferences directly or optimize question selection via other means.

• Synthetic Data Generation: The paper's methodology for generating its own training data using a larger LLM (Gemini 2.0) to simulate the forward process and then fine-tuning a smaller LLM (Gemma 7B) in the reverse process is a practical and effective way to create supervised learning signals for a complex conversational task.

• Emphasis on Profile Reconstruction: The explicit objective is to maximize the probability of reconstructing the complete user profile through sequential question-answering, which provides a clear and measurable goal for the PE process.

  In essence, this paper provides a principled, diffusion-inspired framework to instill a strategic questioning behavior (funneling, non-repetitive) into LLMs for preference elicitation, going beyond generic LLM capabilities.

4. Methodology

4.1. Principles

The core idea of the method is to optimize a Large Language Model (LLM) to ask effective clarifying questions for preference elicitation by drawing inspiration from diffusion models in discrete spaces. The theoretical basis is that the process of gradually revealing user preferences can be modeled as denoising a corrupted (incomplete) user profile.

The intuition is as follows: Imagine you have a complete picture of a user's preferences. If you gradually obscure parts of this picture, and at each step, you ask a question that would help reveal the obscured part, you are essentially performing a forward (corruption) process. Conversely, if you start with an obscured (empty) picture and learn to ask the right questions to fill in the missing details, you are performing a reverse (denoising) process. This reverse process is what the LLM is trained to do: reconstruct the user's full profile by asking a sequence of questions. A key principle is that these questions should follow a funnel pattern, starting with general inquiries and progressing to more specific ones, mirroring natural human conversation and efficient information gathering.

4.2. Core Methodology In-depth (Layer by Layer)

The proposed model involves two main phases: a forward process for profile corruption (generating training data) and a reverse process for profile reconstruction (fine-tuning the LLM to ask questions).

The overall framework is illustrated in Figure 1, depicting how corrupted user profiles are addressed and reconstructed through clarifying questions. The forward process generates questions and answers to create noisy profiles, and the reverse process learns to denoise by asking effective questions.

该图像是示意图，展示了我们的方法用于处理受损用户档案以及通过澄清问题进行重构的流程。图中包含两个阶段的过程：前向过程和反向过程。前向过程通过生成一系列问题获取用户反馈，而反向过程则通过学习有效的问题来“去噪”用户档案，从而改善推荐系统的个性化效果。

Figure 1: Our model for addressing corrupted user profiles and reconstruction through clarifying questions.

4.2.1. Reverse Process: Profile Reconstruction by Asking Questions

The goal in the reverse process is to train an LLM (referred to as the Questioner) to transform an initial empty user profile $P_0 = \emptyset$ into a final, complete ground-truth profile $P_n$ by asking a sequence of questions and receiving answers. This process involves intermediate profiles $P_1, \ldots, P_{n-1}$. Each profile $P_t$ represents the state after $t$ question-answer interactions. Formally, $P_t = \{ (Q_i, A_i) \}_{i=0}^{t-1}$, where $Q_i$ is the question asked and $A_i$ is its corresponding answer.

The generative process aims to learn how the complete profile $P_n$ is formed. Using the chain rule and assuming conditional independence (for simplification), the probability of generating the entire series of profile versions is modeled as:

$ {\mathfrak{p}} ( P ) = \prod _ { i = 0 } ^ { n } {\mathfrak{p}} ( P _ { i } | P _ { 0 } , \dots , P _ { i - 1 } ) $

This general formula for a series of profile versions $P = (P_0, P_1, \ldots, P_n)$ is from the Background section of the paper. For the reverse process where we build $P_n$ from $P_0$, the probability of a profile $P_n$ (which is effectively the sequence of questions and answers leading to it) can be broken down as:

$ p _ { \theta , \phi } ( P _ { n } ) = \prod _ { t = 1 } ^ { n } p ( P _ { t } | P _ { t - 1 } ; \theta , \phi ) $

Here, $p_{\theta, \phi}(P_n)$ is the probability of obtaining the complete profile $P_n$ given the learned parameters $\theta$ and $\phi$. The product iterates from $t=1$ to $n$, representing each step where the profile is updated. $p(P_t | P_{t-1} ; \theta, \phi)$ is the probability of transitioning from the partial profile $P_{t-1}$ to $P_t$. This transition probability is further decomposed into three components:

$ \begin{array} { r l } & { \ p ( P _ { t } \mid P _ { t - 1 } ; \theta , \phi ) = p _ { \theta } ( Q _ { t - 1 } \mid P _ { t - 1 } ) } \ & { \qquad \quad \times p _ { \phi } ( A _ { t - 1 } \mid Q _ { t - 1 } , P _ { t - 1 } ) } \ & { \qquad \quad \times p ( P _ { t } \mid P _ { t - 1 } , Q _ { t - 1 } , A _ { t - 1 } ) } \end{array} $

Let's break down each component:

• $p _ { \theta } ( Q _ { t - 1 } \mid P _ { t - 1 } )$: This is the probability that the Questioner (an LLM parameterized by $\theta$) generates question $Q_{t-1}$ given the current partial profile $P_{t-1}$. This component reflects the Questioner's ability to choose an appropriate clarifying question based on what is already known about the user.

• $p _ { \phi } ( A _ { t - 1 } \ | \ Q _ { t - 1 } , P _ { t - 1 } )$: This is the probability that the user (or more precisely, a user simulator parameterized by $\phi$) provides answer $A_{t-1}$ to question $Q_{t-1}$, given $Q_{t-1}$ and the partial profile $P_{t-1}$. This component models the user simulator's response behavior.

• $p ( P _ { t } \mid P _ { t - 1 } , Q _ { t - 1 } , A _ { t - 1 } )$: This is the probability of generating the next state $P_t$ (the updated profile) given the previous state $P_{t-1}$, the question $Q_{t-1}$, and the answer $A_{t-1}$. This component is deterministic and does not have any learnable parameters. It's defined as:

  $ \begin{array} { r } { p ( P _ { t } | P _ { t - 1 } , Q _ { t - 1 } , A _ { t - 1 } ) = \left{ \begin{array} { r l } { 1 } & { \mathrm { ~ i f ~ } P _ { t } = P _ { t - 1 } \cup { ( Q _ { t - 1 } , A _ { t - 1 } ) } } \ { 0 } & { \mathrm { ~ o t h e r w i s e } } \end{array} \right. } \end{array} $ This means the new profile $P_t$ is simply the previous profile $P_{t-1}$ with the new question-answer pair $(Q_{t-1}, A_{t-1})$ added to it. The paper notes that including questions along with answers helps the model avoid repetitive queries and improves performance.

The overall objective is to maximize the probability of generating the complete user profile across all users by asking effective clarifying questions. This is formalized as:

$ \operatorname* { m a x } _ { \theta , \phi } \sum _ { i = 1 } ^ { | I | } \log ( p _ { \theta , \phi } ( P _ { n } ^ { i } ) ) , $

where $P_n^i$ is the complete profile for user $i$, and $|I|$ is the total number of users. This objective is optimized by fine-tuning two LLMs: one for the Questioner (to learn $\theta$) and one for the user simulator (to learn $\phi$).

4.2.2. Forward Process: Profile Corruption (Generating Training Data)

The forward process is crucial for generating the training data used to fine-tune the Questioner and user simulator in the reverse process. It starts with a complete user profile and gradually corrupts it by removing information, analogous to adding noise in diffusion models.

1. Structured User Profile Creation: Given a user profile $P^u$ in raw text format (e.g., "user likes action movies"), it's first converted into a structured JSON format using an LLM. This JSON format $JP^u$ allows for easier manipulation and querying of specific pieces of information. $ J P ^ { u } : = \mathrm { L L M } ( P ^ { u } ) $ For example, $P^u$ could be "User likes action movies and sci-fi. Director Christopher Nolan is a favorite." This might be converted to $JP^u = \{(\text{'Genre'}, \text{'action movies'}), (\text{'Genre'}, \text{'sci-fi'}), (\text{'Director'}, \text{'Christopher Nolan'}) \}$. The JSON profile is represented as $J P ^ { u } ~ = ~ \{ \left( t _ { i } , c _ { i } \right) \} _ { i = 1 } ^ { m }$, where $t_i$ is a tag (e.g., 'Genre', 'Director') and $c_i$ is its content (e.g., 'action movies', 'Christopher Nolan').

2. Generating Funnel Questions: The core idea here is to generate a sequence of questions that follow a funnel pattern: starting general and becoming specific. To achieve this, two constraints are imposed:

   • Generality Ordering: Questions should start with easier, more straightforward inquiries and progress to more specific ones.
   • Dependency Handling: Broader aspects should be asked before more specific ones (e.g., movie genre before a specific director). An LLM is used to rank the tags in $JP^u$ from general to specific. Then, another LLM is prompted to generate funnel questions $Q_i$ and their corresponding answers $A_i$ based on this ranked JSON profile: $ ( { \mathcal Q } _ { 0 } , A _ { 0 } ) , \ldots , ( Q _ { n - 1 } , A _ { n - 1 } ) \ = \ \mathrm { L L M } ( J P ^ { u } , { t _ { 1 } , t _ { 2 } , t _ { 3 } , \ldots , t _ { m } } ) $ Here, $Q_i$ is the generated question and $A_i$ is its answer derived from $JP^u$. For example, $Q_i = \text{'Do you like action movies?'}$ and $A_i = \text{'yes'}$. A mapping $\mathcal { T } ( Q _ { i } , A _ { i } )$ identifies the set of tag-content pairs from $JP^u$ that are addressed by $Q_i$ and $A_i$. For example, if $Q_i = \text{'Do you like action movies?'}$ and $A_i = \text{'yes'}$, then $\mathcal{T}(Q_i, A_i)$ might be $\{(\text{'Genre'}, \text{'The user likes action movies'})\}$. The questions $Q_i$ are ordered such that $Q_0$ is general and $Q_{n-1}$ is specific. $n$ is the total number of questions.

3. Iterative Profile Corruption (Data Generation): The forward process then iteratively removes information from the complete profile to create partial profiles at each step. Since the questions $Q_i$ are generated in a funnel manner (from general to specific, i.e., $Q_0$ is most general, $Q_{n-1}$ is most specific), the corruption process starts by removing the information related to the most specific question ($Q_{n-1}$) first. This way, when going in reverse (training the Questioner), the LLM will learn to ask the most general questions first to reconstruct the profile.

   The partial user profile at step $t$ is represented as $J P _ { t } ^ { u }$, which is the complete profile $J P ^ { u }$ minus the information covered by questions $Q_t$ through $Q_{n-1}$. $ J P _ { t } ^ { u } = J P ^ { u } \setminus \bigcup _ { i = t } ^ { n - 1 } \mathcal { T } _ { i } $ The index $t$ ranges from $n$ down to 0.

   • If $t=n$, the union is empty, so $J P _ { n } ^ { u } = J P ^ { u }$ (the full profile).
   • If $t=0$, information from all questions $Q_0 \ldots Q_{n-1}$ has been removed, resulting in $J P _ { 0 } ^ { u } = \emptyset$ (the empty profile). $J P _ { t } ^ { u }$ represents the partial user profile available just before asking question $Q_t$.

   The training data $D_u$ for user $u$ for the reverse process consists of pairs of (question, partial profile): $ D _ { u } = { ( Q _ { n - 1 } , J P _ { n - 1 } ^ { u } ) , ( Q _ { n - 2 } , J P _ { n - 2 } ^ { u } ) , . . . , ( Q _ { 0 } , J P _ { 0 } ^ { u } ) } . $ In each pair $(Q_i, JP_t^u)$, the model is given $JP_t^u$ as input and trained to generate $Q_i$ as the target. The full dataset $D$ for fine-tuning is the union of $D_u$ for all users $u \in I$.

   Algorithm 1 summarizes this forward process:

   Algorithm 1 Forward process: Profile Corruption

   Input: A user profile $P_u$ in text format. Output: Training data $D_u$, comprising question-partial user profile pairs for various partial profiles.

   1: Convert $P_u$ into a JSON format $JP^u$. 2: Sort tags $\{ t_1, t_2, \ldots, t_m \}$ from $JP^u$ based on notion of generality. 3: Generate Funnel Questions $\{ (Q_0, A_0), (Q_1, A_1), ..., (Q_{n-1}, A_{n-1}) \}$ based on the extracted tags. 4: $t \gets n - 1$ 5: while $t \geq 0$ do 6: $\quad$ Create partial profile $JP^{u_t}$ by using Equation (7). 7: $\quad D_u \gets D_u \cup \{ (Q_t, JP^{u_t}) \}$ 8: $\quad t \gets t - 1$ 9: end while 10: return $D_u$

   An example illustrating the full process (likely referring to the generation of questions and removal of information) is shown in Figure 2.

   该图像是一个示意图，展示了通过前向过程生成用户偏好的清晰问题。图中包含不同的用户档案及对应的问题示例，如'你对音乐剧有兴趣吗？'和'你喜欢什么类型的电影？'等。

   F inormation. The revere process then reconstructs the profle byiterativly answerin theelicitation qustions

4.2.3. User Simulation

For evaluating the trained Questioner, an environment is needed where it can interact with a user simulator. This simulator's role is to answer questions based on a ground-truth user profile. The user simulator is also an LLM. Given a question $Q$ and the ground-truth user profile $P$, it tries to find the answer: $A = \mathrm{LLM}(P, Q)$. If an answer cannot be found in the profile, it responds with "I don't know", implying the user has no specific preference regarding that question.

To enhance the user simulator's ability to answer questions effectively, it is also fine-tuned. The training data $\hat{D}_u$ for the user simulator is constructed from the forward process: $ \begin{array} { r l } & { \hat { D } _ { u } = { ( \mathcal { T } _ { n - 1 } , Q _ { n - 1 } , J P ^ { u } ) , ( \mathcal { T } _ { n - 2 } , Q _ { n - 2 } , J P ^ { u } ) , } \ & { \qquad \ldots , ( \mathcal { T } _ { 0 } , Q _ { 0 } , J P ^ { u } ) } } \end{array} $ Each tuple $(\mathcal{T}_i, Q_i, JP^u)$ serves as a training instance. Given question $Q_i$ and the full user profile $JP^u$, the model is trained to generate the corresponding answer (implicitly, by identifying the relevant $\mathcal{T}_i$ which contains the tag-content pairs, and generating the content $A_i$).

This two-stage process of data generation (forward) and model fine-tuning (reverse for Questioner and user simulator) forms the complete methodology.

5. Experimental Setup

5.1. Datasets

• Movielens Dataset: This is a widely used public dataset in recommender systems research, consisting of movie ratings from users. The paper utilizes it as the domain for preference elicitation.
• User Profiles: The ground-truth user profiles used in the experiments are not directly from the raw Movielens dataset. Instead, they are derived from Jeong et al. [15] and Tennenholtz et al. [29, 30]. These profiles were specifically generated using an LLM and the complete raw history of ratings from each user. The original authors (Tennenholtz et al.) evaluated these LLM-generated profiles and found them to be predictive of user ratings. The use of LLM-generated profiles for training ensures consistency with the LLM-centric approach of the paper and provides rich, textual preference data.

5.2. Evaluation Metrics

The quality of the generated questions and the effectiveness of preference elicitation are measured using two standard Natural Language Processing (NLP) metrics: ROUGE and BLEU. These metrics assess the similarity between the generated user profile (reconstructed by the Questioner) and the target (ground-truth) user profile.

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation):

  • Conceptual Definition: ROUGE is a set of metrics used for evaluating automatic summarization and machine translation. It works by comparing an automatically produced summary or translation against a set of human-produced reference summaries or translations. It primarily measures the overlap of n-grams (sequences of $n$ words), word sequences, and word pairs between the generated text and the reference text. Higher ROUGE scores indicate greater similarity to the reference, typically implying higher quality. The "Recall-Oriented" aspect emphasizes how much of the reference information is captured by the generated text.
  • Mathematical Formula (ROUGE-N): $ \mathrm{ROUGE-N} = \frac{\sum_{S \in {\text{Reference Summaries}}} \sum_{\text{n-gram} \in S} \mathrm{Count_{match}(\text{n-gram})}}{\sum_{S \in {\text{Reference Summaries}}} \sum_{\text{n-gram} \in S} \mathrm{Count}(\text{n-gram})} $
  • Symbol Explanation:
    • $\mathrm{ROUGE-N}$: The ROUGE score based on matching n-grams of length $N$.
    • $S \in \{\text{Reference Summaries}\}$: Iterates through each reference summary (in this case, the ground-truth user profile).
    • $\text{n-gram}$: A sequence of $N$ words.
    • $\mathrm{Count_{match}(\text{n-gram})}$: The number of times a particular n-gram appears in both the generated profile and the reference profile.
    • $\mathrm{Count}(\text{n-gram})$: The total number of times a particular n-gram appears in the reference profile. The paper likely uses variants like ROUGE-1 (unigrams), ROUGE-2 (bigrams), or ROUGE-L (longest common subsequence), which are common in NLP tasks. The abstract mentions ROUGE generally, implying a composite score or one of these common variants.

• BLEU (Bilingual Evaluation Understudy):

  • Conceptual Definition: BLEU is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. It measures the precision of n-grams between the candidate text (generated) and the reference text. It also includes a brevity penalty to discourage very short outputs. Higher BLEU scores indicate closer resemblance to professional human translations, suggesting better quality.
  • Mathematical Formula: $ \mathrm{BLEU} = \mathrm{BP} \cdot \exp \left( \sum_{n=1}^{N} w_n \log p_n \right) $ where $\mathrm{BP}$ is the brevity penalty and $p_n$ is the n-gram precision. $ p_n = \frac{\sum_{\text{n-gram} \in \text{candidate}} \mathrm{Count_{clip}}(\text{n-gram})}{\sum_{\text{n-gram} \in \text{candidate}} \mathrm{Count}(\text{n-gram})} $ $ \mathrm{BP} = \begin{cases} 1 & \text{if } c > r \ e^{1 - r/c} & \text{if } c \le r \end{cases} $
  • Symbol Explanation:

    • $\mathrm{BLEU}$: The BLEU score.

    • $\mathrm{BP}$: Brevity Penalty.

    • $N$: The maximum n-gram order considered (typically 4).

    • $w_n$: The weight for the n-gram precision $p_n$ (often uniform, $1/N$).

    • $p_n$: The n-gram precision for n-grams of length $n$.

    • $\mathrm{Count_{clip}}(\text{n-gram})$: The count of an n-gram in the candidate that also appears in the reference, clipped to its maximum count in any single reference sentence.

    • $\mathrm{Count}(\text{n-gram})$: The total count of an n-gram in the candidate text.

    • $c$: Length of the candidate (generated) text.

    • $r$: Effective reference corpus length (closest reference length to candidate length).

      The evaluation process for these metrics involves the following steps, as detailed in Algorithm 2:

Algorithm 2 Evaluation Process

Input: A corrupted profile $P_t$, target profile $P_n$, parameters $\theta, \phi$, maximum question number $T$. Output: A sequence of questions and answers that transforms $P_t$ to $P_n$.

1: Initialize profiles: $P_{\mathrm{current}} \gets P_t$ 2: Initialize question count: $\mathrm{count} \gets 0$ 3: while $(P_{\mathrm{current}} \neq P_n)$ and $(\mathrm{count} < T)$ do 4: $\quad$ Generate question $Q_{t-1}$ using the fine-tuned model as the Questioner. 5: $\quad$ Query the user simulator model, which accesses the target profile $P_n$ to determine an answer $A_{t-1}$: 6: $\quad$ if answer is found in $P_n$ then 7: $\quad \quad$ Set $A_{t-1}$ to the corresponding value in $P_n$. 8: $\quad$ else 9: $\quad \quad$ Set $A_{t-1}$ to "No Preference". 10: $\quad$ end if 11: $\quad$ Update: 12: $\quad P_{\mathrm{current}} \gets P_{\mathrm{current}} \cup \{ (Q_{t-1}, A_{t-1}) \}$ 13: $\quad$ Increment question count: $\mathrm{count} \gets \mathrm{count} + 1$ 14: end while 15: return $P_{\mathrm{current}}$

This algorithm simulates an interaction session where the Questioner (trained in the reverse process) asks questions, and the user simulator provides answers based on the ground-truth profile $P_n$. The process continues until the current profile $P_{\mathrm{current}}$ matches $P_n$ or a maximum number of questions $T$ (set to 10 in experiments) is reached. The final $P_{\mathrm{current}}$ is then compared to $P_n$ using ROUGE and BLEU.

5.3. Baselines

The paper compares its proposed fine-tuned Gemma model (as Questioner and user simulator) against several baselines to demonstrate the effectiveness of its approach:

• Non-fine-tuned Gemma model: This serves as a baseline Questioner to show the impact of the proposed diffusion-inspired fine-tuning on question generation.
• Non-fine-tuned Gemini model as a user simulator: This is used to compare against the fine-tuned Gemma user simulator. Gemini 2.0 is a larger, more capable LLM, so using it in a non-fine-tuned capacity for simulation helps assess if fine-tuning a smaller model can outperform a larger, general-purpose LLM for specific tasks. The comparisons are structured to evaluate the quality of the Questioner (fine-tuned vs. non-fine-tuned) and the user simulator (fine-tuned Gemma vs. non-fine-tuned Gemini) both individually and in combination.

5.4. Training Details

• LLMs:
  • Questioner and fine-tuned user simulator: Gemma LLM (7B version, 28 layers). Chosen for public availability of weights and strong performance for its size.
  • Forward process data generation: Gemini 2.0. A larger, more capable LLM used to ensure high quality of the synthetic training data.
• Fine-tuning technique: Parameter-Efficient Fine-Tuning (PEFT) specifically using Low-Rank Adaptation (LoRA) [13]. This technique is used to efficiently adapt the pre-trained Gemma model.
• Hyperparameters:
  • Batch size: 64
  • Learning rate: 0.001

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate the significant impact of the proposed diffusion-inspired fine-tuning on the LLM's ability to generate effective clarifying questions and the benefit of fine-tuning the user simulator.

6.1.1. Effect of Fine-Tuning (Questioner and User Simulator)

The following are the results from Figure 3(a) of the original paper:

该图像是两个柱状图，展示了使用不同问题设置的平均得分和未回答问题的百分比。在图(a)中，蓝色和红色柱子分别表示Bleu和Rouge得分，显示了Fine-tuned和Non-Fine-tuned问题的比较。图(b)则展示了不同设置下未回答问题的百分比，其中Non-Fine-tuned问题的比例最高，达到0.46。

e p Questioners with different user simulators. (b) Percentage of unanswered questions for models.

Figure 3(a) compares the performance (measured by ROUGE and BLEU scores) of different configurations of Questioners and user simulators.

• Impact on Questioner: Comparing non-finetuned questions to finetuned questions (assuming a finetuned simulation as the user simulator), there is a substantial improvement. The ROUGE score increases from approximately 0.4 to 0.68, and BLEU increases from 0.28 to 0.49. This indicates that the fine-tuning process, using the data generated by the forward process, makes the Questioner LLM significantly better at asking questions that lead to accurate profile reconstruction.

• Impact on User Simulator: By comparing the finetuned questions with finetuned simulation (green bar) against finetuned questions with non-finetuned Gemini simulation (light green bar), we observe that the finetuned simulator leads to better performance. This suggests that a specially fine-tuned LLM as a user simulator, even if smaller (Gemma) than a general-purpose large LLM (Gemini), can provide more effective and accurate answers within the experimental setup. This highlights the importance of matching the simulation environment to the task.

  The following are the results from Figure 3(b) of the original paper:

Figure 3(b) further clarifies these findings by showing the percentage of unanswered questions across different model settings.

• The non-finetuned questions combined with non-finetuned Gemini as a simulator result in the highest percentage of unanswered questions (0.46). This suggests that a general-purpose LLM (even a large one like Gemini) struggles to answer questions accurately from a profile without specific fine-tuning for this task, and a non-fine-tuned Questioner asks less precise questions.
• Fine-tuning the user simulator (comparing the first two bars to the last two bars) significantly reduces the unanswered questions, even for non-finetuned Questioners. This confirms that a well-tuned user simulator is crucial for reliable evaluation and effective interaction. The finetuned questions with finetuned simulation results in the lowest percentage of unanswered questions, indicating a highly effective interaction where questions are well-formed and answers are accurately retrieved.

6.1.2. Effect of Number of Questions

The following are the results from Figure 4 of the original paper:

该图像是一个图表，展示了BLEU（左图）和ROUGE（右图）分数与问题数量的关系。不同的曲线代表了不同的模型和微调策略，随着问题数量的增加，分数普遍呈现上升趋势。

Figure 4: BLEU (left) and ROUGE (right) scores vs. number of questions.

Figure 4 illustrates how BLEU and ROUGE scores evolve as the number of questions asked increases, for different Questioner-simulator combinations.

• Worst Performance: The non-finetuned Questioner with the Gemini simulator (blue lines) consistently performs the worst. This is attributed to both the Questioner asking ineffective questions and the Gemini simulator failing to provide relevant answers.
• Improved Simulator: Replacing the Gemini simulator with a finetuned simulator (orange lines, non-finetuned Q + finetuned sim) boosts performance. This shows that some of the questions asked by the non-fine-tuned model are useful, but their effectiveness is amplified when responded to by a more capable simulator.
• Improved Questioner: Using a finetuned Questioner with the Gemini simulator (green lines, finetuned Q + Gemini sim) also yields a performance boost compared to the non-finetuned Questioner with Gemini simulator. This directly demonstrates the value of fine-tuning the Questioner to ask more effective clarifying questions.
• Best Performance and Funneling: The best model, the finetuned Questioner paired with the finetuned simulator (red lines), shows superior performance. Crucially, the curves indicate a funneling effect: the model gathers broader information in the initial ~5 turns, leading to a rapid increase in scores, and then shifts to more specific questions in later turns (6-7), further refining the profile. This behavior aligns with the design goal of funnel questions.

6.1.3. Effect of Adding Question History

The following are the results from Figure 5 of the original paper:

该图像是一个柱状图，展示了在不同条件下（Finetune与Non-Finetune以及是否使用Q-H）评估模型生成效果的平均得分，包括Bleu和Rouge两种指标。可见，Finetune情况下的得分普遍高于Non-Finetune，且在使用Q-H时效果尤为显著。

(a) Comparing the overall performance of models by integrating questions and answers (Q-H) into user profiles.

Figure 5 shows the impact of including the question history (Q-H) along with answers in the partial user profiles during the interaction.

• Fine-tuned Model: For the finetuned Questioner, adding question history (Finetune with Q-H) significantly increases performance compared to not including it (Finetune without Q-H). This is because the history helps the model avoid asking repetitive questions and enables it to generate more effective follow-up questions based on the established context.

• Non-fine-tuned Model: In contrast, for the non-finetuned Questioner, adding question history (Non-Finetune with Q-H) decreases performance compared to Non-Finetune without Q-H. This is because while Q-H might reduce repetitiveness, the non-fine-tuned model is not trained to generate effective non-repetitive questions. It might start asking less relevant or less structured questions when forced to be non-repetitive without proper training.

  The following are the results from Figure 6 of the original paper:

  该图像是一幅柱状图，展示了不同条件下重复提问的百分比。数据分为四个类别：未微调且不使用 Q-H、未微调并使用 Q-H、已微调且不使用 Q-H、已微调并使用 Q-H，分别对应的百分比为 0.37、0.44、0.21 和 0.03。

This figure (VLM description referring to "Percentage of repetitive questions") likely corresponds to a different aspect than Figure 5(a) but is not explicitly referenced in the text as Figure 6. Assuming it is relevant, it would show how adding Q-H reduces repetitive questions, especially for the finetuned model. The "Finetune with Q-H" shows the lowest percentage of repetitive questions (0.03), validating the claim that Q-H helps avoid repetition in fine-tuned models.

6.1.4. Impact of Fine-Tuning Steps on Model Performance

The following are the results from Figure 7 of the original paper:

该图像是一个柱状图，展示了Questioner模型在不同微调步骤下（未微调、4000步、28000步和40000步）的BLEU和ROUGE评分。蓝色柱表示BLEU评分，红色柱表示ROUGE评分，结果显示随着微调步骤的增加，模型的评分显著提升。

Figure 6: BLEU and ROUGE scores of the Questioner model at different fine-tuning steps (0, 4000, 28000, and 40000)

Figure 6 (labeled as Figure 7 in the VLM description, but 6 in the paper text) illustrates how BLEU and ROUGE scores change with an increasing number of fine-tuning steps for the Questioner model.

• Improvement with Steps: Both BLEU and ROUGE scores show a clear upward trend as the fine-tuning steps increase from 0 (non-finetuned) to 4000, 28000, and 40000. This indicates that continued training helps the Questioner learn to ask better, more effective follow-up questions, which is essential for preference elicitation. The largest jump in performance typically occurs early in training (e.g., from 0 to 4000 steps), with continued but diminishing returns for further steps.

6.1.5. Analyzing the Questions Asked by the Model (Funneling)

The following are the results from Figure 8 of the original paper:

该图像是一个条形图，展示了基于加权排名（总和 >= 300）的类别预期值。图中列出了各种偏好的预期价值，如逃避、怀旧、幽默以及视觉效果等，反映了用户在推荐过程中可能关注的不同方面。

rW values correspond to concepts typically addressed later.

To confirm that the fine-tuned Questioner indeed asks questions in a funnel format, the paper analyzes the weighted rank (WR) of concepts (keywords from the JSON profile) across conversations. The WR for each concept is calculated as: $ W R = \sum _ { i = 1 } ^ { T } i \times p ( i ) $ where $T$ is the maximum number of questions the model can ask, and p(i) is the probability that the concept appears in position $i$ (i.e., at turn $i$ in the conversation). A lower WR means the concept is typically asked earlier, indicating a more general concept. A higher WR means the concept is typically asked later, indicating a more specific concept.

Figure 7 (labeled Figure 8 in the VLM description, but Figure 7 in paper text) shows the WR values for various concepts.

• Early Concepts (Lower WR): Concepts like Genre, Film Era, and Decade have lower WR values. This indicates that the Questioner tends to ask about these broader aspects earlier in the conversation.

• Mid-Conversation Concepts (Intermediate WR): Concepts such as Directors, Visual Style, and Tone have intermediate WR values, suggesting they are introduced after the initial broad questions.

• Late Concepts (Higher WR): Highly detailed concepts like Special Effects, Humor, and Atmosphere exhibit higher WR values, meaning they are typically addressed later in the conversation.

  This progression strongly confirms that the Questioner successfully learns to ask questions in a funnel-like manner, starting from general concepts and gradually moving to more specific details, consistent with the training strategy derived from the forward process.

6.2. Data Presentation (Tables)

The paper primarily presents its results through figures and descriptive text. No specific data tables are provided in the main body of the paper that require transcription using Markdown or HTML table formats. All numerical results are either embedded in the text (e.g., "Rouge from 0.4 to 0.68") or visually represented in the provided figures.

6.3. Ablation Studies / Parameter Analysis

The paper implicitly performs some ablation studies by comparing different configurations:

• Effect of Fine-Tuning: Comparing non-fine-tuned vs. fine-tuned Questioner (Figure 3a, 4, 6) shows the effectiveness of the fine-tuning process.

• Effect of User Simulator: Comparing fine-tuned vs. non-fine-tuned user simulator (Figure 3a, 3b, 4) highlights the importance of an effective simulation environment.

• Effect of Question History: The comparison of models with and without question history (Figure 5) serves as an ablation to understand the role of conversational context.

• Effect of Fine-Tuning Steps: Figure 6 (VLM 7) explicitly analyzes the impact of the number of fine-tuning steps, demonstrating that more training improves performance.

  These comparisons act as ablation studies by demonstrating the individual contributions and interactions of different components and training choices to the overall performance. They confirm that the fine-tuning approach, an effective user simulator, including question history, and sufficient training steps are all crucial for the model's success in generating effective preference elicitation questions.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces a novel and effective method for training Large Language Models (LLMs) to ask sequential, funnel-shaped clarifying questions for preference elicitation in conversational recommendation systems. Inspired by diffusion models, the approach involves a forward process to corrupt complete user profiles by systematically removing information (answers to generated questions) and a reverse process where an LLM (Questioner) is fine-tuned to denoise (reconstruct) these profiles by learning to ask the appropriate questions. Key findings include a significant improvement in the LLM's ability to generate domain-specific and contextually relevant follow-up questions, leading to more accurate user profile reconstruction. The method successfully teaches the LLM to adopt a funneling strategy, starting with general questions and progressing to more specific ones, mimicking natural conversational flow. Additionally, fine-tuning an LLM to act as a user simulator was shown to be crucial for robust evaluation and improved interaction quality.

7.2. Limitations & Future Work

The authors highlight the potential of their model to "advance personalized user interactions and open new avenues for adaptive learning in LLMs." While the paper doesn't explicitly list limitations within a dedicated section, several implicit limitations and future work directions can be inferred:

• Domain Specificity: The experiments are conducted solely in the movie recommendation domain. While the authors claim applicability "across various domains," further validation would be needed to confirm this generalizability without significant re-training or adaptation.

• Synthetic User Simulation: The evaluation relies on an LLM-based user simulator rather than real human users. While this is a practical approach for research, LLM simulators might not perfectly capture the nuances, inconsistencies, or evolving nature of real human preferences and conversational behavior. This could lead to an optimistic evaluation.

• Data Generation Reliance: The forward process for generating training data depends on a powerful LLM (Gemini 2.0). The quality and biases of this initial LLM will directly impact the quality of the generated training data and, consequently, the performance of the fine-tuned Questioner.

• Fixed Question Limit: The evaluation process sets a limit of 10 questions. While practical, real-world PE might require more or fewer questions depending on the complexity of preferences and user engagement.

• Definition of "Good Question": While ROUGE and BLEU measure textual similarity to a ground-truth profile, they don't fully capture all aspects of a "good" preference elicitation question, such as user satisfaction, perceived helpfulness, or efficiency in real user interactions.

  Future work could involve:

• Conducting user studies with real users to validate the effectiveness and user experience of the Questioner in real-world scenarios.

• Exploring the generalizability of the method to diverse domains beyond movie recommendations.

• Investigating alternative noise mechanisms or diffusion processes for discrete data, potentially leading to more robust or efficient PE strategies.

• Integrating user feedback into the training loop, possibly through reinforcement learning from human feedback (RLHF), to make the LLM questions even more aligned with human preferences.

• Developing more sophisticated user simulators that can model dynamic user behavior or uncertainty more accurately.

7.3. Personal Insights & Critique

This paper presents a highly inspiring and innovative application of diffusion models to a critical problem in conversational AI and recommender systems. The idea of framing preference elicitation as a denoising process is elegant and conceptually powerful.

Strengths:

• Novelty: The diffusion-inspired framework for sequential question generation is a significant methodological contribution, offering a structured way to train LLMs for a complex conversational task.
• Addressing Cold-Start: By enabling effective PE, the method directly tackles the cold-start problem and enhances personalization, which is a long-standing challenge in RSs.
• Structured Questioning: The funneling effect achieved by the Questioner is a major win. It reflects a more intuitive and efficient conversational strategy for gathering information, improving user experience.
• Practical Training Data Generation: The use of a larger LLM to generate synthetic training data for a smaller LLM is a clever and scalable approach to developing specialized models without requiring extensive human annotation.
• Impact of User Simulator: The clear demonstration of how a fine-tuned user simulator improves evaluation validity and model performance is an important finding for future research in dialogue systems.

Critique/Areas for Improvement:

• Real User Validation: The most significant missing piece is validation with real human users. While LLM simulators are useful, they operate within a defined ground-truth and might not fully capture the variability, cognitive load, or subjective nature of human responses to clarifying questions. A user study would be crucial to confirm perceived helpfulness, satisfaction, and actual preference coverage.

• Complexity of Profile Representation: While JSON is structured, the conversion from natural language to JSON and the subsequent mapping $\mathcal{T}(Q_i, A_i)$ could introduce ambiguities or loss of nuanced information. The quality of this initial LLM-driven structuring is paramount.

• Bias Propagation: If the LLM used for initial profile generation or question generation (Gemini 2.0) carries biases, these could be propagated through the training data and affect the fine-tuned Questioner's questioning strategy, potentially leading to biased preference elicitation.

• Interpretability of Weighted Rank: While the Weighted Rank analysis for funneling is insightful, further qualitative analysis (e.g., examples of early vs. late questions generated by the model) could strengthen the argument for human-like conversational flow.

  Transferability: The core methodology of generating structured conversational training data via a forward (corruption) process and then fine-tuning an LLM in a reverse (denoising) process could be applied to various other dialogue system tasks where sequential information gathering is crucial. Examples include:

• Diagnostic Chatbots: Training chatbots to ask sequential diagnostic questions for medical or technical troubleshooting.

• Customer Service Automation: Enabling LLMs to efficiently gather necessary information from customers before escalating to human agents.

• Educational Tutors: Creating LLM-based tutors that ask adaptive questions to assess a student's understanding and guide their learning path.

  Overall, this paper provides a robust and promising framework that pushes the boundaries of LLM capabilities in conversational preference elicitation, opening exciting avenues for more intelligent and adaptive recommendation systems.