1. Bibliographic Information

1.1. Title

Query Understanding in LLM-based Conversational Information Seeking

1.2. Authors

• Yifei Yuan (University of Copenhagen, Denmark)
• Zahra Abbasiantaeb (University of Amsterdam, The Netherlands)
• Yang Deng (Singapore Management University, Singapore)
• Mohammad Aliannejadi (University of Amsterdam, The Netherlands)

1.3. Journal/Conference

Companion Proceedings of the ACM Web Conference 2025 (WWW Companion '25), April 28-May 2, 2025, Sydney, NSW, Australia.

The Web Conference (WWW) is a highly prestigious and top-tier conference in the fields of the World Wide Web, information retrieval, and related technologies. Publication at WWW signifies significant contributions and relevance to the web community. The "Companion Proceedings" typically include tutorials, workshops, and posters, which are integral parts of the conference, showcasing emerging topics and offering educational content.

1.4. Publication Year

2025

1.5. Abstract

The abstract outlines that Query Understanding in Conversational Information Seeking (CIS) is about accurately interpreting user intent using context-aware interactions. This involves resolving ambiguities, refining queries, and adapting to evolving information needs. Large Language Models (LLMs) significantly improve this process by interpreting complex language and adapting dynamically, leading to more relevant and precise search results in real-time. This tutorial will explore advanced techniques for enhancing query understanding in LLM-based CIS systems. It will cover LLM-driven methods for developing robust evaluation metrics for multi-turn interactions, strategies for building more interactive systems, and applications such as proactive query management and query reformulation. The tutorial also addresses key challenges in integrating LLMs for query understanding in conversational search and suggests future research directions. The primary goal is to deepen the audience's understanding of LLM-based conversational query understanding and stimulate discussions for continuous advancement in the field.

1.6. Original Source Link

• Original Source Link: https://arxiv.org/abs/2504.06356 (arXiv preprint, indicating it's a version submitted before or alongside peer review)
• PDF Link: https://arxiv.org/pdf/2504.06356v1.pdf (Direct PDF access on arXiv)

2. Executive Summary

2.1. Background & Motivation (Why)

The paper, a tutorial proposal, addresses the critical challenge of Query Understanding in Conversational Information Seeking (CIS) systems. Query understanding refers to a system's ability to accurately interpret a user's underlying intent, even when the query is incomplete, vague, or ambiguous. This is particularly complex in CIS because users often start with imprecise queries, refine them over multiple turns, ask follow-up questions, and may even shift context mid-conversation. Traditional Information Retrieval (IR) systems struggle with these dynamic, context-dependent interactions.

The emergence of Large Language Models (LLMs) (e.g., GPT-4) has revolutionized Natural Language Understanding (NLU) and IR, offering unprecedented capabilities in processing natural language and understanding context. LLMs have shown great promise in enhancing query understanding in CIS through areas like conversational context understanding, query clarification, user simulation, and query reformulation.

Despite these advancements, several significant gaps and challenges remain:

• (i) Robust Evaluation Metrics: It is difficult to effectively measure how well a system understands user intent across dynamic, multi-turn conversations.

• (ii) Improving Conversational Interaction: Making user-system exchanges smoother and more natural is still an open problem.

• (iii) Increasing User Proactivity: Encouraging users to actively refine and clarify their searches remains a challenge.

• (iv) Handling Ambiguity: LLMs need to balance providing appropriate responses with requesting clarifications when faced with vague or incomplete queries.

  This tutorial aims to address these critical challenges by providing advanced techniques and insights for improving query understanding specifically within LLM-based CIS systems.

2.2. Main Contributions / Findings (What)

As a tutorial proposal, the "contributions" are the key topics and structured knowledge it aims to deliver to its audience. The primary contributions of this tutorial are:

• Comprehensive Exploration of LLM-based Query Understanding: Providing a deep dive into how LLMs can be leveraged to enhance query understanding in CIS, moving beyond static Information Retrieval.

• Advanced Evaluation Techniques: Presenting LLM-driven methods for developing robust evaluation metrics to assess query understanding quality in multi-turn interactions, including both end-to-end evaluation and LLM-based relevance assessment.

• Strategies for Interactive Systems: Discussing LLM-based user simulation and multimodal conversational interactions to build more dynamic and natural CIS systems.

• Applications in Proactive Query Management: Covering techniques for unanswerable query mitigation (e.g., providing partial information, explanations, or useful query suggestions), uncertain query clarification (e.g., teaching LLMs to ask clarifying questions), and strategies for balancing user and system initiatives.

• Techniques for Query Enhancement: Exploring methods for resolving ambiguity in queries (e.g., query expansion, query refinement, follow-up question suggestion) and conversational query rewriting (e.g., handling low-resource scenarios, multimodal content, generating LLM-based answers).

• Identification of Key Challenges and Future Directions: Outlining current limitations in integrating LLMs for query understanding (e.g., multilingual and cross-cultural understanding, real-time adaptation to evolving user intent) and inspiring future research.

  The overall goal is to provide a structured, in-depth understanding of the current state and future potential of LLM-based conversational query understanding, fostering further discussions and advancements in the field.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully appreciate the content of this tutorial, a foundational understanding of several key concepts is beneficial:

• Information Retrieval (IR): The field concerned with organizing, storing, retrieving, and evaluating information, particularly from unstructured text documents. It's about finding relevant documents from a large collection in response to a user's query.
• Natural Language Processing (NLP): A subfield of Artificial Intelligence (AI) that deals with the interaction between computers and human language. It involves tasks like Natural Language Understanding (NLU) (interpreting meaning) and Natural Language Generation (NLG) (producing human-like text).
• Query Understanding: The process by which an IR system interprets the intent and context of a user's search query to provide more accurate and relevant results. This goes beyond simple keyword matching to grasp the deeper meaning.
• Conversational Information Seeking (CIS): An interactive form of IR where users engage in a dialogue with a system (e.g., a chatbot or virtual assistant) over multiple turns to refine their information needs. Unlike traditional IR, CIS involves context building, clarification, and adaptation throughout the conversation.
• Large Language Models (LLMs): These are highly advanced NLP models, typically based on the Transformer architecture, trained on massive amounts of text data. They can understand, generate, and process human-like text, demonstrating capabilities in tasks like question answering, summarization, translation, and more. Examples include GPT-3, GPT-4, LLaMA, etc. The Transformer architecture, for instance, introduced the self-attention mechanism, which allows the model to weigh the importance of different words in the input sequence when processing each word. $$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$ Where:
  • $Q$ (Query), $K$ (Key), $V$ (Value) are matrices derived from the input embeddings.
  • $Q K^T$ represents the dot product of the Query and Key matrices, capturing similarity.
  • $\sqrt{d_k}$ is a scaling factor to prevent large values in the dot product from pushing the softmax function into regions with tiny gradients.
  • $\mathrm{softmax}$ normalizes the scores to produce a distribution of weights.
  • The result is a weighted sum of the Value vectors, emphasizing relevant parts of the input.
• Context-aware Interactions: In CIS, the system needs to remember and utilize previous turns of a conversation to interpret the current query correctly. This implies maintaining a conversational context that evolves with each interaction.
• Ambiguity Resolution: The process of identifying and clarifying vague or unclear terms in a user's query. For example, if a user asks "What is the capital?", the system needs to understand "capital of what?" and might ask clarifying questions.
• Query Reformulation: The process of modifying or rephrasing a user's original query to improve the chances of retrieving more relevant information. This can involve adding keywords, removing irrelevant terms, or rephrasing the entire query based on conversational context.

3.2. Previous Works

The tutorial proposal references several key areas of prior research, highlighting how LLMs are building upon or transforming these existing challenges.

3.2.1. Query Understanding Evaluation

• End-to-End Evaluation: This involves using human-judged benchmarks to assess the relevance of query-passage pairs in CIS. Notable datasets and benchmarks include:
  • QReCC [10]: A large-scale open-domain conversational question-answering dataset.
  • TopioCQA [4]: Another large-scale open-domain conversational question-answering dataset with topic switching.
  • TREC CAsT 19-22 [19, 35]: The Text REtrieval Conference (TREC) Conversational Assistance Track, which focuses on complex, knowledge-intensive conversations.
  • TREC iKAT 23 [5]: The Interactive Knowledge Assistant Track at TREC, designed to evaluate personalized conversational search systems.
• LLM-based Relevance Assessment: Leveraging LLMs to evaluate the relevance of retrieved information to a user's query [2, 27, 33]. This is a newer approach but comes with challenges such as non-reproducibility, unpredictable outputs, and potential data leakage between training and inference stages [37].

3.2.2. LLM-based Conversational Interaction

• LLM-based User Simulation: Simulating diverse user behaviors, intents, and query patterns helps LLMs anticipate real-world scenarios, improving their ability to handle complex queries [52, 53]. This is crucial for evaluating systems across various domains like information-seeking dialogues [40, 43], conversational question-answering [3], and task-oriented dialogues [44].
• Multimodal Conversational Interactions: Expanding CIS beyond text to include other content types like images or audio [24, 48, 58, 60]. This allows LLMs to interpret and respond across different media, enhancing applications in e-commerce, healthcare, and spatial analysis.

3.2.3. LLM-based Proactive Query Management

This area moves CIS systems from passive to proactive responses.

• Unanswerable Query Mitigation: Instead of just responding with "No Answer" [16], proactive systems can provide partially relevant information [55], explanations for unanswerability [22], or suggest useful alternative queries [41, 47].
• Uncertain Query Clarification: Systems can ask clarifying questions when uncertain about user intent [8, 9, 61]. LLMs can be trained to do this through methods like in-context learning [20], self-learning [11], reinforcement learning [15], contrastive learning [14], and in multimodal scenarios [60].
• Balancing User and System Initiatives: Determining "when" a system should take the initiative (e.g., asking a clarifying question or offering a suggestion) is critical, as taking the initiative inappropriately can harm user experience [62] or not improve retrieval [28]. Research explores predicting when to take initiative [32, 50] and simulating user-system interactions [6] to understand these dynamics.

3.2.4. LLM-based Query Enhancement

Modifying user queries to improve retrieval performance.

• Resolving Ambiguity: Techniques like query expansion [25, 51], query refinement [23], and follow-up question suggestion [12] help clarify vague queries.
• Conversational Query Rewriting: Rephrasing queries within a conversational context to improve accuracy and relevance [39, 49]. LLMs enhance this by handling low-resource (few-shot or zero-shot) scenarios [31, 56, 57], incorporating multimodal content [59], and generating LLM-based answers for better retrieval [1].

3.3. Technological Evolution

The field of Information Retrieval has evolved significantly. Initially dominated by keyword-based search and Boolean logic, it progressed to statistical models (like TF-IDF), then to learning-to-rank methods and neural IR models. The advent of deep learning and particularly Transformer models paved the way for Large Language Models.

Before LLMs, Query Understanding relied on techniques like query expansion using thesauri, named entity recognition, and part-of-speech tagging. Conversational Search systems were often rule-based or employed simpler dialogue state tracking mechanisms. The breakthrough of LLMs brought a paradigm shift: their ability to understand nuance, generate coherent text, and learn from massive data allowed for a more flexible, context-aware, and human-like interaction. They moved query understanding from rule-based or statistical pattern matching to semantic understanding and dynamic adaptation, greatly enhancing the potential for truly intelligent CIS systems.

3.4. Differentiation

The authors explicitly differentiate this tutorial from related ones presented at recent conferences:

• Conversational Information Seeking: Theory and Application (SIGIR22) [18]

• Proactive Conversational Agents in the Post-ChatGPT World (SIGIR23) [30]

• Large Language Model Powered Agents in the Web (WWW24) [21]

• Tutorial on User Simulation for Evaluating Information Access Systems on the Web (WWW24) [13]

  While these tutorials cover broad aspects of conversational IR and agent-based interactions, the distinguishing factor of the proposed tutorial is its specific and focused emphasis on enhancing query understanding within LLM-based conversational IR systems and beyond. This narrower, yet deeper, focus allows for a more detailed exploration of the mechanisms and challenges of how LLMs interpret and refine user queries in dynamic, multi-turn contexts, which is a core component of effective CIS.

4. Methodology

This paper is a tutorial proposal, not a research paper presenting novel methodology or empirical findings. Therefore, the "methodology" section describes the structure and content that the authors plan to present in their tutorial.

4.1. Principles

The core principle underpinning this tutorial is that Large Language Models (LLMs) offer significant advancements for Query Understanding in Conversational Information Seeking (CIS). The tutorial aims to systematically explore and explain how LLMs can be leveraged to address existing challenges in CIS, focusing on improving the accuracy and relevance of information retrieval by deeply interpreting user intent within dynamic, context-rich conversations. The intuition is that by understanding the nuances of language and adapting to evolving user needs, LLMs can make CIS systems much more effective and user-friendly.

4.2. Steps & Procedures

The tutorial is structured as a lecture-style presentation covering the latest advancements. The proposed schedule, outlining the flow of topics and their allocated time, serves as the "steps and procedures" of the tutorial delivery:

• Introduction (20 min):
  • Ad-hoc search: Briefly introduce traditional, single-turn search.
  • Preliminary of query understanding: Define query understanding in its basic form.
  • Adapting LLMs in query understanding: Explain how LLMs bring new capabilities to query understanding in conversational settings.
• Part I: Conversational Query Understanding Evaluation (30 min):
  • End-to-end evaluation: Discuss how CIS systems are evaluated using human-judged benchmarks that assess the relevance of query-passage pairs across conversations.
  • LLM-based relevance assessment: Explore methods of using LLMs themselves to judge the relevance of retrieved information, along with their associated challenges.
• Part II: LLM-based Conversational Interaction (30 min):
  • LLM-based user simulation: Detail how LLMs can simulate diverse user behaviors to test and improve CIS systems.
  • Multimodal conversational interaction: Discuss integrating non-textual data (e.g., images, audio) into CIS interactions and how LLMs handle these.
• Part III: LLM-based Proactive Query Management (40 min):
  • Unanswerable query mitigation: Explain strategies for systems to handle queries for which direct answers are not available, such as providing partial information or explanations.
  • Uncertain query clarification: Describe how LLMs can be taught to ask clarifying questions when user intent is ambiguous.
  • Balancing user and system initiatives: Discuss the critical aspect of determining when a CIS system should proactively intervene versus letting the user drive the conversation.
• Part IV: LLM-based Query Enhancement (30 min):
  • Resolving ambiguity in queries: Cover techniques like query expansion, query refinement, and follow-up question suggestion powered by LLMs.
  • Conversational query rewrite techniques: Present methods for LLMs to rephrase or modify user queries within a conversation to improve retrieval accuracy.
• Summary and Outlook (30 min):
  • Open challenges and beyond: Discuss current limitations and significant future research directions in LLM-based conversational query understanding.

4.3. Mathematical Formulas & Key Details

As this is a tutorial proposal and not a research paper detailing a specific method, it does not present new mathematical formulas or algorithms. The content described in the "Steps & Procedures" outlines the topics that will be discussed during the tutorial, which would likely involve explaining existing techniques, models, and evaluation metrics, some of which may include mathematical formulations. However, the tutorial proposal itself does not introduce or reproduce any specific mathematical formulas.

5. Experimental Setup

As this document is a tutorial proposal, it does not involve an experimental setup in the traditional sense of a research paper. Tutorial proposals outline educational content and do not present new empirical results from experiments. Therefore, there are no sections on Datasets, Evaluation Metrics, or Baselines to report here. The paper discusses evaluation metrics and datasets within the context of what will be covered during the tutorial (e.g., QReCC, TREC CAsT for evaluation), but it does not present a specific experimental setup or results using them.

6. Results & Analysis

Similar to the Experimental Setup section, this tutorial proposal does not present any empirical results or analyses of experimental data. Its purpose is to outline the scope, structure, and content of an educational tutorial on LLM-based conversational query understanding. Therefore, there are no Core Results, Data Presentation (Tables), or Ablations / Parameter Sensitivity to report from this document. The tutorial itself, when delivered, would likely refer to results from existing research papers to illustrate points, but these are not contained within the proposal document itself.

7. Conclusion & Reflections

7.1. Conclusion Summary

This tutorial proposal clearly outlines a timely and relevant deep-dive into Query Understanding within LLM-based Conversational Information Seeking (CIS). It systematically covers various facets, from evaluation methodologies to practical applications like proactive query management and query enhancement, all powered by the capabilities of Large Language Models (LLMs). The authors aim to provide a comprehensive overview of existing techniques, discuss the challenges that LLMs bring, and inspire future research directions. The tutorial's detailed schedule promises a structured exploration of how LLMs are transforming the way systems interpret and respond to user intent in conversational contexts, ultimately leading to more accurate and relevant information retrieval.

7.2. Limitations & Future Work

The authors acknowledge several open challenges in integrating LLMs for query understanding in conversational search systems, which also serve as critical future research directions:

• Multilingual and Cross-Cultural Query Understanding: While LLMs perform well in English, their capabilities need to be expanded to effectively support diverse languages and cultural nuances. This is essential for fostering inclusive and accurate search experiences globally.

• Real-time Adaptation to Evolving User Intent: Developing models that can dynamically adjust search strategies in real-time as user intent shifts throughout a conversation remains a significant challenge. Instructing LLMs to accurately detect and adapt to these shifts is a crucial future direction for truly adaptive CIS systems.

  These limitations highlight that despite the powerful capabilities of LLMs, significant research is still needed to achieve truly robust, globally applicable, and dynamically adaptive conversational information seeking.

7.3. Personal Insights & Critique

This tutorial proposal is well-structured and addresses a highly relevant and rapidly evolving area in Information Retrieval and NLP. The focus on query understanding as the core mechanism for successful CIS is appropriate, given its foundational role in delivering relevant results.

The choice of LLMs as the central technology is timely, reflecting the current state of AI research and deployment. By breaking down the complex problem into evaluation, interaction, proactive management, and enhancement, the tutorial provides a holistic view of the query understanding pipeline in CIS. The explicit differentiation from other related tutorials is valuable, ensuring that the audience understands the unique contribution and focus of this particular offering.

One area that could be particularly impactful for future work, building on the tutorial's foundation, is the ethical considerations and potential biases in LLM-based query understanding. As LLMs are trained on vast datasets, they can inadvertently perpetuate societal biases, which might lead to unfair or discriminatory search results or clarifications. While not explicitly listed as a challenge, addressing fairness and interpretability in LLM-driven CIS could be a crucial next step for the field.

Additionally, the practical aspects of LLM deployment in real-world CIS systems, such as computational cost, latency, and scalability for real-time interactions, are important considerations. The tutorial touches on real-time adaptation, but deeper discussions around efficient inference and deployment strategies for LLMs in CIS could also be beneficial.

Overall, this tutorial promises to be a valuable resource for anyone working in or interested in the intersection of LLMs, conversational AI, and information seeking, potentially driving significant advancements in how humans interact with information systems on the web. The expertise of the presenters, as evidenced by their biographies and extensive publication records in these fields, further solidifies the potential impact and quality of this tutorial.