1. Bibliographic Information

• Title: Knowledge Boundary of Large Language Models: A Survey
• Authors: Moxin Li, Yong Zhao, Wenxuan Zhang, Shuaiyi Li, Wenya Xie, See-Kiong Ng, Tat-Seng Chua, and Yang Deng.
• Affiliations: The authors are affiliated with several prominent research institutions, including the National University of Singapore, Singapore Management University, Singapore University of Technology and Design, The Chinese University of Hong Kong, and the University of Minnesota - Twin Cities.
• Journal/Conference: This paper is a preprint available on arXiv. It has not yet been formally published in a peer-reviewed journal or conference at the time of this analysis.
• Publication Year: 2024 (v2 submitted in December 2024).
• Abstract: The paper addresses the limitations of Large Language Models (LLMs) in memorizing and using knowledge, which leads to untruthful or inaccurate responses. It argues that the concept of an LLM's "knowledge boundary" is not well-defined in current research. The authors propose a new, comprehensive definition and a formalized four-type taxonomy of knowledge. Based on this framework, the survey systematically reviews the field by examining the motivations for studying knowledge boundaries, the methods for identifying them, and the strategies for mitigating the problems they cause. The paper concludes by discussing open challenges and future research directions, aiming to provide a foundational overview for the research community.
• Original Source Link:
  • arXiv Link: https://arxiv.org/abs/2412.12472v2
  • PDF Link: http://arxiv.org/pdf/2412.12472v2

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: Despite their impressive capabilities, Large Language Models (LLMs) frequently generate factually incorrect information (hallucinations), are easily misled by false context, and exhibit overconfidence when faced with questions they don't know the answer to. These failures stem from a fundamental lack of awareness of their own knowledge limitations.
  • Gaps in Prior Work: Previous attempts to define the "knowledge boundary" of LLMs have been largely conceptual and lack formalization (e.g., the "Know-Unknow Quadrant"). Existing surveys on related topics are either too narrow, focusing only on specific mitigation strategies like abstention, or lack a clear, structured definition of the boundary itself.
  • Innovation: This paper introduces a formalized, multi-layered definition of the LLM knowledge boundary. This provides a rigorous framework to systematically categorize knowledge, understand the root causes of LLM failures, and organize existing research on identification and mitigation strategies.

• Main Contributions / Findings (What):

  • A Novel Taxonomy of Knowledge Boundaries: The paper defines three nested boundaries:
    1. Universal Knowledge Boundary: All knowledge known to humanity.
    2. Parametric Knowledge Boundary: Knowledge abstractly encoded within an LLM's parameters.
    3. Outward Knowledge Boundary: Knowledge that an LLM can reliably and correctly generate in response to queries.
  • A Formalized Four-Type Knowledge Categorization: Based on the boundaries, knowledge is classified into four types:
    1. Prompt-Agnostic Known Knowledge (PAK): Reliably known, regardless of prompt phrasing.
    2. Prompt-Sensitive Known Knowledge (PSK): Stored in parameters but only accessible with the right prompt.
    3. Model-Specific Unknown Knowledge (MSU): Knowledge that is known to humans but not stored in a specific LLM (e.g., recent or domain-specific facts).
    4. Model-Agnostic Unknown Knowledge (MAU): Knowledge unknown to humanity itself (e.g., future events, unsolved problems).
  • A Systematic Literature Review: The survey is structured around three key research questions, providing a comprehensive overview of:
    1. Why study knowledge boundaries (analyzing undesired behaviors like hallucinations and context-misled responses).
    2. How to identify these boundaries (using techniques like uncertainty estimation, confidence calibration, and internal state probing).
    3. How to mitigate the issues they cause (summarizing strategies tailored to each knowledge type).
  • Future Research Directions: The paper highlights critical open challenges, including the need for better benchmarks, the potential to utilize knowledge boundary awareness to improve LLM performance, and the unintended side effects of mitigation strategies like over-refusal.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Large Language Model (LLM): A type of artificial intelligence model trained on vast amounts of text data to understand and generate human-like language. LLMs "store" knowledge implicitly within their billions of numerical parameters.
  • Knowledge Boundary: A conceptual limit defining what an LLM knows, what it thinks it knows, and what it doesn't know. Understanding this boundary is crucial for ensuring LLM reliability.
  • Hallucination: A phenomenon where an LLM generates text that is factually incorrect, nonsensical, or not grounded in its training data, yet presents it as factual.
  • Prompt Engineering: The practice of carefully designing input text (prompts) to elicit desired or more accurate responses from an LLM. This is particularly relevant for accessing Prompt-Sensitive Known Knowledge.
  • Uncertainty Estimation (UE): A set of techniques used to quantify how "sure" a model is about its prediction. High uncertainty can signal that the model is operating outside its knowledge boundary.
  • Confidence Calibration: The process of aligning a model's predicted confidence score (e.g., the probability it assigns to an answer) with its actual correctness rate. A well-calibrated model's confidence is a reliable indicator of its accuracy.
  • Retrieval-Augmented Generation (RAG): A technique that enhances LLMs by allowing them to retrieve information from an external knowledge base (like a database or the internet) before generating a response. This helps address Model-Specific Unknown Knowledge.

• Previous Works: The paper positions its contribution by critiquing existing frameworks:

  • The Know-Unknow Quadrant (Yin et al., 2023): This framework categorizes knowledge into four types: known-knowns, known-unknowns, unknown-knowns, and unknown-unknowns. The authors argue this is a useful high-level concept but lacks the formal, mathematical definition needed for rigorous analysis.
  • Work by Yin et al. (2024): This work introduced a formalized definition but was criticized for focusing only on a specific LLM and lacking comprehensiveness.
  • Other Surveys (Li et al., 2024e; Wen et al., 2024b): The authors note that these surveys either lack a clear definition of the knowledge boundary or focus too narrowly on a single mitigation strategy like abstention (refusing to answer).

• Differentiation: This survey's primary innovation is its comprehensive and formalized taxonomy. Unlike prior work, it doesn't just categorize knowledge conceptually but provides mathematical definitions that distinguish between knowledge universally available, knowledge stored in the model, and knowledge the model can actually use. This structured approach allows for a more organized and insightful review of the existing literature on identifying and mitigating knowledge-related failures in LLMs.

4. Methodology (Core Technology & Implementation)

The core methodological contribution of this survey is its novel, formalized definition of the LLM knowledge boundary and the resulting four-part knowledge taxonomy. This framework serves as the foundation for the entire paper.

• Principles: The central idea is that an LLM's knowledge exists in nested layers. The boundaries between these layers define what is accessible and what is not. The paper formalizes this intuition by defining three distinct boundaries.

• The Three Knowledge Boundaries:

  Image 3: A visual representation of the proposed knowledge taxonomy. (a) shows the nested structure of the three knowledge boundaries and the four resulting knowledge types. (b) provides example queries for each knowledge type, illustrating how an LLM's response differs based on where the required knowledge falls within the boundaries.

  1. Universal Knowledge Boundary: This is the outermost boundary, encompassing the entire set of abstract knowledge known to humanity, denoted as $\mathcal{K}$. It represents everything that could theoretically be answered.
  2. Parametric Knowledge Boundary: This boundary encloses the knowledge that is abstractly stored within a specific LLM's parameters ($\theta$). Knowledge within this boundary can be verified by at least one possible prompt.
  3. Outward Knowledge Boundary: This is the innermost and most practical boundary. It defines the knowledge that an LLM can consistently generate correct outputs for, given a limited set of test prompts ($\hat{Q}_k$). This is what is empirically observable.

• The Four-Type Knowledge Taxonomy: Based on these boundaries, the paper provides formal definitions for four types of knowledge. Let $k$ be a piece of knowledge, $Q_k$ be the set of all possible question-answer pairs (q, a) expressing $k$, $\hat{Q}_k \subseteq Q_k$ be a test set of such pairs, and $P_{\theta}(a|q) > \epsilon$ mean the LLM correctly answers question $q$ with high confidence.

  1. Prompt-Agnostic Known Knowledge (PAK): Knowledge that is robustly known and can be correctly recalled regardless of how it's prompted (within the test set). $$K _ { \mathsf { P A K } } = \{ k \in \mathcal { K } | \forall ( q _ { k } ^ { i } , a _ { k } ^ { i } ) \in \hat { Q } _ { k } , P _ { \theta } ( a _ { k } ^ { i } | q _ { k } ^ { i } ) > \epsilon \}$$

     • Explanation: For a piece of knowledge $k$ to be PAK, the LLM must generate the correct answer for all (∀) tested prompts in $\hat{Q}_k$.

  2. Prompt-Sensitive Known Knowledge (PSK): Knowledge that exists within the LLM's parameters but is "brittle." The LLM fails on some prompts but can succeed if prompted in just the right way. $$\begin{array} { r } { K \mathsf { { p s x } } = \{ k \in \mathcal { K } | ( \exists ( q _ { k } ^ { i } , a _ { k } ^ { i } ) \in Q _ { k } , P _ { \theta } ( a _ { k } ^ { i } | q _ { k } ^ { i } ) > \epsilon ) } \\ { \land ( \exists ( q _ { k } ^ { i } , a _ { k } ^ { i } ) \in \hat { Q } _ { k } , P _ { \theta } ( a _ { k } ^ { i } | q _ { k } ^ { i } ) < \epsilon ) \} } \end{array}$$