1. Bibliographic Information

1.1. Title

The central topic of the paper is "CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAG". It focuses on improving the robustness of Retrieval-Augmented Generation (RAG) models against misinformation present in external documents by dynamically adjusting the influence of these documents based on their credibility.

1.2. Authors

The authors of the paper are:

• Boyi Deng (University of Science and Technology of China)

• Wenjie Wang (National University of Singapore)

• Fengbin Zhu (National University of Singapore)

• Qifan Wang (Meta AI)

• Fuli Feng (University of Science and Technology of China)

  Their affiliations indicate a collaboration between academic institutions and a leading AI research company, suggesting a blend of theoretical rigor and practical relevance. Wenjie Wang and Fengbin Zhu are marked with an asterisk, typically denoting co-corresponding authors.

1.3. Journal/Conference

The paper does not explicitly state the journal or conference it was published in, but the presence of an arXiv link and a publication UTC date suggests it might be a preprint or submitted to a major conference in the field of Natural Language Processing (NLP) or Machine Learning. Given the topic, venues like ACL, EMNLP, NeurIPS, or ICML would be relevant and influential in this domain.

1.4. Publication Year

The paper was published at (UTC): 2025-04-11T00:00:00.000Z, indicating a publication year of 2025.

1.5. Abstract

The abstract introduces Retrieval-Augmented Generation (RAG) as a method to mitigate Large Language Model (LLM) hallucinations by referencing external documents. However, it highlights a crucial problem: misinformation in these external documents can mislead LLMs. To address this, the paper proposes "credibility-aware RAG," where LLMs dynamically adjust the influence of retrieved documents based on their credibility scores. The authors introduce a plug-and-play method called Credibility-aware Attention Modification (CrAM). CrAM works by identifying influential attention heads within LLMs and modifying their attention weights according to the credibility of the associated documents, thereby diminishing the impact of low-credibility information. Experiments conducted on the Natural Questions (NQ) and TriviaQA datasets using Llama2-13B, Llama3-8B, and Qwen1.5-7B demonstrate that CrAM significantly improves RAG performance against misinformation by over 20%, even outperforming supervised fine-tuning (SFT) methods.

1.6. Original Source Link

The original source link for the paper is /files/papers/690cd61f0de225812bf9335e/paper.pdf. This appears to be a direct link to the PDF file, likely hosted on a repository like arXiv or a conference proceeding archive. The publication status is "preprint" or "published in proceedings" based on the URL structure and the academic context.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the vulnerability of Retrieval-Augmented Generation (RAG) systems to misinformation present in external knowledge sources. While RAG effectively reduces hallucinations in Large Language Models (LLMs) by providing factual context, the quality of this context is paramount. If the retrieved documents contain misinformation, LLMs can be misled into generating unfaithful or incorrect responses. This is a significant concern as misinformation pollution is prevalent in online data, as demonstrated by instances like Microsoft's Bing being misled and research showing that LLM-generated misinformation can degrade RAG performance.

Prior research has focused on misinformation detection to measure document credibility. A straightforward approach would be to simply discard low-credibility documents. However, the paper points out that directly discarding documents might lead to a loss of relevant and important information, potentially degrading overall performance. This highlights a crucial gap: while credibility scores can be obtained, effective mechanisms for LLMs to utilize these scores without outright discarding information are underdeveloped.

The paper's innovative entry point is to explore "credibility-aware RAG," where LLMs can automatically adjust the influence of retrieved documents based on their credibility scores, rather than a binary accept/reject decision. This allows for a more nuanced handling of potentially compromised information. Previous attempts in this direction relied on supervised fine-tuning (SFT), which is resource-intensive and requires specialized training data, limiting its broad applicability. The paper therefore seeks a non-SFT, plug-and-play solution.

2.2. Main Contributions / Findings

The paper makes several primary contributions to address the challenge of misinformation in RAG systems:

• Exploration of Credibility-Aware RAG without Fine-tuning: The authors formally define and explore the task of credibility-aware RAG as a way to alleviate misinformation pollution without requiring computationally expensive and data-intensive fine-tuning of LLMs. This approach aims for a more practical and adaptable solution.
• Introduction of CrAM (Credibility-aware Attention Modification): They propose a novel plug-and-play method called CrAM. This method enhances LLMs with credibility-aware RAG capabilities by:
  1. Identifying Influential Attention Heads: CrAM selects specific attention heads within the LLM that have a significant impact on generating incorrect answers when misinformation is present. This is achieved using a modified causal tracing approach.
  2. Modifying Attention Weights: For these identified influential heads, CrAM adjusts their attention weights based on the credibility scores of the retrieved documents. This mechanism reduces the attention paid to low-credibility documents, effectively mitigating their misleading influence.
• Extensive Experimental Validation and Superior Performance: The paper conducts comprehensive experiments on two open-domain Question Answering (QA) datasets (Natural Questions and TriviaQA) using three popular LLMs (Llama2-13B, Llama3-8B, and Qwen1.5-7B).

  • Key Findings: CrAM significantly improves the Exact Match (EM) and F1 Score performance of RAG systems by over 20% compared to vanilla RAG when facing misinformation.

  • Outperformance of SFT-based Methods: Notably, CrAM often surpasses supervised fine-tuning (SFT)-based methods (like CAG) in most scenarios, demonstrating its efficiency and effectiveness without the need for extensive retraining.

  • Robustness: CrAM shows robustness to varying numbers of low-credibility documents and minor sensitivity to the size of the dataset used for identifying influential heads.

    These contributions offer a practical and effective strategy for making RAG systems more resilient to misinformation, which is a critical step towards building more trustworthy AI applications.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully understand the CrAM paper, it is essential to grasp several foundational concepts in Large Language Models (LLMs) and Natural Language Processing (NLP).

3.1.1. Large Language Models (LLMs)

Large Language Models (LLMs) are advanced artificial intelligence models, typically based on the transformer architecture, that have been trained on vast amounts of text data. They are capable of understanding, generating, and processing human language for a wide range of tasks, such as question answering, text summarization, translation, and creative writing. LLMs like GPT-3/4, Llama, and Qwen are characterized by their massive scale (billions of parameters) and emergent abilities.

3.1.2. Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is a technique that enhances LLMs by providing them with access to external, up-to-date, and domain-specific information. Instead of relying solely on the knowledge memorized during their pre-training, RAG systems first retrieve relevant documents or passages from a large corpus (e.g., Wikipedia, a company's internal documents) based on a user's query. These retrieved documents are then fed to the LLM along with the original query, allowing the model to generate more accurate, factual, and contextually rich responses. This process helps alleviate hallucinations (generating false or nonsensical information) and keeps the LLM's knowledge current.

3.1.3. Hallucinations in LLMs

Hallucinations in LLMs refer to the phenomenon where the model generates information that is factually incorrect, nonsensical, or unfaithful to the provided source context, despite presenting it confidently. This can arise from limitations in their training data, biases, or simply from the generative nature of the models, which prioritize fluency and coherence over strict factual accuracy. RAG was developed as a primary method to combat these hallucinations by grounding responses in external, verifiable information.

3.1.4. Attention Mechanism

The attention mechanism is a core component of transformer models, which are the backbone of most LLMs. It allows the model to weigh the importance of different parts of the input sequence when processing a specific part of the sequence. Instead of processing all input tokens equally, attention enables the model to focus on the most relevant tokens for a given task or context.

The most common form is self-attention, which calculates the attention weights between all pairs of tokens in a single input sequence. It works by computing three vectors for each token:

• Query (Q): Represents the current token being processed.

• Key (K): Represents all other tokens in the sequence.

• Value (V): Represents the actual information content of all other tokens.

  The attention score between a query token and a key token indicates how much attention the query token should pay to the key token. These scores are then normalized (typically with a softmax function) to create attention weights, which are then used to compute a weighted sum of the value vectors.

The standard scaled dot-product attention formula is: $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ Where:

• $Q$ is the matrix of queries.
• $K$ is the matrix of keys.
• $V$ is the matrix of values.
• $Q^T$ is the transpose of the matrix of queries.
• $d_k$ is the dimension of the key vectors, used for scaling to prevent very large dot products that could push the softmax function into regions with tiny gradients.
• softmax is an activation function that normalizes the scores into probabilities (weights) that sum to 1.

3.1.5. Multi-Head Attention

Multi-head attention is an extension of the attention mechanism where the query, key, and value vectors are projected multiple times (into different "heads") using different learned linear transformations. Each attention head then independently computes its own attention output. The outputs from all attention heads are then concatenated and linearly transformed again to produce the final output. This allows the model to capture different types of relationships or focus on different aspects of the input simultaneously, similar to how different filters in a Convolutional Neural Network (CNN) might detect different features. Different attention heads can specialize in different tasks, e.g., some might focus on syntactic relationships, others on semantic ones.

3.1.6. Supervised Fine-Tuning (SFT)

Supervised Fine-Tuning (SFT) is a common technique in LLM training where a pre-trained LLM is further trained on a smaller, task-specific dataset with labeled examples. The goal of SFT is to adapt the general knowledge of the pre-trained LLM to a specific downstream task (e.g., sentiment analysis, summarization, or, in this context, credibility-aware RAG). While powerful, SFT requires significant computational resources (GPUs) and a meticulously curated, high-quality labeled dataset for the specific task.

3.1.7. Causal Tracing

Causal tracing is a technique developed to understand the internal workings of neural networks, particularly transformers. It aims to quantify the causal contribution of specific internal states (e.g., hidden states, attention outputs) to the model's final output. It typically involves running the model twice: once normally to get a baseline probability for a target output, and once with noise injected into a specific internal state (or "intervention") to observe how the target output's probability changes. The difference in probabilities then quantifies the indirect effect (IE) or contribution of that internal state. It helps pinpoint which parts of the network are responsible for specific behaviors or predictions.

3.2. Previous Works

The paper contextualizes its work by reviewing existing research in misinformation detection and combating misinformation in RAG.

3.2.1. Misinformation Detection

• Non-LLM-based methods: These methods train models specifically to identify false or misleading information. Examples include using BERT (Devlin et al. 2019) to score document credibility (Kaliyar, Goswami, and Narang 2021) or Graph Neural Networks for detection (Vaibhav, Mandyam, and Hovy 2019).
• LLM-based methods: More recent approaches leverage LLMs themselves, often without additional training, to assess credibility. For instance, GPT-4 has been used for document credibility scoring (Pelrine et al. 2023), and LLM agents for iterative verification (Quelle and Bovet 2024). The CrAM paper adopts a similar LLM-based approach (using gpt-3.5-turbo-0125) for generating credibility scores in its experimental setup.

3.2.2. Combating Misinformation in RAG

The paper acknowledges that RAG's vulnerability to misinformation has been identified (Zou et al. 2024; Pan et al. 2023b,a), leading to various mitigation strategies:

• Query Augmentation and Voting: CAR (Weller et al. 2024) retrieves a larger set of documents and uses a voting mechanism to reduce misinformation impact. This approach, however, often involves multiple rounds of model inference, leading to inefficiency.
• Independent Response Aggregation: RobustRAG (Xiang et al. 2024) generates LLM responses for each document independently and then aggregates them using keyword-based or decoding-based algorithms. Similar to CAR, this can be inefficient due to multiple inferences.
• Supervised Fine-Tuning (SFT) with Credibility Scores: Several works (Hong et al. 2024; Pan et al. 2024) propose assigning credibility scores to retrieved documents and then fine-tuning LLMs to understand and leverage these scores during generation. An example is CAG (Pan et al. 2024), which directly incorporates credibility scores and documents into prompts for fine-tuning. While effective, SFT methods are resource-intensive and require specialized training data.
• Knowledge Conflict Resolution: $CD^2$ (Jin et al. 2024) involves training two LLMs—one for truthful answers and one for misleading answers—to better distinguish conflicting information. This also falls under fine-tuning approaches.

3.3. Technological Evolution

The evolution of LLM research related to factual accuracy can be broadly traced as follows:

1. Early LLMs: Primarily focused on language generation and understanding, often suffering from hallucinations due to reliance on internal, sometimes outdated or biased, parametric knowledge.
2. Emergence of RAG: To combat hallucinations and provide up-to-date information, RAG systems were developed. These systems augment LLMs with external knowledge retrieval capabilities, improving factual grounding.
3. Discovery of RAG's Vulnerability: Researchers soon identified that RAG systems are susceptible to misinformation pollution in their external corpora, turning a solution into a new problem.
4. Early Mitigation Strategies: Initial efforts focused on misinformation detection (pre-filtering documents) or SFT methods to teach LLMs how to weigh document credibility. However, pre-filtering risks losing relevant information, and SFT is resource-heavy.
5. CrAM's Contribution: This paper fits into the latest stage by proposing a non-SFT, plug-and-play method that leverages the existing attention mechanism within LLMs to dynamically adjust document influence based on credibility scores. This represents a more efficient and adaptable solution compared to prior SFT-based approaches.

3.4. Differentiation Analysis

Compared to the main methods in related work, CrAM offers distinct innovations:

• Non-SFT (Plug-and-Play) vs. Supervised Fine-Tuning (SFT): The primary differentiation is CrAM's plug-and-play nature, meaning it does not require fine-tuning the LLM. This stands in stark contrast to CAG (Pan et al. 2024), Hong et al. (2024), and Jin et al. (2024), which necessitate extensive computational resources and carefully curated training data for SFT. CrAM's approach is more practical for real-world deployment where fine-tuning may be prohibitive.

• Dynamic Influence Adjustment vs. Document Exclusion: Unlike "Exclusion" methods that discard documents below a certain credibility threshold, CrAM dynamically scales down the influence of low-credibility documents while retaining them. This prevents the loss of potentially useful information that might be present alongside misinformation, as highlighted in the paper (Yoran et al. 2024).

• Targeted Attention Modification vs. Prompt Engineering: While "Prompt Based" methods attempt to inform the LLM about credibility via prompts, CrAM directly intervenes in the LLM's internal attention mechanism. By identifying and modifying influential attention heads, CrAM offers a more granular and potentially more effective control over how the LLM processes document credibility, moving beyond surface-level prompt instructions.

• Efficiency vs. Multiple Inferences: Compared to methods like CAR (Weller et al. 2024) and RobustRAG (Xiang et al. 2024) that require multiple rounds of LLM inference or complex aggregation strategies, CrAM operates within a single inference pass by modifying the attention weights of existing attention heads, making it more computationally efficient.

• Leveraging Internal LLM Structure: CrAM uniquely exploits the heterogeneous roles of different attention heads within the transformer architecture. By identifying influential heads that contribute to misinformation-induced errors, it applies modifications precisely where they are most impactful, a nuance not explicitly addressed by other methods that either fine-tune the entire model or rely on external filtering.

  In essence, CrAM provides an efficient, adaptable, and targeted approach to credibility-aware RAG by modifying the LLM's internal attention process, circumventing the limitations of SFT and brute-force document filtering.

4. Methodology

The CrAM method is designed to enable Large Language Models (LLMs) to automatically adjust their reliance on retrieved documents based on their credibility scores, particularly reducing the impact of low-credibility documents. It operates in a plug-and-play manner, meaning it does not require fine-tuning the LLM. The core idea revolves around modifying the attention weights of specific, influential attention heads within the LLM.

4.1. Principles

The fundamental principle behind CrAM is that not all parts of an LLM's attention mechanism contribute equally to processing information, especially when dealing with misinformation. Some attention heads might be more susceptible to misinformation or play a larger role in incorporating document information into the generated output. By identifying these influential attention heads and then dynamically scaling their attention weights based on the credibility scores of the documents, CrAM can selectively reduce the LLM's focus on low-credibility content without discarding it entirely. This targeted intervention aims to nudge the LLM towards generating more factual responses.

4.2. Core Methodology In-depth (Layer by Layer)

The CrAM methodology can be broken down into two main phases: Influential Head Identification and Attention Weight Modification, followed by its application in the CrAM Workflow.

4.2.1. Credibility-Aware RAG Formal Definition

The paper formally defines the objective of credibility-aware RAG. Given an LLM $L$, a user query $x$, and a set of relevant documents $\mathcal{D} = \{d_1, d_2, \dots, d_n\}$ associated with credibility scores $\mathcal{S} = \{s_1, s_2, \dots, s_n\}$, the goal is to enable LLMs to automatically adjust the influence of these documents on the generated output $y$ based on their credibility scores $\mathcal{S}$. This is formally expressed as:

$$\operatorname* { m a x } \mathrm { ~ M e t r i c } ( \operatorname { C o m b i n e } ( L , x , \mathcal { D } , \mathcal { S } ) )$$

Where:

• $L$: Represents the Large Language Model being used.
• $x$: Denotes the user query.
• $\mathcal{D}$: Is the set of retrieved documents relevant to the query $x$.
• $\mathcal{S}$: Is the set of credibility scores corresponding to each document in $\mathcal{D}$.
• $\operatorname{Combine}(L, x, \mathcal{D}, \mathcal{S})$: Represents the method or mechanism through which the credibility scores are integrated into the LLM's generation process. For CrAM, this involves the attention modification.
• $\operatorname{Metric}(\cdot)$: Is a function that assesses the quality of the generated output, implicitly measuring how well LLMs adjust to document credibility. In this work, the accuracy of Question Answering (QA) tasks is used to approximate this metric, assuming that reduced impact from low-credibility documents should lead to higher QA accuracy.

4.2.2. Attention Weight Modification

The attention weight modification is the core mechanism by which CrAM regulates the influence of documents. It directly manipulates the attention weights within the LLM based on the credibility scores of the input documents.

1. Tokenization and Credibility Score Normalization: First, the user query $x$ and the set of relevant documents $\mathcal{D} = \{d_1, d_2, \dots, d_n\}$ are concatenated and tokenized into a single token sequence $\mathcal{T}(x, \mathcal{D}) = \{t_1, t_2, \ldots, t_m\}$, where $t_k$ is the $k$-th token. Each document $d_i$ has an associated credibility score $s_i$. To ensure these scores are suitable for scaling attention weights, they are normalized to a range of [0, 1]. For any token $t_k$ belonging to a document $d_i$, its normalized credibility score $\bar{s}_k$ is calculated as:

   $$\bar { s } _ { k } = \left\{ \begin{array} { l l } { \frac { s _ { i } - \operatorname* { m i n } ( \mathcal { S } ) } { \operatorname* { m a x } ( \mathcal { S } ) - \operatorname* { m i n } ( \mathcal { S } ) } } & { \mathrm { i f ~ } t _ { k } \mathrm { ~ b e l o n g s ~ t o ~ } d _ { i } } \\ { 1 } & { \mathrm { o t h e r w i s e } } \end{array} \right.$$

   Where:

   • $s_i$: The raw credibility score of document $d_i$ to which token $t_k$ belongs.
   • $\operatorname{min}(\mathcal{S})$: The minimum credibility score among all documents in the set $\mathcal{S}$.
   • $\operatorname{max}(\mathcal{S})$: The maximum credibility score among all documents in the set $\mathcal{S}$.
   • If $t_k$ belongs to $d_i$, its score is scaled to [0, 1] based on the min-max range of all document scores. This ensures that a document with the lowest credibility gets a normalized score of 0, and the highest gets 1.
   • otherwise: This condition applies to tokens that are part of the original query $x$ (not from the retrieved documents). Query tokens are always assigned a normalized score of 1, indicating they should always be fully attended to, as their credibility is implicitly assumed to be high (from the user). The resulting vector $\bar{\mathbf{s}} = [\bar{s}_1, \ldots, \bar{s}_m] \in \mathbb{R}^{1 \times m}$ contains the normalized credibility scores for the entire token sequence.

2. Modified Attention Weight Matrix: For each attention head $h$ in the LLM, let $\mathbf{A}_h$ represent its original attention weights matrix. This matrix typically has dimensions $m \times m$, where $m$ is the sequence length. Each row $(\mathbf{A}_h)_k$ corresponds to the attention weights from token $t_k$ to all other tokens in the sequence. To modify these weights, CrAM performs an element-wise multiplication with the normalized credibility scores vector $\bar{\mathbf{s}}$:

   $$( \mathbf { A } _ { h } ) _ { k } ^ { * } = \operatorname { N o r m } ( ( \mathbf { A } _ { h } ) _ { k } \odot { \bar { \mathbf { s } } } ) , k \in \{ 1 , \ldots , m \}$$

   Where:

   • $(\mathbf{A}_h)_k$: Represents the $k$-th row vector of the original attention weights matrix for head $h$. This vector indicates how much token $t_k$ attends to every other token in the sequence.

   • $\odot$: Denotes the element-wise multiplication (Hadamard product) of vectors. By multiplying $(\mathbf{A}_h)_k$ with $\bar{\mathbf{s}}$, the attention weight from token $t_k$ to any other token $t_j$ is scaled by $\bar{s}_j$ (the normalized credibility score of the document containing $t_j$). If $t_j$ is from a low-credibility document (low $\bar{s}_j$), the attention it receives is reduced.

   • $\operatorname{Norm}(\cdot)$: Refers to $\ell_1$ normalization. After scaling, the sum of attention weights in each row might no longer be one. $\ell_1$ normalization re-normalizes each row vector so that its elements sum to one, maintaining the probabilistic interpretation of attention weights.

   • $(\mathbf{A}_h)_k^*$: The $k$-th row vector of the modified attention weights matrix for head $h$.

     The overall effect is that tokens from low-credibility documents will receive lower attention from other tokens (both query and document tokens), thereby diminishing their influence on the LLM's subsequent computations and generated output.

   The following figure (Figure 2 from the original paper) illustrates the CrAM mechanism:

   
   该图像是示意图，展示了在检索增强生成（RAG）与信任度感知注意力修改（CrAM）中，如何处理不同文档对大型语言模型（LLM）输出的影响。上部分展示了RAG模型如何基于两个文档生成回答，强调了不准确的结果。下部分强调CrAM通过调整低可信文档的注意力权重，提升可信文档的影响力，从而更准确地产生回答。图中定义了不同的注意力头和对应的可信度得分。

   Fur lutiAM.CRAGAMrstte ea ndthe attention weights based on the credibility scores of each document.

4.2.3. Influential Head Identification

The paper notes that different attention heads have varying patterns and functions, and thus different impacts on the LLM's output. The goal here is to identify which attention heads are most responsible for incorporating misinformation and thus should be subject to attention weight modification. CrAM adapts causal tracing (Meng et al. 2022) for this purpose.

The contribution of an attention head $h$ is quantified using an indirect effect (IE) measure:

1. Baseline Probability Calculation ($P_0$): Given an LLM $L$, a user query $x$, and a set of documents $\mathcal{D}$ that includes one misinformation document $d_{mis}$ (e.g., $\mathcal{D} = \{d_{mis}, d_1, d_2, \ldots, d_n\}$), and an incorrect answer $a_{wrong}$ to $x$ that is supported by $d_{mis}$. The first step is to calculate the generation probability of this incorrect answer $a_{wrong}$ by the LLM without any modifications:

   $$P _ { 0 } = P _ { L } ( a _ { w r o n g } \mid x , \mathcal { D } )$$

   Where $P_L(\cdot)$ denotes the probability of generating $a_{wrong}$ by LLM $L$. This $P_0$ represents the LLM's propensity to generate the incorrect answer when exposed to misinformation.

2. Modified Probability Calculation ($P_1$): Next, a specific attention head $h$ is targeted for modification. The attention weights of only this head are modified using the method described in Section 4.2.2. For this specific step, credibility scores $\mathcal{S} = \{0, 1, 1, \ldots, 1\}$ are used, where the misinformation document $d_{mis}$ is assigned a score of 0 (lowest credibility), and all other documents $d_1, d_2, \ldots, d_n$ are assigned a score of 1 (highest credibility). This simulates maximally suppressing the misinformation document's influence through head $h$. Then, the generation probability of $a_{wrong}$ is recalculated with this modified LLM (denoted $L_h^*$):

   $$P _ { 1 } = P _ { L _ { h } ^ { * } } ( a _ { w r o n g } \mid x , \mathcal { D } )$$

   Where $L_h^*$ signifies the LLM $L$ with the attention weights matrix of head $h$ modified as per Equation (1).

3. Quantifying Contribution (IE): The contribution of attention head $h$ to generating the incorrect answer is then quantified as the difference between these two probabilities, known as the indirect effect (IE):

   $$\mathrm { I E } _ { h } = P _ { 0 } - P _ { 1 }$$

   A positive $\mathrm{IE}_h$ means that modifying head $h$ decreased the probability of generating the incorrect answer, indicating that head $h$ originally contributed to the generation of the incorrect answer by attending to the misinformation. A larger positive $\mathrm{IE}_h$ implies a greater original contribution to the error.

To ensure robustness, this IE calculation is performed over a small, dedicated dataset (separate from test data) containing examples of misinformation leading to incorrect answers. The average IE for each attention head is then computed across this dataset. Attention heads are then ranked by their average IE in descending order, and the top-ranked ones are selected as influential attention heads.

4.2.4. CrAM Workflow

The overall CrAM workflow integrates these two components:

1. Offline Influential Head Identification:

   • A small dataset containing misinformation-polluted documents (where misinformation leads to incorrect answers) is used.
   • For each attention head in the LLM, the average IE is calculated as described above (Section 4.2.3).
   • All attention heads are ranked by their average IE in descending order.
   • The top-ranked heads are selected as the influential attention heads that will be modified during inference. The number of heads to select is a hyperparameter determined on a validation set.

2. Online Inference with Attention Modification:

   • Given any user query, along with the retrieved documents and their credibility scores.

   • The attention weights of only the previously identified influential attention heads are modified using the method described in Section 4.2.2.

   • The LLM then generates its final answer using these modified attention weights. This process aims to significantly reduce the impact of low-credibility documents on the generated output.

     This workflow ensures that the costly influential head identification step is performed only once offline, and the attention modification during online inference is efficient, targeting only the most relevant attention heads.

5. Experimental Setup

5.1. Datasets

The experiments are conducted on two widely used open-domain Question Answering (QA) datasets:

• Natural Questions (NQ) (Kwiatkowski et al. 2019): This dataset consists of real user questions issued to Google search, paired with answers found in Wikipedia articles. It focuses on finding short or long answers directly from a provided document.

• TriviaQA (Joshi et al. 2017): This dataset contains questions from trivia and quiz-league websites, paired with evidence documents from Wikipedia and other web sources. It is known for its complex questions and requires aggregating information from multiple sources.

  These datasets are well-suited for evaluating RAG performance because they require LLMs to retrieve and synthesize information from external documents to answer questions. They are also standard benchmarks in the QA field, allowing for fair comparison with other methods.

5.1.1. Document Preparation

The paper carefully prepares both high-credibility and low-credibility (misinformation) documents to evaluate the robustness of the proposed method.

1. High-credibility documents: These are collected by retrieving relevant documents from an external corpus.

   • bge-large-en-v1.5 is used as an embedding model to retrieve an initial set of candidate documents from a Wikipedia dump (specifically, December 30, 2018, as used in Karpukhin et al. 2020).
   • bge-reranker-large is then applied to rank these candidates, and the top four documents are selected as high-credibility inputs. This ensures that these documents are genuinely relevant and factually accurate according to a reliable source.

2. Low-credibility documents (Misinformation): These are specifically generated to contain misinformation.

   • gpt-3.5-turbo-0125 is used to generate these documents.
   • The LLM is prompted to create news-style pieces that contain misinformation supporting an incorrect answer to a given question.
   • For each question, three distinct low-credibility documents are generated, all supporting the same incorrect answer. This controlled generation allows for precise study of misinformation impact.
   • Example of misinformation (from the paper's discussion): "The first person to win the Nobel Prize in Physics was not Roentgen, but Einstein." This type of misinformation includes both an incorrect assertion and a denial of correct information.

In-context corpus composition: Instead of directly injecting low-credibility documents into the entire RAG corpus, the study combines generated low-credibility documents with retrieved high-credibility documents for the LLM's input. This approach, referred to as 4 high + 1 low (e.g., four high-credibility documents plus one low-credibility document), provides granular control over the amount of misinformation and allows for a more focused evaluation of its impact.

5.2. Evaluation Metrics

For Question Answering (QA) tasks, the paper employs two standard metrics: Exact Match (EM) and F1 Score.

5.2.1. Exact Match (EM)

• Conceptual Definition: Exact Match (EM) is a strict metric that measures whether the LLM's generated answer is identical to one of the ground-truth answers. It is case-insensitive and ignores leading/trailing whitespace and common punctuation. It indicates whether the model can produce a perfectly correct answer.
• Mathematical Formula: $ \mathrm{EM} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(\text{predicted_answer}_i \in \text{gold_answers}_i) $ Where:
  • $N$: The total number of questions in the dataset.
  • $\text{predicted\_answer}_i$: The answer generated by the model for question $i$.
  • $\text{gold\_answers}_i$: The set of acceptable ground-truth answers for question $i$.
  • $\mathbb{I}(\cdot)$: The indicator function, which returns 1 if the condition inside is true, and 0 otherwise.
• Symbol Explanation: For each question, if the predicted answer perfectly matches any of the ground-truth answers, it scores 1; otherwise, it scores 0. The EM score is the average of these binary scores across all questions.

5.2.2. F1 Score

• Conceptual Definition: The F1 Score is a more lenient metric than EM. It treats both the predicted answer and the ground-truth answer as "bags of words" (sets of tokens) and calculates the overlap between them. It is the harmonic mean of Precision and Recall, where Precision measures how many of the predicted tokens are correct, and Recall measures how many of the correct tokens were captured by the prediction. It is particularly useful when answers can be phrased in multiple ways or contain partial correct information.
• Mathematical Formula: $ \mathrm{F1} = \frac{1}{N} \sum_{i=1}^{N} \frac{2 \cdot \mathrm{Precision}_i \cdot \mathrm{Recall}_i}{\mathrm{Precision}_i + \mathrm{Recall}_i} $ Where:
  • $\mathrm{Precision}_i = \frac{\text{Number of overlapping tokens}}{\text{Number of tokens in predicted answer}}$
  • $\mathrm{Recall}_i = \frac{\text{Number of overlapping tokens}}{\text{Number of tokens in gold answer}}$
  • $N$: The total number of questions.
  • For each question $i$, Precision and Recall are calculated based on the overlap of tokens between the predicted answer and the best matching ground-truth answer (if multiple exist).
• Symbol Explanation: The F1 Score balances Precision (avoiding false positives) and Recall (avoiding false negatives). A higher F1 Score indicates a better balance between correctly identifying relevant tokens and not including irrelevant ones.

5.3. Credibility Scores Generation

The paper uses two methods to assign credibility scores to documents:

• Ideal Setting: This represents a perfect scenario where credibility scores are known definitively.

  • High-credibility documents are assigned a score of 10.
  • Low-credibility documents (containing misinformation) are assigned a score of 1. This binary-like assignment (after normalization, they become 1 and ~0 respectively) allows for clear evaluation of the method's potential under optimal conditions.

• GPT Setting: This simulates a more realistic scenario where credibility scores are estimated by another LLM.

  • gpt-3.5-turbo-0125 is employed to directly generate a credibility score for each document.
  • Prompts are designed to instruct GPT to provide these scores. The distribution of these GPT-generated scores is provided in Appendix C of the original paper, showing a more continuous and less binary range of scores.

5.4. Baselines

The CrAM model is compared against four types of methods to assess its performance:

1. Naive RAG: This is the standard RAG pipeline. It simply retrieves documents and feeds them to the LLM without any mechanisms to account for or combat misinformation. It serves as a strong baseline to show the performance degradation when misinformation is present.
2. Prompt Based: This is a non-SFT method that attempts to inform the LLM about document credibility by including the credibility scores directly in the prompt alongside the documents. The LLM is then expected to implicitly adjust its behavior based on these prompt instructions.
3. Exclusion: This is another non-SFT method where documents with credibility scores below a certain threshold are completely removed before being fed to the LLM. This method is not compared in the Ideal Setting because the binary nature of ideal scores (10 or 1) would make thresholding trivial (e.g., threshold > 1 would discard all misinformation, making it an ideal RAG scenario without misinformation).
4. CAG (Credibility-Aware Generation): Proposed by Pan et al. (2024), CAG is an SFT-based method. It directly incorporates credibility scores and documents into prompts and then fine-tunes an LLM (specifically, Llama2-13B in their work) to explicitly learn how to leverage these scores for better understanding and generation in the presence of misinformation. This serves as a comparison against fine-tuning approaches.

5.5. Hyperparameters

• Data points for IE calculation: 100 randomly selected data points from each dataset are used to calculate the average Indirect Effect (IE) for all attention heads during the influential head identification phase.
• Validation set for head selection: Another validation set of 100 data points from each dataset is used to determine the optimal number of top-ranked influential attention heads to include in the final modified set. This helps tune the CrAM model's configuration.

6. Results & Analysis

6.1. Core Results Analysis

The experiments aimed to evaluate CrAM's effectiveness in mitigating misinformation in RAG across different LLMs and settings.

6.1.1. Comparison with Non-SFT Methods

The following are the results from Table 1 and Table 2 of the original paper, comparing CrAM with other non-SFT methods in both Ideal and GPT credibility settings. The common experimental setup is $4√ + 1x$, meaning four high-credibility documents and one low-credibility document.

The following are the results from Table 1 of the original paper:

The following are the results from Table 2 of the original paper:

Observations:

• Significant Gains over Baselines: In both Ideal and GPT settings, CrAM consistently and significantly outperforms Naive RAG and Prompt Based methods across all LLMs (Qwen1.5-7B, Llama2-13B, Llama3-8B) and datasets (NQ, TriviaQA). For instance, in the Ideal Setting on TriviaQA, CrAM with Llama2-13B achieves a remarkable 31.90% increase in EM over Prompt Based.
• Effectiveness with Realistic Scores: Even with GPT-generated credibility scores (a more realistic scenario), CrAM maintains its superiority over Naive RAG and Prompt Based, showing its practical applicability. It also outperforms Exclusion which discards documents, demonstrating the benefit of nuanced attention adjustment over hard filtering.
• Surpassing $4√ Naive RAG$: Notably, under the Ideal Setting with $4√ + 1x$ documents, CrAM's performance sometimes exceeds that of Naive RAG with $4√$ (no misinformation). This counter-intuitive result is explained by the generated misinformation sometimes containing denials of correct information, allowing LLMs to reuse the correct information after CrAM suppresses the misleading denial. This highlights CrAM's ability to effectively neutralize misinformation while still allowing the LLM to extract truth.

6.1.2. Comparison with SFT-based Method

The following figure (Figure 3 from the original paper) presents the performance comparison between CrAM and the SFT-based CAG-13B model, regarding the varying number of low-credibility documents under the ideal setting.

该图像是图表，展示了在不同数量的误导性文档下，CrAM和CAG 13B在Natural Questions（NQ）和TriviaQA任务上的F1得分变化。左侧为NQ的结果，右侧为TriviaQA的结果，随着误导性文档数量的增加，F1得分呈下降趋势。

Figure 3: Performance comparison of CrAM and CAG-13B regarding the varying number of documents containing misinformation under ideal setting.

Observations:

• Consistent Outperformance: CrAM (specifically, Llama2-13B based CrAM) consistently and remarkably outperforms CAG-13B (which is also Llama2-13B based) in terms of F1 Score across both NQ and TriviaQA datasets, even as the number of low-credibility documents increases from 1 to 3.
• Efficiency and Effectiveness: This finding is crucial because CAG requires supervised fine-tuning, which is computationally expensive and data-intensive. CrAM, being a non-SFT method, achieves superior results without these overheads, demonstrating its efficiency and effectiveness.

6.2. Ablation Studies / Parameter Analysis

6.2.1. Effect of Number of Low-credibility Documents

The paper investigates how varying the quantity of misinformation affects CrAM's performance. The following figure (Figure 4 from the original paper) shows the performance change on NQ regarding the varying number of documents with misinformation.

该图像是一个对比图，展示了在理想设置和GPT设置下，CrAM、基于提示和天真的RAG在处理包含误信息的文档数量时的表现。图中显示不同设置下的EM值随误信息文档数量变化的趋势，同时标注了各方法的表现差异。

Figure 4: Performance change on NQ regarding the varying number of documents with misinformation.

Observations:

• Robustness to Misinformation Load: CrAM consistently outperforms Prompt Based and Naive RAG as the number of low-credibility documents (1x to 3x) increases, in both Ideal and GPT settings.
• Smaller Performance Drop: CrAM exhibits a significantly smaller performance degradation compared to other models when more low-credibility documents are introduced. This demonstrates CrAM's robustness and scalability in handling increasing amounts of misinformation.

6.2.2. Effect of Dataset Size on Attention Heads Selection

The process of identifying influential attention heads uses a small subset of the data. The following figure (Figure 5 from the original paper) shows the performance on NQ and TriviaQA regarding the dataset size for determining the influential attention head changes.

该图像是图表，展示了在不同数据集大小下，CrAM方法在自然问题(NQ)和TriviaQA上的EM得分。左侧图表显示了NQ的结果，右侧图表显示了TriviaQA的结果。随着数据集大小的增加，CrAM在这两个数据集上的EM得分保持相对稳定。

Figure 5: Performance on NQ and TriviaQA regarding the dataset size for determining the influential attention head changes.

Observations:

• Minor Impact of Dataset Size: While there are minor fluctuations in performance, the overall impact of the number of data points used for influential head identification is not substantial (maximum difference of 4% in EM). This indicates that CrAM's head selection mechanism is relatively stable and does not require a massive dataset, contributing to its efficiency.

6.2.3. Analysis on Number of Selected Attention Heads

The selection of influential attention heads is a critical component of CrAM. The following figure (Figure 6 from the original paper) shows the performance on NQ in ideal setting regarding the varying number of selected attention heads.

该图像是一个图表，展示了在自然问答（NQ）数据集上，不同数量的修改最高排名注意力头对EM（Exact Match）得分的影响。随着修改的头数增加，EM得分呈现一定波动，最高达到约0.35。

Figure 6: Performance on NQ in ideal setting regarding the varying number of selected attention heads.

Observations:

• Sensitivity at Extremes: The model's performance (EM) drops sharply when very few or all attention heads are selected for modification. This suggests that a targeted approach is necessary, as modifying too few might miss critical influential heads, and modifying all might interfere with beneficial attention patterns.

• Stable Performance in Mid-Range: There's a relatively stable performance range when a moderate number of attention heads are selected. This implies that only a subset of heads are truly influential for misinformation handling, and once these are covered, additional modifications have diminishing returns or even negative impacts.

  To understand why this happens, the paper analyzes the distribution of Indirect Effect (IE) values for all attention heads. The following figure (Figure 7 from the original paper) shows the density distribution of IE of all the attention heads in Llama3-8B.

  该图像是一个密度分布图，展示了IE值的分布情况。横轴表示IE值，纵轴表示密度，该图形中展现了IE值在不同区间的分布趋势，主要集中在接近0的区域。

Figure 7: Density distribution of IE of all the attention heads in Llama3-8B.

Observations:

• Normal-like Distribution of IE: The density distribution of IE values (contributions to incorrect answers) approximates a normal distribution centered around 0.
• Sparse Influence: The majority of attention heads have IE values concentrated near 0, meaning most heads have a minor impact on whether misinformation leads to an incorrect answer. Only heads with IE values significantly far from zero (either positive or negative) have a substantial impact. This supports the rationale for selective attention modification, as only a few influential heads need to be targeted.

6.2.4. Ablation Study

To validate the design choices of CrAM, an ablation study is conducted. The following are the results from Table 3 of the original paper:

Variants:

• CrAM-all: This variant removes the influential head identification step and applies attention weight modification to all attention heads in the LLM.
• Naive RAG: This is equivalent to disabling the attention weight modification mechanism in CrAM entirely.

Observations:

• Necessity of Influential Head Selection: CrAM-all shows noticeable performance drops compared to the full CrAM model across all LLMs and datasets. For Llama3-8B, the decrease is substantial (e.g., 14.5% on NQ). This empirically validates the importance of identifying and targeting only the influential attention heads, supporting the idea that indiscriminate modification can harm performance.

• Necessity of Attention Weight Modification: Disabling the attention weight modification (i.e., Naive RAG) leads to a dramatic performance drop (e.g., over 27.5% on TriviaQA for all three LLMs) compared to CrAM. This strongly confirms that dynamically adjusting attention weights based on credibility scores is crucial for combating misinformation.

  In summary, the ablation study conclusively demonstrates that both components of CrAM—the identification of influential attention heads and the credibility-aware attention weight modification—are essential for its superior performance.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces CrAM (Credibility-aware Attention Modification), a novel and effective plug-and-play method designed to combat misinformation within Retrieval-Augmented Generation (RAG) systems for Large Language Models (LLMs). CrAM addresses the critical challenge of misinformation pollution in external documents without requiring extensive fine-tuning. Its core contribution lies in its two-stage approach: first, it identifies influential attention heads within the LLM that are most susceptible to misinformation using a modified causal tracing technique; second, it modifies the attention weights of these specific heads based on the credibility scores of retrieved documents, thereby reducing the impact of low-credibility information.

Extensive experiments on Natural Questions (NQ) and TriviaQA datasets, utilizing Llama2-13B, Llama3-8B, and Qwen1.5-7B, demonstrate CrAM's significant efficacy. It improves Exact Match (EM) performance by over 20% compared to vanilla RAG, even outperforming supervised fine-tuning (SFT)-based methods like CAG. The method also exhibits robustness to varying amounts of misinformation and sensitivity analyses confirm the importance of its targeted attention modification strategy. CrAM offers an efficient and practical solution for enhancing the trustworthiness of LLMs in RAG settings.

7.2. Limitations & Future Work

The paper does not include a dedicated "Limitations" section, but some points can be inferred:

• Reliance on Credibility Estimators: CrAM's effectiveness inherently relies on the quality of the credibility scores provided. If the credibility estimator (e.g., GPT-3.5 in the GPT Setting) is inaccurate or biased, CrAM's performance would be compromised. The paper shows performance differences between the Ideal and GPT settings, highlighting this dependency. Future work could focus on improving robust and reliable credibility score generation methods.
• Computational Cost of Influential Head Identification: While plug-and-play for inference, the influential head identification step involves calculating Indirect Effects (IEs) for all attention heads over a dataset. Although performed offline and on a small dataset, this process still requires computational resources and specific data containing misinformation-answer pairs. Optimizing this identification process or making it more adaptive could be a direction.
• Generalizability of Influential Heads: The paper implicitly assumes that the influential attention heads identified for QA tasks on NQ and TriviaQA (and with specific misinformation types) are generalizable across different tasks, LLM variants, and types of misinformation. Further investigation into the task-specificity or domain-specificity of these influential heads could be valuable.
• Understanding Attention Head Specialization: While the paper leverages the idea that different attention heads have different functions, a deeper, more mechanistic understanding of why certain heads become influential in propagating misinformation could lead to more sophisticated and potentially model-agnostic intervention strategies.

7.3. Personal Insights & Critique

The CrAM paper presents an elegant and practically significant solution to a pressing problem in LLM deployment.

• Elegance of the Solution: The idea of credibility-aware attention modification is intuitively appealing. Rather than complex fine-tuning or blunt document discarding, directly manipulating the LLM's internal attention mechanism to reflect external credibility scores is a smart and targeted approach. It respects the existing LLM architecture while adding a crucial layer of control. The plug-and-play nature is a huge advantage for real-world applications.

• Leveraging Interpretability Research: The work cleverly builds upon prior research into attention head interpretability and causal tracing. By identifying influential heads, CrAM avoids modifying the entire network, leading to efficiency and potentially preserving other beneficial behaviors of the LLM. This demonstrates a valuable synergy between LLM interpretability and robustness research.

• Nuanced Handling of Misinformation: The ability to scale down influence rather than simply discard documents is a key strength. Misinformation is rarely black and white; a document might contain accurate information alongside a misleading claim. CrAM's method allows the LLM to potentially still extract value from the credible parts of a document while downplaying the unreliable parts. This is supported by the observation that CrAM with misinformation can sometimes outperform Naive RAG without misinformation, implying effective suppression and extraction.

• Potential for Broader Application: The core idea of modifying attention based on external signals has broader implications beyond credibility. One could imagine similar mechanisms for sarcasm detection, sentiment weighting, or source reliability in other NLP tasks. For instance, if an LLM is processing text from multiple sources, some known to be biased, a similar attention modification could be applied.

• Critique on IE Calculation: While causal tracing is a powerful tool, its application in determining IE relies on specific choices (e.g., assigning 0 to misinformation and 1 to others for calculating $P_1$). The robustness of this IE calculation to different types of misinformation, different LLM architectures, or varying baseline credibilities could be further explored. Also, the choice of "incorrect answer supported by misinformation" for IE calculation is specific; how would CrAM behave if misinformation leads to a subtly biased but not strictly "incorrect" answer?

• Real-world Credibility Score Generation: The GPT Setting is a step towards realism, but real-world credibility scoring is an active research area with its own challenges (e.g., bias in GPT itself, scalability, domain specificity). The practical success of CrAM will largely depend on the advancements in generating these credibility scores accurately and efficiently.

  Overall, CrAM presents a compelling and practical advancement in making RAG systems more resilient to misinformation, marking a significant step towards more trustworthy AI systems.