1. Bibliographic Information

1.1. Title

UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models

1.2. Authors

Qizhou Chen, Dakan Wang, Taolin Zhang, Zaoming Yan, Chengsong Vou, Chengyu Wang, Xiaofeng He. Affiliations include East China Normal University, Exacity Inc., Alibaba Group, and Hefei University of Technology.

1.3. Journal/Conference

The paper is published at (UTC) 2025-05-18T10:19:01.000Z, indicating a recent publication. While the specific conference or journal is not explicitly named in the provided text, the NeurIPS paper checklist suggests it might be submitted or accepted for NeurIPS 2025, a highly reputable conference in machine learning.

1.4. Publication Year

2025

1.5. Abstract

Model editing aims to enhance the accuracy and reliability of large language models (LLMs) by efficiently adjusting their internal parameters without full retraining. Existing LLM editing datasets are often limited to narrow knowledge domains and offer restricted evaluation scopes, frequently overlooking the diverse demands of editing and the ripple effects that edits can cause. To address these issues, this paper introduces UniEdit, a unified benchmark for LLM editing, grounded in open-domain knowledge. The benchmark is constructed by selecting entities from 25 common domains across five major categories, leveraging extensive triple knowledge from open-domain knowledge graphs like Wikidata to ensure comprehensive domain coverage. To tackle generality and locality concerns in editing, the authors designed a Neighborhood Multi-hop Chain Sampling (NMCS) algorithm. This algorithm samples subgraphs based on a given knowledge piece, enabling the evaluation of comprehensive ripple effects. Finally, proprietary LLMs are used to convert the sampled knowledge subgraphs into natural language text, ensuring grammatical accuracy and syntactical diversity. Extensive statistical analysis confirms the scale, comprehensiveness, and diversity of the UniEdit benchmark. Comprehensive experiments are conducted across multiple LLMs and editors, analyzing their performance to highlight strengths and weaknesses in editing across open knowledge domains and various evaluation criteria, thus offering valuable insights for future research.

1.6. Original Source Link

https://arxiv.org/abs/2505.12345v3 The paper is available as a preprint on arXiv (version 3), indicating it is publicly accessible and has undergone revisions.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the limitations of existing model editing benchmarks for Large Language Models (LLMs). While LLMs possess powerful natural language processing capabilities, they often struggle to provide accurate and real-time information, especially in rapidly changing environments or when new facts emerge. Model editing techniques offer a solution to efficiently update the internal knowledge of these models without the computational burden and risk of catastrophic forgetting associated with full retraining.

The problem is important because LLMs are increasingly deployed in high-stakes industries like medicine, finance, and education, where inaccurate information can have significant consequences. Existing benchmarks for evaluating model editing methods have several shortcomings:

• Narrow knowledge domains: Most datasets are confined to a limited set of topics or relations.

• Limited evaluation scope: They often focus only on whether the edited fact is recalled (reliability) and its paraphrased versions (generality), but overlook the broader ripple effects or how unrelated facts are preserved (locality).

• Lack of integration: Different benchmarks construct data based on isolated evaluation criteria, preventing a comprehensive assessment of combined scenarios.

• Small scale: Many datasets are too small to adequately train or evaluate advanced editing methods.

  The paper's entry point is to address these gaps by creating a unified, large-scale, and open-domain knowledge editing benchmark that can comprehensively evaluate various generality and locality criteria, including their combinations.

2.2. Main Contributions / Findings

The primary contributions of the paper are:

• A Unified Open-Domain Benchmark (UniEdit): Introduction of the first open-domain knowledge editing benchmark, UniEdit, designed to simulate real-world editing challenges comprehensively. It leverages Wikidata, the largest open-source knowledge graph, covering 25 domains across five major categories.

• Novel Sampling Algorithm (NMCS): Development of the Neighborhood Multi-hop Chain Sampling (NMCS) algorithm. This algorithm unifies and extends various evaluation criteria, enabling the generation of diverse and challenging generality and locality samples, including multi-hop chains and combinations of criteria.

• Comprehensive Experimental Analysis: Extensive experiments are conducted on UniEdit using multiple LLM backbones and various editing methods. The analysis provides valuable insights into the performance and limitations of existing LLM editors.

  Key conclusions and findings from the paper include:

• Generality Challenge: Existing editors, particularly those following the Locate-then-Edit (L&E) paradigm, exhibit significant limitations in handling complex generality evaluations within UniEdit.

• Domain Sensitivity: Editing performance varies across different knowledge domains, highlighting the critical need for improved low-resource knowledge editing capabilities.

• Complexity Impact: Increasing sample complexity (e.g., multi-hop reasoning, combined criteria) generally increases the difficulty of generality tasks, but can, counter-intuitively, sometimes ease locality evaluation by reducing the likelihood of overlapping components with edited knowledge.

• Data Scale for Edit Training: The scale and domain coverage of training data significantly influence the performance of editors that rely on edit training.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a reader should be familiar with the following core concepts:

• Large Language Models (LLMs): These are advanced artificial intelligence models, typically based on the transformer architecture, trained on vast amounts of text data. They can understand, generate, and translate human-like text, and perform various natural language processing (NLP) tasks. Examples include GPT-3, LLaMA, and Deepseek.
• Knowledge Editing: The process of updating or correcting specific factual information stored within an LLM's parameters without undergoing full retraining. This is crucial for maintaining model accuracy and relevance over time, especially as real-world knowledge changes.
• Knowledge Graphs (KGs): Structured representations of information that organize entities (e.g., people, places, concepts) and their relationships. KGs typically store facts as triples in the format (subject, relation, object), e.g., (Eiffel Tower, located in, Paris). Wikidata is an example of a large, open-domain KG.
• Triple (s, r, o): The fundamental unit of information in a Knowledge Graph. It consists of a subject (head entity) $s$, a relation (predicate) $r$, and an object (tail entity) $o$. For example, in the triple (Paris, capital of, France), Paris is the subject, capital of is the relation, and France is the object.
• Reliability: In the context of model editing, reliability refers to the ability of an edited LLM to correctly recall the specific new or modified fact it was trained to learn. If an LLM is edited to state that "The capital of France is Lyon", it should consistently answer "Lyon" when asked about "The capital of France".
• Generality: Generality evaluates whether an edited LLM can apply the learned knowledge to related queries or different contexts, beyond the exact phrasing of the original edit. This includes rephrased questions, multi-hop reasoning, or queries involving aliases of the edited entities. For example, if the LLM learns "The capital of France is Lyon", it should also respond correctly to "What is the primary city of France?" if "primary city" is a related concept.
• Locality: Locality (also known as consistency or unintended side effects) assesses whether the edits made to an LLM do not negatively impact its performance on unrelated knowledge or queries. If the LLM learns about "Lyon", it should still correctly answer questions about unrelated topics like "The highest mountain in the world". A good editing method should preserve the model's original capabilities on untouched knowledge.
• Catastrophic Forgetting: A phenomenon in machine learning where a model, when updated with new information, tends to forget previously learned information. This is a major challenge in continuous learning and model editing, as retraining an entire LLM for each new fact is computationally expensive and risks losing existing knowledge.

3.2. Previous Works

The paper categorizes previous works into Knowledge Editing Methods and Knowledge Editing Benchmarks.

3.2.1. Knowledge Editing Methods

These methods aim to modify LLMs' internal parameters to incorporate new knowledge. They generally fall into two categories:

• Locate-then-Edit (L&E) methods: These approaches first identify specific parts of the LLM (e.g., layers, neurons) that are responsible for storing the knowledge to be edited, and then modify only those parts.

  • ROME [16]: Identifies edit-sensitive layers using causal tracing and updates their weights. Causal tracing is a technique to pinpoint which model components (e.g., specific neurons or attention heads) are causally responsible for a model's output on a given input.
  • MEMIT [17] and WILKE [18]: Enhance ROME by distributing parameter changes across multiple layers, aiming for more stable edits.
  • PMET [19]: Uses information extraction patterns of attention layers for precise updates.
  • AlphaEdit [33]: Extends L&E to lifelong editing by projecting updates into the null space of preserved knowledge, meaning changes are made in directions that do not interfere with existing, correct knowledge.
  • UnKE [28] and AnyEdit [34]: Explore adapting L&E to unstructured knowledge editing, where the knowledge isn't a neat (s,r,o) triple but embedded in free text.

• External module-based strategies: These methods keep the core LLM parameters fixed and introduce an external module that handles the edits or redirects queries.

  • KE [35]: Trains an LSTM-based hyper-network to predict parameter updates.
  • MEND [36]: Improves on KE by using the first-order gradient of the edit knowledge to enhance the editing signal.
  • SERAC [37]: Trains a counterfactual model for query redirection. When a query is related to an edited fact, a separate "counterfactual" model provides the response; otherwise, the original LLM is used.
  • T-Patcher [38]: Incorporates additional neurons specifically for edited knowledge.
  • GRACE [39]: Remaps edit-related representations based on edit distance thresholds using discrete key-value adaptors.
  • RECIPE [40]: Creates continuous prefixes for dynamic editing through prompt learning.
  • LEMOE [41]: Enables lifelong editing using a Mixture of Experts (MoE) with expert routing, where different experts handle different knowledge domains or editing tasks.

• Other notable early efforts:

  • ENN [42]: Investigated model editing through meta-learning.
  • [43]: Explored partial parameter tuning for editing large transformers.
  • Knowledge Neurons [15]: Proposed the concept and studied how factual knowledge is stored in pre-trained transformers, providing the theoretical basis for L&E.
  • IKE [44]: Uses in-context learning to guide LLMs with editing instructions, where editing examples are provided directly in the prompt.

3.2.2. Knowledge Editing Benchmarks

These datasets and evaluation frameworks are designed to test the effectiveness of editing methods.

• ZSRE [20]: Uses WikiReading to generate QA editing data, evaluating reliability and simple rephrasing (Rep).
• CounterFact [16]: Constructs counterfactual data (e.g., changing a known fact) to increase editing difficulty, also focusing on reliability and rephrasing.
• MQuAKE [21] and BAKE [24]: Extend evaluation to include multi-hop reasoning (MH) and relational reversal (RR).
• RippleEdit [23]: Refines multi-hop definition and introduces 1-N forgetfulness, entity aliasing (SA, OA), and relation specificity (RS). 1-N forgetfulness refers to forgetting other facts related to a subject when one specific fact is edited.
• ReCoE [25]: Investigates entity reasoning.
• EVOKE [22]: Assesses the overfitting problem of L&E methods.
• CliKT [30], HalluEditBench [31], WikiBigEdit [32]: Focus on specific knowledge types or problems, such as biomedical long-tail knowledge, LLM hallucinations, and recently updated Wikidata knowledge, respectively.
• AKEW [27], UnKEBench [28], AnyEdit [34]: Address unstructured editing in texts.

3.3. Technological Evolution

The evolution of knowledge editing benchmarks has progressed from simple reliability and rephrasing evaluations (e.g., ZSRE, CounterFact) to increasingly complex criteria. Early benchmarks focused on single-fact edits and their direct paraphrases. The field then recognized the need to evaluate ripple effects, leading to the inclusion of multi-hop reasoning, relation reversal, and entity aliasing (MQuAKE, BAKE, RippleEdit). More recent works have started to address specific challenges like overfitting (EVOKE), long-tail knowledge (CliKT), or hallucinations (HalluEditBench).

However, a persistent limitation has been the narrow knowledge domains and the isolated nature of evaluation criteria across these benchmarks. Most are sampled from a limited number of KG triples or relations, or refined from other datasets, which may not generalize to the diverse knowledge an LLM possesses.

3.4. Differentiation Analysis

Compared to the main methods and benchmarks in related work, UniEdit introduces several core differences and innovations:

• Open-Domain Scope: Unlike most previous benchmarks confined to narrow domains, UniEdit is built upon Wikidata and encompasses 25 common domains across five major categories, ensuring broad and diverse knowledge coverage.
• Unified and Comprehensive Evaluation: UniEdit integrates and extends almost all existing evaluation criteria for generality and locality (Rep, MH, RR, SER, SA, OA, SS, RS, OS, 1-NF), including novel combinations of these criteria. This contrasts with benchmarks that focus on one or a few isolated criteria.
• Structured Ripple Effect Sampling: The NMCS algorithm is a key innovation, allowing for the systematic sampling of multi-hop chains that precisely define generality and locality structures. This provides a more rigorous and challenging evaluation of ripple effects compared to ad-hoc methods.
• Scale and Diversity: UniEdit is significantly larger than many previous benchmarks, comprising 311K entries (each with an edit, generality, and locality sample), enhancing its utility for training and evaluating data-hungry editing methods.
• Real-world Relevance: By grounding itself in open-domain knowledge and simulating complex ripple effects, UniEdit aims to better reflect the challenges of knowledge editing in real-world scenarios.

4. Methodology

4.1. Principles

The core idea behind UniEdit is to create a comprehensive and unified benchmark for LLM knowledge editing that addresses the limitations of narrow domain coverage, limited evaluation criteria, and small scale in previous datasets. The principles guiding its construction are:

1. Open-Domain Coverage: Utilize a large, open-source knowledge graph (Wikidata) to ensure a broad and diverse range of factual knowledge across multiple domains.
2. Comprehensive Evaluation: Integrate and extend existing generality and locality evaluation criteria, including their complex combinations, to thoroughly assess the ripple effects of knowledge edits.
3. Structured Data Generation: Employ a systematic algorithm (NMCS) to sample knowledge subgraphs (specifically, multi-hop chains) that precisely define the generality and locality relationships to the edited fact.
4. Natural Language Conversion: Convert the structured knowledge into diverse and grammatically correct natural language prompts and targets using powerful proprietary LLMs (e.g., Deepseek-V3).
5. Scalability: Generate a large number of diverse samples suitable for rigorous testing and potential training of editing methods.

4.2. Core Methodology In-depth (Layer by Layer)

The data construction process for UniEdit involves five main steps, as illustrated in Figure 2.

The following figure (Figure 2 from the original paper) shows the data construction pipeline of UniEdit.

该图像是一个示意图，展示了UniEdit数据构建的流程。图中包括数据准备与清理、领域实体检索、编辑三元组采样、一般性和局部子图采样及最终数据生成等步骤。特别是步骤4中的多跳QA链采样方法(NMCS)，用于确保编辑的广度和连续性。

4.2.1. Step 1: Data Preparation and Cleaning

The process begins with the raw Wikidata dump (latest-all.json), which contains an enormous amount of data: 113.7 million entities and 12,300 properties (relations). Each entity has an ID, label, description, aliases, and claims (which represent the triples where the entity acts as the head).

• Entity Filtering: Entities with no English labels are removed. Low-utility keywords (e.g., "point of time") in descriptions are filtered out, reducing the total to 29.9 million entities.
• Property Filtering: Properties are filtered based on data type and manual verification. Non-linguistic or low-editorial-value properties (e.g., those pointing to images, IDs, URLs) are removed, retaining 2.4 thousand properties. These retained properties fall into seven types: wikibase-item (pointing to other entities), string, quantity, time, math, globe-coordinate, and monolingual text.
• Indexing: The cleaned entities are ingested into a search engine (like Elasticsearch) to facilitate efficient retrieval and sampling in subsequent steps.

4.2.2. Step 2: Entity Retrieval with Domains

To ensure balanced coverage across diverse fields, entities are categorized into five major sectors: Natural Sciences, Humanities, Social Sciences, Applied Sciences, and Interdisciplinary Studies. These sectors collectively span 25 distinct domains.

• Keyword Generation: Domain-specific keywords are generated for each domain using a proprietary LLM (specifically, GPT-4). These keywords help in identifying relevant entities. The prompt used for GPT-4 generation asks for derivative vocabulary and terminology (nouns and adjectives), with instructions to avoid polysemous words and use double quotation marks.

• Entity Retrieval: Relevant entities are retrieved from the Elasticsearch index based on their labels and descriptions matching the domain-specific keywords.

• Relevance Filtering: An additional filtering step applies exact string matching to filter out noisy results (e.g., "black hole" only matching "black") to improve relevance.

  The following figure (Figure 3a from the original paper) shows the data distribution across domains.

  该图像是图表，展示了UniEdit数据统计，包括各领域的数据分布（a）、结构统计（b）、描述中的名词统计（c）以及一般性和局部性标准的数据统计（d，e）。

The following figure (Figure 7 from the original paper) shows the word cloud distributions of head entity descriptions in the edit samples across different domains.

该图像是一个词云，展示了不同领域的主要实体描述。图中包含了农业、天文学、生物学等多个领域的关键词，关键词的大小反映其在各领域中的重要性和频率。

The following table (Table 5 from the original paper) shows partial keywords of each domain and count of retrieved entities.

4.2.3. Step 3: Edit Triples Sampling

From the domain-specific entity sets ($E$), head entities are sampled to form the edit triples. To ensure diversity and avoid over-sampling semantically similar items, a sequential weighted sampling approach is used, dynamically adjusting sampling weights.

The probability $p_{e_i}$ for sampling an entity $e_i$ from the set $E$ is given by: $$p _ { e _ { i } } = \frac { w _ { i } } { \sum _ { j } w _ { j } } \mathrm { ~ s.t. ~ } w _ { i } = \left\{ \begin{array} { l l } { 0 , } & { \mathrm { if ~ } e _ { i } \in S , } \\ { f _ { \mathrm { i w } } ( e _ { i } ) / \gamma ^ { \psi ( e _ { i } , S ) } , } & { \mathrm { else. } } \end{array} \right.$$ Where:

• $p_{e_i}$ is the probability of sampling entity $e_i$.

• $w_i$ is the adjusted weight for entity $e_i$.

• $S$ is the set of already sampled head entities. If $e_i$ is already in $S$, its weight becomes 0 to prevent re-sampling.

• $f_{\mathrm{iw}}(e_i) = f_{\mathrm{es}}(e_i) f_{\mathrm{em}}(e_i)$ is the initial sampling weight, balancing between:

  • $f_{\mathrm{es}}(e_i)$: The ElasticSearch retrieval score for $e_i$.
  • $f_{\mathrm{em}}(e_i)$: The exact match count of domain keywords in $e_i$'s description.

• $\gamma$ is the decay base, set to 1.05, controlling how quickly the sampling probability decreases.

• $\psi(e_i, S) = \sum_{s \in S} \text{sim}(e_i, s)$ is the decay factor, which accumulates the similarity of $e_i$ to all entities already in $S$. This down-weights entities that are similar to already sampled ones.

  The similarity function $\text{sim}(e_i, s)$ is defined as: $$\sin ( e _ { i } , s ) = \sum _ { u _ { e _ { i } } \in f _ { \mathrm { d w } } ( e _ { i } ) } \sum _ { u _ { s } \in f _ { \mathrm { d w } } ( s ) } \frac { \mathbb { I } ( u _ { e _ { i } } = u _ { s } ) } { \| f _ { \mathrm { d w } } ( e _ { i } ) \| } \delta ( u ) \mathrm { ~ s.t. ~ } \delta ( u ) = \left\{ \begin{array} { l l } { \delta _ { \mathrm { i n } } , } & { \mathrm { ~ if ~ } u \in U , } \\ { \delta _ { \mathrm { o u t } } , } & { \mathrm { ~ else. ~ } } \end{array} \right.$$ Where:

• $\mathbb{I}(\cdot)$ is the indicator function, which is 1 if the condition is true, and 0 otherwise.

• $f_{\mathrm{dw}}(e)$ denotes the set of word segments extracted from the description of entity $e$.

• $U$ is the set of domain keywords.

• $\delta(u)$ is the decay weight for a word segment $u$. Words in the domain keyword set $U$ are assigned a lower decay weight ($\delta_{\mathrm{in}} = 0.2$) to mitigate the impact of sampling decay on domain relevance, while other words have a higher decay weight ($\delta_{\mathrm{out}} = 1$). This means that similarity based on domain-specific words is less penalized than similarity based on general words.

  A total of 30,000 head entities are sampled per domain. For each sampled head entity $s_{\varepsilon}$, an edit triple $t_{\varepsilon} = \sigma(f_{\mathrm{twh}}(s_{\varepsilon}), \mathcal{U})$ is generated, where $f_{\mathrm{twh}}(s_{\varepsilon})$ retrieves all triples with $s_{\varepsilon}$ as the head, and $\mathcal{U}$ represents a uniform distribution for selection among them.

4.2.4. Step 4: Generality and Locality Subgraphs Sampling

This is the most critical step, where the Neighborhood Multi-hop Chain Sampling (NMCS) algorithm is introduced to create diverse generality and locality samples. The distinction lies in whether the sampled subgraph includes the entire edit triple $t_{\varepsilon} = (s_{\varepsilon}, r_{\varepsilon}, o_{\varepsilon})$.

• Initial Triple Selection:

  • For generality samples, sampling always starts with the edit triple $t_{\varepsilon}$.
  • For locality samples, the initial triple $t_l$ is chosen from one of four options, selected uniformly: head entity $s_{\varepsilon}$, relation $r_{\varepsilon}$, tail entity $o_{\varepsilon}$ (only if $o_{\varepsilon}$ is an entity), or a random entity $\tilde{e}$ from the full set of filtered entities $\tilde{E}$. The generation of $t_l$ is formalized as: $$t _ { l } = \left\{ \begin{array} { l l } { \sigma ( f _ { \mathrm { t w r } } ( x ) , \mathcal { U } ) , } & { \mathrm { i f } x = r _ { \varepsilon } , } \\ { \sigma ( f _ { \mathrm { t w } ^ { * } } ( x ) , \mathcal { U } ) , } & { \mathrm { e l s e } . } \end{array} \right. \ \mathrm { s.t. } \ x = \sigma ( \{ s _ { \varepsilon } , o _ { \varepsilon } , r _ { \varepsilon } , \tilde { e } \} , \mathcal { U } ) \ , \ f _ { \mathrm { t w } ^ { * } } = \sigma ( \{ f _ { \mathrm { t w h } } , f _ { \mathrm { t w t } } \} , \mathcal { U } )$$ Where:
  • $x$ is the initial component chosen uniformly from $\{s_{\varepsilon}, o_{\varepsilon}, r_{\varepsilon}, \tilde{e}\}$.
  • $f_{\mathrm{twr}}(x)$ retrieves all triples where $x$ appears as the relation.
  • $f_{\mathrm{twh}}(x)$ retrieves all triples where $x$ appears as the head entity.
  • $f_{\mathrm{twt}}(x)$ retrieves all triples where $x$ appears as the tail entity.
  • $f_{\mathrm{tw}^*}$ is chosen uniformly between $f_{\mathrm{twh}}$ and $f_{\mathrm{twt}}$.
  • If the generated $t_l$ happens to be identical to the edit triple $t_{\varepsilon}$, resampling is performed to ensure distinctness for locality samples.

• NMCS Algorithm: The NMCS algorithm is then applied uniformly to these initial triples to obtain multi-hop reasoning chains.

  • For generality: $\mathcal{T}_g = \mathtt{NMCS}(t_{\varepsilon}, \varnothing, 3, 4, \tilde{E})$

  • For locality: $\mathcal{T}_l = \mathtt{NMCS}(t_l, \{t_{\varepsilon}\}, 3, 4, \tilde{E})$

    Here is the full transcription of Algorithm 1: Neighborhood Multi-hop Chain Sampling (NMCS):

Algorithm 1 Neighborhood Multi-hop Chain Sampling (NMCS)
Input: Initial triple $t_0 = (s_0, r_0, o_0)$, blacklist $B$, max_hops $h$, max_trials $m$, entity_set $\tilde{E}$
Output: Multi-hop chains $\mathcal{T}$

1: $T = \{t_0\}$ # Subgraph triple set
2: $E_{\mathrm{add}} = \{s_0\}$ # Added nodes
3: if $o_0 \in \tilde{E}$ then $E_{\mathrm{add}} = E_{\mathrm{add}} \cup \{o_0\}$
4: $E_{\mathrm{end}} = \mathsf{clone}(E_{\mathrm{add}})$ # End nodes
5: # Expand both sides of $t_0$ to sample a
   chain of neighboring triples
6: while $\mathtt{len}(T) < h$ and $\mathtt{len}(E_{\mathrm{end}}) > 0$ do
7:     `e = \sigma(E_{\mathrm{end}}, \mathcal{U})`
8:     for $i = 1$ to $m$ do
9:         $f_{\mathrm{tw}^*} = \sigma(\{f_{\mathrm{twh}}, f_{\mathrm{twt}}\}, \mathcal{U})$
10:        $t = \sigma(f_{\mathrm{tw}^*}(e), \mathcal{U})$
11:        if $t = \emptyset$ or $t \in T$ then continue
12:        $(s, r, o) = t$
13:        if $\{s, o\} \cap E_{\mathrm{add}} = \{e\}$ then
14:            break # Acyclic, finish sampling
15:    $E_{\mathrm{end}} = E_{\mathrm{end}} \setminus \{e\}$
16:    if $t = \emptyset$ then continue
17:    $T = T \cup \{t\}$
18:    # Update added nodes and end nodes
19:    if $f_{\mathrm{tw}^*} = f_{\mathrm{twt}}$ then
20:        $E_{\mathrm{add}}, E_{\mathrm{end}} = E_{\mathrm{add}} \cup \{s\}, E_{\mathrm{end}} \cup \{s\}$
21:    else
22:        $E_{\mathrm{add}} = E_{\mathrm{add}} \cup \{o\}$
23:        if $o \in \tilde{E}$ then $E_{\mathrm{end}} = E_{\mathrm{end}} \cup \{o\}$
24: # Map entities to triples
25: `M = \text{defaultdict(list)}`
26: for $t$ in $T$ do
27:    $(s, r, o) = t$
28:    $M[s].\mathsf{append}(t), M[o].\mathsf{append}(t)$
29: # Randomly select object e and expand both sides
   to construct valid multi-hop QA chains
30: for $e$ in $\mathsf{shuffle}(\mathtt{list}(E_{\mathrm{add}}))$ do
31:    $\tau = [[t] \text{ for } t \text{ in } M[e]]$
32:    for $C$ in $\tau$ do
33:        $e_{\mathrm{ce}} = e$ # Current end to extend chain
34:        while true do
35:            $(s, r, o) = C[-1]$
36:            $e_{\mathrm{ce}} = s \text{ if } s \neq e_{\mathrm{ce}} \text{ else } o$
37:            if $\mathtt{len}(M[e_{\mathrm{ce}}]) = 1$ then
38:                break # Endpoint
39:            $t_1, t_2 = M[e_{\mathrm{ce}}]$
40:            $(s, r, o) = t = t_1 \text{ if } t_1 \neq C[-1] \text{ else } t_2$
41:            if $e_{\mathrm{ce}} = s$ and $\|f_{\mathrm{twr}}(r, o)\| > 1$ then
42:                break # Avoid multi-valued hop
43:            else if $e_{\mathrm{ce}} = o$ and $\|f_{\mathrm{twhr}}(s, r)\| > 1$ then
44:                break # Avoid multi-valued hop
45:            $C.\text{append}(t)$
46:    if any($[t_0 \text{ in } C \text{ for } C \text{ in } \mathcal{T}]$) then
47:        break # $t_0$ should in $\mathcal{T}$
48: # Reverse the order of triples in each chain
49: $\mathcal{T} = [C.\mathrm{reverse}() \text{ for } C \text{ in } \tau]$
50: return $\mathcal{T}$

Let's break down the NMCS algorithm:

• Input:

  • $t_0 = (s_0, r_0, o_0)$: The initial triple (either $t_{\varepsilon}$ for generality or $t_l$ for locality).
  • $B$: A blacklist of triples to avoid (e.g., $t_{\varepsilon}$ for locality samples).
  • $h$: Maximum number of hops (length of the chain).
  • $m$: Maximum number of trials for sampling a neighboring triple.
  • $\tilde{E}$: The full set of filtered entities.

• Part 1: Expanding the Initial Triple (Lines 1-23)

  • Initializes $T$ with $t_0$, E_add with $s_0$ (and $o_0$ if it's an entity), and E_end as a clone of E_add. E_add tracks all nodes (entities) that have been part of the sampled subgraph, while E_end specifically tracks nodes from which the chain can be extended.
  • A while loop (Line 6) continues as long as the current subgraph has fewer than max_hops triples and there are still end nodes to expand from.
  • In each iteration, an entity $e$ is uniformly sampled from E_end (Line 7).
  • A for loop (Line 8) attempts to find a new triple $t$ connected to $e$.
    • It uniformly chooses whether to sample a triple where $e$ is the head ($f_{\mathrm{twh}}$) or the tail ($f_{\mathrm{twt}}$) (Line 9).
    • It samples a triple $t$ using the chosen function (Line 10).
    • It checks if $t$ is empty or already in $T$ (Line 11); if so, it skips to the next trial.
    • It checks for acyclic expansion: if both $s$ and $o$ of $t$ are already in E_add and only one of them is $e$, it means adding $t$ would create a cycle and connect two previously separated parts of the graph, so it breaks (Line 13-14). This ensures a chain-like structure.
  • If no suitable triple is found after $m$ trials, $e$ is removed from E_end (Line 15).
  • If a suitable triple $t$ is found, it's added to $T$ (Line 17).
  • E_add and E_end are updated with the new entity from $t$ (Lines 19-23).

• Part 2: Constructing Multi-hop QA Chains (Lines 24-47)

  • $M$ is a dictionary that maps each entity to a list of triples it participates in within the sampled subgraph $T$ (Lines 25-28).
  • It iterates through each entity $e$ in E_add (Line 30) and attempts to form QA chains starting from triples connected to $e$.
  • For each triple $t$ connected to $e$, it starts a chain $C$ (Line 31).
  • A while loop (Line 34) extends the chain until an endpoint is reached or a multi-valued hop is avoided.
    • e_ce (current end) tracks the entity at the current end of the chain being extended (Line 33, 36).
    • If e_ce is only connected to one triple in $M$, it's an endpoint, and the chain extension stops (Line 37-38).
    • Otherwise, it finds the next triple $t$ connected to e_ce that is not already in the chain (Lines 39-40).
    • Crucially, it avoids multi-valued hops for intermediate nodes (Lines 41-44). A multi-valued hop would occur if, for example, an entity $s$ and relation $r$ could point to multiple objects, making the "next hop" ambiguous. $\|f_{\mathrm{twr}}(r, o)\| > 1$ checks if the relation $r$ and object $o$ can be a tail for multiple subjects. $\|f_{\mathrm{twhr}}(s, r)\| > 1$ checks if the subject $s$ and relation $r$ can point to multiple objects. This maintains the clarity of the multi-hop prompt.
    • The new triple $t$ is appended to the chain $C$ (Line 45).
  • After forming chains starting from $e$, it checks if any of these chains $\mathcal{T}$ contains the original initial triple $t_0$ (Line 46). If so, it breaks, ensuring that $t_0$ is part of the final set of chains.

• Finalization: The order of triples in each chain $C$ is reversed (Line 49) to prepare for natural language generation, and the set of chains $\mathcal{T}$ is returned.

  This NMCS algorithm systematically generates diverse multi-hop chains, incorporating various structural patterns that correspond to MH, RR, SER, and 1-NF evaluation criteria, including their combinations.

4.2.5. Step 5: Final Data Generation

The sampled structured data (edit triples, generality chains, locality chains) are converted into natural language text to form the final UniEdit dataset.

• LLM Conversion: Deepseek-V3 [48], a proprietary LLM, is used for this conversion. For each multi-hop sample, Deepseek-V3 first generates single-hop sentences for each triple, which are then merged into a coherent multi-hop natural language sentence.
• Prompt Engineering: Specific prompts are designed to guide Deepseek-V3 in converting structured knowledge into cloze-style sentences (where the object is left blank for prediction). These prompts emphasize grammatical accuracy, syntactical diversity, and avoidance of information leakage.
• Quality Control:

  • Automated Checks: Each generated prompt is checked to ensure it contains the subject and correctly points to the object.

  • Human Evaluation: A sample of the generated data undergoes human evaluation to assess fluency and logical consistency. The human assessment uses a 1-5 scale for both fluency and logical consistency, and Krippendorff's alpha is used to measure inter-rater agreement.

    This comprehensive process ensures that UniEdit provides a large-scale, diverse, and high-quality benchmark for evaluating LLM knowledge editing.

5. Experimental Setup

5.1. Datasets

The primary dataset used for experimentation and evaluation is UniEdit itself.

• Source: Constructed from Wikidata, the largest open-source knowledge graph.

• Scale: Comprises 311,142 entries. Each entry includes:

  • One editing sample.
  • One generality sample.
  • One locality sample. This results in a total of 933,426 samples (Union in Table 6).

• Domain Characteristics: Covers 25 common domains categorized into five sectors (Natural Sciences, Humanities, Social Sciences, Applied Sciences, Interdisciplinary Studies), ensuring broad and diverse knowledge coverage.

• Structural Diversity: The NMCS algorithm ensures a wide range of structural patterns for generality and locality samples, including various combinations of criteria like Multi-Hop (MH), Relation Reversal (RR), Same Entity Recognition (SER), Subject Alias (SA), Object Alias (OA), Subject Specificity (SS), Relation Specificity (RS), Object Specificity (OS), and 1-N Forgotten (1-NF).

• Data Types: The tail entities in UniEdit are not limited to wikibase-items (entities) but also include other data types such as string, quantity, time, math, globe-coordinate, and monolingual text.

  The following table (Table 6 from the original paper) shows data count statistics of UniEdit across different data types.

  Types	Data	Entity	Relation	String	Quantity	Time	Math	Coord.	MNLT
  Edit	311,142	363,014	1,770	13,434	29,211	26,669	2,377	4,940	167
  Generality	311,142	440,772	1,864	15,220	35,889	33,416	2,637	7,810	192
  Locality	311,142	394,889	1,784	16,126	31,417	31,427	1,730	19,506	128
  Union	933,426	703,282	1,934	44,780	96,517	91,512	6,744	32,256	487

Coord. refers to globe-coordinate and MNLT refers to monolingual text. The "Entity" column counts entities as tail entities in triples.

For evaluating general performance after sequential editing, additional general-purpose benchmarks were used:

• CSQA [56]: CommonsenseQA, evaluates commonsense knowledge.
• ANLI [57]: Adversarial NLI, measures reasoning ability.
• MMLU [58]: Measuring Massive Multitask Language Understanding, assesses exam-level proficiency.
• SQuAD-2 [59]: Stanford Question Answering Dataset 2.0, focuses on reading comprehension.

5.2. Evaluation Metrics

The evaluation metrics for UniEdit are based on the three core criteria for model editing, as defined in Section 3 and detailed in Appendix A: Reliability, Generality, and Locality.

5.2.1. Core Editing Metrics

• Reliability (Rel.): Measures if the edited LLM ($f'_{\mathrm{llm}}$) can correctly recall the specific edited knowledge itself.

  • Conceptual Definition: This metric checks whether the model successfully learned the new or updated fact it was explicitly instructed to acquire.
  • Formula: Not explicitly provided in the paper, but conceptually, it's typically an accuracy score. Given an editing request $\varepsilon_i = (q_{\varepsilon_i}, o_{\varepsilon_i})$, reliability is 1 if $f'_{\mathrm{llm}}(q_{\varepsilon_i}) = o_{\varepsilon_i}$ and 0 otherwise. The overall reliability is the average over all editing requests.
  • Symbol Explanation:
    • $f'_{\mathrm{llm}}$: The Large Language Model after editing.
    • $q_{\varepsilon_i}$: The input query corresponding to the $i$-th editing request.
    • $o_{\varepsilon_i}$: The expected output (object) for the $i$-th editing request.

• Generality (Gen.): Measures if the edited LLM can adjust its responses for queries related to the edited samples.

  • Conceptual Definition: This metric assesses the model's ability to generalize the learned edit to different phrasings, contexts, or multi-hop reasoning paths that logically derive from the edited fact.
  • Formula: Not explicitly provided, but typically accuracy on a set of generality queries $\mathcal{G}(\mathcal{E})$.
  • Symbol Explanation:
    • $f'_{\mathrm{llm}}$: The Large Language Model after editing.
    • $q_g$: A query from the set of generality queries.
    • $o_g$: The expected output for $q_g$.
    • $\mathcal{G}(\mathcal{E})$: The relevant neighborhood of the edit collection $\mathcal{E}$, representing queries related to the edited facts.

• Locality (Loc.): Measures if the edited LLM maintains consistency with the initial model ($f_{\mathrm{llm}}$) on queries unrelated to previously edited knowledge.

  • Conceptual Definition: This metric ensures that the editing process does not inadvertently alter correct, pre-existing knowledge or introduce undesirable side effects on unrelated facts.
  • Formula: Not explicitly provided, but typically accuracy on a set of locality queries $\mathcal{L}(\mathcal{E})$. Specifically, it checks if $f'_{\mathrm{llm}}(q_l) = f_{\mathrm{llm}}(q_l)$.
  • Symbol Explanation:
    • $f'_{\mathrm{llm}}$: The Large Language Model after editing.
    • $f_{\mathrm{llm}}$: The Large Language Model before editing.
    • $q_l$: A query from the set of locality queries.
    • $o_l$: The expected output for $q_l$.
    • $\mathcal{L}(\mathcal{E})$: The sample distribution independent of $\mathcal{E}$, excluding $\mathcal{E} \cup \mathcal{G}(\mathcal{E})$.

5.2.2. Specific Generality Criteria (from Appendix A.1)

The paper uses the NMCS algorithm to generate samples corresponding to various generality criteria:

• Rephrase (Rep): Evaluates if the LLM recalls the edited content with different syntactic structures.
• Multi-Hop (MH): Checks if the LLM can infer a final entity through a chain of related facts, starting from the edited knowledge.
• Relation Reversal (RR): Assesses if the LLM can infer the subject when given the object and the inverse relation of the edited fact.
• Same Entity Recognition (SER): Determines if the LLM can correctly identify two different prompts as referring to the same entity, especially relevant for double-chain prompts.
• Subject Alias (SA): Evaluates if the LLM can recognize an alias of the subject and produce the correct response.
• Object Alias (OA): Assesses if the LLM can predict an alias of the object.

5.2.3. Specific Locality Criteria (from Appendix A.2)

The locality criteria are defined by their overlap with the edit triple $t_{\varepsilon} = (s_{\varepsilon}, r_{\varepsilon}, o_{\varepsilon})$:

• Completely Unrelated (W/O): Queries where the subject, object, and relation are entirely different from the edit triple components.
• Subject Specificity (SS): Queries involving the edit subject $s_{\varepsilon}$ but a different relation ($r \neq r_{\varepsilon}$).
• Relation Specificity (RS): Queries involving the edit relation $r_{\varepsilon}$ but different subject and object (not overlapping with $s_{\varepsilon}, o_{\varepsilon}$).
• Object Specificity (OS): Queries involving the edit object $o_{\varepsilon}$ but a different relation ($r \neq r_{\varepsilon}$).
• 1-N Forgotten (1-NF): For a one-to-many relation, checks if the LLM forgets other valid objects when one specific object is edited out. This corresponds to the subject-relation-crossed case. An object-relation-crossed case (inverse of 1-NF) is also introduced.

5.2.4. Evaluation Methodology

• For reliability, generality, and locality scores on cloze-style questions: The predicted probability distribution over object candidates is obtained. The metric checks if each token of the expected object appears within the top-5 predictions.
• For judgment-type queries (e.g., SER): Evaluation is based on the top-1 prediction (e.g., "Yes" or "No").
• For multi-hop queries: If the LLM doesn't know the non-edited hops, single-hop samples are temporarily edited into the model to bridge the multi-hop queries, to isolate the multi-hop reasoning ability.

5.2.5. General Performance Metrics (for sequential editing evaluation)

For evaluating general performance after sequential editing on CSQA, ANLI, MMLU, and SQuAD-2:

• CSQA, ANLI, MMLU: Accuracy of multiple-choice selections. $$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$
• SQuAD-2: Inverse of Perplexity (PPL) on the answer text.
  • Conceptual Definition: Perplexity measures how well a probability model predicts a sample. A lower perplexity indicates a better fit for the language model. The inverse is used so that a higher score indicates better performance, reflecting the model's confidence in generating correct answers.
  • Formula: Perplexity (PPL) for a sequence of $N$ tokens $W = (w_1, w_2, \dots, w_N)$ given a language model $M$ is defined as: $$\text{PPL}(W) = \left( \prod_{i=1}^N \frac{1}{P(w_i | w_1, \dots, w_{i-1})} \right)^{1/N} = \exp\left( - \frac{1}{N} \sum_{i=1}^N \log P(w_i | w_1, \dots, w_{i-1}) \right)$$ The inverse of Perplexity would then be: $$\text{Inverse PPL}(W) = \frac{1}{\text{PPL}(W)} = \exp\left( \frac{1}{N} \sum_{i=1}^N \log P(w_i | w_1, \dots, w_{i-1}) \right)$$
  • Symbol Explanation:
    • $W$: A sequence of tokens ($w_1, w_2, \dots, w_N$).
    • $N$: The total number of tokens in the sequence.
    • $P(w_i | w_1, \dots, w_{i-1})$: The probability of token $w_i$ given the preceding tokens, as assigned by the language model $M$.
    • $\log$: Natural logarithm.
    • $\exp(\cdot)$: Exponential function ($e^{(\cdot)}$).

5.3. Baselines

The paper evaluates the proposed benchmark against various LLM backbones and editing methods.

5.3.1. LLM Backbones

• GPT2-XL (1.5B): A 1.5 billion parameter transformer model, an earlier generation of large language models.

• GPT-J (6B): A 6 billion parameter transformer model.

• LLaMa-3.1 (8B): A more recent 8 billion parameter transformer model from the LLaMA family.

  These backbones represent different scales and architectural variations of LLMs.

5.3.2. Editors

The evaluated editors span both parameter modification and external module-based approaches:

• Fine-Tuning (FT): A basic parameter modification method where an intermediate layer of the LLM is fine-tuned directly on the edit samples until a maximum number of iterations. It tends to overfit.
• ROME [16]: Locate-then-Edit method that uses causal tracing to find influential layers and performs a rank-one update on the weight matrix.
• AlphaEdit [33]: An improvement on ROME that projects updates into the null space of preserved knowledge to enhance locality.
• SERAC [37]: An external module-based method that maintains edit samples in memory and uses a scope classifier (e.g., multi-qa-mpnet-base-dot-v1) to identify relevant inputs, routing them to a counterfactual model (e.g., 0PT125) for modified responses.
• T-Patcher [38]: Parameter modification method that incorporates and trains extra neurons within the FFN (Feed-Forward Network) of the LLM's final layer for edited knowledge.
• GRACE [39]: External module-based method that introduces retrieval-based adapters for continual editing, using a dictionary-style structure to create new mappings for representations needing modification based on token-based linear distance retrieval.
• IKE [44]: In-context learning method that provides training samples as contextual information in the prompt, allowing the LLM to adapt its responses without internal parameter changes. It does not support sequential edits.

5.3.3. Experimental Environment

• Hardware: High-performance computing platform with dual Intel Xeon Gold 5320 CPUs (52 cores) and two NVIDIA A800 GPUs.
• Software: Ubuntu 20.04.6 LTS operating system, Python 3.11.9.
• Hyperparameters: Specific hyperparameters for each editor (iterations, optimizers, learning rate, modified layers) are detailed in Table 8 in Appendix D.

6. Results & Analysis

6.1. Core Results Analysis

The experiments conducted on UniEdit provide significant insights into the capabilities and limitations of current LLM editing methods.

6.1.1. Overall Performance

The following table (Table 2 from the original paper) presents the overall editing performance on UniEdit, with "W/O" indicating results of pre-edit LLMs. "Rel.", "Gen.", and "Loc." are the abbreviations of reliability, generality, and locality, respectively.

• Pre-edit LLM Performance (W/O): The unedited LLMs (W/O) show low Reliability and Generality scores (28-52%), which is expected given the long-tail distribution of domain knowledge and the need for new information. Their Locality is 100% because they haven't been modified yet, thus maintaining consistency with their original state on unrelated facts.
• High Reliability: Most editors achieve high Reliability scores (close to 100%), indicating they effectively inject the intended edits. Fine-Tuning (FT) even achieves a perfect 100% Reliability, but this often comes at the cost of generality and locality.
• Struggle with Generality: A key finding is that editors generally struggle with the challenging generality evaluation in UniEdit.
  • L&E-based methods (ROME, AlphaEdit) and others like T-Patcher and GRACE show relatively low generality scores (28-57%). This suggests they are effective at direct edits but fail to generalize that knowledge to broader contexts or related queries. The paper attributes this to their focus on direct backpropagation through edit statements, often overlooking wider applicability.
  • IKE and SERAC achieve the best generality performance (76-89%). IKE leverages in-context learning, while SERAC benefits from edit training to learn priors. However, this emphasis on generality can sometimes lead to slightly lower locality scores.
• Locality Trade-offs:
  • Methods like ROME, T-Patcher, GRACE, and AlphaEdit generally maintain high locality (93-99%), meaning they largely preserve existing knowledge. GRACE stands out with near-perfect locality due to its token-based linear distance retrieval mechanism preventing interference. However, this mechanism also limits its generality.
  • FT, IKE, and SERAC show slightly lower locality scores (80-93%), indicating a trade-off where improving reliability and generality can sometimes affect unrelated knowledge.

6.1.2. Performance Across Domains

The following figure (Figure 4 from the original paper) illustrates the editing performance on UniEdit across domains, with each metric representing the average result across three post-edit backbones.

该图像是一个表格，展示了不同编辑方法在 UniEdit 基准测试中的编辑性能，涵盖多个领域。各项指标的数值代表三个后编辑架构的平均结果，色带表示可靠性（绿色）、一般性（蓝色）和局部性（红色），且数值已在各领域中标准化。

• Reliability Consistency: Editor performance on reliability shows minimal variation across domains, consistently high.
• Generality Variation: All editors exhibit a consistent pattern for generality: higher scores in Natural Sciences and Humanities, and lower scores in Social Sciences and Applied Sciences. The hypothesis is that this stems from a distributional bias in LLMs' pretraining corpora, where knowledge in well-represented domains generalizes better.
• Locality Inconsistency: Locality performance across domains is less consistent between editors. However, all editors achieve relatively high scores in Humanities, possibly due to the models' greater exposure to literary content during pretraining.
• Implication: These observations highlight the importance of open-domain knowledge editing, especially for underrepresented or low-resource domains that receive less attention in existing pretraining corpora.

6.1.3. Performance Across Evaluation Criteria

The following figure (Figure 5 from the original paper) shows the editing performance across combinations of generality and locality evaluation criteria.

该图像是图表，展示了不同模型在一般性和局部性评估标准上的编辑性能。图中左半部分的雷达图显示单一标准的评估结果，右半部分则反映了与其他标准结合后的结果。

• Generality Difficulty with Complexity: For generality, most editors show lower scores on more complex evaluations, such as combinations of Rep, OA, and SA, or combinations like RR, MH, OA, and SA, compared to single criteria. This suggests that the more intricate the prompt structure (i.e., when the edit information is part of a complex natural language sentence covering multiple criteria), the harder it is for the injected knowledge to be recognized and applied.
  • An exception is IKE's performance on OA and the combination of RR, MH, OA. This is attributed to a sampling bias in UniEdit where this combination might be more frequent than standalone OA, leading to better performance through in-context learning demonstrations.
• Locality and Complexity: For locality, adding MH to the evaluation (e.g., SS vs. $SS + MH$) does not necessarily lead to a performance decline; sometimes, performance even improves.
  • This is counter-intuitive but explained by the underlying principle: complex locality sentences reduce the likelihood of overlapping components with the edited knowledge, thereby preventing interference with the model's original response. If the locality query is sufficiently complex and distinct, it's less likely to be mistakenly identified as related to the edit.
  • An exception is the combination of OS and RS, which creates dual overlap with the edit sample, making the evaluation more challenging than standalone OS.
• Overall Impact of Complexity: The paper concludes that adding complexity significantly increases the challenge for generality tasks more than for locality tasks.

6.2. Ablation Studies / Parameter Analysis

6.2.1. Domain Generalization of Edit Training

The following figure (Figure 6 from the original paper) shows the editing performance of SERAC trained on five domains from different sectors in UniEdit, using GPT2-XL as the backbone.

该图像是一个表格，展示了SERAC在UnIEDIT中五个不同领域的编辑性能。表格中以颜色带表示可靠性（绿色）、一般性（蓝色）和局部性（红色），以便于不同领域的性能比较。

• Domain-Specific Training Benefits: The first five columns of Figure 6 clearly show that training SERAC on a specific domain (e.g., Chemistry) results in better performance when tested on that corresponding domain.
• Cross-Domain Transfer: Similar or overlapping training and testing domains tend to yield better results in reliability and generality. For example, SERAC trained on Chemistry performs better on Biology, and training on Data Science yields better results on Computer Science. This suggests some degree of knowledge transfer between related domains.
• Locality Robustness: For locality, the results show minimal variation across different training domains. This is expected because locality samples are usually designed to have limited relevance to any specific domain, involving only a small portion of domain-specific elements.
• Impact of Training Data Scale: Compared to the overall performance of SERAC (Table 2), where it was likely trained on a more diverse and larger dataset, its performance (especially generality) decreases significantly when trained only on five specific domains. This finding underscores the critical importance of the scale and breadth of training data for the effectiveness of edit training-based editors.

6.2.2. Sequential Editing Performance

The following figure (Figure 10 from the original paper) shows the sequential editing performance of different editors on UniEdit across three backbones.

该图像是图表，展示了不同编辑器在 UniEdit 基准测试中对三种语言模型的顺序编辑性能。图表包括 GPT2-XL、GPT-1 和 LLaMa 3.1 的准确率随编辑次数变化的趋势，横轴为顺序编辑次数，纵轴为准确率，同时包含 W/O 情况下的表现。

• Performance Degradation: As the number of edits increases, editing performance generally declines across most editors and backbones. This highlights the challenge of continually updating LLMs without degrading their performance.
• ROME's Vulnerability: ROME shows the most severe drop in performance during sequential editing. This is often attributed to accumulated weight updates causing harmful parameter norm growth and disrupting model stability.
• AlphaEdit's Robustness: AlphaEdit, by leveraging null-space projection and cached updates, significantly improves robustness against increasing edit counts compared to ROME-style methods.
• Retrieval-Based Robustness: GRACE and SERAC, which incorporate retrieval mechanisms, demonstrate the highest robustness to sequential edits. GRACE's performance remains nearly unchanged even after a large number of edits, indicating its effectiveness in isolating edits. However, its strong assumption of a linear semantic structure limits its generality (scores are similar to the unedited model). SERAC benefits from edit training, facilitating the retrieval of semantically related knowledge and leading to strong generality and robustness.
• Importance of Edit Training Datasets: The robust performance of SERAC emphasizes the importance of constructing effective edit training datasets to enhance knowledge editing, especially in sequential scenarios.

6.2.3. General Performance after Sequential Editing

The following table (Table 9 from the original paper) shows the General Performance of LLaMA-3 (8B) after 1,000 edits on UniEDIT, tested on four benchmarks: CSQA, ANLI, MMLU, and SQuAD-2.

• Catastrophic Forgetting in L&E: L&E-type methods like ROME and T-Patcher suffer significant performance degradation on general-purpose benchmarks after 1,000 edits. Their average scores drop drastically (e.g., ROME from 50.41% to 19.07%), especially on SQuAD-2, indicating severe catastrophic forgetting due to accumulated weight updates affecting fundamental model capabilities.
• AlphaEdit Mitigation: AlphaEdit mitigates this issue by projecting updates into the null space, showing a much smaller drop in general performance (from 50.41% to 49.24%), demonstrating improved locality and stability.
• External Module Robustness: External module-based methods (SERAC, GRACE) generally perform well, maintaining near pre-edit performance on general benchmarks. This is because they can bypass inputs semantically distant from edited knowledge, thus avoiding interference with the base model's general capabilities.
• Fine-Tuning Behavior: Surprisingly, Fine-Tuning (FT) preserves general performance better than some L&E methods in sequential editing, although it still shows a noticeable drop (from 50.41% to 38.82%).
• Correlation with Locality: A positive correlation is observed between general performance degradation and locality degradation. This reinforces the idea that general evaluation samples can often be viewed as a type of locality evaluation if they are truly independent of the edited samples.

6.3. Instance Analysis

The paper provides case studies (Tables 10, 11, and 12 in Appendix D) to illustrate the behavior of editors on specific UniEdit instances. These tables show GPT2-XL outputs for Reliability, Generality (with MH or RR and OA), and Locality (OS or SS) tasks.

The following table (Table 10 from the original paper) shows GPT2-XL outputs after applying various editors to a representative astronomy domain case in UNIEDIT.

The following table (Table 11 from the original paper) shows GPT2-XL outputs after applying various editors to a representative art domain case in UniEdit.

The following table (Table 12 from the original paper) shows GPT2-XL outputs after applying various editors to a representative computer science domain case in UNIEDIT.

• Pre-edit Failure: In all three cases, GPT2-XL fails to produce correct answers before editing, as expected.
• Reliability Success: After editing, most editors successfully enable the model to follow the edit instructions, yielding high reliability.
• Locality Preservation: Most editors also generally preserve the original model's output on locality samples. However, IKE sometimes shows poor locality (e.g., in Table 10), where its output includes parts of the in-context learning instruction, indicating interference.
• Generality Divergence: The most significant differences among editors appear in their generality performance.
  • Multi-hop Generality (Tables 10 & 11): Even when intermediate hops are edited into the model, only IKE consistently predicts the final answer correctly for multi-hop generality. This highlights a common weakness in other editors to integrate and leverage multiple related edits into a coherent multi-hop reasoning chain.
  • Non-Multi-hop Generality (Table 12): For non-multi-hop generality (e.g., Relation Reversal), most editors (except SERAC) still fail to generalize the reversed relational fact, producing tokens identical to the original model. SERAC, while producing the correct answer, then generates repetitive or meaningless tokens, suggesting that the quality of its counterfactual model is crucial for good responses.
• Conclusion: The instance analysis confirms the overall findings: generality remains a substantial challenge for most editors, particularly for complex and multi-hop scenarios, while locality is generally better preserved by many methods, especially those with strong isolation mechanisms. IKE and SERAC show promise for generality but can have locality issues or generate repetitive outputs, respectively.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces UniEdit, a novel and comprehensive benchmark for Large Language Model (LLM) knowledge editing, grounded in open-domain knowledge. By leveraging Wikidata and proposing a unified Neighborhood Multi-hop Chain Sampling (NMCS) algorithm, UniEdit effectively integrates and extends various existing evaluation criteria for generality and locality, including their complex combinations. This approach significantly increases the challenge for LLM editing evaluation. The extensive experimental analysis across multiple LLMs and editing methods yields several key insights:

1. Generality remains a major hurdle: Editors, particularly those based on the Locate-then-Edit (L&E) paradigm, exhibit notable limitations in handling complex generality tasks.
2. Domain-specific performance: Editing performance varies across different knowledge domains, underscoring the necessity for improved low-resource knowledge editing.
3. Complexity's dual effect: Higher sample complexity (e.g., multi-hop reasoning) increases the difficulty of generality but can, paradoxically, ease locality evaluation by reducing interference.
4. Training data impact: The scale and diversity of training data are crucial for the performance of edit training-based editors.

7.2. Limitations & Future Work

The authors acknowledge several limitations and propose future research directions:

• Language Scope: UniEdit currently focuses solely on English and lacks evaluations for other languages.
  • Future Work: Expanding the benchmark to include multilingual knowledge editing is a clear next step.
• Modality Limitation: The benchmark emphasizes a single language modality and does not include challenging evaluations for other modalities, such as vision LLM editing.
  • Future Work: Leveraging multimodal content from Wikidata (e.g., videos, images) to develop more comprehensive multimodal editing benchmarks.
• Granularity and Coverage: While UniEdit is open-domain, it could further explore more fine-grained, long-tail domains and incorporate even more diverse evaluation criteria.
  • Future Work: Deeper investigation into niche knowledge areas and broader exploration of editing nuances.

7.3. Personal Insights & Critique

UniEdit is a highly valuable contribution to the field of LLM knowledge editing. Its strength lies in its unified, open-domain, and comprehensive nature, directly addressing the fragmented and narrow scope of previous benchmarks. The NMCS algorithm is particularly innovative, providing a systematic way to generate complex generality and locality samples that are crucial for truly testing the robustness of editing methods. The detailed statistical analysis confirming the scale and diversity of the benchmark further strengthens its utility.

The findings highlight a critical gap: while many editing methods can reliably inject specific facts, their ability to generalize that knowledge to related contexts or to integrate multiple edits into a coherent knowledge base is still limited. This suggests that future research in model editing should move beyond simple reliability and rephrasing towards more sophisticated knowledge reasoning and integration capabilities. The observation that locality can sometimes improve with increased query complexity for unrelated facts is an intriguing nuance, suggesting that sufficiently complex, distinct contexts can naturally shield models from unintended edits.

A potential area for future critique or investigation could be the inherent biases of the source data (Wikidata) and the proprietary LLMs used for keyword generation and text conversion. While the authors discuss bias propagation from Wikidata (e.g., disproportionate Indian street addresses, attribute richness imbalance) and mitigate some issues through targeted filtering and sampling decay, the reliance on GPT-4 for keyword generation introduces distributional biases from its pretraining corpus. Similarly, Deepseek-V3's role in natural language conversion, while constrained, still involves a trade-off between adherence and freedom. While the authors' discussion is transparent, the extent to which these LLM-induced biases might subtly shape the benchmark's characteristics and evaluation challenges is worth deeper exploration.

The toolkit and pipeline provided could be highly transferable. Researchers could adapt it to create benchmarks for other languages by integrating multilingual KGs or translating existing Wikidata content. It could also be used to explore editing within specific sub-domains of interest or even adapted for tasks beyond factual editing, such as policy adherence or ethical guideline enforcement in LLMs. Overall, UniEdit sets a new standard for evaluating LLM knowledge editing and provides a fertile ground for developing more intelligent and robust editing solutions.