Harmful fine-tuning (HFT), performed directly on open-source LLMs or through Fine-tuning-as-a-Service, breaks safety alignment and poses significant threats. Existing methods aim to mitigate HFT risks by learning robust representation on alignment data or making harmful data unlearnable, but they treat each data sample equally, leaving data vulnerability patterns understudied. In this work, we reveal that certain subsets of alignment data are consistently more prone to forgetting during HFT across different fine-tuning tasks and exhibit lower robustness compared to other subsets. Inspired by these findings, we propose Vulnerability-Aware Alignment (VAA), which calculates data vulnerability, partitions data into "vulnerable" and "invulnerable" groups, and encourages balanced learning using a group distributionally robust optimization (Group DRO) framework. Specifically, VAA learns an adversarial sampler that samples examples from the currently underperforming group and then applies group-dependent adversarial perturbations to the data during training, aiming to encourage a balanced learning process across groups. Experiments across four fine-tuning tasks demonstrate that VAA significantly reduces harmful scores while preserving downstream task performance, outperforming state-of-the-art baselines.

1. Bibliographic Information

• Title: Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
• Authors: Liang Chen, Xueting Han, Li Shen, Jing Bai, Kam-Fai Wong. Their affiliations are not explicitly listed in the header, but the acknowledgments mention CUHK (The Chinese University of Hong Kong).
• Journal/Conference: The paper is available on OpenReview, which is a platform commonly used for submissions to top-tier conferences like ICLR, NeurIPS, and ICML. The link suggests it was submitted for review, and the content format is typical of a conference paper.
• Publication Year: The paper cites sources from 2024, and the OpenReview link is active, suggesting the work is from 2024 or later.
• Abstract: The paper addresses the problem of Harmful Fine-Tuning (HFT), where the safety alignment of Large Language Models (LLMs) is compromised during fine-tuning. The authors observe that existing methods treat all alignment data equally, ignoring that certain subsets are more "vulnerable" to being forgotten. They propose Vulnerability-Aware Alignment (VAA), a method that first identifies and partitions data into "vulnerable" and "invulnerable" groups. It then uses a Group Distributionally Robust Optimization (Group DRO) framework to ensure balanced learning across these groups. VAA employs an adversarial sampler to focus on the underperforming group and applies group-specific perturbations to enhance robustness. Experiments show VAA significantly reduces harmfulness while maintaining performance on downstream tasks, outperforming existing methods.
• Original Source Link: The paper is available at https://openreview.net/forum?id=EMHED4WTHT, with the PDF at https://openreview.net/pdf?id=EMHED4WTHT. Its status appears to be under review or recently accepted at a conference.

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: The safety measures built into aligned Large Language Models (LLMs) are fragile. They can be easily broken through a process called Harmful Fine-Tuning (HFT), where a user fine-tunes the model on a custom dataset. This is a major threat, as even fine-tuning on purely benign (harmless) data can degrade safety alignment, a phenomenon known as catastrophic forgetting.
  • Importance & Gaps: As open-source LLMs and Fine-tuning-as-a-Service platforms become ubiquitous, protecting against HFT is critical. Previous methods, known as "alignment-stage" defenses, try to make the initial safety alignment more robust. However, they have a key limitation: they treat every piece of alignment data as equally important and equally robust. The paper argues this is a flawed assumption and that some data points are inherently more likely to be "forgotten" during fine-tuning. This pattern of uneven forgetting is an understudied vulnerability.
  • Fresh Angle: This paper introduces a data-centric perspective. Instead of focusing solely on the model's parameters or representations, it investigates the characteristics of the alignment data itself. The core innovation is to identify which data points are "vulnerable" and then design an alignment process that specifically strengthens the model's learning on this vulnerable subset.

• Main Contributions / Findings (What):

  • Empirical Discovery of Uneven Vulnerability: The paper provides strong evidence that certain subsets of alignment data are consistently more prone to being forgotten during HFT. This vulnerability is data-dependent and transferable across different fine-tuning tasks and even across different model architectures. Vulnerable examples are also shown to be less robust to model weight perturbations.
  • Vulnerability-Aware Alignment (VAA) Framework: The paper proposes VAA, a novel alignment-stage defense method. VAA operates in two stages:
    1. Group Estimation: It partitions the alignment data into "vulnerable" and "invulnerable" groups based on their forgetting behavior during a simulated HFT process.
    2. Vulnerability-Aware Training: It uses a Group Distributionally Robust Optimization (Group DRO) framework to force the model to learn both groups equally well, focusing on the worst-performing group at each step. This is implemented as a two-player game between an adversarial sampler and the LLM.
  • Superior Performance: Experiments on four fine-tuning tasks and two LLMs (Llama2-7B, Qwen2.5-7B) show that VAA significantly reduces the model's harmfulness after HFT while preserving its performance on the intended downstream task. It consistently outperforms state-of-the-art baselines like Vaccine, RepNoise, and Booster.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Large Language Models (LLMs): These are deep learning models with billions of parameters, trained on massive text data to understand and generate human-like language (e.g., GPT-4, Llama).
  • Fine-Tuning: The process of taking a pre-trained LLM and further training it on a smaller, task-specific dataset to adapt its behavior.
  • Safety Alignment: A crucial step after pre-training where an LLM is trained to be helpful and harmless. This usually involves techniques like Supervised Fine-Tuning (SFT) on safety data and Reinforcement Learning from Human Feedback (RLHF) to avoid generating toxic, biased, or dangerous content.
  • Harmful Fine-Tuning (HFT): A malicious or unintentional process where fine-tuning an aligned LLM erodes its safety alignment, making it susceptible to generating harmful responses. This can happen even if the fine-tuning dataset is small or contains only benign examples.
  • Distributionally Robust Optimization (DRO): A machine learning optimization paradigm. Instead of minimizing the average loss over the training data (known as Empirical Risk Minimization or ERM), DRO aims to minimize the worst-case loss over a set of possible data distributions. This makes the model more robust to shifts or imbalances in the data.
  • Group DRO: A specific type of DRO where the data is partitioned into known groups (e.g., based on demographics, or in this case, vulnerability). The goal is to minimize the loss of the worst-performing group, ensuring fairness and balanced performance across all groups.

• Previous Works: The paper categorizes HFT mitigation methods into three stages:

  1. Alignment-Stage Methods: These methods modify the initial safety alignment process to make the model inherently more robust before it is handed over for fine-tuning. This is the category VAA belongs to.
     • Vaccine (Huang et al., 2024d): Adds perturbations to the model's hidden embeddings during alignment to reduce "embedding drift" and prevent forgetting.
     • RepNoise (Rosati et al., 2024a): Makes harmful data "unlearnable" by manipulating the model's representations.
     • Booster (Huang et al., 2024b): Uses a regularizer to slow down the rate at which the model learns from harmful examples.
  2. Fine-Tuning-Stage Methods: These methods regulate the fine-tuning process itself, for example, by adding constraints or regularizers during user-led fine-tuning.
  3. Post-Fine-Tuning-Stage Methods: These methods attempt to "repair" a model after it has been compromised by HFT.

• Differentiation: While previous alignment-stage methods like Vaccine and Booster focus on making the model's parameters or representations robust, they treat all alignment data points as equally important. VAA's key innovation is its data-centric approach. It is the first to systematically study the uneven vulnerability of alignment data. By identifying and explicitly modeling this vulnerability using a Group DRO framework, VAA directly tackles the root cause of why some safety knowledge is forgotten more easily than others, leading to a more targeted and effective defense.

4. Methodology (Core Technology & Implementation)

The paper's core idea is to first identify which alignment examples are easily "forgotten" during HFT and then use a robust training procedure to force the model to learn these "vulnerable" examples just as well as the "invulnerable" ones. This is done in two main stages.

Stage 1: Group Estimation by Vulnerability

The first step is to partition the alignment dataset into two groups: vulnerable and invulnerable.

该图像是示意图，展示了论文中对齐数据集被划分为两个子集的过程：G1（无弱点组，Invulnerable）和G2（有弱点组，vulnerable），并展示了各组在经验分布上的占比，体现了数据脆弱性分组的核心思想。

• Defining and Calculating Data Vulnerability: The paper defines vulnerability as the tendency of an alignment example to be "forgotten" during HFT. Forgetting is measured by tracking the Harmful Score (HS) of each example throughout a simulated fine-tuning process. The HS for an example is a binary value: 1 if the model's output for it is harmful, 0 otherwise. An example is considered "forgotten" at a time step if its HS increases from its initial value (e.g., from 0 to 1).
• The ForgotNum Metric: To quantify vulnerability, the authors introduce ForgotNum. For each alignment example $i$, they count the total number of times it is forgotten over $T$ training steps of HFT: $$\mathrm { F o r g o t N u m } _ { i } = \sum _ { t = 1 } ^ { T } \left( \mathbb { I } ( \mathbf { H } \mathbf { S } _ { i } ^ { t } > \mathbf { H } \mathbf { S } _ { i } ^ { 0 } ) \right)$$ where $\mathrm{HS}_i^t$ is the harmful score of example $i$ at step $t$, $\mathrm{HS}_i^0$ is the initial score, and $\mathbb{I}(\cdot)$ is the indicator function. A higher ForgotNum indicates greater vulnerability.
• Proxy Fine-Tuning for Grouping: Since the actual downstream fine-tuning data is unknown, the authors use a proxy dataset (Alpaca dataset mixed with 10% harmful data) to simulate HFT. They fine-tune an aligned model on this proxy data and calculate the ForgotNum for every example in the original alignment set.
• Partitioning: Examples with ForgotNum > 0 are classified as vulnerable, and those with ForgotNum = 0 are classified as invulnerable. This grouping serves as a prior for the next stage.

Empirical Motivation: Data-Dependent and Transferable Forgetting

The paper provides strong empirical justification for this grouping strategy.

该图像是图表，展示了不同中毒率和多种微调任务下的遗忘行为分析。(a)部分展示了SST2任务在中毒率0%、10%、20%时，遗忘（Forget）、共同遗忘（Common Forget）、未遗忘（Unforget）、共同未遗忘（Common Unforget）四类数据所占比例；(b)部分则展示了固定10%中毒率下，SST2、GSM8K和AGNews三个任务中的遗忘模式对比。数据显示部分样本存在一致的遗忘倾向。

As shown in Figure 2, the analysis reveals:

1. Forgetting is Data-Dependent: In Figure 2(a), even as the proportion of harmful data (poison rate) increases, a significant portion of the forgotten examples (the shaded red "Common Forget" region) is the same across all settings. This shows that certain examples are inherently prone to being forgotten.

2. Forgetting is Transferable: In Figure 2(b), when fine-tuning on completely different tasks (SST2, GSM8K, AGNews), there is still a substantial overlap in which alignment examples are forgotten. This transferability justifies using a single proxy dataset for grouping.

3. Vulnerable Data is Less Robust:

   该图像是三维损失曲面图，展示了模型在不同数据组上的鲁棒性行为。左图对应易受攻击的“易损”数据，右图对应“不易损”数据。结果表明模型对“易损”数据的扰动更敏感，鲁棒性较差。

Figure 3 shows the loss landscape for vulnerable vs. invulnerable data. The landscape for vulnerable data (left) is much "sharper" and more chaotic, while for invulnerable data (right) it is "flatter." A flatter loss landscape indicates greater robustness to changes in model weights. This suggests vulnerable examples are less robustly learned during standard alignment.

Stage 2: Vulnerability-Aware Alignment (VAA)

With the data partitioned, VAA uses a Group DRO framework to ensure balanced learning. This is framed as a two-player game between an adversarial sampler and the LLM.

该图像是论文中描述Vulnerability-Aware Alignment方法的示意图，展示了包含5个步骤的训练流程：Step 1通过对抗采样获得分组权重$G_i$，Step 2对各组数据施加组别相关的对抗扰动$\varepsilon_i$，Step 3计算损失，Step 4执行镜像上升优化，Step 5进行梯度下降更新LLM模型参数，体现了分组鲁棒优化框架下的平衡学习过程。

• Principles: Standard training, or Empirical Risk Minimization (ERM), optimizes the average loss across all data: $$\widehat { \theta } _ { \mathtt { E R M } } : = \arg \operatorname* { m i n } _ { \theta \in \Theta } \mathbb { E } _ { ( x , y ) \sim \hat { P } } [ \ell ( \theta ; ( x , y ) ) ]$$ This is problematic because the invulnerable group is typically much larger, so its gradients dominate, leading to under-learning of the vulnerable group (a phenomenon called "gradient starvation"). Group DRO addresses this by minimizing the worst-case loss across the groups: $$\hat { \theta } _ { \mathrm { D R O } } = \arg \operatorname* { m i n } _ { \theta \in \Theta } \Big \{ \operatorname* { s u p } _ { G _ { i } \in \mathcal { Q } } \mathbb { E } _ { ( x , y ) \sim G _ { i } } \big [ f _ { i } ( \theta ; ( x , y ) ) \big ] \Big \}$$ where $G_i$ is a data group (vulnerable or invulnerable) and $f_i(\theta)$ is a robust objective function for that group. This forces the model to perform well on all groups, especially the one it currently finds most difficult.

• Steps & Procedures (The Two-Player Game):

  1. Adversarial Sampler (Player 1): At each training step, a sampler chooses a group to train on. It maintains a probability distribution $q$ over the groups. It adversarially updates $q$ to sample more frequently from the group with the highest current loss (the "hard" group).
  2. LLM (Player 2): The LLM receives a batch of data from the group selected by the sampler. It then updates its parameters $\theta$ to minimize its loss on this challenging data.

• Mathematical Formulas & Key Details:

  1. The Robust Objective $f_i(\theta)$: To proactively handle the parameter shifts caused by HFT, VAA uses a robust objective that combines the standard loss with a perturbed loss. $$f _ { i } ( \theta ) = ( 1 - \lambda ) \ell _ { i } ( \theta ) + \lambda \ell _ { i } ( \theta + \epsilon _ { i } )$$

  • $\ell_i(\theta)$: The standard loss (e.g., cross-entropy) for group $G_i$.
  • $\epsilon_i$: A group-dependent adversarial perturbation applied to the model's weights $\theta$. It is calculated to maximize the loss for that group.
  • $\lambda$: A hyperparameter that balances standard learning and robust learning. It is gradually increased from 0 to 1 during training (curriculum learning).

  2. Learning the Adversarial Sampler $q$: The sampler's distribution $q$ is updated using mirror ascent, a technique for optimization on a probability simplex. This results in an update rule similar to the EXP3 algorithm from adversarial bandits: $$q _ { i } ^ { ( t ) } = \frac { q _ { i } ^ { ( t - 1 ) } \exp { \left( \eta _ { q } f _ { i } \big ( \theta ^ { ( t - 1 ) } \big ) \right) } } { Z }$$

  • $q_i^{(t)}$: The probability of sampling group $i$ at step $t$.
  • $f_i(\theta^{(t-1)})$: The loss of group $i$ from the previous step. This acts as the "reward" signal.
  • $\eta_q$: The step size for the sampler.
  • $Z$: A normalization constant to ensure probabilities sum to 1. This rule increases the sampling probability for groups that currently have a higher loss, forcing the LLM to focus on them.

  3. Training the LLM $\theta$: Once a group $G_i$ is sampled, the LLM performs a standard gradient descent step on the robust objective $f_i(\theta)$ for a batch of data from that group.

This iterative process continues until convergence, at which point the LLM has learned to perform equally well on both vulnerable and invulnerable data, making its safety alignment more robust.

5. Experimental Setup

• Datasets:

  • Alignment Dataset: 2,000 safe samples from an enriched version of the BeaverTails dataset.
  • Grouping Proxy Dataset: The Alpaca dataset mixed with 10% harmful data (from BeaverTails, but non-overlapping with fine-tuning or test sets).
  • Fine-tuning Datasets: Four diverse tasks to simulate user fine-tuning:
    • SST-2: Sentiment classification.
    • AG News: News topic classification.
    • GSM8K: Grade-school math reasoning.
    • AlpacaEval: Instruction following. For HFT simulation, these datasets were mixed with harmful examples from BeaverTails at a poison ratio $p$. Default settings were $p=10\%$ and $n=1,000$ total samples.

• Evaluation Metrics:

  1. Harmful Score (HS):
     • Conceptual Definition: Measures the model's safety. It is the percentage of harmful responses generated by the model when tested on a set of unseen malicious instructions. A lower HS is better.
     • Calculation: The model generates responses to 1,000 malicious prompts from the BeaverTails test set. A separate moderation model classifies each response as harmful or not. The HS is the proportion of harmful responses.
  2. Fine-tuning Accuracy (FA):
     • Conceptual Definition: Measures the model's utility or performance on the intended downstream task. A higher FA is better.
     • Calculation: This is the standard accuracy metric for the given task, calculated on the task's test set (e.g., classification accuracy for SST-2 and AG News).

• Baselines:

  • SFT: Standard Supervised Fine-Tuning. This is the vanilla approach with no special defense.
  • Vaccine: An alignment-stage defense that adds perturbations to hidden embeddings.
  • RepNoise: An alignment-stage defense that makes harmful data unlearnable.
  • Booster: An alignment-stage defense that uses a regularizer to slow down learning from harmful data.

• Models: Llama2 7B and Qwen2.5 7B.

6. Results & Analysis

The paper's experiments demonstrate VAA's effectiveness across various settings. The following tables are transcribed from the paper.

Core Results

Generalization to Fine-tuning Datasets

This experiment evaluates VAA on four different tasks with a 10% poison ratio.

Table 1: Performance analysis for different fine-tuning tasks. Best in bold, second-best underlined. This is a manual transcription of Table 1 from the paper.

• Analysis: VAA achieves the lowest (best) Harmful Score (HS) across all four diverse fine-tuning tasks, often by a large margin. On average, its HS is 24.82, significantly lower than the next best baseline (Vaccine at 31.28). Importantly, this safety improvement does not come at the cost of utility; VAA maintains or improves the Fine-tuning Accuracy (FA), achieving the highest average FA. Baselines like RepNoise and Booster struggle on complex tasks like GSM8K and AlpacaEval, whereas VAA remains effective, showing its robustness.

Robustness to Harmful Ratio

This experiment varies the percentage of harmful data ($p$) in the fine-tuning set for the SST2 task.

Table 2: Performance analysis for different harmful ratio. This is a manual transcription of Table 2 from the paper.

• Analysis: VAA consistently provides the best protection (lowest HS) regardless of the poison ratio. A key finding is its strong performance at p=0%. This shows that VAA not only defends against malicious HFT but also mitigates the safety degradation caused by fine-tuning on purely benign data, a critical and often overlooked problem.

Robustness to Harmful Fine-tuning Epochs

This experiment shows how safety degrades as HFT continues for more epochs on the SST2 task.

Table 3: Performance analysis for different harmful fine-tuning epochs. This is a manual transcription of Table 3 from the paper.

• Analysis: As expected, longer fine-tuning leads to higher harmfulness for all methods. However, VAA maintains a significantly lower HS at every checkpoint, demonstrating sustained robustness over the course of HFT.

Generalization to Different LLMs

This experiment tests if VAA's benefits apply to a different model, Qwen2.5-7B. Crucially, the vulnerable/invulnerable group assignments were derived from LLaMA2 and transferred directly without re-computation.

Table 4: Performance analysis for different harmful fine-tuning epochs on Qwen2.5-7B. This is a manual transcription of Table 4 from the paper.

• Analysis: VAA again achieves the best safety performance on a completely different model architecture. This result strongly supports the hypothesis that data vulnerability is a fundamental property that transfers across models, making the VAA approach generalizable.

Ablations / Discussion

Impact of Example Grouping and Sampling

These experiments validate VAA's core design choices on the SST2 task.

Table 5: Impact of examples grouping. This is a manual transcription of Table 5 from the paper.

• Analysis: Removing the grouping (- w/o group) significantly worsens the HS, proving that explicitly modeling vulnerability is crucial. When 10% of the group labels are randomized (- w noisy group), performance degrades only slightly, showing that the GDRO framework is robust to imperfect group assignments.

Table 6: Impact of sampling strategies. This is a manual transcription of Table 6 from the paper.

• Analysis: The dynamic, adversarial sampling of VAA is superior. Sampling only from the vulnerable group is better than only from the invulnerable group (confirming vulnerable data is more informative for robustness), but both are suboptimal. Standard importance sampling (Imp. sampling), a common technique for imbalanced data, is also less effective than VAA's adaptive approach.

Computational Overhead

VAA is computationally efficient. It requires 1.5x the backpropagation steps of standard SFT, which is more efficient than Vaccine (2x) and Booster (3x), while delivering better performance.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully identifies and addresses the problem of uneven forgetting in harmful fine-tuning. It demonstrates that certain alignment examples are consistently vulnerable and that this vulnerability is transferable. The proposed method, VAA, leverages this insight by using a Group DRO framework to promote balanced learning between vulnerable and invulnerable data. Experimental results confirm that VAA significantly enhances safety against HFT across various tasks and models without compromising utility, outperforming existing alignment-stage defenses.

• Limitations & Future Work: The authors acknowledge several limitations:

  • The data partitioning method is simple and requires a proxy fine-tuning step. Future work could explore more sophisticated, continuous measures of vulnerability that don't need this extra stage.
  • VAA reduces but does not completely eliminate safety breakdowns. It could be combined with other techniques like AI watermarking for more comprehensive protection.

• Personal Insights & Critique:

  • Novelty and Impact: The paper's primary strength is its shift to a data-centric view of alignment robustness. Identifying "vulnerable" data points is a powerful and intuitive concept that moves beyond treating alignment data as a monolithic block. This perspective could inspire a new class of defense mechanisms.
  • Practicality: The one-time cost of grouping data and the modest overhead of VAA training make it a practical solution for developers of open-source models or providers of fine-tuning services. The demonstrated transferability of vulnerability across models is a particularly strong result, as it means the expensive grouping step doesn't need to be repeated for every new model architecture.
  • Potential Weaknesses: The definition of "vulnerable" (ForgotNum > 0) is binary and depends on a single proxy fine-tuning run. The choice of proxy dataset could influence which examples are flagged. A more nuanced, probabilistic, or multi-proxy approach might yield even more robust groupings.
  • Future Directions: This work opens up interesting questions. What linguistic or semantic properties characterize vulnerable examples? Are they more complex, ambiguous, or related to topics where safety boundaries are subtle? Answering these could lead to even better methods for generating or identifying robust alignment data from the start. Furthermore, applying this "vulnerable group" concept to other domains, such as mitigating bias or improving out-of-distribution generalization, seems like a promising avenue for future research.