1. Bibliographic Information

1.1. Title

From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

1.2. Authors

Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, Jing Xiao. Affiliations: Ping An Technology (Shenzhen) Co., Ltd., China (for Ming Li, Yong Zhang, Zhitao Li, Ning Cheng, Jianzong Wang, Jing Xiao) and University of Maryland (for Ming Li, Jiuhai Chen, Lichang Chen, Tianyi Zhou). Jianzong Wang from Ping An Technology (Shenzhen) Co., Ltd. and Tianyi Zhou from the University of Maryland are the corresponding authors.

1.3. Journal/Conference

The paper does not explicitly state a journal or conference. It is available as a preprint on arXiv, indicated by the original source link typically pointing to a PDF. The Published at (UTC): 2024-01-01T00:00:00.000Z suggests a publication date, likely for the preprint. Given the nature of LLM research, it's common for such work to first appear as a preprint before formal peer-review at a prominent AI/NLP conference (e.g., NeurIPS, ICML, ACL, EMNLP) or journal.

1.4. Publication Year

2024 (as per the provided Published at (UTC): 2024-01-01T00:00:00.000Z).

1.5. Abstract

This paper introduces a self-guided methodology for Large Language Models (LLMs) to automatically select high-quality data, termed cherry samples, from open-source datasets for instruction tuning. This approach aims to reduce the need for manual curation and associated costs. The core innovation is the Instruction-Following Difficulty (IFD) metric, which quantifies the discrepancy between an LLM's expected response and its intrinsic generation capability, thereby identifying the most impactful training samples. Empirical results on Alpaca and WizardLM datasets demonstrate that this strategy achieves improved LLM performance with significantly less data (as little as 10% of the original input), highlighting a transformative leap in efficiency and resource-conscious advancements for LLM instruction tuning. The authors make their code, data, and models available.

1.6. Original Source Link

/files/papers/695333960394820b7e46522f/paper.pdf (This link indicates it's a direct PDF file, consistent with a preprint or an internally hosted paper. It is not an officially published journal/conference link).

2. Executive Summary

2.1. Background & Motivation

The rapid advancement of Large Language Models (LLMs) has highlighted the critical role of instruction tuning in refining their ability to follow specific guidelines and produce desired outputs. Initially, the common belief was that accumulating vast datasets was paramount for effective instruction tuning. However, seminal works like LIMA challenged this notion, suggesting that data quality, rather than sheer quantity, is the dominant factor in enhancing an LLM's instruction-following capabilities. While LIMA underscored the importance of high-quality data, it also brought forth a significant challenge: the lack of automated methods to identify such high-quality data from the enormous pool of available datasets, often relying on labor-intensive and expensive manual curation.

The core problem the paper aims to solve is this gap in automatically identifying high-quality (or cherry) instruction data for LLM instruction tuning. This problem is crucial because manual curation is costly, time-consuming, and does not scale well with the ever-growing size of datasets. Prior research often relied on external, fully-trained models or extensive statistical analysis for data curation, which could be computationally expensive, neglect the intrinsic abilities of the base model, or be difficult to adapt. The paper's innovative idea is to leverage the LLM itself in a self-guided manner to discern the difficulty of instruction data, thus enabling autonomous selection of the most impactful samples.

2.2. Main Contributions / Findings

The paper makes several primary contributions to the field of LLM instruction tuning:

• Self-Guided Data Selection Methodology: The authors propose a novel self-guided approach that empowers LLMs to autonomously identify and select cherry data from large open-source datasets. This significantly minimizes manual curation efforts, thereby reducing costs and streamlining the training process. This is a key innovation for resource-conscious advancements in LLM development.

• Instruction-Following Difficulty (IFD) Metric: A pivotal contribution is the introduction of the Instruction-Following Difficulty (IFD) score. This metric quantifies how much a given instruction aids the model in generating its corresponding response, by comparing the cross-entropy loss of generating a response with and without the instructional context. A higher IFD score indicates greater difficulty for the model in aligning its response with the instruction, identifying data samples that are particularly valuable for training. The IFD is model-specific, providing a tailored view of instruction difficulty.

• Empirical Validation and Efficiency: Through extensive experiments on popular instruction tuning datasets like Alpaca and WizardLM, the proposed strategy demonstrates remarkable efficiency. The paper shows that models trained with a mere 5% (for Alpaca) or 10% (for WizardLM) of the original data, selected using the IFD metric, consistently outperform models trained on the full dataset. This highlights a transformative impact by enabling the training of powerful LLMs with significantly reduced data requirements and computational resources.

• Insights into Data Characteristics: The study provides insights into the characteristics of cherry data. Visualization via t-SNE and verb-noun parsing reveals that high IFD samples are not uniformly distributed but cluster around more complex, creative, and knowledge-intensive tasks (e.g., "write story", "explain concept"), rather than simple tasks (e.g., "rewrite sentence", "edit text"). This suggests that IFD effectively identifies instructions that push the model to access and rearrange its intrinsic knowledge.

  These findings collectively address the challenge of automated high-quality data selection, paving the way for more efficient and effective instruction tuning of LLMs.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a reader should be familiar with the following fundamental concepts:

• Large Language Models (LLMs): These are advanced artificial intelligence models, like GPT-3, GPT-4, and LLaMA, that are pre-trained on vast amounts of text data to understand, generate, and process human language. Their core architecture typically relies on the Transformer (Vaswani et al., 2017) architecture, which uses self-attention mechanisms to weigh the importance of different parts of the input sequence.

  • Self-Attention: A mechanism in Transformer models that allows the model to weigh the importance of different words in an input sequence when encoding a particular word. It calculates Query (Q), Key (K), and Value (V) matrices from the input embeddings. The attention score is then computed as the softmax of the dot product of $Q$ and $K$, scaled by the square root of the dimension of $K$ ($d_k$), and then multiplied by $V$. $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ Where:
    • $Q$: Query matrix.
    • $K$: Key matrix.
    • $V$: Value matrix.
    • $d_k$: Dimension of the key vectors.
    • $QK^T$: Dot product of Query and Key, representing similarity.
    • $\mathrm{softmax}$: Normalization function to get attention weights.

• Instruction Tuning: A fine-tuning technique applied to pre-trained LLMs where the model is trained on a dataset of (instruction, output) pairs. The goal is to make the LLM better at following natural language instructions and generating responses that align with those instructions. This process helps the model specialize its knowledge and adapt its behavior to user prompts, moving beyond just predicting the next word in a general corpus.

• Cross-Entropy Loss: A commonly used loss function in machine learning, particularly for classification and language modeling tasks. It measures the difference between two probability distributions: the true distribution (e.g., the actual next word in a sequence) and the predicted distribution (e.g., the model's probability distribution over all possible next words). In language models, it quantifies how well the model predicts the next token in a sequence given the preceding tokens. A lower cross-entropy loss indicates better model performance.

  • For a single target token $y$ and a predicted probability distribution $P$ over all possible tokens, the cross-entropy loss is typically defined as: $ L = - \sum_{i=1}^{C} y_i \log(p_i) $ Where:
    • $C$: Total number of classes (vocabulary size).
    • $y_i$: A binary indicator (0 or 1) if class $i$ is the correct class. In one-hot encoding, $y_i=1$ for the true class and 0 otherwise.
    • $p_i$: The predicted probability of class $i$.
  • In the context of sequence generation, this loss is often averaged over all tokens in the generated sequence, as seen in the paper's Conditioned Answer Score and Direct Answer Score.

• KMeans Clustering: An unsupervised machine learning algorithm used to partition $n$ observations into $k$ clusters. The goal is to group data points such that each point belongs to the cluster with the nearest mean (centroid). It's used in this paper to ensure diversity when selecting initial pre-experienced samples by grouping similar instruction embeddings and then sampling from each group.

• t-SNE (t-Distributed Stochastic Neighbor Embedding): A non-linear dimensionality reduction technique used for visualizing high-dimensional data, typically by mapping it to a two- or three-dimensional space. It's particularly good at preserving local structures within the data, meaning that points that are close together in the high-dimensional space remain close in the low-dimensional visualization. In this paper, it's used to visualize instruction embeddings and observe the distribution of cherry data.

3.2. Previous Works

The paper contextualizes its contributions by referencing several key prior studies:

• Early Instruction Tuning (Wei et al., 2022; Longpre et al., 2023): Initially, instruction tuning was thought to heavily rely on the quantity of data. Datasets like Super-NaturalInstructions (Wang et al., 2022) were developed to amass vast collections of instructions for various NLP tasks.
• LIMA (Zhou et al., 2023): This seminal work challenged the quantity-over-quality paradigm. LIMA demonstrated that even a limited set of manually curated, high-quality instruction data could significantly improve an LLM's instruction-following capabilities. This paper builds on LIMA's insight, but tackles the unaddressed challenge of automatically identifying such high-quality data.
• Self-Instruct (Wang et al., 2023b): An approach to generate instruction data by leveraging a large language model (like GPT-3) to generate instructions and then their corresponding outputs. The Alpaca dataset (Taori et al., 2023), used in this paper, was created using the self-instruct methodology.
• EvolInstruct (Xu et al., 2023): An algorithm that uses an LLM (e.g., ChatGPT) to iteratively evolve simple instructions into more complex ones, thereby improving the quality and diversity of instruction data. The WizardLM dataset, also used here, was generated using EvolInstruct.
• Coreset Selection (Tsang et al., 2005; Har-Peled and Kushal, 2005; Munteanu et al., 2018; Toneva et al., 2018; Paul et al., 2021; Mindermann et al., 2022): This field aims to select a small, representative subset (coreset) of data to speed up training while maintaining performance. This paper's goal of cherry-picking high-quality data aligns with the spirit of coreset selection, but specifically for instruction tuning. Examples include using expected loss gradient norm scores (Paul et al., 2021) or Bayesian probability theory (Mindermann et al., 2022) to estimate data point impact.
• Instruction Data Selection (Cao et al., 2023; Chen et al., 2023a): More recent work directly addressing instruction data selection.
  • Instruction Mining (Cao et al., 2023) evaluates various indicators and uses statistical regression models to select data, often requiring training numerous models.
  • ALPAGASUS (Chen et al., 2023a) uses an external, fully-trained LLM (like ChatGPT) to score each sample for quality.
• Pointwise Mutual Information (PMI) (Holtzman et al., 2021; Wiegreffe et al., 2023; Mou et al., 2016; Zhou et al., 2019): A metric in NLP that measures the statistical association between two events or words. IFD shares conceptual similarities with PMI in assessing correlations between questions and answers, but IFD specifically focuses on the model's difficulty in aligning responses given instructional context.

3.3. Technological Evolution

The evolution of instruction tuning has moved from:

1. Massive Data Collection: Early efforts focused on simply gathering as much instruction data as possible, believing that more data inherently leads to better models.
2. Distillation and Self-Generation: Techniques like Self-Instruct and EvolInstruct emerged to automatically generate instruction data from powerful teacher models (e.g., GPT-3, ChatGPT), reducing manual labor but not necessarily guaranteeing optimal quality for target models.
3. Quality-Centric Approaches: The LIMA paper marked a shift, demonstrating that even a small amount of high-quality, human-curated data could yield superior instruction-following models. This sparked the current research focus on data quality.
4. Automated Quality Identification (Current Paper): This paper fits into the latest phase by introducing an automated, self-guided mechanism for the target LLM itself to identify high-quality data, moving beyond manual curation or reliance on external, potentially misaligned, teacher models.

3.4. Differentiation Analysis

Compared to the main methods in related work, the core differences and innovations of this paper's approach are:

• Self-Guided and Model-Intrinsic: Unlike ALPAGASUS which relies on external, fully-trained LLMs (like ChatGPT) for scoring data quality, this paper proposes a self-guided methodology where the target LLM itself (specifically, a pre-experienced version of it) evaluates the difficulty of instruction data using the IFD metric. This makes the data selection process model-specific and intrinsic, potentially leading to better alignment with the target model's current capabilities and learning needs.
• Novel IFD Metric: The IFD metric is a unique contribution that explicitly disentangles the instruction-following difficulty from the inherent difficulty of generating the answer itself. By comparing Conditioned Answer Score (loss with instruction) with Direct Answer Score (loss without instruction), IFD isolates the utility of the instruction, a nuance not directly captured by simpler metrics like raw loss or perplexity (which High CA Scores baseline attempts to use). This allows for a more precise identification of instructions that truly challenge and improve the model's alignment capabilities.
• Efficiency and Cost-Effectiveness: While other methods like Instruction Mining involve training numerous models or ALPAGASUS incurs API costs from external powerful LLMs, this method relies on a briefly pre-trained version of the target model. This makes the data filtering process more resource-conscious and efficient compared to approaches that necessitate significant external computational resources or extensive statistical model training.
• Distributional Insights: The paper provides deeper insights into the characteristics of selected cherry data, showing they tend to be complex and creative rather than merely diverse or easy to generate. This informs future instruction data generation efforts by highlighting what types of instructions are most valuable.

4. Methodology

4.1. Principles

The core idea behind the proposed method is that a Large Language Model (LLM) can, through a brief initial exposure to instruction data, develop a basic understanding of instructions. This pre-experienced LLM can then be leveraged to self-guide the selection of the most impactful instruction-response pairs for its subsequent, more focused training. The theoretical basis is rooted in the observation that not all instruction data is equally effective for instruction tuning. By quantifying how much an instruction truly helps the model generate a correct response, we can pinpoint difficult but valuable samples (cherry data) that force the model to better align its intrinsic knowledge with the provided instructions. This moves beyond simply identifying correct or diverse data, to finding data that maximizes the learning gain for the specific model.

4.2. Core Methodology In-depth (Layer by Layer)

The methodology is structured into three sequential phases, as illustrated in Figure 1 of the original paper: Learning from Brief Experience, Evaluating Based on Experience, and Retraining from Self-Guided Experience.

4.2.1. Phase 1: Learning from Brief Experience

This initial phase aims to imbue the base LLM with a foundational ability to follow instructions. This is crucial because a completely untrained model might not be able to meaningfully evaluate the difficulty of an instruction.

1. Dataset Preparation:

   • The process begins with an initial full target dataset, denoted as $D_0$. This dataset contains $n$ triplets, where each triplet $x$ is structured as (Instruction, [Input], Answer).
   • A Question string is formed by mapping the Instruction and optional [Input] components. The specific map function used is aligned with the format of the original target dataset (e.g., $Instruction\nInput: [Input]\nResponse:$).
   • Each word within the Question ($Q$) and Answer ($A$) strings is denoted as $w_i^Q$ and $w_i^A$ respectively.

2. Instruction Embedding Generation:

   • For each sample $x_j$ in the dataset, the pre-trained base LLM model ($LLM_{\theta_0}$) is used to obtain embeddings for its Question part. The base LLM's initial weights are represented by $\theta_0$.
   • The embeddings for each word $w_{j,i}^Q$ in the Question $j$ are extracted as its corresponding last hidden state $h_{j,i}^Q$. This is given by: $ [ h _ { j , 1 } ^ { Q } , . . h _ { j , m } ^ { Q } ] = L L M _ { \theta _ { 0 } } ( w _ { j , 1 } ^ { Q } , . . w _ { j , m } ^ { Q } ) $ Where:
     • $w_{j,i}^Q$: The $i$-th word of Question $j$.
     • $h_{j,i}^Q$: The corresponding last hidden state (embedding) for the $i$-th word of Question $j$.
     • $m$: The number of words in Question $j$.
     • $LLM_{\theta_0}$: The pre-trained base LLM with initial weights $\theta_0$.
   • These word embeddings are then aggregated, typically by averaging, to form a single instruction embedding $h_j^Q$ for each Question: $ h _ { j } ^ { Q } = \frac { \sum _ { i = 1 } ^ { m } { h _ { j , i } ^ { Q } } } { m } $ Where:
     • $h_j^Q$: The aggregated embedding for Question $j$.

3. Diverse Sample Selection:

   • To ensure that the initial training exposes the model to a wide range of instructions, KMeans clustering is applied to these instruction embeddings $h_j^Q$.
   • The paper sets $k=100$ clusters. From each of these 100 clusters, 10 instances are sampled, resulting in a total of $100 \times 10 = 1000$ pre-experienced samples. This sampling strategy aims to maximize the diversity of instructions seen by the model initially.

4. Brief Pre-training:

   • The initial LLM ($LLM_{\theta_0}$) is then trained for only 1 epoch using these 1000 pre-experienced samples.

   • This brief training phase produces a brief pre-experienced model (let's denote its weights as $\theta_{pre}$), which possesses a basic ability to follow instructions without being extensively fine-tuned on the entire dataset.

     The following figure (Figure 1 from the original paper) shows the system architecture:

     该图像是示意图，展示了我们提出的方法概览。分为三个部分：1) 从简短经验中学习，2) 基于经验进行评估，3) 从自我指导经验中进行再训练。每部分描述了LLM与指令数据之间的关系，以及如何优化模型训练过程。

4.2.2. Phase 2: Evaluating Based on Experience

In this phase, the brief pre-experienced model ($LLM_{\theta_{pre}}$) is used to calculate the Instruction-Following Difficulty (IFD) score for every sample in the full dataset $D_0$. This score quantifies how challenging each instructional sample is for the model to follow.

1. Conditioned Answer Score ($s_{\theta}(A|Q)$):

   • This score measures the model's ability to generate the ground-truth Answer ($A$) when conditioned on the Question ($Q$). It is calculated as the averaged cross-entropy loss of predicting each token in the Answer given the Question and all preceding tokens of the Answer.
   • The formula for the Conditioned Answer Score (denoted as $s_{\theta}(A|Q)$) is: $ L _ { \theta } ( { \cal { A } } | Q ) = - \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \log P ( w _ { i } ^ { A } | Q , w _ { 1 } ^ { A } , w _ { 2 } ^ { A } , \dots , w _ { i - 1 } ^ { A } ; \theta ) $ Where:
     • $N$: The total number of words (tokens) in the ground-truth Answer $A$.
     • $w_i^A$: The $i$-th word of the ground-truth Answer $A$.
     • $Q$: The Question (instruction + input) provided as context.
     • $w_1^A, \dots, w_{i-1}^A$: The preceding words of the Answer that have already been generated.
     • $P(\dots ; \theta)$: The probability assigned by the LLM (with weights $\theta$, specifically $\theta_{pre}$ from the pre-experienced model) to the next token $w_i^A$, given the context.
     • $\log P(\dots)$: The natural logarithm of this probability.
     • The negative sign ensures that lower probabilities (worse predictions) result in higher loss values.
     • The $\frac{1}{N}$ averages the loss across all tokens in the Answer.

2. Direct Answer Score ($s_{\theta}(A)$):

   • This score measures the LLM's inherent ability to generate the Answer ($A$) without any instructional context ($Q$). It gauges the intrinsic difficulty of the answer string itself, based on the model's pre-trained knowledge.
   • The formula for the Direct Answer Score is: $ s _ { \theta } ( A ) = - \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \log P ( w _ { i } ^ { A } | w _ { 1 } ^ { A } , \dots , w _ { i - 1 } ^ { A } ; \theta ) . $ Where:
     • All symbols are the same as above, but crucially, $Q$ is absent from the conditioning context. This means the model only relies on the previously generated tokens of the Answer itself to predict the next one.

3. Instruction-Following Difficulty (IFD) Score ($\mathrm{IFD}_{\theta}(Q, A)$):

   • The IFD score is the core metric. It is calculated as the ratio between the Conditioned Answer Score and the Direct Answer Score. This ratio aims to isolate the contribution of the instruction by normalizing the loss with instructional context by the inherent difficulty of the answer string.
   • The formula for the IFD score is: $ \operatorname { I F D } _ { \theta } ( Q , A ) = { \frac { s _ { \theta } ( A | Q ) } { s _ { \theta } ( A ) } } $ Where:
     • $s_{\theta}(A|Q)$: The Conditioned Answer Score (loss with instruction).
     • $s_{\theta}(A)$: The Direct Answer Score (loss without instruction).
     • $\theta$: The weights of the brief pre-experienced model ($LLM_{\theta_{pre}}$), as the IFD is model-specific.
   • Interpretation of IFD:
     • A high IFD score indicates that the instruction provides little benefit (or even hinders) the model in generating the correct response, relative to its ability to generate the response intrinsically. This suggests the instruction is difficult for the model to follow or align with its internal knowledge. These are the cherry samples that can provide significant learning opportunities.
     • A low IFD score indicates that the instruction greatly helps the model in generating the response, or that the model already finds the response easy to generate and the instruction further facilitates it. These samples might be less impactful for further instruction tuning.
   • Misalignment Filter: A threshold of 1 is set for IFD scores. Typically, providing context (the instruction) should make prediction easier, meaning $s_{\theta}(A|Q)$ should be less than $s_{\theta}(A)$. Therefore, if $\mathrm{IFD}_{\theta}(Q, A) > 1$, it implies that the instruction provides no useful context or is misaligned with the response, as the loss with instruction is even higher than without it. Such samples are considered poor quality and are filtered out.

4. Efficiency Note: The paper mentions Superfiltering (Li et al., 2024b) as a follow-up work which suggests that good prompting can alleviate the need for a pre-experienced model, and that IFD scores calculated by weak language models are consistent with strong models. This implies potential future optimizations where even smaller models or direct use of the base model might be sufficient for IFD calculation, further boosting efficiency.

4.2.3. Phase 3: Retraining from Self-Guided Experience

After calculating the IFD scores for all samples in the target dataset using the brief pre-experienced model, the final instruction tuning phase begins.

1. Cherry Data Selection:

   • Samples with relatively large IFD scores are selected as cherry data. The exact percentage (e.g., top 5%, 10%, 15%) is determined empirically. The intuition is that these are the most challenging and thus most beneficial examples for the model to learn from.
   • Any samples with $IFD scores > 1$ (indicating misalignment) are filtered out prior to selection.

2. Final Model Training:

   • The base LLM ($LLM_{\theta_0}$) is then fine-tuned using only this carefully selected subset of cherry data.
   • The resulting model is termed a cherry model. This model is expected to achieve superior instruction-following performance compared to models trained on larger, unfiltered datasets, due to the high quality and targeted difficulty of the training samples.

5. Experimental Setup

5.1. Datasets

The study uses a combination of training datasets for instruction tuning and diverse test datasets for evaluation.

5.1.1. Training Datasets

• Alpaca Dataset (Taori et al., 2023):
  • Source/Generation: Developed using the self-instruct approach (Wang et al., 2023b) with text-davinci-003 (a proprietary OpenAI model).
  • Scale: Encompasses 52,002 instruction-following samples.
  • Characteristics: Contains a wide variety of instructions, designed to teach general instruction-following capabilities. The paper notes that its dependence on text-davinci-003 raised concerns about data quality.
  • Domain: General-purpose, open-domain instructions.
• WizardLM Dataset (Xu et al., 2023):
  • Source/Generation: Leverages the EvolInstruct algorithm, incorporating ChatGPT during reformulation to improve data quality.
  • Scale: The paper utilizes WizardLM70K, implying around 70,000 samples. After filtering out "AI censure" instances (following Vicuna strategy), a streamlined subset of 63,655 entries was used.
  • Characteristics: Designed to provide more complex and higher-fidelity instruction data compared to Alpaca due to the EvolInstruct process.
  • Domain: General-purpose, open-domain instructions, often more intricate.

5.1.2. Test Datasets

To ensure a comprehensive and unbiased assessment, five diverse test sets were employed. These sets contain approximately 1,000 human-curated instructions in total, covering both open-domain and closed-domain tasks from various sources.

• Vicuna (Chiang et al., 2023): A popular benchmark for evaluating chatbot performance, known for its diverse conversational prompts. It provides specific sub-categories for in-depth analysis.
• Koala (Vu et al., 2023): Another chatbot benchmark, which can also provide insights into conversational abilities.
• WizardLM (Xu et al., 2023): Used as a test set in addition to a training set, to evaluate generalization on its own instruction types. It also provides specific sub-categories.
• Self-instruct (Wang et al., 2023b): A benchmark derived from the self-instruct method, testing the model's ability to follow instructions generated by a large LLM.
• LIMA (Zhou et al., 2023): A dataset known for its high-quality, human-curated instructions, emphasizing the importance of quality over quantity.

5.1.3. Example of a Data Sample

The paper does not provide an explicit raw data sample from Alpaca or WizardLM in the main text, but in Appendix F, it shows examples of Instruction, Input (optional), and Response triplets used for IFD score analysis.

Positive Examples:

• Instruction: Create a poem that tells the story of a struggle against an unseen force. Input: (empty) Response: A distant enemy looms on the horizon, Selves of the light fight despite fear and fatigue. ... Peace of their struggle found in the morning's glow, Rising higher than any unseen force's ebb and flow.
• Instruction: Give a brief description of the coronavirus. Input: (empty) Response: Coronavirus is a type of virus that can cause illnesses ranging from the common cold to severe diseases such as Middle East Respiratory Syndrome (MERS) and Severe Acute Respiratory Syndrome (SARS). ...

Negative Examples:

• Instruction: What emotion is expressed in this tweet? Input: I'm so frustrated right now!! Response: Frustration
• Instruction: Identify a famous quote from the given book Input: Book: Pride and Prejudice by Jane Austen Response: "It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife."

These examples illustrate the typical instruction-input-response format used in the datasets.

5.2. Implementation Details

• Base Models:
  • LLaMA-7B (Touvron et al., 2023a)
  • LLaMA2-7B (Touvron et al., 2023b)
  • LLaMA2-13B (Touvron et al., 2023b)
• Codebases:
  • For LLaMA-7B experiments on Alpaca and WizardLM: The original Alpaca codebase was utilized.
  • For LLaMA2-7B and LLaMA2-13B experiments: The Vicuna codebase (Chiang et al., 2023) was utilized.
• Optimizer: Adam optimizer (Kingma and Ba, 2017).
• Learning Rate: $2 \times 10^{-5}$.
• Batch Size: 128.
• Training Epochs:
  • Cherry models (final models): Trained for 3 epochs.
  • Pre-experienced models: Trained for only 1 epoch.
• Max Input Length:
  • Alpaca dataset training: 512 tokens.
  • WizardLM dataset training: 1024 tokens (the original WizardLM model used 2048, an inherent disadvantage for the authors' reimplemented model).
  • LLaMA2 models: 2048 tokens, enabled by Flash Attention mechanism (Dao et al., 2022).
• Data Filtering: For WizardLM, "AI censure" instances were filtered out, resulting in a dataset of 63,655 entries, aligning with the Vicuna strategy.
• Prompting: For LLaMA2 models, the instruction prompt format from Vicuna (Chiang et al., 2023) was used.

5.3. Evaluation Metrics

The paper employs a multi-faceted evaluation strategy, combining human-like judgments, established benchmarks, and direct human feedback.

5.3.1. Pair-wise Comparison

This method leverages powerful LLMs to judge the quality of responses from different models.

• Conceptual Definition: Two models' responses to the same instruction are presented to an independent, more advanced LLM (the "judge"). The judge rates each response based on attributes like relevance, accuracy, and helpfulness, and then compares them to determine which model performed better. This aims to simulate human evaluation while being more scalable.
• Judge Models: GPT-4 and ChatGPT were used as judging models.
• Scoring: Each model's response is rated on a scale of 1 to 10.
• Positional Bias Mitigation: To counter positional bias (where the order of presentation might influence the judge's preference, Ko et al., 2020; Wang et al., 2023a), responses from the two models are sent to the judge twice, with their order swapped.
• Win/Tie/Loss Definition: A model is declared to Win if:
  • It outperforms the competitor in both orderings, OR
  • It wins in one ordering and ties in the other.
  • A Tie occurs if:
  • It ties in both orderings, OR
  • It wins in one ordering and loses in the other.
  • A Loss occurs if:
  • It lags (performs worse) in both orderings, OR
  • It ties in one ordering and loses in the other.
• Winning Score: For aggregated results, a winning score is calculated as $\frac { \mathrm { Num(Win) - Num(Lose) } } { \mathrm { Num(All) } } + 1$. A score greater than 1.0 indicates that the model performs better than the comparison baseline.

5.3.2. Benchmarks

Two widely recognized benchmarks for LLMs are used to assess performance on established tasks.

• Huggingface Open LLM Leaderboard:
  • Conceptual Definition: An open evaluation framework (Gao et al., 2021) that tests generative language models on various NLP tasks. It provides a standardized way to compare LLMs across multiple capabilities.
  • Evaluation Tasks (Sub-metrics):
    • ARC (AI2 Reasoning Challenge) (Clark et al., 2018): A question-answering dataset requiring multi-step reasoning.
    • HellaSwag (Zellers et al., 2019): A commonsense reasoning task that evaluates models' ability to predict plausible next sentences in ambiguous contexts.
    • MMLU (Massive Multitask Language Understanding) (Hendrycks et al., 2021): A broad set of multiple-choice questions across 57 subjects, designed to measure a model's world knowledge and problem-solving abilities.
    • TruthfulQA (Lin et al., 2022): Assesses whether a model generates truthful answers to questions that elicit false statements from humans.
  • Mathematical Formula/Symbol Explanation: Each sub-metric typically reports accuracy or F1-score. The overall leaderboard score is often an average of these. No specific formulas are provided in the paper, but these are standard NLP evaluation metrics.
• AlpacaEval Leaderboard (Dubois et al., 2023; Li et al., 2023b):
  • Conceptual Definition: An LLM-based automatic evaluation framework built on the AlpacaFarm evaluation set. It compares model responses with Davinci003 responses using GPT-4 as a judge. It provides a score indicating how often a model's response is preferred over Davinci003.
  • Mathematical Formula/Symbol Explanation: The score typically represents the win rate against Davinci003. No specific formula is provided in the paper, but it implies a comparison mechanism similar to the Pair-wise Comparison described above.

5.3.3. Human Evaluation

• Conceptual Definition: Direct human assessment of model responses, considered the gold standard for qualitative evaluation, though labor-intensive.
• Procedure:
  1. A new random test set of 100 instructions was created by sampling 20 instructions from each of the five test sets.
  2. Three human participants were asked to compare responses generated by the models.
  3. For each comparison, participants chose one of three options: Win, Tie, or Loss (from the perspective of the model being evaluated).
  4. Final results were determined by majority voting among the three participants.

5.4. Baselines

The paper compares its cherry models against several baselines to demonstrate the efficacy of its self-guided data selection and IFD metric:

• Official Alpaca Model: The model trained on the full Alpaca dataset (52,002 samples). This is the primary baseline for the Alpaca experiments.
• Reimplemented WizardLM Model: A WizardLM model trained by the authors using their own configuration (LLaMA-7B, 1024 max input length, filtered "AI censure" instances) on the full WizardLM dataset (63,655 samples). This serves as a fair baseline for WizardLM experiments under consistent training conditions.
• Data Randomly Selected: Models trained on subsets of data chosen purely at random (e.g., 5%, 10%, 15% of the original dataset). This baseline verifies that the performance gains are due to intelligent selection, not just reduced data size.
• Data with Diversity: Models trained on data selected by KMeans clustering (similar to the pre-experience phase, but for the main training data selection) to maximize diversity, without considering IFD scores. This tests whether diversity alone is sufficient.
• Data with Low IFD Score: Models trained on data with the lowest IFD scores (the antithesis of the proposed method). This ablation directly validates that higher IFD scores correlate with more impactful training data.
• Data with High CA Scores: Models trained on data selected based on high Conditioned Answer (CA) scores (equivalent to high loss or perplexity). This is a common heuristic for identifying "hard" samples. This baseline helps to show the unique benefit of normalizing CA with DA via IFD.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results consistently validate the effectiveness and efficiency of the proposed self-guided data selection method, particularly the IFD metric.

6.1.1. Main Pairwise Comparison

The paper's primary findings from pairwise comparison using GPT-4 as the judge are compelling:

• Alpaca Dataset: Our model, trained with only approximately 5% of the original Alpaca data, outperforms the official Alpaca model which was trained with the full dataset.

• WizardLM Dataset: Our model, trained with approximately 10% of the original WizardLM data, outperforms the reimplemented WizardLM model (trained under the same configuration with full data).

  The following figure (Figure 2 from the original paper) presents these main results:

  该图像是图表，展示了我们的方法与 Alpaca（100%）和 WizardLM（100%）的比较。图(a)比较我们使用5% Alpaca数据与官方模型的结果，图(b)比较我们使用10% WizardLM数据与重实现的模型的结果。每个水平条代表特定测试集的比较。 Figure 2(a) shows that our model with 5% Alpaca data wins more often against the official Alpaca model across all five test sets (Vicuna, Koala, WizardLM, SInstruct, LIMA). Figure 2(b) similarly demonstrates that our model with 10% WizardLM data performs better or comparably against the reimplemented WizardLM model across the test sets, with notable wins in Vicuna and Koala.

6.1.2. Performance Across Data Growth

To further analyze the impact of data quantity, models were trained on subsets containing 5%, 10%, 15%, and 20% of the training datasets. The winning score (calculated as $\frac { \mathrm { Num(Win) - Num(Lose) } } { \mathrm { Num(All) } } + 1$) shows a consistent trend:

• With merely 10% of selectively chosen data, our models consistently exceed the results of models trained on the full dataset for both Alpaca and WizardLM. This highlights the substantial efficiency gains of the method.

  The following figure (Figure 3 from the original paper) illustrates the winning score changes over data growth:

  该图像是图表，展示了在不同训练数据百分比下，我们的模型与全数据模型的获胜分数变化。获胜分数计算为 $\frac{\mathrm{Num(Win) - Num(Lose)}}{Num(All)} + 1$，当值大于1时，意味着该模型表现优于对比模型。

The plot clearly shows that for both Alpaca and WizardLM, the winning score quickly surpasses 1.0 (indicating better performance than the full-data model) at 5-10% data, and generally maintains this advantage or improves further with slightly more data.

6.1.3. Benchmark Results

The effectiveness of our automatically selected data is also validated on public benchmarks.

The following are the results from Table 1 of the original paper:

• Our cherry model using 5% Alpaca data outperforms the official Alpaca model on both the Huggingface Open LLM Leaderboard (52.06 vs 50.21 average) and AlpacaEval (34.74 vs 26.46).
• Our cherry model using 10% WizardLM data shows a close performance compared to the reimplemented WizardLM model on both benchmarks (e.g., 51.59 vs 52.79 average on Open LLM, and 61.44 vs 61.99 on AlpacaEval), despite using significantly less data.

6.1.4. Human Evaluation

• Cherry Alpaca (5%) vs. Alpaca (100%): Our model achieved 49/100 wins, 25/100 ties, and 26/100 losses, indicating a clear preference for our model.
• Cherry WizardLM (10%) vs. Reimplemented WizardLM (100%): Our model showed 37/100 wins, 32/100 ties, and 31/100 losses, demonstrating comparable or slightly better performance.

6.2. Ablation Studies / Parameter Analysis

6.2.1. Ablation on Data Selection Mechanism

This section compares IFD-based selection against other data selection strategies, using ChatGPT as the judge.

The following figure (Figure 4 from the original paper) presents the overall winning score changes by comparing models using different data selection strategies with the official Alpaca model:

该图像是图表，展示了不同模型与官方Alpaca模型的获胜分数变化。随着训练数据百分比的增加，各种数据选择策略模型的表现差异显著，特别是使用自我引导选择策略的模型（蓝色线）在低数据量时表现优越。文中数据表明，该策略在节省数据的同时提升了模型性能。

The following are the results from Table 5 of the original paper:

(Note: The Winning Score in Table 5 is likely from the perspective of the ablation model against the official Alpaca in human evaluation, which contradicts the text saying "It represents our model better when the Win count is greater than the Lose count." Given that Ours 5% has the highest Open LLM Avg and the text in 4.2.1-4.2.4 implies other methods underperform, the Winning Score for ablation models is probably the inverse of the main results' winning score.)

• Data Randomly Selected (Random): Models trained on 5-15% random data consistently underperformed against the official Alpaca model. Our method, with equivalent data, surpasses random selection, confirming the value of targeted selection.
• Data with Diversity (Diversity): Selecting data solely based on diversity (using K-means clustering) resulted in subpar performance, similar to random selection. This indicates that diversity alone is insufficient for effective instruction tuning, emphasizing the need for difficulty-aware selection.
• Data with Low IFD Score (Low IFD score): Training on data with the lowest IFD scores yielded the least performance among all methods. This directly validates the IFD metric, showing a clear positive correlation between higher IFD scores and improved model performance.
• Data with High CA Scores (High CA score): Models relying solely on high Conditioned Answer (CA) scores (raw loss/perplexity) performed significantly worse than the official Alpaca model. This highlights the crucial role of the Direct Answer Score in the IFD metric to factor out the LLM's intrinsic ability to fit the answer string, which CA alone neglects.

6.2.2. Ablation on Pre-Experienced Data

6.2.2.1. Number of Pre-Experience Data

The paper investigates the impact of the number of pre-experienced samples (0, 100, 300, and 500) on the final cherry model performance.

• 0 Pre-experienced samples: When no pre-experienced model is used (i.e., IFD calculated directly on the raw base model), performance is the lowest. However, it still outperforms the Alpaca model when using 10% of the data, showing IFD's inherent effectiveness.

• 100 Pre-experienced samples: Slightly better than 0 samples, but still insufficient for the model to acquire basic instruction-following ability.

• 300 Pre-experienced samples: A distinct performance gain is observed, suggesting this amount is sufficient for equipping the model with basic instruction-following capability.

• 500 Pre-experienced samples: Further increasing the number of samples beyond 300 does not significantly improve performance.

  The following figure (Figure 5 from the original paper) illustrates the overall winning score changes with different numbers of pre-experienced samples:

  该图像是图表，展示了不同预先经验样本数与官方Alpaca模型的整体获胜得分变化。横轴为训练数据百分比，纵轴为获胜得分。每条线代表不同数量的预先经验样本，随着训练数据增加，得分表现具有一定波动。

The plot shows a clear increase in winning score as the number of pre-experienced samples increases from 0 to 300, after which the performance plateaus or slightly declines.

6.2.2.2. Distribution of Pre-Experience Data

Experiments were conducted to see if the selection strategy for the 1000 pre-experienced samples (Difficulty, Diversity, Random) impacts the final cherry model performance.

The following are the results from Table 2 of the original paper:

All three strategies (Difficulty, Diversity, Random) for selecting the initial 1000 pre-experienced samples lead to cherry models that surpass the Alpaca model and are comparable to each other. This suggests that the existence of a pre-experience process itself is more critical than the specific sampling strategy for this initial phase, and the IFD metric is robust across these variations.

6.3. Results on LLaMA2 Models

To demonstrate the generalizability of the method, experiments were conducted on newer LLaMA2-7B and LLaMA2-13B models. IFD scores were calculated directly based on the corresponding LLaMA2 pre-trained models.

The following are the results from Table 3 of the original paper:

On both LLaMA2-7B and LLaMA2-13B models, our cherry models trained with much less data (e.g., 5% or 10% Alpaca data) outperform the models trained with the original full data across the Huggingface Open LLM Leaderboard and AlpacaEval. This further confirms the consistent advantages and generalizability of the proposed method across different base LLMs.

6.4. Cherry Data Characteristics

6.4.1. Distribution Characteristics

• t-SNE Visualization: The paper visualized instruction embeddings of the Alpaca dataset using t-SNE. Samples with the top 5% IFD scores (red points) and the least 5% IFD scores (blue points) were highlighted.

• Key Finding: Contrary to the belief that high-quality data should be uniformly scattered or maximize diversity across all instruction types, the cherry data did not scatter uniformly. Instead, clear boundaries existed between samples of high and low difficulty, forming distinct clusters. This suggests that high-difficulty instructions represent specific, challenging regions in the instruction embedding space.

• Manual Examination: Clusters with high IFD scores were found to contain deeper, more intricate tasks such as storytelling or elucidation of phenomena. Conversely, clusters with low IFD scores were replete with rudimentary tasks like editing punctuation, words, or sentences. This supports the hypothesis that the method identifies tasks that compel LLMs to rearrange and access their intrinsic knowledge repositories.

  The following figure (Figure 6 from the original paper) shows the t-SNE visualization:

  该图像是使用 t-SNE 可视化的指令嵌入图。红色点代表 IFD 分数最高的 $5\%$ 样本，蓝色点则表示 IFD 分数最低的 $5\%$ 样本，灰色点为其他样本。

The figure visually confirms the clustering, with red and blue points forming somewhat distinct groupings rather than being evenly mixed, especially in certain regions of the 2D space.

6.4.2. Pattern Characteristics

To understand the linguistic patterns, the Berkeley Neural Parser was used to identify verb-noun structures in instructions from top 5% and least 5% IFD score data in Alpaca.

The following are the results from Table 4 of the original paper:

• High IFD Data: Predominantly involves creative and complex instructions such as "Write Story", "Generate List", "Explain Concept", "Create Poem", etc. These tasks require substantial creativity, thinking skills, and deep understanding from the LLM.
• Low IFD Data: Focuses more on rule-following and less creative tasks like "Rewrite Sentence", "Edit Sentence", "Change Sentence", "Classify Sentence", "Replace Word", etc. These tasks demand less generative creativity.
• Conclusion: The IFD metric effectively identifies instructions that require more creativity and deep understanding, which are crucial for aligning LLMs.

6.5. Additional Discussions (from Appendix G)

6.5.1. Fully-trained Model as Pre-Experienced Model?

The paper explored whether a fully-trained Alpaca model could serve as the pre-experienced model for selecting cherry data.

The following are the results from Table 9 of the original paper:

Results show that using a fully-trained model for IFD calculation hardly surpasses the baseline Alpaca and underperforms our models across different data scales. This suggests that the overly large distribution gap between a fully-trained model and a raw model makes it inappropriate for selecting samples meant to guide the initial raw model's learning. A briefly pre-experienced model is crucial because it better reflects the current learning state of the model.

6.5.2. How Many Cherry Samples are Required?

While the method provides flexibility, the optimal percentage of cherry data to select depends on various factors (absolute IFD values, distribution of hard examples, original dataset size). Based on empirical study, the paper suggests that selecting samples with the top 10% IFD scores is a safe and reasonable choice for good performance.

6.6. Detailed Main Comparison (from Appendix I)

• Comparison with Official Alpaca (Figure 11): Our cherry models consistently outperform the official Alpaca (7B) model across all test sets and data scales (5% to 15% of data), as judged by ChatGPT.
• Comparison with Reimplemented WizardLM (Figure 12): Our cherry models begin outperforming the reimplemented WizardLM (7B) model from the 10% data scale, showing strong performance across test sets.
• Comparison with Official WizardLM (Figure 13): Despite inherent disadvantages (e.g., max token size 1024 vs 2048 for official), our cherry model achieves comparable performance with the official WizardLM model when using 40% of the data. This highlights the method's robustness even under challenging conditions.

6.7. Detailed Ablation Comparison (from Appendix J)

• Data Randomly Selected (Figure 14): Our cherry models consistently outperform models trained with randomly selected data across all tested percentages (5% to 15%), reinforcing the value of intelligent data selection.
• Data with Low IFD Score (Figure 15): Models trained with low IFD scores consistently show worse performance than our cherry models, further emphasizing that high IFD identifies valuable learning samples.
• Data with High CA Scores (Figure 16): Our cherry models consistently outperform models trained with high conditioned answer scores, demonstrating the superiority of IFD's nuanced approach over raw perplexity.
• Number of Pre-Experienced Data (Figure 17): This figure visually supports the conclusion from 6.2.2.1, showing performance improvement up to 300-500 pre-experienced samples, and then plateauing.
• Distribution of Pre-Experience Data (Figure 18): This figure shows that various strategies for selecting pre-experienced data (difficulty, diversity) lead to comparable performance gains over the baseline, reinforcing that the pre-experience process itself is key.
• Fully-trained Model as Pre-Experienced Models (Figure 19): This figure graphically illustrates that using a fully-trained Alpaca model as the pre-experienced model does not yield superior results, aligning with the conclusions in 6.5.1.

6.8. Cherry Data General Characteristics (from Appendix E)

An additional evaluation using ChatGPT to score instructions on six aspects (Scope, Complexity, Clarity, Depth, Simplicity, Knowledge Required) for top 5% and least 5% IFD samples revealed:

• High IFD samples scored higher in Scope, Complexity, Depth, and Knowledge Required.

• Low IFD samples scored higher in Clarity and Simplicity.

• Simplicity showed the most pronounced discrepancy.

  The following figure (Figure 8 from the original paper) displays this comparison:

  该图像是图表，展示了Alpaca数据中评分最高5%和最低5% IFD分数的数据实例在范围、复杂性、清晰度、深度、简单性和知识需求六个维度上的对比情况。图中红色和蓝色线分别代表最高和最低评分群体在各维度的表现。

This analysis further confirms that the IFD score successfully identifies more intricate and challenging instructions, which are beneficial for instruction tuning.

7. Conclusion & Reflections

7.1. Conclusion Summary

This study successfully demonstrates a novel self-guided approach for Large Language Models (LLMs) to autonomously select high-quality instruction tuning data that is precisely tailored to the model's learning needs. The key innovation, the Instruction-Following Difficulty (IFD) score, effectively quantifies the actual difficulty an instruction poses to a model by decoupling the instruction's helpfulness from the inherent complexity of the response. Through empirical validation on Alpaca and WizardLM datasets, the paper establishes that models trained with a significantly smaller subset of cherry data (as low as 5-10% of the original) can outperform models trained on the entire, uncurated datasets. This highlights a transformative path toward more efficient, resource-conscious, and effective instruction tuning of LLMs, moving decisively from a quantity-centric to a quality-centric paradigm. Furthermore, analysis of the cherry data reveals that high IFD scores correspond to instructions demanding greater creativity, depth, and knowledge integration, providing valuable insights for future data generation efforts.

7.2. Limitations & Future Work

The authors acknowledge the primary limitation of their method:

• Inconvenience of Training the Pre-Experienced Model: While the IFD score concept is simple and effective, the necessity of training a brief pre-experienced model adds a step to the pipeline, which might be inconvenient for direct real-world deployment.

• Trade-off between Research and Real-world: The authors note that from a research perspective, the pre-experienced phase is valuable as it equips base models with a basic instruction-following ability, making Conditioned Answer Score calculations more reasonable. However, for real-world implementation, directly using the base model (as explored in LLaMA2 experiments) might be more efficient, albeit with potentially a slight performance trade-off.

  Future work directions implied by the paper and related discussions include:

• Optimizing the Pre-experience Phase: Further research into simplifying or potentially eliminating the pre-experienced model training (e.g., through advanced prompting strategies as hinted by Superfiltering) could enhance efficiency.

• Adaptive Data Selection: Exploring dynamic data selection during training, rather than a static pre-selection, could offer further improvements.

• Generalization to Other Domains/Tasks: Investigating the applicability of IFD and self-guided selection to more specialized LLM tasks or specific domains where data quality is paramount.

• Instruction Data Generation: Utilizing the IFD metric to guide the automatic generation of new, high-quality instruction data that specifically targets the model's identified areas of difficulty.

7.3. Personal Insights & Critique

This paper presents a highly intuitive and impactful approach to instruction tuning. The IFD metric is a simple yet powerful concept that effectively captures the model-specific difficulty of instructions, a crucial factor often overlooked by static quality metrics or external judging models. The demonstration that a tiny fraction of carefully selected data can outperform full datasets is a significant practical contribution, especially in an era where LLM training costs are exorbitant.

Insights and Applications:

• Cost Reduction: The method's ability to drastically reduce the amount of training data needed translates directly into substantial savings in computational resources (GPU hours) and time, making advanced LLM tuning more accessible.
• Targeted Learning: The IFD allows for targeted learning, focusing the model on what it actually struggles with, rather than redundant or easily mastered instructions. This could lead to more robust and specialized models.
• Data Augmentation Guidance: The discovered characteristics of cherry data (complexity, creativity) are invaluable for designing intelligent data augmentation or instruction generation strategies. Instead of generating more random instructions, future systems could be guided to create instructions that align with high IFD patterns.
• Transferability: The core idea of model-intrinsic difficulty assessment could be transferred to other machine learning domains beyond LLMs, where dataset quality is critical (e.g., active learning, curriculum learning for vision models).

Potential Issues/Areas for Improvement:

• Threshold for IFD > 1: While a threshold of 1 for IFD is intuitive for misalignment, a more nuanced, possibly data-driven or dynamically adjusted threshold could be explored for filtering. Some instructions might inherently be difficult without being misaligned.

• Computational Cost of IFD Calculation: Although more efficient than training full models, calculating Direct Answer Score for every sample still requires a pass through the dataset without instruction, which can be computationally intensive for very large datasets and models. The Superfiltering paper addresses this, indicating an area for future practical optimization.

• "Brief Experience" Definition: The choice of 100 clusters * 10 instances and 1 epoch for the pre-experienced model is somewhat heuristic. While ablations show robustness, a more theoretically grounded approach to defining "brief" or an adaptive mechanism for the pre-experience phase could be beneficial.

• Sub-Category Performance: The appendix shows that cherry models sometimes underperform in specific categories like Math, Coding, or Complex Format, especially when the original dataset was exceptionally rich in these areas or the base model is inherently weak in them. This suggests that while IFD is generally effective, some "data-hungry" categories might still require larger targeted datasets or different quality metrics.

  Overall, this paper offers a significant step forward in making LLM instruction tuning more intelligent and efficient, with broad implications for future research and deployment of large AI models.