1. Bibliographic Information

1.1. Title

The title of the paper is "Self-Improving LLM Agents at Test-Time". The central topic is a novel method for large language model (LLM) agents to enhance their performance on-the-fly during inference, without relying on traditional, expensive large-scale fine-tuning.

1.2. Authors

The paper is published by "Anonymous authors", indicating it is under double-blind review, a common practice in academic conferences to ensure fairness in the review process by concealing author identities. Therefore, specific author research backgrounds and affiliations are not disclosed.

1.3. Journal/Conference

The paper is published on OpenReview, a platform often used for submissions to conferences like ICLR (International Conference on Learning Representations) and NeurIPS (Conference on Neural Information Processing Systems) that employ a transparent peer-review process. The OpenReview platform allows for public discussion and review before final publication decisions. While the specific conference is not explicitly stated in the provided text, ICLR is mentioned in the ETHICS STATEMENT as the relevant Code of Ethics. Both ICLR and NeurIPS are top-tier conferences in machine learning, known for publishing cutting-edge research in artificial intelligence and deep learning.

1.4. Publication Year

The paper is published at (UTC): 2025-10-08T00:00:00.000Z, indicating a publication year of 2025.

1.5. Abstract

The abstract introduces the problem of inefficient and expensive large-scale data collection and fine-tuning for language models, which often fails to guarantee generalization or handle complex scenarios. It highlights that existing methods rarely consider the novelty or redundancy of training samples.

To address these issues, the paper proposes a new test-time self-improvement method for LLM agents, termed Test-Time Self-Improvement (TT-SI). The algorithm operates in three steps:

1. Self-awareness: Identifies samples the model struggles with using an uncertainty function.

2. Self-data augmentation: Generates similar examples from these uncertain samples.

3. Self-learning: Uses these newly generated samples for test-time fine-tuning.

   Two variants are explored: TT-SI, where the same model generates and learns from its own uncertain cases, and Test-Time Distillation (TT-D), where a stronger "teacher" model generates examples for the "student" model to adapt from.

Empirical evaluations on agent benchmarks show that TT-SI achieves a $+5.36$ absolute gain in average accuracy compared to standard learning methods, while using 68x fewer training samples. TT-D further enhances performance in more challenging scenarios requiring diverse training signals. The findings suggest that TT-SI offers a cost-effective and generalizable new paradigm for building self-evolving LLM agents in complex environments.

1.6. Original Source Link

The original source link is: https://openreview.net/forum?id=M1zSTXY1xr. This link points to the paper's page on OpenReview, where its status is typically "under double-blind review" or "accepted/rejected". The PDF link is https://openreview.net/pdf?id=M1zSTXY1xr.

2. Executive Summary

2.1. Background & Motivation

The paper addresses fundamental challenges in the current paradigm of fine-tuning large language models (LLMs), particularly for agentic LLMs (LLMs equipped with capabilities to interact with external tools and environments).

Core Problem: The prevailing approach involves creating massive training datasets, assuming that quantity and diversity will lead to robust generalization. However, this strategy faces several critical issues:

• Inefficiency and Expense: Gathering large datasets is costly and time-consuming, requiring significant computational resources and manual labor.

• Lack of Generalization Guarantee: Despite the investment, there's no guarantee that models trained on these datasets will effectively handle novel, complex scenarios or unseen test distributions (data distributions encountered during testing that might differ from the training data).

• Redundancy and Inefficient Resource Use: Current methods often process all training samples uniformly, without assessing whether a sample provides truly novel information or if it is redundant with knowledge the model has already acquired. This leads to wasted computational effort and can even hinder generalization, especially for long-tail examples (rare occurrences) or adversarial examples (intentionally crafted inputs to trick a model).

• Distributional Shift: Models often struggle when the data distribution they encounter at test time ($\mathcal{P}_{\text{test}}$) differs from the distribution they were trained on ($\mathcal{P}_{\text{train}}$).

• Catastrophic Forgetting: Fine-tuning on new tasks can inadvertently degrade performance on previously learned skills.

• Model Churn: The rapid release of new, more capable base LLMs necessitates costly re-training cycles for downstream tasks (specific tasks the model is adapted for), as the entire fine-tuning process must be repeated.

  Importance in the Current Field: As LLMs become central to agentic AI (AI systems that can perceive, act, and reason in environments), their ability to adapt efficiently and generalize robustly on-the-fly is paramount. The current inductive fine-tuning paradigm (learning general rules from specific examples) is becoming a bottleneck due to its cost, inflexibility, and inability to handle dynamic environments (environments that change over time).

Paper's Entry Point / Innovative Idea: Inspired by human learning—where individuals strategically identify knowledge gaps and focus on targeted practice rather than re-learning everything—the paper proposes a test-time self-improvement method. This method allows LLM agents to adapt on-the-fly to challenging instances (inputs the model struggles with) by identifying them, generating similar synthetic examples (data artificially created), and then performing a lightweight fine-tuning (small, temporary updates to the model's parameters) based on these generated examples. This shifts the paradigm from exhaustive offline training to targeted online adaptation.

2.2. Main Contributions / Findings

The paper makes several significant contributions to the field of LLM agent learning:

• Novel Three-Stage Algorithm: It introduces a test-time self-improvement framework that integrates self-awareness, self-augmentation, and self-learning. This algorithm is motivated by principles of human learning, enabling on-the-fly adaptation:

  1. Self-Awareness: An Uncertainty Estimator (H) identifies uncertain samples (inputs the model is less confident about) during inference.
  2. Self-Augmentation: A Data Synthesis Function (G) generates new, distributionally similar training instances (examples that resemble the original data's characteristics) based on these uncertain samples.
  3. Self-Learning: Test-Time Fine-tuning (T) performs temporary gradient updates (small, localized adjustments to model parameters) on these generated samples.

• Two Practical Variants: The paper studies two concrete implementations of this framework:

  • Test-Time Self-Improvement (TT-SI): The same model (student model) generates its own additional training examples and learns from them.
  • Test-Time Distillation (TT-D): A stronger, more capable model (teacher model) generates the training examples, providing distilled supervision (guidance from a more knowledgeable model) for the student model to adapt.

• Systematic Empirical Study: The authors conduct a comprehensive empirical study, analyzing critical components such as the choice of uncertainty estimator, the learning method at test time (e.g., fine-tuning vs. in-context learning), scaling of generated samples, and other parameter effects.

• Significant Performance Gains with High Efficiency:

  • TT-SI achieves an average absolute accuracy gain of $+5.36$ across three challenging agent benchmarks (NexusRaven, SealTool, API-Bank) compared to standard learning methods.
  • Crucially, TT-SI achieves these improvements with orders of magnitude less data, specifically using 68x fewer training samples than traditional supervised fine-tuning (SFT) on the SealTool benchmark, while still surpassing SFT's accuracy. This highlights its efficiency without compromising effectiveness.
  • TT-D further improves performance, especially in harder scenarios requiring more diverse and higher-quality training signals.
  • The research shows that agentic LLMs can self-improve during inference even from a single training instance per uncertain case.

• Fast, Training-Free Alternative (TT-SI with ICL): When direct fine-tuning is not feasible, TT-SI combined with in-context learning (ICL) (providing examples directly in the prompt) offers a training-free (no weight updates) and low-overhead alternative that can still outperform other standard learning methods in similar conditions, particularly for out-of-distribution (OOD) data (data that differs significantly from what the model was initially trained on).

• Foundational Framework for Self-Evolving Agents: The work pioneers a new algorithmic framework that integrates self-awareness, targeted self-generated data, and iterative self-training, paving the way for self-evolving agents capable of continuous lifelong adaptation (the ability to continuously learn and adapt over time). It opens new research directions in optimizing uncertainty estimation, data synthesis, and continual training for more capable and generalizable agents.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a foundational grasp of several machine learning and natural language processing concepts is essential:

• Large Language Models (LLMs): These are advanced artificial intelligence models trained on vast amounts of text data to understand, generate, and process human language. They typically employ transformer architectures and can perform a wide range of tasks, from translation and summarization to question answering and code generation. Examples include GPT models, Llama, and Qwen.
• Fine-Tuning: After initial pre-training on a massive, general corpus, LLMs are often further trained on smaller, task-specific datasets to adapt them to particular downstream tasks. This process is called fine-tuning. It adjusts the pre-trained model's parameters to improve its performance on the target task.
• Agentic LLMs / LLM Agents: These are LLMs that are not just text generators but can also perceive their environment (e.g., parse a user query), reason (e.g., decide which tool to use), and act (e.g., make an API call or perform a tool operation). They often involve tool-use capabilities, allowing them to interact with external systems or retrieve information.
• Test-Time Training (TTT): This is a paradigm where a model makes small, temporary adjustments to its parameters or behavior during the inference phase (when it's making predictions on new, unseen data). Unlike traditional training, these updates are typically ephemeral (short-lived) and specific to the current test input, aiming to improve generalization (ability to perform well on unseen data) under distributional shifts (differences between training and test data distributions).
• Parameter-Efficient Fine-Tuning (PEFT): Methods designed to fine-tune large models without updating all their parameters, significantly reducing computational cost and memory usage.
  • LoRA (Low-Rank Adaptation): A popular PEFT technique. Instead of fine-tuning all weights in an LLM, LoRA injects small, trainable rank-decomposition matrices into existing layers. This means that for a pre-trained weight matrix $W_0$, the fine-tuned weight matrix becomes $W_0 + \Delta W$, where $\Delta W = BA$. Here, $A$ and $B$ are low-rank matrices, meaning they have significantly fewer parameters than $W_0$. This allows for efficient fine-tuning by only training $A$ and $B$, while $W_0$ remains fixed. After fine-tuning, the adapted model can be represented by $W_0 + BA$, or the changes can be merged back into $W_0$ for deployment.
• Negative Log-Likelihood (NLL): A common loss function (a measure of how well a model is performing) used in classification and language modeling. For a given prediction, NLL measures the negative logarithm of the probability assigned by the model to the correct class or token. A lower NLL indicates a higher probability assigned to the correct output, thus better model confidence and performance.
• Softmax Function: A function that converts a vector of arbitrary real numbers (logits or scores) into a probability distribution. It squashes each value between 0 and 1, and the sum of all values equals 1. For a vector $z = [z_1, z_2, \ldots, z_N]$, the softmax output for component $j$ is: $ \text{softmax}(z)j = \frac{e^{z_j}}{\sum{k=1}^N e^{z_k}} $ This is often used to interpret model outputs as probabilities over possible actions or classes.
• In-Context Learning (ICL): A capability of LLMs where they learn a new task or adapt to a new data distribution simply by being provided with a few examples (demonstrations) within the prompt (the input text given to the model), without any gradient updates (changes to the model's internal parameters). The model leverages its vast pre-training knowledge to understand the pattern from these examples and apply it to a new query.
• Supervised Fine-Tuning (SFT): The traditional fine-tuning process where a model is trained on a dataset of input-output pairs ($x_i, y_i$) where $y_i$ are ground-truth labels (correct answers provided by humans or another reliable source). The model learns to map inputs to their correct outputs under human supervision.
• Uncertainty Estimation: The process of quantifying how confident a model is in its predictions. High uncertainty suggests the model might be struggling or that the input is out-of-distribution. Various methods exist, such as softmax confidence (using the highest softmax probability), entropy, or margin-based uncertainty (difference between top two predictions).

3.2. Previous Works

The paper builds upon and differentiates itself from several lines of prior research:

• Standard LM Post-Training and Fine-Tuning:

  • Kumar et al. (2025), Grattafiori et al. (2024), Yang et al. (2025), Zelikman et al. (2022), Guo et al. (2025), Zeng et al. (2024), Chen et al. (2024): These works represent the common paradigm of equipping LLMs with knowledge, reasoning, and agentic capabilities through large-scale fine-tuning on extensive datasets. The paper contrasts its approach with this standard, highlighting its inefficiencies, high cost, and inability to guarantee generalization to complex or novel scenarios.
  • Ouyang et al. (2022), Wang et al. (2023), Zeng et al. (2024): These works involve gathering large datasets (human-curated or LLM-synthesized) for fine-tuning. The current paper argues against the exhaustive nature of these datasets and the implicit assumption that all samples are equally informative.
  • Luo et al. (2025): This work addresses catastrophic forgetting, a problem where fine-tuning on new tasks degrades performance on previously acquired skills. The proposed TT-SI framework, by using temporary, instance-specific updates, aims to inherently mitigate catastrophic forgetting by not permanently altering the base model weights.

• Transductive and Local Learning:

  • Bottou & Vapnik (1992), Joachims (1999): These classical works introduced the concepts of transductive learning (predicting labels for specific, given test examples rather than learning a general function) and local learning (adapting a model based on the local neighborhood of a test point). The paper explicitly grounds its test-time adaptation approach in these principles, aiming to adapt the model on-the-fly based on individual test inputs rather than relying solely on a globally trained inductive model.

• Test-Time Training (TTT) in Deep Learning:

  • Sun et al. (2020): Demonstrated that a simple self-supervised TTT objective can improve robustness of image classifiers under distribution shift. This work laid a foundation for TTT in computer vision, showing its potential for online adaptation.
  • Sun (2023): Provides a comprehensive overview of TTT, outlining its benefits for small, ephemeral parameter updates during inference to condition the model on the current input.
  • Hardt & Sun (2024): Applied TTT to LLMs by fine-tuning on retrieved nearest neighbors to reduce perplexity (a measure of how well a probability model predicts a sample).
  • SIFT (Hübotter et al., 2025): Actively selects diverse, informative neighbors for LLM fine-tuning to limit redundancy.
  • Akyürek et al. (2025): Applied rule-based linear transformations to in-context test examples to generate additional test-time training data for few-shot learning.
  • Differentiation: The current paper distinguishes itself by focusing on general reasoning tasks for LLM-based agents (rather than just perplexity), and by selecting informative test instances, generating and filtering training signals on-the-fly without assuming access to high-quality neighbors or in-context exemplars. It introduces the first language generation-based test-time fine-tuning method applied to LLM agents.

• Self-Instruction and Data Synthesis:

  • Wang et al. (2023): Introduced self-instruction methodologies, where LLMs generate their own instructions and corresponding inputs/outputs for training. The paper leverages this idea in its Data Synthesis Function (G), where the agent itself (or a stronger teacher model) generates new training examples based on an uncertain seed example. This allows for self-augmentation without manual labeling.

3.3. Technological Evolution

The technological evolution in LLM training has moved from:

1. Massive Pre-training: Training very large transformer models on enormous, diverse text corpora to learn general language patterns and world knowledge.

2. Inductive Fine-tuning: Adapting these pre-trained models to specific downstream tasks using large, task-specific, i.i.d. (independent and identically distributed) datasets. This stage is where issues like cost, data redundancy, distributional shift, and catastrophic forgetting become prominent.

3. Parameter-Efficient Fine-Tuning (PEFT): Developing methods like LoRA to make fine-tuning less resource-intensive, but still largely operating within the inductive paradigm.

4. Test-Time Adaptation (TTT): Exploring ways for models to adapt dynamically during inference, moving towards local and transductive learning. This often involves self-supervised objectives or leveraging retrieved examples.

   This paper's work fits into the most recent stage, pushing the boundaries of test-time adaptation specifically for LLM agents. It integrates self-awareness (identifying struggles), self-augmentation (generating targeted data), and self-learning (on-the-fly fine-tuning) into a coherent framework.

3.4. Differentiation Analysis

Compared to existing methods, the core differences and innovations of this paper's approach are:

• Targeted, On-the-Fly Adaptation for Agents: Unlike general LLM TTT methods that might focus on perplexity or simpler tasks, this work specifically applies test-time fine-tuning to LLM-based agents performing complex tool-use and reasoning tasks. This is presented as the first such application of language generation-based TTT for LLM agents.

• Uncertainty-Guided Learning (Self-Awareness): A key innovation is the Uncertainty Estimator (H), which actively identifies challenging or uncertain samples. This is crucial because it allows the model to selectively adapt only when needed, contrasting with approaches that might adapt for all test instances or rely on external signals. This efficiency-first approach minimizes computational overhead.

• Self-Generated Data (Self-Augmentation): Instead of relying on retrieved nearest neighbors (which assume access to a high-quality, pre-indexed dataset) or rule-based transformations, TT-SI uses the LLM agent itself (or a teacher model in TT-D) to synthesize new training examples specifically tailored to the uncertain test instance. This makes the data generation process on-the-fly and adaptive, providing relevant local examples without requiring a large external data store.

• Lightweight, Ephemeral Fine-Tuning (Self-Learning): The framework uses LoRA for test-time fine-tuning, ensuring that updates are computationally efficient and temporary. This is vital for inference-time operations and explicitly avoids catastrophic forgetting by resetting parameters after each instance. This contrasts with inductive SFT, where permanent weight changes can lead to forgetting and costly retraining cycles.

• Efficiency and Generalizability: The results demonstrate that TT-SI achieves comparable or superior accuracy to SFT with significantly fewer training samples (e.g., 68x less), showcasing a substantial improvement in cost-effectiveness and generalization to OOD data.

  In essence, the paper moves beyond the passive consumption of pre-existing training data and towards a more active, metacognitive (thinking about one's own thinking) learning paradigm for LLM agents, where the agent intelligently identifies its weaknesses, generates its own relevant learning material, and adapts locally and temporarily.

4. Methodology

4.1. Principles

The core idea of the proposed Test-Time Self-Improvement (TT-SI) method is inspired by principles of human learning, particularly self-regulated learning. Just as a student preparing for an exam might metacognitively reflect on their knowledge, identify specific gaps (self-awareness), generate targeted practice questions to address those deficiencies (self-augmentation), and practice repeatedly (self-learning), TT-SI empowers LLM agents to perform a similar process during inference.

This approach shifts from the conventional inductive fine-tuning paradigm—which attempts to build a single, comprehensive model to cover all tasks from a large, static dataset—to a more local and transductive learning approach. The model adapts on-the-fly to challenging test instances by focusing computational and learning resources precisely where they are most needed, rather than processing redundant or already mastered information. This aims to create more effective and generalizable agentic LLMs in a cost-efficient manner.

4.2. Core Methodology In-depth (Layer by Layer)

The TT-SI framework is structured as a three-step algorithm that operates for each test sample. Let's deconstruct Algorithm 1 along with the mathematical formulations and detailed explanations for each component.

The overall process is summarized in Algorithm 1:

Algorithm 1 Test-Time Self-Improvement Framework
Require: Test dataset D_test, model M, data generation prompt P, temporary dataset size K, initial
    model parameters θ_0
    for each x_i ∈ D_test do
        Step 1: Uncertainty Estimator (H)
        Compute uncertainty (softmax-difference):
            ℓ_n = -log P_M(a_n | x_i), ∀ a_n  ▷ Negative Log-Likelihood (NLL) for candidate action
            p_n = exp(ℓ_n - max_j ℓ_j) / Σ_k exp(ℓ_k - max_j ℓ_j)  ▷ Apply Relative Softmax Scoring (RSS) normalization
            u(x_i) = p^(1) - p^(2)  ▷ Highest minus second-highest RSS scores
        Step 2: Data Synthesis Function (G)
        if u(x_i) < τ then ▷ Check uncertainty
            Generate K synthetic samples using LLM:
                D_i ← L_gen(x_i, K)  ▷ Equation (5)
        Step 3: Test-Time Fine-tuning (T)
            Learn temporary model parameters θ_i^* via LoRA:
                θ_i^* ← arg min_{θ_0} Σ_{(x', y') ∈ D_i} ℓ(M(x'; θ_0), y')  ▷ Equation (7)
            Perform inference with adapted parameters θ_i^* :
                ŷ_i ← M(x_i; θ_i^*)
            Reset model parameters:
                θ_i^* → θ_0  ▷ Restore original parameters
        else
            Perform inference directly:
                ŷ_i ← M(x_i; θ_0)
        end if

Let's break down each step:

4.2.1. Step 1: Self-Awareness (Uncertainty Estimator H)

This step focuses on identifying which input samples the model $\mathcal{M}$ is uncertain about. The goal is to selectively target only the informative and challenging cases for adaptation, thereby enhancing efficiency.

Definition: Given an input $x_i$ (e.g., a task query), the Uncertainty Estimator (H) determines the model's confidence in its potential actions. The model $\mathcal{M}$ can choose from a set of available actions $a_1, ..., a_n ∈$\mathcal{A}

(e.g., different `API calls`). For each input $x_i$ and a candidate action $a_n$, the confidence score

\mathcal{C}_i is computed as: $ \mathcal{C}_{i}=\mathrm{H}\left(x_{i}, a_{n}, \mathcal{M}\right) $ Here, \mathcal{C}_i represents the confidence score for input $x_i$, $a_n$ is a candidate action, and \mathcal{M}

is the model. This estimation happens without using the true label $y_i$, which ensures it can be applied during inference when labels are unavailable. An input $x_i$ is considered `uncertain` if its confidence score

\mathcal{C}_i

falls below a predefined threshold

\tau

.

**Selecting Uncertain Samples (Algorithmic Details):**
To quantify uncertainty, the method uses a `margin-based confidence estimator`. This involves three sub-steps:

1.  **Compute Negative Log-Likelihood (NLL):** For each possible action $a_n$ out of $N$ available actions, the `NLL` is calculated. `NLL` represents the negative logarithm of the probability $P_M(a_n | x_i)$ that the model

\mathcal{M}

assigns to action $a_n$ given input $x_i$. A lower `NLL` means the model considers that action more likely.
    $\ell_{n}=-\log P_{\mathcal{M}}\left(a_{n} \mid x_{i}\right), \quad \forall n \in 1,2, \ldots, N$
    Here,

\ell_n

denotes the `NLL` for action $a_n$.

2.  **Apply Relative Softmax Scoring (RSS):** `Raw NLL scores` are unbounded and difficult to interpret directly as confidence. To address this, a `Relative Softmax Scoring (RSS)` mechanism transforms `NLL` scores into a normalized `confidence distribution`. This involves exponentiating the negative `NLL` scores (which can be interpreted as `logits` or `unnormalized log-probabilities`), shifting them by the maximum `NLL` score for numerical stability, and then normalizing them using the `softmax` function.
    $p^{n}=\frac{\exp \left(\ell_{n}-\max _{j} \ell_{j}\right)}{\sum_{k=1}^{N} \exp \left(\ell_{k}-\max _{j} \ell_{j}\right)}, \quad \text { where } \quad \ell_{n}=-\operatorname{NLL}\left(a_{n} \mid x_{i}\right)$
    In this formula:
    *

p^n

is the `RSS confidence score` for action $a_n$.
    *

\ell_n

is the `negative log-likelihood` score for action $a_n$.
    *

\max_j \ell_j

is the maximum `NLL` score among all candidate actions. This term is crucial for `numerical stability` by preventing `overflow` when `exponentiating` large negative values. It effectively re-centers the scores before exponentiation.
    *   The `exp()` function converts log-probabilities back to probabilities.
    *   The denominator ensures that the sum of all `RSS` scores

\sum_{k=1}^{N} p^k

equals 1, forming a `probability distribution`.

3.  **Compute Softmax-Difference Uncertainty:** To quantify the `prediction uncertainty`, the method calculates the difference between the highest (

p^{(1)}

) and second-highest (

p^{(2)}

) `RSS scores`. A smaller difference indicates higher uncertainty (the model is nearly equally confident about two or more actions), while a larger difference indicates higher confidence in the top prediction.
    $u\left(x_{i}\right)=p^{(1)}-p^{(2)}$
    Here:
    *

u(x_i) is the uncertainty score for input $x_i$. * p^{(1)}

is the highest `RSS score` among all candidate actions.
    *

p^{(2)}

is the second-highest `RSS score` among all candidate actions.

        Finally, an input $x_i$ is selected as `uncertain` if

u(x_i) < \tau

, where

\tau

is a `user-defined threshold`. If

u(x_i) \geq \tau

, the model performs inference directly using its current parameters without any adaptation.

### 4.2.2. Step 2: Self-Augmentation (Data Synthesis Function G)

If Step 1 identifies an input $x_i$ as `uncertain` (i.e.,

u(x_i) < \tau

), the `Data Synthesis Function (G)` is triggered to generate new training examples.

**Definition:**
When $x_i$ is flagged as uncertain, $G$ generates $K$ new `synthetic training examples` along with their corresponding labels. These examples

(x_{ij}^{\prime}, y_{ij}^{\prime})_{j=1}^{K}

are designed to be `semantically similar` to the original uncertain $x_i$ but with slight variations. The original $x_i$ acts as a `seed example` to guide the generation process.
$\mathbf{G}: x_{i} \rightarrow\left\{\left(x_{i j}^{\prime}, y_{i j}^{\prime}\right)\right\}_{j=1}^{K}$
Here:
*

\mathbf{G}

is the `Data Synthesis Function`.
*

x_i

is the original uncertain input.
*   `$K$` is a `user-defined hyperparameter` specifying the number of synthetic samples to generate.
*

(x_{ij}^{\prime}, y_{ij}^{\prime})

represents the $j$-th generated input-output pair for the uncertain sample $x_i$.

    These $K$ generated pairs immediately form a `temporary, query-specific dataset`

\mathcal{D}_i: $ \mathcal{D}_{i}=\left\{\left(x_{i j}^{\prime}, y_{i j}^{\prime}\right)\right\}_{j=1}^{K} $ This dataset \mathcal{D}_i

is specifically created for `localized adaptation` of the model's parameters for the current `uncertain input` $x_i$.

**Generating Samples (Algorithmic Details):**
The implementation of

\mathbf{G}

utilizes the `agent itself` for `data synthesis`, denoted as

\mathcal{L}_{\text{gen}}

(this is `self-augmentation`). For each generation instance,

\mathcal{L}_{\text{gen}}$$is provided with: * A `carefully hand-crafted prompt`\mathcal{P}$$ (see Figure 6). This prompt instructs the LLM on how to generate variants.

• The uncertain inputx_i

as a `direct seed`, *crucially without its corresponding label $y_i$* (because $y_i$ is unknown at inference time).
*   The specified number of samples $K$ to generate.

    The `LLM` then produces $K$ new input-output pairs

{(x_{ij}^{\prime}, y_{ij}^{\prime})}_{j=1}^{K}

. This `seed-based generation` process, inspired by `self-instruction methodologies`, guides

\mathcal{L}_{\text{gen}}

to create `variants` that maintain the `core semantic meaning` and `task relevance` of $x_i$ while introducing `controlled surface-level variations`. This `on-the-fly data synthesis` enables `targeted` and `timely model adaptation`.

The prompt for data generation (Figure 6) explicitly instructs the LLM to:
1.  Create distinct variants by altering names, context, or wording, ensuring no duplication of the original.
2.  Adhere to a specific `JSON` output format for both the overall response and the `function calling` structure within the `"output"` field.

### 4.2.3. Step 3: Self-Learning (Test-Time Fine-Tuning T)

After detecting uncertainty and generating a `temporary dataset`

\mathcal{D}_i

, the final step is `Test-Time Fine-tuning (T)`, which adapts the model

\mathcal{M}

using these new samples.

**Definition:**
The initial model parameters

\theta_0

are optimized to minimize a `loss function`

\mathcal{L}(\mathcal{D}_i; \theta_0)

on the generated dataset

\mathcal{D}_i

. This optimization yields `temporarily updated parameters`

\theta_i^*

specifically for predicting the target task for the current input $x_i$. A critical aspect is that *after* making a prediction, the model's parameters are `restored` to the original

\theta_0

for the next iteration (processing $x_{i+1}$). This ensures that adaptations are `instance-specific` and do not `permanently alter the base model`, thereby preventing `catastrophic forgetting`.

**Test-Time Fine-Tuning (Algorithmic Details):**
The primary goal is to temporarily adjust the model

\mathcal{M}'s parameters $\theta$ to better handle the current uncertain samplex_i

. This is achieved by `fine-tuning` the model on the newly synthesized dataset

\mathcal{D}i = {(x{ij}^{\prime}, y_{ij}^{\prime})}_{j=1}^{K}

.

1.  **Loss Minimization:** The adaptation involves minimizing a `task-specific loss function`

\mathcal{L}_{\text{task}}

over the samples in

\mathcal{D}_i

. For each self-generated sample

(x_{ij}^{\prime}, y_{ij}^{\prime}) \in \mathcal{D}_i

, the loss is computed as

\ell(\mathcal{M}(x_{ij}^{\prime}; \theta), y_{ij}^{\prime})

. The objective function for adapting parameters

\theta

using

\mathcal{D}_i is: $ \theta_{i}^{*}=\arg \min _{\theta^{\prime}} \sum_{\left(x_{i j}^{\prime}, y_{i j}^{\prime}\right) \in \mathcal{D}_{i}} \ell\left(\mathcal{M}\left(x_{i j}^{\prime} ; \theta^{\prime}\right), y_{i j}^{\prime}\right) $ Here: * \theta_i^*

represents the `adapted parameters` specifically for the context of $x_i$.
    *

\arg \min_{\theta^{\prime}}

means finding the parameter set

\theta^{\prime}

that minimizes the subsequent sum.
    *

\ell(\mathcal{M}(x_{ij}^{\prime}; \theta^{\prime}), y_{ij}^{\prime})

is the `loss` incurred by the model

\mathcal{M}

(with parameters

\theta^{\prime}

) on the generated input $x_{ij}^{\prime}$ compared to its generated label $y_{ij}^{\prime}$. The `summation` aggregates losses over all $K$ generated samples in

\mathcal{D}_i

.

2.  **Efficient Updates with LoRA:** To ensure `computational efficiency` during these `inference-time updates`, the method employs `Low-Rank Adaptation (LoRA)`. `LoRA` allows for `fine-tuning` with a small number of additional parameters, making the updates fast and memory-efficient.

3.  **Inference and Parameter Reset:** Once

\theta_i^ is learned, inference is performed on the original uncertain input $x_i$ using these adapted parameters: \hat{y}_i \leftarrow \mathcal{M}(x_i; \theta_i^)

. Immediately after, the model parameters are `reset` to their original state

\theta_0

. This crucial step ensures that the adaptation is temporary and local, preventing any accumulated changes from affecting subsequent, unrelated test instances.

### 4.2.4. Variants: TT-SI and TT-D

The paper explores two variants of this framework:
*   **Test-Time Self-Improvement (TT-SI):** In this variant, the `Data Synthesis Function (G)` (specifically

\mathcal{L}_{\text{gen}}

) is implemented using `the same model`

\mathcal{M}

that is being adapted. This means the model generates additional training examples from its own uncertain cases and then learns from them. This is a purely `self-contained` approach.
*   **Test-Time Distillation (TT-D):** In this variant, $G$ uses a `stronger, more capable model` (referred to as a `teacher model`, e.g., `GPT-5-mini`) to generate similar examples for the uncertain cases identified by the `student model`

\mathcal{M}

. The `student model` then adapts using this `distilled supervision` from the `teacher`. This approach aims to provide higher-quality and potentially more diverse training signals, especially for complex scenarios.

    This iterative process of `detection`, `synthesis`, and `adaptation` is performed for each sample identified as uncertain, providing a dynamic and efficient way for `LLM agents` to improve `on-the-fly`.

# 5. Experimental Setup

## 5.1. Datasets

The experiments evaluate the `TT-SI` approach on three distinct `agent benchmarks`, each designed to test different aspects of `LLM agent capabilities`. These benchmarks are publicly available and widely used, ensuring transparency and reproducibility.

1.  **NexusRaven (Srinivasan et al., 2023):**
    *   **Description:** A `function-calling benchmark` specifically designed to test an agent's ability to execute single, `nested`, and `parallel function calls` of varying complexity. It focuses on realistic software operation tasks, particularly in `cybersecurity` and `enterprise applications`.
    *   **Characteristics:** Features `long` and `diverse tool invocations` across `65 distinct APIs`.
    *   **Scale:** Total of `318 samples`.
    *   **Purpose:** Tests high-fidelity function execution in business scenarios.
    *   **Data Sample:**
        The following figure (Figure 7 from the original paper) shows a sample example from the NexusRaven test data:

        ![img-7.jpeg](images/nexusraven_sample.jpeg)

        This example demonstrates a user request to "standardize this address" and the expected `tool_call` output using the `standardizeUSAddress` function with appropriate parameters.

2.  **SealTool (Wu et al., 2024):**
    *   **Description:** A `self-instruct dataset` for `tool learning`, measuring precision in `tool selection`, adherence to `output formats`, and `adaptability` across diverse scenarios. Its latest version is designed to minimize `potential data leakage`.
    *   **Characteristics:** Comprises `4,076 APIs` spanning various domains.
    *   **Scale:** A `curated test set` of `294 samples` is used in the experiments. It also provides an `official training split` of approximately `13k samples` for comparison with `SFT` baselines.
    *   **Purpose:** Serves as a robust benchmark for `tool-use evaluation`.
    *   **Data Sample:**
        The following figure (Figure 8 from the original paper) shows a sample example from the SealTool test data:

        ![img-8.jpeg](images/sealtool_sample.jpeg)

        This example shows a user asking for "statistics for the Real Madrid team" and the `tool_call` output using the `getTeamStats` function.

3.  **API-Bank (Li et al., 2023):**
    *   **Description:** Evaluates `multi-turn user-agent dialogues`, requiring agents to `track conversational state`, make `informed tool calls` at each turn, and handle `realistic conditions` such as `noisy` or `incomplete inputs`.
    *   **Characteristics:** Contains `314 multi-turn conversations` with `753 distinct API calls`.
    *   **Scale:** The experiments focus on `316 samples` from `Levels 1 and 2` to balance `task complexity` and `data availability`.
    *   **Purpose:** Assesses `LLM` agents' capabilities in `conversational tool use`.
    *   **Data Sample:**
        The following figure (Figure 9 from the original paper) shows a sample example from the API-Bank test data:

        ![img-9.jpeg](images/apibank_sample.jpeg)

        This example illustrates a `multi-turn conversation` where the user wants to "modify my appointment" and, after further clarification, the `tool_call` for `ModifyRegistration` is generated.

**Why these datasets were chosen:** These datasets represent a diverse range of `agentic tasks`, from single function calls to complex `multi-turn dialogues` and various `API` types. They are well-established and challenging, making them suitable for validating the `efficacy`, `generalizability`, and `efficiency` of new `agent learning` methods like `TT-SI`.

## 5.2. Evaluation Metrics

The primary evaluation metric used across all benchmarks for assessing agent performance is `Accuracy`. Additionally, for evaluating the `Uncertainty Estimator (H)` itself, several other metrics are used: `True Positive Rate (TPR)`, `False Positive Rate (FPR)`, `F1 Score`, and `Youden's J statistic`.

1.  **Accuracy:**
    *   **Conceptual Definition:** Accuracy measures the proportion of correct predictions (function calls) made by the model relative to the total number of predictions. For `agentic tasks`, a prediction is considered correct only if all components (function name, arguments, and their values/types) `exactly match` the `ground truth`.
    *   **Mathematical Formula:**
        $\text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}}$
    *   **Symbol Explanation:**
        *   `Number of correct predictions`: The count of instances where the model's output (function name, arguments, and values) perfectly matches the expected `ground-truth` output.
        *   `Total number of predictions`: The total number of test instances processed by the model.

2.  **True Positive Rate (TPR) / Sensitivity / Recall:**
    *   **Conceptual Definition:** `TPR` measures the proportion of actual `positive cases` (e.g., truly uncertain/erroneous predictions by the base model) that were correctly identified as positive. In the context of `uncertainty estimation`, it tells us how many of the model's actual errors were successfully flagged as `uncertain`.
    *   **Mathematical Formula:**
        $\text{TPR} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$
    *   **Symbol Explanation:**
        *   `True Positives`: Cases where the base model made an incorrect prediction, and the `Uncertainty Estimator (H)` correctly identified it as uncertain.
        *   `False Negatives`: Cases where the base model made an incorrect prediction, but $H$ failed to identify it as uncertain.

3.  **False Positive Rate (FPR):**
    *   **Conceptual Definition:** `FPR` measures the proportion of actual `negative cases` (e.g., correct predictions by the base model) that were incorrectly identified as positive. In `uncertainty estimation`, it indicates how many of the model's correct predictions were mistakenly flagged as `uncertain`, leading to unnecessary adaptation.
    *   **Mathematical Formula:**
        $\text{FPR} = \frac{\text{False Positives}}{\text{False Positives} + \text{True Negatives}}$
    *   **Symbol Explanation:**
        *   `False Positives`: Cases where the base model made a correct prediction, but $H$ incorrectly identified it as uncertain.
        *   `True Negatives`: Cases where the base model made a correct prediction, and $H$ correctly identified it as *not* uncertain.

4.  **F1 Score:**
    *   **Conceptual Definition:** The `F1 Score` is the `harmonic mean` of `precision` and `recall` (which is equivalent to `TPR`). It is a balanced metric that considers both `false positives` and `false negatives`, providing a single score that is useful when classes are unevenly distributed or when both minimizing `false positives` and `false negatives` is important. For `uncertainty estimation`, it reflects the overall effectiveness of correctly identifying uncertain cases.
    *   **Mathematical Formula:**
        First, we define `Precision` and `Recall`:
        $\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$
        $\text{Recall} = \text{TPR} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$
        Then, `F1 Score` is:
        $\text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$
    *   **Symbol Explanation:**
        *   `True Positives`, `False Positives`, `False Negatives`: As defined above.
        *   `Precision`: The proportion of instances identified as uncertain that were *actually* incorrect predictions by the base model.
        *   `Recall`: The proportion of actual incorrect predictions by the base model that were correctly identified as uncertain (same as `TPR`).

5.  **Youden's J Statistic:**
    *   **Conceptual Definition:** `Youden's J statistic` (also known as `Youden's index`) is a single measure that captures the overall diagnostic effectiveness of a `binary classification test` (like our uncertainty estimator). It maximizes the sum of `sensitivity` (`TPR`) and `specificity` (`True Negative Rate`, or $1 - FPR$), effectively balancing the `true positive rate` against the `false positive rate`. A value of 1 indicates a perfect test, while 0 indicates no better than random chance.
    *   **Mathematical Formula:**
        $J = \text{TPR} - \text{FPR}$
    *   **Symbol Explanation:**
        *   `TPR`: `True Positive Rate`, as defined above.
        *   `FPR`: `False Positive Rate`, as defined above.

## 5.3. Baselines

The paper compares `TT-SI` and `TT-D` against several standard learning methods to demonstrate their advantages:

1.  **Standard Prompting (Base / w/o TT-SI):** This is the most basic baseline, where the `LLM agent` (`Qwen2.5-1.5B-Instruct`) performs `zero-shot inference` (making predictions without any examples or fine-tuning specifically for the task in the prompt) directly on the test samples. This represents the model's inherent capability without any adaptation. The paper reports results for both `Input/Output` (direct prediction) and `Majority Vote` (using `self-consistency` by generating multiple outputs and taking the majority decision).

2.  **In-Context Learning (ICL):** A common `few-shot learning` technique where the `LLM` is given a few example input-output pairs directly within the `prompt` to guide its prediction for a new query. The paper tests `1-shot ICL` where one example is provided. This baseline demonstrates how well the model can adapt without any parameter updates. The paper also includes a variant `TT-SI with ICL`, where the `self-generated examples` are inserted into the `prompt` instead of being used for fine-tuning.

3.  **Supervised Fine-Tuning (SFT):** This is the traditional `fine-tuning` approach where the `LLM` is trained on a large, `human-annotated` or `synthesized dataset` for the `downstream task`. The paper specifically uses the `official training split` of `SealTool` (approximately `13k samples`) for this baseline. `SFT` represents the conventional method for adapting `LLMs` to specific tasks and serves as a strong benchmark for `accuracy` and `generalization`. The comparison against `SFT` highlights `TT-SI`'s `data efficiency`.

    These baselines are representative because they cover the spectrum of common `LLM deployment strategies`, from `zero-shot` to `few-shot` (`ICL`) and `full fine-tuning` (`SFT`), allowing for a comprehensive evaluation of `TT-SI`'s performance relative to established techniques.

# 6. Results & Analysis

## 6.1. Core Results Analysis

The empirical evaluations demonstrate that `TT-SI` consistently improves `LLM agent` performance at `test-time`, often with significantly higher `data efficiency` than conventional methods.

**Insight 1: Agents can self-improve at test-time even when training on just one sample.**
The main results presented in Table 1 show `TT-SI`'s effectiveness across three agentic benchmarks (NexusRaven, SealTool, API-Bank) under two inference settings: `Input/Output` (direct prediction) and `Majority Vote` (using `self-consistency` to improve robustness).
*   **TT-SI's Improvement:** `TT-SI` improves the baseline (Base) by an average of $+5.36$ absolute gain in direct inference (`60.26%` to `65.62%`) and $+3.52$ with `majority vote` (`63.42%` to `66.94%`). This is achieved by generating and learning from only `one synthetic sample per uncertain case`. This finding is crucial as it demonstrates that minimal, `uncertainty-guided adaptation` can substantially boost `inference-time performance`.
*   **TT-D's Further Gains:** `Test-Time Distillation (TT-D)`, where a `stronger teacher model` (`GPT-5-mini`) generates the synthetic data, further improves performance over `TT-SI` by $+2.71$ (direct inference) and $+1.17$ (majority vote). This indicates that higher-quality, potentially more diverse `training signals` from a superior model can provide additional, consistent gains, particularly in complex scenarios.

**Insight 2: TT-SI outperforms inductive SFT with orders of magnitude less data.**
Figure 2 (left) compares `TT-SI` against `in-context learning (ICL)` and `supervised fine-tuning (SFT)` on `SealTool`, which offers an official training split of approximately `13,000 samples`.
*   `TT-SI` (with SFT as the learning method) achieves `72.43%` accuracy, surpassing all three baselines.
*   Notably, it exceeds `standard inductive SFT` (`70.20%`) by $+2.23$ accuracy.
*   This improvement is achieved using only `190 uncertain cases` (each with one synthetic example), which amounts to `68x` fewer samples than the `12,000 samples` used for the full `SFT` training set. This highlights `TT-SI`'s remarkable `efficiency` and makes it an effective alternative to conventional `inductive learning` approaches that require massive datasets.

**Insight 3: When training is infeasible, Test-Time Self-Improvement with ICL offers a fast alternative.**
As shown in Figure 2 (left), when `TT-SI` is used with `in-context learning (ICL)` (inserting generated examples directly into the prompt without fine-tuning), it achieves a slight improvement over the base model (`66.37%` to `66.38%`). More importantly, `TT-SI` with `ICL` also outperforms the `standard ICL baseline` (`67.74%`) that leverages `SealTool's training split`. This suggests that even without `fine-tuning`, `TT-SI's uncertainty-guided generation` of `demonstrations` can `boost model confidence` and `accuracy`, especially when a dedicated training split or fine-tuning is impractical. This offers a `training-free`, `low-overhead` alternative.

**Insight 4: Uncertainty filtering balances accuracy and efficiency.**
The `Uncertainty Estimator (H)` plays a critical role in `TT-SI` by identifying `uncertain samples` for `targeted adaptation`, while `certain ones` are passed directly. The paper evaluates a variant `TT-SI w/o H` (without the uncertainty estimator) where *all* test inputs are treated as uncertain and subjected to adaptation.
*   `TT-SI w/o H` achieves `73.47%` accuracy, a marginal $+1.04$ gain over `TT-SI` with $H$ (`72.43%`).
*   However, this requires adapting on all `294 test samples` instead of just `190 samples` (an additional `104 LoRA weights` to learn).
*   The slight accuracy gain is `outweighed by the significant efficiency loss` due to the increased computational cost. This `underscores the importance of uncertainty filtering` for practical `test-time adaptation`, demonstrating that intelligently selecting *which* samples to adapt on is crucial for an optimal `cost-performance trade-off`.

## 6.2. Data Presentation (Tables)

The following are the results from [Table 1] of the original paper:

| Inference | Method | NexusRaven | SealTool | API-Bank | Avg. | $\Delta \%$
| :--: | :--: | :--: | :--: | :--: | :--: | :--:
| Input/Output | w/o TT-SI (Base) | 44.03 | 66.67 | 70.08 | 60.26 | -
|  | w. TT-SI | 50.08 | 72.43 | 74.34 | 65.62 | $\uparrow 5.36$
|  | w. TT-D | 52.52 | 75.17 | 77.29 | 68.33 | $\uparrow 8.07$
| Majority Vote | w/o TT-SI (Base) | 46.56 | 69.73 | 73.96 | 63.42 | -
|  | w. TT-SI | 52.20 | 72.93 | 75.68 | 66.94 | $\uparrow 3.52$
|  | w. TT-D | 54.53 | 72.25 | 77.56 | 68.11 | $\uparrow 4.69$

This table presents the `accuracy` results for the `baseline prompting` (`w/o TT-SI (Base)`), `TT-SI`, and `TT-D` methods across three `agentic benchmarks` (NexusRaven, SealTool, API-Bank) under two different `inference settings` (`Input/Output` and `Majority Vote`).

\Delta

denotes the average absolute improvement over the corresponding baseline, and

\uparrow

indicates a performance increase.

The following are the results from [Table 2] of the original paper:

| $\tau /$ Setting | TPR | FPR | Unc. $(\Delta \%)$ | Acc.
| :-- | :--: | :--: | :--: | :--:
| Base | - | - | - | 66.37
| 0.35 | 0.42 | 0.09 | $51(17 \%)$ | 68.10
| 0.95 | 0.96 | 0.53 | $190(65 \%)$ | 72.43
| No Unc. (all) | 1.00 | 1.00 | $294(100 \%)$ | 73.47

This table illustrates the impact of the `uncertainty threshold`

\tau

on `TT-SI`'s performance and efficiency on the `SealTool` benchmark. It shows the `True Positive Rate (TPR)`, `False Positive Rate (FPR)`, the number of `uncertain samples` identified (Unc., with percentage of total test samples), and the resulting `accuracy`.

## 6.3. Ablation Studies and Analysis

**Insight 5: Data scaling on OOD data highlights the limits of SFT and the strength of TT-SI.**
Figure 2 (right) illustrates the `data scaling behavior` for `standard SFT` and `TT-SI` on `out-of-distribution (OOD)` data using the `xLAM function-calling dataset`.
*   `TT-SI` consistently outperforms `SFT` across all tested scales (1, 2, 4, 8 samples), with improvements becoming more pronounced as more `uncertain examples` are incorporated. This underscores the value of `uncertainty-guided data` and `targeted test-time learning` when dealing with `OOD` scenarios.
*   Even the `training-free variant of TT-SI with ICL` surpasses `standard SFT` under these conditions, using the same data amounts per scale. This suggests that `test-time approaches` can be highly effective even without a dedicated training split or `fine-tuning` when confronting `OOD` data.

**Insight 6: Optimal $\tau$ improves efficiency with minimal accuracy loss.**
Table 2 details the impact of the `uncertainty threshold`

\tau

on `TT-SI`'s performance and efficiency.
*   Regardless of

\tau

, `TT-SI` consistently improves `accuracy` over the `Base` model (`66.37%`).
*   A `high`\tau

(e.g., 0.95 or No Unc. (all) where $\tau$ effectively approaches 1 and selects all samples) leads to higher accuracy (72.43% at $\tau=0.95$, 73.47% for No Unc.). However, selecting all samples (No Unc. (all)) requires updates for all 294 instances, incurring substantial computational overhead for only marginal additional gains.

• At $\tau=0.95$, TT-SI achieves 72.43% accuracy with only 190 updates (65% of total samples), meaning 35% fewer updates than adapting on all samples, while preserving near-optimal performance. This represents an effective balance between accuracy and efficiency.
• A low\tau

(e.g., `0.35`) minimizes $False Positives (FPR=0.09)$, meaning few correct predictions are unnecessarily flagged. However, it also misses many errors ($TPR=0.42$), resulting in lower accuracy (`68.10%`).
*   The paper concludes that

\tau=0.95

offers the best trade-off, capturing most errors ($TPR=0.96$) while avoiding `redundant updates` ($FPR=0.53$ is higher than

\tau=0.35

, but the overall `Youden's J` (balance of `TPR-FPR`) is optimized here, as discussed in Section D of the Appendix).

    The following figure (Figure 4 (Right) from the original paper) summarizes the performance of the uncertainty estimator compared to other baselines.

    ![img-4.jpeg](images/uncertainty_perf.jpeg)

This table shows that the proposed `Uncertainty Estimator (H)` (labeled as `Ours`) significantly outperforms `Random`, `Trivial` (always uncertain), and `Perplexity`-based methods across all metrics (`TPR`, `FPR`, `F1`, `Youden's J`), achieving the highest `F1` (`61.19`) and `Youden's J` (`63.29`). This empirical validation confirms the effectiveness of the `softmax-difference` method for identifying challenging samples.

**Insight 7: TT-SI improves both small and large Qwen models, with larger relative gains for smaller models.**
The experiments evaluate `TT-SI`'s `scalability` across different model capacities using `Qwen2.5-1.5B-Instruct` (smaller) and `Qwen2.5-7B-Instruct` (larger).
*   For the smaller model, `TT-SI` yields a substantial $+5.76$ absolute gain (`66.67%` to `72.43%`).
*   For the larger model, it delivers a $+3.02$ gain (`80.95%` to `83.97%`).
*   These results indicate that `TT-SI` consistently enhances `performance` regardless of `model size`, supporting its `architectural generality`.
*   Interestingly, the `relative boost` is more pronounced for `smaller models`. This suggests that `TT-SI` can be an `efficiency-oriented strategy` for deploying `compact agentic models` by extracting more performance from them.

## 6.4. Visualizations

The following figure (Figure 1 (Left) from the original paper) provides an overview of the `Test-Time Self-Improvement (TT-SI)` framework.

![img-0.jpeg](/files/papers/690387b07ee60e235c4a1054/images/1.jpeg)
*该图像是流程示意图及柱状图，展示了文中提出的测试时自我提升方法。左侧流程图包含输入、自我认知、自我增强及自我提升四步，右侧柱状图展示TT-SI方法在准确率上的提升，最高达73.47%。*

This flowchart visually depicts the three core steps: `Self-Awareness` (identifying challenging samples with $H$), `Self-Data Augmentation` (generating similar variants with $G$), and `Self-Improvement` (lightweight update via `Test-Time Fine-tuning (T)`). The right panel shows the performance comparison on `SealTool` that supports `Insight 2`.

The following figure (Figure 2 (Left and Right) from the original paper) presents the experimental results on `SealTool`.

![img-1.jpeg](/files/papers/690387b07ee60e235c4a1054/images/2.jpeg)
*该图像是包含两部分的图表，展示了不同训练样本数量下TT-SI和TT-D方法的准确率对比。左图为条形图，显示基线、ICL、SFT、TT-SI和TT-D的准确率及其样本数；右图为曲线图，比较GPT-5-mini模型在不同样本规模下的性能表现。*

The left bar chart visually compares `TT-SI` against `baselines` and its variants, reinforcing `Insight 2` and `Insight 3`. The right line chart shows `scaling behavior` under different `adaptation strategies` with varying numbers of samples, supporting `Insight 5`.

The following figure (Figure 3 from the original paper, not provided in markdown but referenced in text for Insight 7) would likely show bar charts or line graphs comparing the accuracy of `Qwen2.5-1.5B-Instruct` and `Qwen2.5-7B-Instruct` with and without `TT-SI`, visually supporting `Insight 7` about `TT-SI` improving both small and large models with higher relative gains for smaller ones.

The following figure (Figure 4 (Left) from the original paper) depicts the `Uncertainty Estimator Performance` with `Qwen-2.5-1.5B-Instruct` on `Nexus-Raven`.

![img-2.jpeg](/files/papers/690387b07ee60e235c4a1054/images/3.jpeg)
*该图像是一个折线图，展示了不同阈值下模型的真实正例率（TPR）和假正例率（FPR）变化趋势。图中对比了本方法（Ours）和随机基线（Random）的表现，并标注了最大Youden指数$I=63.0\%$，对应阈值0.95时的指标。*

This line graph illustrates the effect of varying the `softmax-difference threshold`

\tau

on `TPR` and `FPR`. It shows how increasing

\tau

increases both `TPR` (correctly identifying errors) and `FPR` (mistakenly flagging correct predictions). The `Youden's J` statistic, which balances `sensitivity` and `specificity`, peaks at

\tau=0.95

, confirming this as an optimal threshold for the estimator.

The following figure (Figure 5 from the original paper) visualizes the `Self-Generated Data`.

![img-3.jpeg](/files/papers/690387b07ee60e235c4a1054/images/4.jpeg)
*该图像是一个散点图，展示了测试样本和生成样本在二维空间中的分布情况。图中多种颜色和形状区分了不同类别的原始样本和对应的生成样本，突出显示了生成样本围绕着对应的测试样本聚集，反映了自我数据增强的效果。*

This scatter plot, using `UMAP` for 2D semantic embedding, shows `all test samples` (circles), an `uncertain input` $x_i$ (star), and `10 self-generated synthetic queries` (triangles) for that $x_i$. The `tight clustering` of the `generated samples` around $x_i$ demonstrates the `distributional alignment` and `semantic faithfulness` of the `Data Synthesis Function (G)`, effectively expanding the local decision boundary with relevant data.

# 7. Conclusion & Reflections

## 7.1. Conclusion Summary

This paper introduces a novel `test-time self-improvement (TT-SI)` framework for `language-based agents`. It addresses the critical limitations of conventional `LLM fine-tuning`, such as high cost, data redundancy, and guaranteed generalization, by enabling `on-the-fly adaptation`. The `TT-SI` algorithm operates in three phases:
1.  **Self-Awareness**: An `Uncertainty Estimator (H)` intelligently identifies `challenging` or `uncertain samples` during inference, allowing for `targeted intervention`.
2.  **Self-Augmentation**: A `Data Synthesis Function (G)` generates new, `semantically similar training instances` based on these uncertain samples, `expanding the model's local decision boundary`.
3.  **Self-Learning**: `Test-Time Fine-tuning (T)` performs `lightweight, temporary parameter updates` using `LoRA` on these generated samples, and then resets the model's parameters, preventing `catastrophic forgetting`.

    The paper demonstrates through extensive empirical evaluations across `NexusRaven`, `SealTool`, and `API-Bank` benchmarks that `TT-SI` consistently improves `agent performance` during `inference`. It achieves significant `absolute accuracy gains` (e.g., $+5.36$ on average) while using `orders of magnitude less training data` (e.g., `68x` fewer samples than `SFT` on `SealTool`) and outperforming other standard inductive learning baselines. The `Test-Time Distillation (TT-D)` variant further improves performance in more complex scenarios by leveraging a `stronger teacher model` for data generation. The findings highlight `TT-SI` as a promising new paradigm for developing `self-evolving LLM agents` that are more `cost-effective`, `generalizable`, and adaptable to `complex real-world scenarios`.

## 7.2. Limitations & Future Work

The authors openly acknowledge several limitations of `TT-SI` and propose future research directions:

**Limitations:**
1.  **Uncertainty Threshold ($\tau$) Sensitivity:** Identifying uncertain samples requires setting a `threshold`\tau

. While ablation studies show consistent gains across different $\tau$ values, optimal performance is sensitive to this choice. Developing principled methods for autonomously learning this threshold within the uncertainty calibration domain remains an open challenge. 2. Base Model Capacity Boundary: TT-SI is inherently limited by the capacity and pre-trained knowledge of the base model. If the fundamental knowledge required to solve a task is entirely absent (e.g., a completely new, unlearned medical concept), self-improvement alone cannot recover it. In such cases, external knowledge integration (e.g., through retrieval or search mechanisms) would be necessary. This points towards the need for truly self-evolving agents that can acquire new knowledge.

Future Work:

1. Self-Evolving Agents & Continual Learning: A key direction is to move beyond self-improvement (temporary, instance-specific adaptation) to self-evolving agents capable of continual and transferable learning. Currently, LoRA updates in TT-SI are temporary and discarded. Future work should investigate whether improvements from one uncertain sample can transfer to others and how agents can self-assess when such transfer is possible.
2. Adaptive Data Generation: Instead of relying on fixed hyperparameters for $K$ (the number of synthetic examples), future research could explore adaptive data generation where the model itself determines how many synthetic examples are needed for a given uncertain case, optimizing the signal-to-noise ratio and computational cost.
3. Co-evolutionary Setup (Dual Learning): The current framework optimizes only the agent. A co-evolutionary setup, akin to dual learning, where both the agent and the data generator adapt to each other (e.g., through mutual learning), could further enhance performance.
4. Domain Extension: Extending TT-SI to other complex domains like mathematics or medicine could reveal how domain-specific uncertainty and knowledge structures interact with self-improvement mechanisms, offering new insights and applications.

7.3. Personal Insights & Critique

This paper presents a highly insightful and practical approach to enhancing LLM agent capabilities by drawing inspiration from human learning principles. The shift from costly, exhaustive offline fine-tuning to efficient, on-the-fly test-time adaptation is a critical step towards more agile and resource-conscious AI systems.

Inspirations and Broader Applicability:

• Metacognitive AI: The self-awareness component, particularly the uncertainty estimator, is a significant step towards metacognitive AI—systems that can reason about their own knowledge and limitations. This capability could be transformative, enabling AI systems to request clarification, seek additional information, or even defer to human experts more intelligently.
• Edge AI and Resource-Constrained Environments: The efficiency gains (68x fewer samples, significant speed-up) and the ability to work with smaller models make TT-SI highly relevant for edge AI deployments or resource-constrained environments where large-scale fine-tuning is infeasible. This could democratize access to advanced LLM agent capabilities.
• Personalized AI: The instance-specific adaptation opens doors for more personalized AI experiences. Imagine an agent that subtly adapts its behavior or knowledge to a particular user's interaction style or specific domain of expertise, without requiring a full re-training.
• Robustness to Distributional Shift: The test-time adaptation inherently makes the model more robust to distributional shifts, as it can locally correct for discrepancies between its training data and the current test input.

Potential Issues, Unverified Assumptions, and Areas for Improvement:

• Quality of Self-Generated Data: While the UMAP visualization (Figure 5) suggests semantic faithfulness, the quality and diversity of self-generated data (especially for TT-SI where the same model generates its own data) for extremely complex or novel scenarios remain a potential bottleneck. If the base model fundamentally misunderstands a concept, its self-generated examples might perpetuate or even amplify that misunderstanding. The reliance on GPT-5-mini in TT-D highlights this concern, suggesting a need for high-quality teacher signals.

• Prompt Sensitivity for Data Generation: The Data Generation Prompt (Figure 6) is hand-crafted. The effectiveness of $G$ likely depends heavily on the quality and specificity of this prompt. Developing adaptive or meta-learning approaches for prompt engineering itself could be a valuable extension.

• Computational Overhead for Real-Time Applications: Although TT-SI is significantly more efficient than SFT, the 10.24 seconds per uncertain sample (combining H, G, T, inference) might still be too slow for truly real-time, low-latency applications. Optimizing the LoRA update speed (e.g., faster PEFT methods, hardware acceleration) or exploring alternatives like hypernetwork-based adaptation could be crucial. The stated exclusion of I/O operation overheads (checkpoint loading, merging, saving) in the timings is a relevant detail, as these can be substantial in practice.

• Generalizability of Uncertainty Estimator: While the softmax-difference estimator performs well in the presented benchmarks, its generalizability across highly diverse agentic tasks or LLM architectures needs further validation. The optimal $\tau$ might also vary significantly across tasks and models.

• Trade-off between Local Adaptation and Global Coherence: Resetting parameters after each instance prevents catastrophic forgetting, but it also means that instance-specific improvements are not retained or transferred globally. The transition to self-evolving agents would need to address how to selectively consolidate useful knowledge while maintaining adaptability without forgetting.

  Overall, this paper provides a robust and well-motivated framework that pushes the boundaries of LLM agent adaptation. It offers a clear path towards more intelligent, efficient, and human-like learning in AI systems, making it a valuable contribution to the field.