Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely on outcome-based rewards that are only provided at the final answer. This reward sparsity becomes particularly problematic in multi-turn settings, where long trajectories exacerbate two critical issues: (i) advantage collapse, where all rollouts receive identical rewards and provide no useful learning signals, and (ii) lack of fine-grained credit assignment, where dependencies between turns are obscured, especially in long-horizon tasks. In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's probability of producing the correct answer. Unlike prior process-level reward approaches that depend on external reward models or costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense reward trajectories. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines in multi-turn scenarios, achieving higher accuracy and improved sample efficiency.

1. Bibliographic Information

• Title: Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents
• Authors: Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying.
• Affiliations: The authors are from Ant Group, Renmin University of China, and an Individual Author. This mix of industry and academic researchers suggests a focus on both foundational research and practical application.
• Journal/Conference: The paper is available on arXiv, which is a preprint server. This means it has not yet undergone formal peer review for a conference or journal. The arXiv ID 2510.14967v1 and the frequent citation of papers from 2025 are unusual, suggesting the authors may have used placeholder dates for a future submission.
• Abstract: The abstract introduces the problem of training Large Language Model (LLM) agents for multi-turn tasks using Reinforcement Learning (RL). It highlights that standard RL approaches suffer from sparse rewards, where feedback is only given at the final step. This leads to two critical issues in long interactions: advantage collapse (no useful learning signal when all attempts fail or succeed identically) and poor credit assignment (difficulty in attributing success or failure to specific actions). The paper proposes Information Gain-based Policy Optimization (IGPO), a new RL framework. IGPO provides dense, intrinsic rewards at each turn by measuring the marginal increase in the model's probability of generating the correct answer. This turn-level reward is combined with the final outcome-based reward. The authors claim that experiments show IGPO significantly outperforms strong baselines in accuracy and sample efficiency.
• Original Source Link:
  • arXiv Link: https://arxiv.org/abs/2510.14967v1
  • PDF Link: https://arxiv.org/pdf/2510.14967v1.pdf
  • GitHub: https://github.com/GuoqingWang1/IGPO

2. Executive Summary

• Background & Motivation (Why): LLM-based agents are increasingly capable of performing complex tasks by using external tools (like web search) over multiple turns. Training these agents with Reinforcement Learning (RL) is a promising direction. However, the standard approach is to reward the agent only based on the correctness of its final answer. In multi-turn tasks, this reward sparsity is a major bottleneck. If an agent makes multiple search queries before answering, it's hard to know which queries were helpful and which were not. This often leads to "advantage collapse," where all sampled action sequences (rollouts) get the same reward (e.g., zero for all incorrect answers), providing no gradient for the model to learn from. Existing attempts to provide intermediate (step-wise) rewards often rely on costly external models or high-variance estimation techniques, making them unstable or hard to scale.

• Main Contributions / Findings (What): The paper introduces Information Gain-based Policy Optimization (IGPO), a simple and effective framework to address this problem.

  1. Dense, Intrinsic Rewards: IGPO defines a novel turn-level reward based on "information gain." This reward measures how much an action in a given turn increases the model's confidence (i.e., probability) in producing the ground-truth answer. This reward is intrinsic because it's derived directly from the model's own belief state, requiring no external reward model. It's dense because it's calculated at every turn, not just at the end.
  2. Combined Reward Signal: IGPO combines these dense, turn-level information gain rewards with the traditional sparse, outcome-based reward for the final answer. This ensures the agent is guided at each step while still being optimized for the ultimate task objective.
  3. Superior Performance and Efficiency: The authors demonstrate through extensive experiments that IGPO consistently outperforms state-of-the-art baselines (including prompt-based, outcome-reward RL, and other step-reward RL methods) on various question-answering benchmarks. It achieves higher accuracy and significantly improves sample efficiency, meaning it learns faster and with less data. This is particularly beneficial for smaller models, which struggle more with sparse rewards.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • LLM Agents: These are LLMs enhanced with the ability to perform actions in an environment. A common action is "tool use," such as calling a web search API to find up-to-date information, executing code, or querying a database.
  • Reinforcement Learning (RL): A machine learning paradigm where an agent learns to make decisions by performing actions in an environment to maximize a cumulative reward. Key terms include:
    • Policy ($\pi$): The agent's strategy, which maps states to actions. In LLMs, this is the model itself, which generates text (actions).
    • Rollout/Trajectory: A sequence of states, actions, and rewards resulting from the agent's interaction with the environment.
    • Reward ($r$): A numerical signal indicating how good an action or sequence of actions was.
    • Advantage ($A$): A measure of how much better a specific action is compared to the average action in a given state. It's used to guide policy updates.
  • Group Relative Policy Optimization (GRPO): An RL algorithm popular for training LLMs. Instead of learning a separate value function to estimate advantage (like in PPO), GRPO samples a group of rollouts for a given prompt, calculates their rewards, and computes a "relative advantage" for each rollout by normalizing its reward against the mean and standard deviation of the group's rewards. This is simpler and often more stable.
  • Reward Sparsity: A common problem in RL where rewards are only given infrequently (e.g., only at the end of a long game or task). This makes it difficult for the agent to figure out which of its many actions were responsible for the final outcome.
  • Advantage Collapse: The paper's term for a consequence of reward sparsity in the GRPO context. When all rollouts in a group receive the same outcome reward (e.g., all are wrong, F1 score = 0), their normalized advantages all become zero. This means the training step provides no learning signal, wasting computation and data.

• Previous Works:

  • Prompt-based Methods: Early approaches like Chain-of-Thought (CoT) and Retrieval-Augmented Generation (RAG) used clever prompting to elicit tool use and reasoning from LLMs without any extra training. While simple, they often lack robustness and generalization.
  • Outcome-Reward RL Methods: This is the dominant paradigm, represented by methods like GRPO and DeepResearcher. They train the agent using RL with a reward based solely on the final answer's correctness. They perform better than prompting but are hampered by reward sparsity.
  • Process-Reward RL Methods: Recent work has tried to solve the sparsity problem with intermediate rewards.
    • ReasoningRAG uses Monte Carlo simulations to estimate the value of intermediate steps, but this can have high variance.
    • StepSearch and GiGPO use external knowledge or heuristics to judge intermediate steps. This can be costly, introduce bias, or lack generalizability.

• Differentiation: IGPO distinguishes itself from previous process-reward methods by proposing a reward that is:

  • Intrinsic: It is calculated using the policy model's own probabilities, without needing an external reward model or oracle.
  • Ground-Truth-Aware: The reward is tied directly to the ground-truth answer, ensuring the agent is rewarded for actions that genuinely move it closer to the correct solution, preventing "reward hacking."
  • Simple and Stable: It avoids the high variance of Monte Carlo estimation and the complexities of external models, offering a straightforward and reliable supervision signal.

4. Methodology (Core Technology & Implementation)

The core of IGPO is its novel reward design and how it's integrated into a policy optimization framework.

• Task Formulation: The agent interacts with an environment (e.g., a web search engine) over multiple turns. A full interaction, or rollout $o$, is a sequence of turns $(\tau_1, \tau_2, \dots, \tau_T)$.

  • For intermediate turns ($t < T$), each turn $\tau_t$ consists of three parts:

    1. [think]: The model generates its reasoning process.
    2. [tool call]: The model generates a request to an external tool (e.g., a search query).
    3. [tool response]: The environment returns the output of the tool (e.g., search results).

  • The final turn $\tau_T$ consists of a [think] step followed by an [answer] step, which produces the final answer $\hat{a}$.

    The overall training pipeline is visualized in Figure 2.

    该图像是论文中图2的示意图，展示了IGPO的训练流程。上部描述通过计算转折点地面真值概率的变化，获得基于信息增益的即时奖励 $R^{IG}$ 并与结果奖励结合形成优势。下部展示每次rollout包含最多 T-1 个交互回合，每回合包括推理、工具调用及响应，最终输出答案，工具响应部分的损失被屏蔽。

• Motivation: The Problem of Advantage Collapse Standard GRPO assigns a single reward to an entire rollout. In difficult tasks, all rollouts in a group might fail, leading to identical rewards and zero advantage, thus stalling the learning process. Figure 1 from the paper illustrates this problem vividly.

  该图像是一张图表，展示了训练过程中Qwen2.5-7B/3B-Instruct模型在IGPO和GRPO方法下，零优势组比例的变化趋势。

  The plot shows that for GRPO (red and blue lines), a significant fraction (up to 32%) of training steps suffer from zero-advantage groups. In contrast, for IGPO (pink line), this ratio is always zero because its dense, turn-level rewards always provide a learning signal.

• Information Gain-based Turn-Level Reward (The Core Idea) IGPO treats each turn as a step in an information-gathering process. The reward for a turn is defined as the "information gain" it provides about the ground-truth answer $a$.

  1. First, the model's confidence in the ground-truth answer $a$ is calculated given the history of the rollout up to turn $t$, denoted as $o_{i, \le t}$. This is computed as the log-probability of generating the tokens of the ground-truth answer $a = (a_1, \dots, a_L)$, conditioned on the preceding context: $$\pi _ { \boldsymbol { \theta } } ( a \mid q , o _ { i , \le t } ) = \exp \left( \frac { 1 } { L } \sum _ { j = 1 } ^ { L } \log \pi _ { \boldsymbol { \theta } } ( a _ { j } \mid q , o _ { i , \le t } , a _ { < j } ) \right)$$

     • Symbol Explanation:
       • $\pi_{\theta}(a|...)$: The probability of generating the ground-truth answer $a$ under the current policy $\pi_{\theta}$.
       • $q$: The initial query.
       • $o_{i, \le t}$: The history of the i-th rollout up to turn $t$.
       • $a_j$: The j-th token of the ground-truth answer.
       • This is essentially a teacher-forcing calculation where we measure the probability the model would have assigned to the correct answer.

  2. The information gain reward $r_{i,t}$ for turn $t$ is the change in this probability from the previous turn: $$r _ { i , t } = \mathrm { I G } ( a \mid q , o _ { i , t } ) = \pi _ { \theta } ( a \mid q , o _ { i , \le t } ) - \pi _ { \theta } ( a \mid q , o _ { i , \le t - 1 } ) , \qquad 1 \le t < T$$

     • Symbol Explanation:
       • $r_{i,t}$: The immediate reward for turn $t$ of rollout $i$.
       • $\mathrm{IG}(...)$: The Information Gain.
       • A positive reward means the action at turn $t$ made the model more confident about the true answer. A negative reward means the action was misleading.

• Integrating Outcome and Turn-level Rewards IGPO creates a dense reward trajectory $\mathbf{r}_i = (r_{i,1}, \dots, r_{i,T})$ for each rollout.

  • For intermediate turns ($t < T$), the reward $r_{i,t}$ is the information gain reward defined above.
  • For the final answer turn ($t = T$), the reward $r_{i,T}$ is the standard outcome-based reward, which is the F1 score between the predicted answer $\hat{a}$ and the ground-truth answer $a$. $$r_{i,T} = \mathrm{F1}(\hat{a}, a)$$

• Policy Optimization with Turn-Level Advantage With a reward for every turn, IGPO can now compute a turn-level advantage.

  1. Turn-level Advantage Estimation:

     • First, all turn-level rewards from all rollouts in the group are collected, $\mathbf{R} = \{ r_{i,t} \}$, and z-normalized to make them comparable. This gives a raw turn-level advantage $A_{i,t}$. $$A _ { i , t } = \frac { r _ { i , t } - \mathrm { m e a n } ( \mathbf { R } ) } { \mathrm { s t d } ( \mathbf { R } ) }$$
     • Next, to account for long-term consequences, a discounted cumulative advantage $\widetilde{A}_{i,t}$ is computed. This propagates rewards from future turns back to the current turn. $$\widetilde { A } _ { i , t } = \sum _ { k = t } ^ { T } \gamma ^ { k - t } A _ { i , k }$$
       • Symbol Explanation:
         • $\gamma \in (0, 1]$ is the discount factor. A value of $\gamma=1$ (as used in the paper) means all future advantages are weighted equally.

  2. IGPO Objective Function: The policy is optimized using a clipped surrogate objective similar to PPO/GRPO, but using the dense, turn-level advantages $\widetilde{A}_{i,t}$. $$\begin{array} { r l } & { \mathcal { J } _ { \mathrm { I G P O } } ( \theta ) = \mathbb { E } _ { ( q , a ) \sim \mathcal { D } , \{ o _ { i } \} \sim \pi _ { \theta _ { \mathrm { old } } } } \Bigg [ \displaystyle \frac { 1 } { G } \sum _ { i = 1 } ^ { G } \frac { 1 } { \vert o _ { i } \vert } \sum _ { t = 1 } ^ { \vert o _ { i } \vert } \operatorname* { m i n } \left( \frac { \pi _ { \theta } \left( o _ { i , t } \mid \dots \right) } { \pi _ { \theta _ { \mathrm { old } } } \left( o _ { i , t } \mid \dots \right) } \widetilde { A } _ { i , t } , \right. } \\ & { \quad \quad \quad \quad \quad \left. \mathrm { c l i p } \left( \frac { \pi _ { \theta } \left( o _ { i , t } \mid \dots \right) } { \pi _ { \theta _ { \mathrm { old } } } \left( o _ { i , t } \mid \dots \right) } , 1 - \epsilon , 1 + \epsilon \right) \widetilde { A } _ { i , t } \right) - \beta \mathbb { D } _ { \mathrm { K L } } ( \pi _ { \theta } \| \pi _ { \mathrm { r e f } } ) \Bigg ] } \end{array}$$

     • Symbol Explanation:
       • This objective encourages actions that have a positive advantage $\widetilde{A}_{i,t}$ by increasing their probability.
       • The min and clip functions are from PPO, used to prevent excessively large policy updates.
       • $\beta \mathbb{D}_{\mathrm{KL}}$ is a regularization term to keep the updated policy from diverging too far from a reference model $\pi_{\mathrm{ref}}$.
       • Crucially, the advantage $\widetilde{A}_{i,t}$ is applied to every decision token within turn $t$, providing fine-grained credit assignment.

5. Experimental Setup

• Datasets: Experiments were conducted on seven question-answering (QA) datasets requiring search.
  • In-Domain (ID): Datasets similar to the training data.
    • NQ (Natural Questions), TQ (TriviaQA), HotpotQA, 2Wiki (2WikiMultiHopQA).
  • Out-of-Domain (OOD): Datasets with different characteristics to test generalization.
    • MusiQue, Bamboogle, PopQA.
• Evaluation Metrics: The primary metric is word-level F1 score.
  1. Conceptual Definition: The F1 score is the harmonic mean of precision and recall. It provides a balanced measure of a model's performance, especially when dealing with text generation where an answer might be partially correct. A higher F1 score is better.
  2. Mathematical Formula: Given a predicted answer $\hat{a}$ and a ground-truth answer $a$, both treated as bags of words: $$\text{Precision} = \frac{|\text{words}(\hat{a}) \cap \text{words}(a)|}{|\text{words}(\hat{a})|}$$ $$\text{Recall} = \frac{|\text{words}(\hat{a}) \cap \text{words}(a)|}{|\text{words}(a)|}$$ $$\text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2 |\hat{a} \cap a|}{|\hat{a}| + |a|}$$
  3. Symbol Explanation:
     • $|\hat{a} \cap a|$: The number of common words between the predicted and ground-truth answers.
     • $|\hat{a}|$: The total number of words in the predicted answer.
     • $|a|$: The total number of words in the ground-truth answer.
• Baselines: IGPO was compared against a comprehensive set of baselines:
  • Prompt-based: CoT, $CoT+RAG$, Search-o1.
  • Outcome-reward RL-based: Search-r1, R1-searcher, DeepResearcher. These are strong competitors that use sparse rewards.
  • Step-reward RL-based: StepSearch, ReasoningRAG, GiGPO. These are direct competitors that also use intermediate rewards but with different mechanisms.
  • General RL Algorithms: RLOO, PPO, GRPO, $Reinforce++$, GSPO. This comparison validates IGPO's reward design against standard RL optimizers.
• Implementation Details:
  • Backbone Model: Qwen2.5-7B-Instruct.
  • Training Framework: verl.
  • Hyperparameters: Discount factor $\gamma = 1$, 32 prompts per step, 16 rollouts per prompt.

6. Results & Analysis

• Core Results: The main results are presented in Table 1 and Table 2. Below are the transcribed tables.

  Table 1: Main results of IGPO compared with different agentic RL baselines across seven datasets.

  | |

  In-domain
  | | | |
  Out-of-domain
  | | | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | Method | NQ | TQ | HotpotQA | 2Wiki | Musique | Bamboogle | PopQA | Avg. | Prompt-based | CoT | 19.8 | 45.6 | 24.4 | 26.4 | 8.5 | 22.1 | 17.0 | 23.4 | CoT+RAG | 42.0 | 68.9 | 37.1 | 24.4 | 10.0 | 25.4 | 46.9 | 36.4 | Search-o1 | 32.4 | 58.9 | 33.0 | 30.9 | 14.7 | 46.6 | 38.3 | 36.4 | Outcome-reward RL-based | Search-r1-base | 45.4 | 71.9 | 55.9 | 44.6 | 26.7 | 56.5 | 43.2 | 49.2 | Search-r1-instruct | 33.1 | 44.7 | 45.7 | 43.4 | 26.5 | 45.0 | 43.0 | 40.2 | R1-searcher | 35.4 | 73.1 | 44.8 | 59.4 | 22.8 | 64.8 | 42.7 | 49.0 | DeepResearcher | 39.6 | 78.4 | 52.8 | 59.7 | 27.1 | 71.0 | 48.5 | 53.9 | Step-reward RL-based | StepSearch-base | - | - | 49.3 | 45.0 | 32.4 | 57.3 | - | 46.0 | StepSearch-instruct | - | - | 50.2 | 43.1 | 31.2 | 53.4 | - | 44.5 | ReasoningRAG | - | - | 48.9 | 50.4 | 20.6 | 45.5 | 46.2 | 42.3 | GiGPO | 46.4 | 64.7 | 41.6 | 43.6 | 18.9 | 68.9 | 46.1 | 47.2 | IGPO | 46.7 | 80.1 | 57.2 | 68.2 | 31.4 | 74.9 | 52.5 | 58.7

  Table 2: Main results of IGPO compared with different RL baselines across seven datasets.

  | |

  In-domain
  | | | |
  Out-of-domain
  | | | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | Method | NQ | TQ | HotpotQA | 2Wiki | Musique | Bamboogle | PopQA | Avg. | RLOO | 40.7 | 72.5 | 49.6 | 55.0 | 24.8 | 62.2 | 43.1 | 49.7 | PPO | 38.7 | 75.4 | 48.6 | 59.7 | 26.2 | 63.4 | 48.7 | 51.5 | GRPO | 40.3 | 77.0 | 48.9 | 57.7 | 25.0 | 65.1 | 49.6 | 51.9 | Reinforce++ | 34.3 | 67.5 | 45.9 | 54.5 | 23.7 | 61.2 | 44.3 | 47.3 | GSPO | 41.5 | 77.7 | 46.3 | 60.1 | 25.4 | 67.6 | 45.4 | 52.0 | IGPO | 46.7 | 80.1 | 57.2 | 68.2 | 31.4 | 74.9 | 52.5 | 58.7
  • Analysis:
    • IGPO is State-of-the-Art: IGPO achieves the highest average score (58.7), a substantial +4.8 points improvement over the best prior method, DeepResearcher (53.9). It shows consistent gains across almost all datasets, both in-domain and out-of-domain.
    • Effectiveness of Dense Rewards: IGPO outperforms all other RL algorithms (PPO, GRPO, etc.), demonstrating that its unique reward structure is more effective than standard optimizers that rely on sparse outcome rewards.

• Ablation Study: This study dissects the contribution of each reward component in IGPO.

  Table 3: Ablation results of IGPO on Qwen2.5-3B/7B-Instruct with different reward designs.

  | |

  In-domain
  | | | |
  Out-of-domain
  | | | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | Method | NQ | TQ | HotpotQA | 2Wiki | Musique | Bamboogle | PopQA | Avg. | Qwen2.5-3B-Instruct | IGPO (w/ F1) | 31.0 | 55.6 | 27.5 | 29.4 | 12.1 | 35.7 | 34.9 | 32.3 | IGPO (w/ IG) | 29.1 | 53.6 | 27.9 | 36.5 | 17.5 | 44.7 | 31.3 | 34.4 | IGPO (w/ F1+IG) | 40.5 | 69.4 | 46.8 | 48.2 | 23.1 | 57.9 | 47.4 | 47.6 | Qwen2.5-7B-Instruct | IGPO (w/ F1) | 40.3 | 77.0 | 48.9 | 57.7 | 25.0 | 65.1 | 49.6 | 51.9 | IGPO (w/ IG) | 37.5 | 75.0 | 51.0 | 61.0 | 28.6 | 69.6 | 47.1 | 52.8 | IGPO (w/ F1+IG) | 46.7 | 80.1 | 57.2 | 68.2 | 31.4 | 74.9 | 52.5 | 58.7
  • Analysis:

    • IGPO (w/ F1) is equivalent to standard GRPO (outcome-reward only).

    • IGPO (w/ IG) uses only the information gain reward.

    • IGPO (w/ F1+IG) is the full model.

    • Synergy is Key: The full model ($w/ F1+IG$) dramatically outperforms using either reward alone. This proves that the dense, intrinsic guidance from Information Gain (IG) and the final task alignment from the outcome reward (F1) are complementary and essential.

    • Robustness of IG Reward: Using only the IG reward performs comparably to or even better than using only the F1 reward (GRPO). This shows the information gain signal is stable and does not lead to "reward hacking," a common issue with intrinsic rewards.

    • Benefit for Smaller Models: The improvement is much more significant for the smaller 3B model (+15.3 points) than for the 7B model (+6.8 points). This confirms that dense rewards are especially crucial for weaker models that suffer more from advantage collapse.

      The training curves in Figure 3 further support these findings, showing that the full IGPO model ($w/ F1+IG$) learns faster and achieves a higher final performance across all datasets.

      该图像是图表，展示了Qwen2.5-7B-Instruct在不同奖励设计下的训练曲线。图中包含NQ、TQ、HotpotQA、2Wiki、Musique、Bamboogle和PopQA七个基准的F1得分随训练步数变化的趋势，比较了IGPO方法中不同奖励组合的表现。

• In-Depth Analysis

  • Ground-truth Entropy Reduction: Figure 4 measures how much the model's uncertainty about the ground-truth answer decreases during a rollout.

    该图像是图表，展示了IGPO和GRPO在训练过程中各训练步数对应的F1分数变化趋势，其中IGPO表现出更显著的性能提升。

    IGPO achieves a much larger entropy reduction than GRPO. This means that the intermediate steps in an IGPO-trained agent are more effective at gathering information and "homing in" on the correct answer, even before generating it.

  • Token Efficiency: Figure 5 plots performance against the number of tokens used for gradient updates.

    该图像是一个折线图，展示了图5中基于令牌数（百万）对比IGPO与GRPO在F1得分上的表现，显示IGPO在训练中表现出更高的准确率和效率。

    IGPO (red line) achieves higher performance with fewer training tokens than GRPO (blue line). This demonstrates superior sample efficiency, a critical advantage when training data is expensive to generate. The dense rewards provide a more potent learning signal for every token processed.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully identifies reward sparsity and advantage collapse as key obstacles in training multi-turn LLM agents. It proposes IGPO, a simple yet powerful solution that provides dense, intrinsic, and ground-truth-aware rewards based on information gain. This method avoids the pitfalls of prior process-reward designs (reliance on external models, high variance) and demonstrably leads to higher accuracy, better generalization, and improved sample efficiency, especially for smaller models.

• Limitations & Future Work: The authors acknowledge a key limitation: IGPO requires access to ground-truth answers during training to calculate the information gain. This makes it unsuitable for open-ended tasks or domains where correct answers are not available. They plan to extend the work to scenarios without explicit supervision in the future.

• Personal Insights & Critique:

  • Novelty and Elegance: The core idea of using the model's own change in belief about the ground truth as a reward signal is highly elegant. It's a clean, self-contained mechanism that directly addresses the credit assignment problem without external dependencies.
  • Practical Implications: The improved sample efficiency is a significant practical benefit, as generating high-quality agent trajectories for RL training is computationally expensive. The strong performance on smaller models suggests this technique could be vital for creating capable, cost-effective agents.
  • Critique of the "Ground-Truth" Dependency: The reliance on ground-truth answers during training is a major constraint. In real-world applications (e.g., a research agent exploring a novel topic), there is no pre-defined "correct answer." Future work could explore using a stronger, "teacher" model's output as a proxy for the ground truth, or developing self-consistency metrics to generate a similar information-gain signal without a known answer.
  • Potential for Generalization: While demonstrated on search-based QA, the principle of rewarding information gain could be applied to other multi-step reasoning tasks, such as mathematical problem solving or code generation, where intermediate steps are crucial for reaching a correct final solution. The key would be defining an appropriate proxy for "ground truth" at each step.