1. Bibliographic Information

1.1. Title

The central topic of the paper is "Learning from Synthetic Data Improves Multi-hop Reasoning".

1.2. Authors

The authors are anonymous, as indicated by "Anonymous authors" and "Paper under double-blind review". This is a common practice in conference submissions to ensure unbiased review.

1.3. Journal/Conference

The paper is published at OpenReview, which is a platform primarily used for managing double-blind peer review processes for academic conferences, particularly in machine learning (e.g., ICLR, NeurIPS). The publication date suggests it is for an upcoming conference in late 2025. This venue is highly influential in the machine learning and artificial intelligence community.

1.4. Publication Year

The paper was published at (UTC): 2025-10-08T00:00:00.000Z, which means the publication year is 2025.

1.5. Abstract

The abstract highlights that Reinforcement Learning (RL) can enhance the reasoning abilities of Large Language Models (LLMs) in various tasks, including multi-hop reasoning. However, RL fine-tuning typically demands extensive, high-quality, and verifiable data, which is usually sourced from expensive human annotations or slow, costly LLM-as-verifier loops. This paper explores the use of synthetic datasets for RL fine-tuning in multi-hop reasoning. The key discovery is that LLMs fine-tuned on synthetic data, even if it contains only fictional knowledge, significantly outperform base models on real-world question-answering benchmarks. By analyzing performance across question difficulty, the authors conclude that synthetic data teaches LLMs to compose knowledge, a fundamental and generalizable reasoning skill. The work thus underscores the effectiveness of synthetic reasoning datasets in boosting LLM reasoning.

1.6. Original Source Link

Official Source Link: https://openreview.net/forum?id=38nYZ5QBui PDF Link: https://openreview.net/pdf?id=38nYZ5QBui This paper is currently under double-blind review, as indicated by the "Anonymous authors" and "Paper under double-blind review" notations.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the scarcity and cost of high-quality, verifiable training data for Reinforcement Learning (RL) fine-tuning of Large Language Models (LLMs) in complex reasoning tasks, particularly multi-hop reasoning.

This problem is crucial in the current field because:

1. RL fine-tuning has proven highly effective in boosting LLM capabilities in domains like math, coding, and multi-hop reasoning.

2. Existing data sources have significant limitations: human-annotated datasets are small and expensive, while LLM-as-verifier loops are slow and costly.

3. LLMs trained on internet-scale data are increasingly prone to data leakage and memorization on benchmarks, making reasoning improvements unreliable.

4. The pace of LLM training is outstripping the availability of high-quality human-written text for reasoning.

   The paper's entry point and innovative idea is to investigate whether LLMs can develop general reasoning capabilities, specifically knowledge composition, solely from synthetic data, without relying on real-world knowledge or expensive human/LLM verification processes. The hypothesis is that reasoning skills might be transferable even from fictional domains.

2.2. Main Contributions / Findings

The primary contributions of the paper are:

1. Scalable and Cost-Effective Data Source: Proposing synthetic multi-hop datasets as a scalable, cost-effective source of reasoning training data that provides infinite, verifiable training signals. This demonstrates that multi-hop reasoning capabilities can be effectively learned from synthetic data, even without factual overlap between training and evaluation domains.

2. Empirical Evidence for Generalization: Providing strong empirical evidence that synthetic reasoning training generalizes to real-world scenarios. The paper shows performance gains across various LLM families and sizes, establishing the practical viability of synthetic data for enhancing reasoning.

3. Analysis of Reasoning Transfer across Difficulty: Studying the transfer of reasoning skills to both synthetic and real-world tasks at different question difficulty levels. The findings indicate that improvements on more complex synthetic tasks consistently lead to enhanced performance on increasingly challenging real-world tasks.

   The key conclusions and findings reached by the paper are:

• LLMs fine-tuned on synthetic data (like PhantomWiki and GSM-∞), which contain only fictional knowledge, perform significantly better on popular real-world question-answering benchmarks (HotpotQA, 2WikiMultihopQA, MuSiQue).

• The performance gains are consistent across different LLM models (Qwen3-0.6B, Qwen3-1.7B, Qwen2.5-1.5B-Instruct, Phi-4-mini-reasoning) and sizes (0.6B to 4B parameters).

• Synthetic data teaches LLMs to compose knowledge – the fundamental ability to integrate information across multiple inferential steps. This skill is shown to be generalizable and transferable across domains, independent of domain-specific factual knowledge.

• Models do not overfit to the synthetic data; performance on real-world benchmarks continues to improve with more training steps/samples on synthetic data.

• Training on synthetic data improves the LLM's ability to generate reasoning traces that include a higher proportion of correct intermediate answers, especially for more challenging questions.

  These findings address the challenge of data scarcity and cost by offering a viable alternative for improving LLM reasoning, potentially accelerating the development of more capable and robust LLMs.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully grasp this paper, a beginner needs to understand several core concepts:

• Large Language Models (LLMs): These are advanced artificial intelligence models designed to understand and generate human-like text. They are typically trained on vast amounts of text data from the internet, learning patterns, grammar, and factual information. Examples include GPT-3/4, Llama, Qwen, etc.
• Reasoning: In the context of LLMs, reasoning refers to the model's ability to process information, draw inferences, and solve problems that require more than simple recall of facts. This often involves understanding relationships between pieces of information, performing calculations, or following logical steps.
• Multi-hop Reasoning: This is a specific type of reasoning where a model needs to combine information from multiple distinct facts or steps to arrive at an answer. It's not enough to find a single piece of information; the model must "hop" between several pieces of knowledge and integrate them logically. For example, to answer "What is the capital of the country where the Eiffel Tower is located?", an LLM first needs to know the Eiffel Tower is in France, and then know the capital of France.
• Reinforcement Learning (RL): A paradigm of machine learning where an intelligent agent learns to make decisions by interacting with an environment. The agent performs actions, receives rewards (or penalties) for those actions, and learns to choose actions that maximize cumulative reward. Unlike supervised learning, RL doesn't require explicit correct answer labels for every input; instead, it learns from feedback on its actions.
• Fine-tuning: The process of taking a pre-trained LLM (which has learned general language patterns on a massive dataset) and further training it on a smaller, specific dataset to adapt it to a particular task or domain. This typically involves adjusting the model's parameters slightly to improve its performance on the new task.
• Reinforcement Learning from Human Feedback (RLHF): A popular fine-tuning technique for LLMs that uses human preferences as a reward signal. Instead of humans directly labeling correct answers, they rank or provide feedback on different LLM outputs. A reward model is then trained on these human preferences, and this reward model guides the RL agent (the LLM) to generate responses that are preferred by humans.
• Supervised Fine-Tuning (SFT): A simpler fine-tuning method where an LLM is trained on a dataset of input-output pairs, with the objective of predicting the next token in a sequence. It's essentially standard supervised learning applied to LLMs for specific tasks, often used as an initial step before RLHF.
• Chain-of-Thought (CoT): A prompting technique that encourages LLMs to generate intermediate reasoning steps before arriving at a final answer. This mimics human problem-solving and often significantly improves LLM performance on complex reasoning tasks by making the reasoning process explicit.
• Synthetic Data: Data that is not collected from real-world events but is artificially generated. In the context of LLMs, synthetic data can include generated questions, answers, or even entire fictional knowledge bases, often created programmatically or by other LLMs. Its advantage is that it can be created in vast quantities and tailored to specific properties.
• Overfitting: A phenomenon in machine learning where a model learns the training data too well, capturing noise and specific details that are not generalizable to new, unseen data. An overfit model performs exceptionally on training data but poorly on test data.

3.2. Previous Works

The paper extensively references prior work, categorizing them into Reasoning in Large Language Models, Training and Fine-tuning Large Reasoning Models, and Leveraging Synthetic Data.

3.2.1. Reasoning in Large Language Models

• Evaluation Benchmarks: LLMs are typically evaluated on their reasoning skills using various benchmarks. These include:
  • Technical/Abstract domains: Mathematics (MATH (Hendrycks et al., 2021), MAA), algorithms, coding (Cobbe et al., 2021), puzzle-solving (Jain et al., 2025; Chollet et al., 2025).
  • Knowledge-intensive domains: Sciences, law (Rein et al., 2024; Sawada et al., 2023).
  • General common sense: Abductive, counterfactual reasoning (Talmor et al., 2019; Zhao et al., 2023; Bhagavatula et al., 2020; Wu et al., 2025a; Hüyük et al., 2025).
  • Natural language question-answering (NLQA): HotpotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), Tang & Yang, 2024; Qi et al., 2021.
  • Interaction with environment: Planning and tool use (Patil et al., 2024; Zhuang et al., 2023; Yao et al., 2024).
• Many of these benchmarks, especially multi-hop reasoning tasks, require LLMs to break down questions into intermediate subproblems and compose them, which is considered a hallmark of effective reasoning (Gong et al., 2025; Xie et al., 2025; Gandhi et al., 2025).

3.2.2. Training and Fine-tuning Large Reasoning Models

• Supervised Fine-tuning (SFT): The simplest approach (Lambert et al., 2025). Variants include instruction fine-tuning (Chung et al., 2024) and Chain-of-Thought (CoT) modeling (Xiang et al., 2025; Zelikman et al., 2022; Hao et al., 2025; Yao et al., 2023; Chen et al., 2023; Wan et al., 2025), which encourage more detailed thinking processes.
• Reinforcement Learning from Human Feedback (RLHF): A more complex, RL-based framework using human preferences (Christiano et al., 2017; Ouyang et al., 2022). Algorithms include Proximal Policy Optimization (PPO) (Schulman et al., 2017), GRPO (Shao et al., 2024), and Direct Preference Optimization (DPO) (Rafailov et al., 2023).
  • Proximal Policy Optimization (PPO): A widely used RL algorithm known for its stability and efficiency. It optimizes a stochastic policy by taking small steps to avoid performance collapse, using a clipping mechanism and a KL divergence penalty to keep the new policy close to the old one. The core objective of PPO involves maximizing a clipped surrogate objective function. For an action $a_t$ taken at state $s_t$ with advantage estimate $\hat{A}_t$, and ratio of new to old policy probabilities $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$, the clipped surrogate objective is: $ L^{CLIP}(\theta) = \mathbb{E}_t\left[\min(r_t(\theta)\hat{A}_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)\right] $ Where $\epsilon$ is a hyperparameter, and $\hat{A}_t$ is the advantage function estimate, which measures how much better an action is than the average action.
• Reinforcement Learning with Verifiable Rewards (RLVR): A variant of RLHF where the human reward model is replaced by a procedural verification function, often applicable in domains with objective ground-truth answers like math or coding (Lambert et al., 2025; Guo et al., 2025a; Abdin et al., 2025). The paper notes that the mechanisms for eliciting novel reasoning patterns through RLVR remain an open research area (Wen et al., 2025; Yue et al., 2025; Shao et al., 2025; Zhao et al., 2025).

3.2.3. Leveraging Synthetic Data

• Challenges in Fine-tuning: Abstract multi-hop reasoning skills are hard to isolate, confounded by other skills or memorization (Wu et al., 2025b; Xie et al., 2024; Yu et al., 2024). LLM benchmarks are prone to test set leakage (Gong et al., 2025; Wu et al., 2025b).
• Synthetic Data as a Solution: Synthetic datasets alleviate these issues by isolating specific reasoning aspects, providing unlimited examples with verifiable rewards.
• Programmatic Generation: Many synthetic reasoning benchmarks are programmatically generated, especially in mathematics (Mirzadeh et al., 2025; Zhou et al., 2025; Wu et al., 2025b), logic puzzles (Xie et al., 2024; Shojaee et al., 2025; Stojanovski et al., 2025), and some NLQA forms (Gong et al., 2025; Guo et al., 2025b; Sinha et al., 2019).
• LLM-generated Examples: Other methods use LLMs to create additional examples and reasoning traces to augment existing datasets (Yang et al., 2025; Goldie et al., 2025; Huang et al., 2025; Saad-Falcon et al., 2024; Li et al., 2025).
• The effectiveness and applicability of synthetic data to real-world reasoning skills, especially beyond RLVR domains, remain underexplored (Yu et al., 2024; Mizrahi et al., 2025; Abbe et al., 2024b;a; Stojanovski et al., 2025), which this paper aims to address.

3.3. Technological Evolution

The field of LLM reasoning has evolved from basic SFT to more sophisticated RL-based fine-tuning techniques like RLHF and RLVR. Initially, LLMs relied heavily on vast real-world datasets for pre-training and fine-tuning. However, as models scale and data quality becomes a bottleneck, the focus has shifted towards synthetic data generation to address issues like data scarcity, cost, and test set leakage. This paper fits into this evolution by exploring the frontier of using purely fictional synthetic data to impart generalized reasoning skills, moving beyond domain-specific RLVR applications. It tests the hypothesis that fundamental reasoning skills like knowledge composition can be learned abstractly, independently of factual knowledge.

3.4. Differentiation Analysis

Compared to the main methods in related work, this paper's core innovations and differences are:

• Fictional Synthetic Data for Generalization: Unlike RLVR, which often focuses on systematically verifiable domains (like math and coding) where synthetic data is used to generate problems within that domain, this work uses synthetic data from fictional domains (PhantomWiki with fictional knowledge, GSM-∞ math problems) to impart reasoning skills that generalize to real-world natural language question-answering tasks. This explicitly rules out direct factual overlap or memorization as the source of improvement.
• Focus on Knowledge Composition: The paper specifically investigates knowledge composition as a fundamental and generalizable reasoning skill, demonstrating its transferability. Many prior works focus on performance on specific benchmarks, while this paper dissects what is learned (composition) and how it transfers.
• Scalable and Verifiable Training Signals: It highlights synthetic datasets as a solution to the data scarcity and cost issues associated with human-annotated data and LLM-as-verifier loops, providing an infinite source of verifiable training signals. This is a practical advancement for LLM fine-tuning.
• Demonstration of Non-Overfitting with Synthetic Data: The paper empirically shows that scaling synthetic training data does not cause overfitting, and performance on real-world benchmarks continues to improve with more synthetic training steps, suggesting robust generalization. This addresses a common concern with synthetic data.

4. Methodology

The methodology section details how the authors conducted their experiments to study the transfer performance from synthetic to real-world datasets. This involved RL fine-tuning various LLMs on carefully selected synthetic datasets and evaluating them on real-world multi-hop reasoning benchmarks.

4.1. Principles

The core idea behind the method is that fundamental reasoning skills, particularly knowledge composition (the ability to chain logical inferences), can be learned from structured, synthetically generated data, even if that data contains no real-world facts. The theoretical basis is that reasoning is a meta-skill independent of domain-specific factual knowledge. By training LLMs with Reinforcement Learning on synthetic datasets that require multi-step reasoning in fictional contexts, the models should learn the process of reasoning, which can then generalize to tasks in real-world knowledge domains. The RL framework provides a mechanism for the model to learn from positive rewards for correct reasoning paths.

4.2. Core Methodology In-depth (Layer by Layer)

The methodology consists of several key components: LLM selection, synthetic training dataset curation, RL fine-tuning algorithm application, and prompt/reward design.

4.2.1. LLM Selection

The study uses LLMs of varying sizes to ensure the generality of the findings:

• Qwen3-0.6B

• Qwen3-1.7B

• Qwen2.5-1.5B-Instruct (Qwen Team, 2024, 2025)

• Phi-4-mini-reasoning (Abdin et al., 2025)

  Each model is fine-tuned for 1 epoch on a random shuffle of the selected synthetic training datasets. This required 4 NVIDIA H100 GPUs for approximately 1 day of training for larger models and longer CoT generations.

4.2.2. Synthetic Training Datasets

Two types of synthetic datasets are chosen for their scalable verification and varying difficulty:

4.2.2.1. GSM-∞ (GSM-Infinity)

• Source: (Zhou et al., 2025)
• Description: This dataset generalizes the GSM8K benchmark (grade school math word problems) to an infinitely extensible version. It constructs random computation graphs to represent ground-truth solutions, which can be augmented with distractor facts. These graphs are then converted into word problems using natural language templates across pre-defined themes (e.g., zoo, teacher-school, movie).
• Relevance: Represents math-based synthetic reasoning and helps investigate generalizability from arithmetic skills to knowledge-intensive real-world tasks.
• Configuration for experiments:
  • Difficulty level: "medium"
  • Number of arithmetic operations: 2 to 20 (defines reasoning complexity).
  • Context length: zero (ensures problems contain only necessary information, simplifying hop identification).
• Data Size: Approximately 600 questions per difficulty level across 19 levels. Half from zoo theme, one-quarter from teacher-school, and one-quarter from movie theme, equally split between forward (addition/multiplication) and reverse (subtraction/division) modes.
• Total Samples: $\approx 12.5 \mathrm{~K}$ samples, with $10 \mathrm{~K}$ for training and the remainder for validation.

4.2.2.2. PhantomWiki

• Source: (Gong et al., 2025)
• Description: This dataset generates on-demand synthetic datasets comprising natural language document corpora and question-answer pairs. It is designed to evaluate LLMs on multi-step and multi-branch reasoning and retrieval. Each dataset describes a random universe of fictional individuals, their attributes, and relationships in Wikipedia-like documents. A context-free grammar and logic programming-based algorithm generate multi-hop reasoning questions (e.g., "Who is the nephew of the friend of the person who likes birdwatching?"). PhantomWiki questions can have multiple answers and require retrieval and knowledge composition across multiple documents.
• Relevance: Directly tests retrieval and knowledge composition skills in a fictional natural language context.
• Configuration for experiments:
  • Relations: Only "easy" relations (immediate family, friends) to simplify conceptual hops.
  • Question type filter: Excludes aggregation questions ("How many...") to focus purely on multi-hop questions ("Who is the of...?", "What is the of...?").
  • Difficulty definition: Answering a question of difficulty $d$ requires hopping through documents of exactly $d$ individuals.
  • Generation: 34 universes of 25 individuals using 100 random seeds. Context-free grammar recursion depth set to 20 for varying difficulties (1 to 9 hops). Ground-truth answers are obtained via PhantomWiki's logic program.
• Total Samples: 330 questions per universe.
• Training/Validation Split: 31 universes for $10 \mathrm{~K}$ training samples, 3 universes ($\approx 1 \mathrm{~K}$ samples) for validation.

4.2.3. RL Fine-tuning for Reasoning

The paper uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as the primary RL fine-tuning algorithm.

4.2.3.1. Group Relative Policy Optimization (GRPO)

• Description: GRPO is a variant of Proximal Policy Optimization (PPO) designed to reduce memory and compute requirements. It replaces the value model-based advantage estimation of PPO with an estimation based on a group of online completions for each prompt.

• Objective Function: Given a question $q$ sampled from a distribution over question set P(Q), GRPO samples a group of $G$ output completions $\{o_{1}, \ldots o_{G}\}$ from the old LLM $\pi$ with parameters $\theta_{\text{old}}$. Each completion is assigned a scalar reward value $\{R_{1}, \ldots, R_{G}\}$. The advantage $\widehat{A}_{i}$ of each completion is estimated by normalizing with respect to the average reward as a baseline. The final objective is:

  $$\begin{aligned} & \mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{[q \sim P(Q),\left\{o_{i}\right\}_{i=1}^{G} \sim \pi_{\theta_{\mathrm{old}}}(O \mid q)]} \\ & {\left[\frac{1}{G} \sum_{i=1}^{G} \frac{1}{\left|o_{i}\right|} \sum_{t=1}^{\left|o_{i}\right|}\left\{\min \left[r_{i, t} \hat{A}_{i, t}, \operatorname{clip}\left(r_{i, t}, 1-\varepsilon, 1+\varepsilon\right) \hat{A}_{i, t}\right]-\beta \mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta} \| \pi_{\mathrm{ref}}\right]\right\}\right] } \\ & \text{where } \widehat{A}_{i, t}=\frac{R_{i}-\operatorname{mean}\left(R_{1}, \ldots, R_{G}\right)}{\operatorname{stdev}\left(R_{1}, \ldots, R_{G}\right)} . \end{aligned}$$

  • Symbol Explanation:
    • $\mathcal{J}_{\mathrm{GRPO}}(\theta)$: The GRPO objective function being maximized with respect to the LLM's current policy parameters $\theta$.
    • $\mathbb{E}_{[\ldots]}$: Expectation taken over questions $q$ sampled from the distribution P(Q) and over groups of output completions $\{o_i\}_{i=1}^G$ generated by the old policy $\pi_{\theta_{\text{old}}}(O \mid q)$.
    • $G$: The size of the group of output completions sampled for each question.
    • $o_i$: The $i$-th output completion (sequence of tokens) generated by the LLM.
    • $|o_i|$: The length of the $i$-th output completion.
    • $t$: An index iterating over tokens within an output completion $o_i$.
    • $r_{i,t}$: The relative weight or importance ratio for token $o_{i,t}$. It's calculated as the ratio of the probability of token $o_{i,t}$ under the current policy $\pi_\theta$ to its probability under the old policy $\pi_{\theta_{\text{old}}}$, given the question $q$ and previous tokens $o_{i,<t}$. $r_{i,t} = \frac{\pi_{\theta}(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}$
    • $\widehat{A}_{i,t}$: The advantage estimate for the $i$-th completion at token $t$. In GRPO, this is simplified to be constant across tokens for a given completion $i$, calculated as the normalized reward of completion $i$.
    • $R_i$: The scalar reward value assigned to the $i$-th output completion.
    • $\operatorname{mean}(R_1, \ldots, R_G)$: The average reward across all $G$ completions in the group.
    • $\operatorname{stdev}(R_1, \ldots, R_G)$: The standard deviation of rewards across all $G$ completions in the group. This normalization helps stabilize training.
    • $\operatorname{clip}(x, \text{lower}, \text{upper})$: A function that clips the value $x$ to be within the range $[\text{lower}, \text{upper}]$. In the objective, it limits the importance ratio to prevent large policy updates that could destabilize training, similar to PPO.
    • $\varepsilon$: A hyperparameter that defines the clipping range for the importance ratio.
    • $\beta$: A hyperparameter controlling the strength of the KL divergence penalty.
    • $\mathbb{D}_{\mathrm{KL}}[\pi_{\theta} \| \pi_{\mathrm{ref}}]$: The Kullback-Leibler (KL) divergence between the current policy $\pi_\theta$ and a reference policy $\pi_{\mathrm{ref}}$ (usually the model's initialization). This penalty encourages the new policy not to deviate too much from the original policy, maintaining stability and avoiding catastrophic forgetting.

• Implementation: The authors use the GRPOTrainer implementation from the open-source Hugging Face TRL library. For their experiments, the KL-divergence penalty hyperparameter $\beta$ is set to 0.

4.2.4. Prompt and Reward Design

The LLMs are trained to perform in-context reasoning and retrieval, meaning all relevant context is provided in the prompt.

• Prompt Structure:

  1. Evidence: For GSM-∞, this is the problem statement. For PhantomWiki, it includes documents for all 25 individuals in the generated universe.
  2. Instruction: Guides the LLM to output the final answer within $<answer>...</answer>$ tags, a standard format for DeepSeek-R1 and Qwen3 models.
  3. CoT Examples: To further guide the output format and reasoning process:
     • GSM-∞: 3 automatically generated ground-truth CoT solutions from the training set.
     • PhantomWiki: 11 CoT examples originally curated by Gong et al. (2025).
  4. Question: The actual question posed to the LLM. The full prompts are detailed in Appendix C.

• Reward Model:

  • Model generations are parsed using regular expressions to extract the last $<answer>...</answer>$ tag.

  • GSM-∞: A binary reward (1 or 0) is given based on whether the correct numeric value is produced.

  • PhantomWiki: An F1 score (between 0 and 1) is used as the reward, as questions can have multiple correct answers.

    The following is an example of a PhantomWiki prompt template (full prompt with many examples is in Appendix C.1):

You are given the following evidence:
(BEGIN EVIDENCE)
{{evidence}}
(END EVIDENCE)

You will be provided a question. Your response must end with the final answer enclosed in tags: <answer>FINAL_ANSWER</answer>

Here, FINAL_ANSWER must be one of the following:

- a name (if there is only one correct answer);
- a list of names separated by ',' (if there are multiple correct answers); or
- numbers separated by ',' (if the answer is numerical); or
- empty string (if there is no answer).

  Here are some examples:
(START OF EXAMPLES)
Example 1:
Question: Who is the sister of Aida Wang?
Answer: Based on the evidence, the sisters of Aida Wang are Barabara Beltran, Vicki Hackworth. <answer>Barabara Beltran, Vicki Hackworth</answer>.

... (many more examples) ...

(END OF EXAMPLES)

Question: {{question}}
Answer: """

An example GSM-∞ prompt template (full prompt with examples is in Appendix C.2):

You are given the following problem:
(BEGIN PROBLEM)
{{problem}}
(END PROBLEM)
You will be provided a question on the above problem. Your
    response must end with the final answer enclosed in tags: <
    answer>FINAL_ANSWER</answer>

Here, FINAL_ANSWER must be a number.
Here are some examples:
(START OF EXAMPLES)
Example 1:
Question: What is the total number of adult animals in Maple Creek ?
Answer: Define adult wolf in Maple Creek as r; so r = 2. Define total number of adult animals in Maple Creek as p; so p = r = 2. <answer>2</answer>.

... (many more examples) ...

(END OF EXAMPLES)

Question: {{question}}
Answer:

5. Experimental Setup

5.1. Datasets

The experiments utilize a combination of synthetic datasets for RL fine-tuning and real-world datasets for evaluation.

5.1.1. Synthetic Training Datasets

• GSM-∞ (Zhou et al., 2025): An infinitely extensible dataset for grade school math word problems. It features a random computation graph converted to word problems via templates, allowing for varying difficulty based on the number of arithmetic operations. The problems are knowledge-free and focus on arithmetic reasoning.
  • Data Sample Example: (from Appendix C.2) Question: What is the total number of adult animals in Maple Creek ? $Answer: Define adult wolf in Maple Creek as r; so r = 2. Define total number of adult animals in Maple Creek as p; so p = r = 2. <answer>2</answer>.$
• PhantomWiki (Gong et al., 2025): Generates on-demand synthetic datasets of fictional document corpora and question-answer pairs for multi-step and multi-branch reasoning and retrieval. It creates universes of fictional individuals and their relationships, then generates multi-hop questions using context-free grammar and logic programming. It specifically tests knowledge composition and retrieval in a natural language context.
  • Data Sample Example: (from Appendix C.1) Question: Who is the sister of Aida Wang? $Answer: Based on the evidence, the sisters of Aida Wang are Barabara Beltran, Vicki Hackworth. <answer>Barabara Beltran, Vicki Hackworth</answer>.$ The evidence would be a set of fictional Wikipedia-like documents describing Aida Wang, Barabara Beltran, Vicki Hackworth, and their relationships.

These synthetic datasets were chosen because they provide scalable verification (ground-truth answers are programmatically generated) and questions of varying difficulty, which are crucial for effective RL fine-tuning. They allow isolating multi-hop reasoning skills without relying on real-world factual knowledge, thus enabling the study of reasoning transfer.

5.1.2. Real-world Evaluation Datasets

The models are evaluated on 500 randomly subsampled questions from the test sets of the following three in-context question answering datasets, which measure multi-hop reasoning capabilities. These were chosen to represent popular, established multi-hop NLQA benchmarks.

• HotpotQA (Yang et al., 2018): A multi-hop question answering dataset with over 100,000 questions, requiring information from two Wikipedia paragraphs. It features a consistent two-hop reasoning structure.
  • Domain: Real-world factual knowledge (Wikipedia).
• 2WikiMultihopQA (Ho et al., 2020): A more recent two-hop dataset with over 190,000 questions, categorized into compositional, inference, comparison, and bridge-comparison. Questions are grounded in Wikidata's knowledge graph, following specific two-hop paths between entities.
  • Domain: Real-world factual knowledge (Wikidata).
• MuSiQue (Trivedi et al., 2022): Evaluates compositional reasoning with 2-4 hop questions created by bridging single-hop questions. Requires joining information from multiple separate paragraphs. The MuSiQue-Answerable split is used to ensure all questions can be answered from the provided context.
  • Domain: Real-world factual knowledge.

5.2. Evaluation Metrics

The paper primarily uses F1 score and binary correctness (accuracy) for evaluation and reward.

5.2.1. F1 Score

• Conceptual Definition: The F1 score is a measure of a model's accuracy on a dataset, often used in information retrieval and natural language processing. It is the harmonic mean of precision and recall, providing a single score that balances both. Precision measures how many of the selected items are relevant, while recall measures how many relevant items are selected. An F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0. It is particularly useful when dealing with imbalanced classes or when both false positives and false negatives are important.
• Mathematical Formula: $ \mathrm{F1} = 2 \times \frac{\mathrm{precision} \times \mathrm{recall}}{\mathrm{precision} + \mathrm{recall}} $ where $ \mathrm{precision} = \frac{\mathrm{true_positives}}{\mathrm{true_positives} + \mathrm{false_positives}} $ and $ \mathrm{recall} = \frac{\mathrm{true_positives}}{\mathrm{true_positives} + \mathrm{false_negatives}} $
• Symbol Explanation:
  • $\mathrm{true\_positives}$: The number of items correctly identified as positive (e.g., correct answers identified by the model).
  • $\mathrm{false\_positives}$: The number of items incorrectly identified as positive (e.g., incorrect answers given by the model that were not ground truth).
  • $\mathrm{false\_negatives}$: The number of items that were actually positive but were incorrectly identified as negative (e.g., ground truth answers that the model missed).

5.2.2. Accuracy (Binary Correctness)

• Conceptual Definition: Accuracy is the ratio of correctly predicted observations to the total observations. It's a straightforward measure of how often the model is correct. In a binary correctness scenario (like GSM-∞), it simply indicates the proportion of questions for which the model produced the exact correct answer.
• Mathematical Formula: $ \mathrm{Accuracy} = \frac{\mathrm{Number\ of\ Correct\ Predictions}}{\mathrm{Total\ Number\ of\ Predictions}} $
• Symbol Explanation:
  • $\mathrm{Number\ of\ Correct\ Predictions}$: The count of instances where the model's output exactly matches the ground-truth answer.
  • $\mathrm{Total\ Number\ of\ Predictions}$: The total count of instances (questions) in the dataset.

5.3. Baselines

The paper primarily compares the performance of the RL fine-tuned LLMs against their base models (i.e., the same LLM before any RL fine-tuning on synthetic data). These base models serve as the reference point to quantify the improvement brought by synthetic data training.

The LLMs used are:

• Qwen3-0.6B

• Qwen3-1.7B

• Phi-4-mini-reasoning (a 4B parameter model, which itself was already trained on some in-house synthetic data (Abdin et al., 2025))

• Qwen2.5-1.5B-Instruct

  The rationale for these choices is to demonstrate consistent performance transfer across a range of LLM families and sizes, from small (0.6B) to medium (4B), and to see if even a model pre-trained for reasoning (Phi-4-mini-reasoning) can further benefit.

5.4. Implementation Details

• Software: Hugging Face TRL library (version 0.21.0) GRPOTrainer, vLLM colocate mode (Kwon et al., 2023), and FlashAttention-2 (Dao, 2024).
• Hardware: 4 NVIDIA H100 GPUs (each with 80GB VRAM).
• Training Time: RL fine-tuning a 1-4B parameter LLM on $10 \mathrm{~K}$ training samples takes $\approx 1$ day. Phi-4-mini-reasoning and Qwen3-1.7B took the full day ($\approx 100 \mathrm{~H} 100$ hours) due to long CoT generations. Qwen2.5-1.5B-Instruct trained faster ($\approx 20 \mathrm{~H} 100$ hours) because it did not generate long CoT.
• Hyperparameters (from Listing 1 in Appendix A):
  • per_device_train_batch_size: 8 (adjusted to 4 for Phi-4-mini-reasoning)
  • gradient_accumulation_steps: 1
  • num_generations: 16 (adjusted to 8 for Phi-4-mini-reasoning)
  • vllm_gpu_memory_utilization: 0.20 (adjusted to 0.25 for Phi-4-mini-reasoning)
  • max_completion_length: 4096
  • temperature: 1.0
  • $top_p$: 1.0
  • $top_k$: null
  • $min_p$: null
  • repetition_penalty: 1.0
  • GRPO algorithm parameters: $\beta: 0.0$, $\varepsilon: 0.2$, importance_sampling_level: "token", scale_rewards: true, loss_type: bnpo, mask_truncated_completions: false
• Prompt Lengths:
  • PhantomWiki: 6000
  • GSM-∞: 2048
  • HotpotQA: 6000
  • 2WikiMultihopQA: 6000
  • MuSiQue: 8000
• Random Seeds: Data generation uses fixed random seeds, and fixed training random seeds are used where possible to ensure reproducibility.
• Evaluation: Models are evaluated with 2 random training seeds, and standard errors are reported.

6. Results & Analysis

The experimental results demonstrate that fine-tuning LLMs on synthetic data significantly improves their performance on real-world multi-hop reasoning benchmarks, even though the synthetic data contains only fictional knowledge. This indicates a successful transfer of knowledge composition skills.

6.1. Core Results Analysis

6.1.1. Performance Transfer from Synthetic to Real-world Datasets

The primary finding is that RL fine-tuning on synthetic datasets (PhantomWiki and GSM-∞) consistently improves F1 scores on real-world multi-hop reasoning benchmarks (HotpotQA, 2WikiMultihopQA, and MuSiQue). This transfer is observed across different LLM families (Qwen, Phi) and sizes (0.6B to 4B parameters).

The following figure (Figure 2 from the original paper) shows the F1 scores on real-world multi-hop reasoning datasets of LLMs finetuned with GRPO on synthetic datasets PhantomWiki and GSM-∞:

Analysis of Figure 2:

• Consistent Improvement: For nearly all LLMs and benchmarks, both PhantomWiki and GSM-∞ training lead to higher F1 scores compared to the base model (labeled "base").
• PhantomWiki's Superiority: PhantomWiki generally yields better performance transfer than GSM-∞ for these multi-hop reasoning tasks. For example, Qwen3-0.6B trained on PhantomWiki shows remarkable relative improvements: $62\%$ on HotpotQA, $63\%$ on 2WikiMultihopQA, and $132\%$ on MuSiQue. This suggests that natural language-based fictional reasoning (PhantomWiki) transfers more effectively to real-world natural language QA than math-based reasoning (GSM-∞).
• Model Family and Size Consistency: The positive trend holds for Qwen3-0.6B, Qwen3-1.7B, Qwen2.5-1.5B-Instruct, and Phi-4-mini-reasoning. Even Phi-4-mini-reasoning, which was already designed for reasoning, benefits from this additional RL fine-tuning. This suggests the approach is robust across different LLM architectures.
• Statistical Significance: The error bars (standard error from 2 random training seeds) indicate that the improvements are generally statistically significant.

6.1.2. Ablation Study on Binary Format Reward

To distinguish between learning proper output formatting and true reasoning capabilities, an ablation study was conducted. LLMs were fine-tuned for 3K training steps solely on a binary reward signal for producing the correct $<answer>...</answer>$ format (1 if correct, 0 if not).

The following are the results from Table 1 of the original paper:

Analysis of Table 1:

• Qwen2.5-1.5B-Instruct Improvement: This model shows a remarkable improvement (e.g., from 0.02 to 0.43 F1 on HotpotQA) with only format reward training. This suggests its base model struggled with the output format, and RL fine-tuning taught it reward hacking to produce the desired format. Therefore, its performance improvements in Figure 2 encompass both formatting and reasoning.
• Other Models' Relative Stability: For Qwen3 family models and Phi-4-mini-reasoning, format reward training does not significantly improve performance, and sometimes even slightly degrades it (e.g., Phi-4-mini-reasoning on 2WikiMultihopQA). This indicates these models already had good output formatting capabilities at initialization.
• Key Takeaway: For models that already format correctly, the improvements observed in Figure 2 are attributed to enhanced knowledge composition, not just formatting. This empirically demonstrates that LLMs can develop knowledge composition from synthetic data alone and apply it in real-world settings, as PhantomWiki and GSM-∞ are purely fictional.

6.1.3. Reasoning Evolution During Training

The paper investigates how reasoning capabilities evolve during training by evaluating intermediate checkpoints saved at every 10% of the total training steps. Since training is for 1 epoch and synthetic data is generated from random universes, this also allows studying the effect of synthetic data scaling.

The following figure (Figure 3 from the original paper) shows F1 scores on real-world multi-hop reasoning datasets of intermediate training checkpoints, when LLMs are finetuned with GRPO on synthetic datasets:

Analysis of Figure 3:

• Continued Improvement: Qwen3 LLMs (Qwen3-0.6B and Qwen3-1.7B) continue to improve on real-world multi-hop reasoning benchmarks with more training steps (or equivalently, more training samples). This is particularly evident with PhantomWiki training.

• No Overfitting: The sustained improvement suggests that models do not overfit to the synthetic training dataset. Instead, learning knowledge composition in fictional worlds consistently translates to gains in real-world contexts.

• Varying Malleability: Different LLMs exhibit varying malleability to RL fine-tuning. Qwen3-0.6B starts lower but shows a steeper upward trend, while Qwen3-1.7B improves slowly. This hints at the role of LLM initialization in RL fine-tuning, an area for future work.

• Similar trends are observed for Phi-4-mini-reasoning in Figure 6 (in Appendix B), while Qwen2.5-1.5B-Instruct does not show such continuous improvement.

  The following figure (Figure 6 from the original paper, located in Appendix B) shows F1 scores on real-world multi-hop reasoning datasets of intermediate training checkpoints for Phi-4-mini-reasoning and Qwen2.5-1.5B-Instruct:

  

  Analysis of Figure 6:

• Phi-4-mini-reasoning: Shows a general improvement trend with training steps, similar to Qwen3 models, indicating that RL fine-tuning on synthetic data is beneficial even for models already designed for reasoning.

• Qwen2.5-1.5B-Instruct: Performance saturates relatively quickly, especially when trained on GSM-∞. This contrasts with the Qwen3 models and Phi-4-mini-reasoning, suggesting Qwen2.5-1.5B-Instruct might be less amenable to learning additional knowledge composition skills through this method or might be primarily benefiting from format learning as seen in the ablation study.

6.1.4. Reasoning Evolution Across Question Difficulty

The synthetic datasets (PhantomWiki and GSM-∞) contain questions of varying difficulties, enabling analysis of model performance as a function of reasoning complexity (e.g., number of hops, number of arithmetic operations).

The following figure (Figure 5 from the original paper) shows reasoning evolution plots of F1 vs question difficulty of intermediate training checkpoints for Qwen3-0.6B and Qwen3-1.7B:

Analysis of Figure 5:

• Learning Across All Difficulties: Qwen3 LLMs learn to correctly answer questions across all difficulty levels as training progresses (indicated by darker lines representing more training steps).

• Transferability of Knowledge Composition: This improvement on validation questions across all difficulties, from universes completely disjoint from the training sets, signifies an enhancement in knowledge composition at all levels simultaneously.

• Impact of Dataset Type: The learning patterns differ between PhantomWiki (natural language, knowledge composition) and GSM-∞ (math, arithmetic operations). PhantomWiki training shows more consistent and broader improvements across difficulty levels for natural language multi-hop QA tasks.

• Similar trends are observed for Phi-4-mini-reasoning in Figure 7 (in Appendix B), while Qwen2.5-1.5B-Instruct saturates quickly, especially on GSM-∞.

  The following figure (Figure 7 from the original paper, located in Appendix B) shows reasoning evolution plots of F1 vs question difficulty of intermediate training checkpoints for Qwen2.5-1.5B-Instruct and Phi-4-mini-reasoning:

  

  Analysis of Figure 7:

• Phi-4-mini-reasoning: Demonstrates continued improvement across varying difficulties for both PhantomWiki and GSM-∞ as training progresses, reinforcing the idea that even specialized reasoning models can further refine their compositional abilities.

• Qwen2.5-1.5B-Instruct: Shows quick saturation, particularly on GSM-∞. The performance improvement plateaus early, suggesting limitations in its ability to acquire deeper knowledge composition skills from these synthetic datasets beyond an initial boost.

6.1.5. Learning to Compose Knowledge in Real-world Tasks

Finally, the paper demonstrates that LLMs learn to compose knowledge by analyzing their reasoning traces on the MuSiQue dataset. MuSiQue questions come with ground-truth intermediate answers.

The following figure (Figure 4 from the original paper) shows reasoning evolution plots on MuSiQue of PhantomWiki training checkpoints:

Analysis of Figure 4:

• Increased Intermediate Answer Generation: As LLMs undergo more training steps on PhantomWiki (darker lines), their generated reasoning traces include a progressively higher proportion of correct intermediate answers (Nth intermediate answer).
• Evidence for Knowledge Composition: This directly supports the claim that LLMs are learning to construct multi-step reasoning paths, not just producing final answers by chance. The ability to correctly generate intermediate steps is a strong indicator of learned knowledge composition.
• Unified Insight: This observation ties together the findings from performance transfer (Figure 2) and synthetic reasoning evolution (Figure 5), confirming that knowledge composition is a fundamental, generalizable skill critical for multi-hop reasoning, transferable across both synthetic and real-world domains.

6.2. Data Presentation (Tables)

The only table in the paper, Table 1, has been fully transcribed in the Ablation Study section above.

6.3. Ablation Studies / Parameter Analysis

The ablation study on binary format reward training (Table 1) effectively disentangles the improvements due to correct output formatting from true gains in reasoning capabilities. It shows that while some models (like Qwen2.5-1.5B-Instruct) initially struggle with formatting and benefit significantly from this reward, others (like Qwen3 and Phi-4-mini-reasoning) already handle formatting well. For the latter, the substantial performance increases observed in the main results are confidently attributed to enhanced knowledge composition rather than mere formatting adherence. This strengthens the paper's core claim about the transferability of reasoning skills.

The study of reasoning evolution during training (Figures 3, 5, 6, 7) also functions as an analysis of the effect of training steps (or synthetic data scaling). It confirms that continued exposure to synthetic data leads to sustained improvements without overfitting, especially for knowledge composition as measured by increasing accuracy on more difficult questions and better intermediate step generation. This implies that the amount of synthetic data can be scaled to further enhance LLM reasoning.

7. Conclusion & Reflections

7.1. Conclusion Summary

This work rigorously evaluates the efficacy of synthetic multi-hop reasoning datasets as a scalable and cost-effective alternative to traditional real-world training data for LLM reasoning. The findings conclusively demonstrate that RL fine-tuning on these synthetic datasets instills transferable compositional inference abilities in LLMs. These acquired skills lead to significant performance gains on diverse real-world question-answering benchmarks like HotpotQA, 2WikiMultihopQA, and MuSiQue, despite a complete absence of factual overlap between the synthetic training data and the real-world evaluation domains. The paper establishes that reasoning, particularly the ability to compose knowledge across multiple logical steps, is a fundamental and generalizable skill that can transfer across different domains and varying levels of complexity. Crucially, the models do not overfit to the synthetic data, showing continuous improvement with more training.

7.2. Limitations & Future Work

The authors acknowledge several limitations and suggest future research directions:

• Extent of Transferability: While the study demonstrates transferability of knowledge composition, the precise extent of this transferability remains an open question. Real-world tasks involve both factual knowledge and knowledge composition.
• Interplay of Factual Knowledge and Synthetic Data: The paper highlights that memorization can degrade performance in counterfactual contexts (Wu et al., 2025a), but models can learn memorization and generalizability simultaneously (Xie et al., 2024). Since synthetic datasets are knowledge-free, further investigation into their interplay with knowledge-intensive real-world datasets is warranted.
• Other Reasoning Capabilities: The current work focuses on knowledge composition through multi-hop reasoning. Future work should explore whether other reasoning capabilities, such as causal reasoning, counterfactual inference, or analogical thinking, can also exhibit similar transferability patterns from synthetic datasets.
• Boundary Conditions for Synthetic-to-Real Transfer: Understanding the specific conditions and mechanisms under which synthetic data effectively transfers to real-world tasks, and extending these methods beyond multi-hop reasoning (Zhao et al., 2023; Wu et al., 2025b; Wang et al., 2024), remains an important area for research.
• LLM Initialization and Malleability: The observed varying malleability of different LLMs to RL fine-tuning (e.g., Qwen3-0.6B showing steep improvement vs. Qwen2.5-1.5B-Instruct saturating quickly) suggests that LLM initialization and its inherent "quality" affect RL fine-tuning outcomes. Analyzing this effect is left for future work.

7.3. Personal Insights & Critique

This paper offers a compelling and highly practical insight into the potential of synthetic data for LLM development. The demonstration that knowledge composition can be learned from purely fictional, verifiable data and then generalized to complex real-world tasks is a significant step forward. It provides a blueprint for mitigating the data scarcity and cost bottlenecks that currently hinder LLM fine-tuning, especially for reasoning tasks. The rigor of testing across multiple LLM sizes and families, and the detailed analysis of reasoning evolution and difficulty stratification, strengthen the claims substantially.

Inspirations and Applications:

• Curriculum Learning with Synthetic Data: The idea of "domain experts as curators of verifiable curricula" for synthetic data is powerful. This could lead to a more principled approach to LLM training, where specific skills are targeted and honed using tailored synthetic environments, much like how humans learn through structured education.
• Robustness to Factual Changes: If LLMs can learn reasoning independently of specific facts, they might become more robust to factual changes or out-of-distribution knowledge, as their core reasoning engine would be more abstract. This could improve LLM performance in rapidly evolving knowledge domains.
• Debugging Reasoning: Synthetic datasets with controllable parameters (like PhantomWiki's number of hops) offer an excellent environment for debugging LLM reasoning. Researchers could pinpoint exactly where a model struggles in a multi-step inference process.

Potential Issues or Unverified Assumptions:

• "Fictional Knowledge" vs. "No Knowledge": While PhantomWiki is fictional, it still uses natural language structures and common relational concepts (family, friends). It's not entirely "knowledge-free" in the sense of abstract symbols. It implicitly carries structural knowledge of how entities and relations are expressed in natural language. The extent to which truly abstract, domain-agnostic reasoning could be learned from symbolic-only synthetic data might be a deeper question.

• Task Specificity of "Knowledge Composition": The multi-hop reasoning tasks primarily test chained inference. While fundamental, knowledge composition might manifest differently in other reasoning paradigms (e.g., causal, abductive). The transferability of compositional ability learned here to those different reasoning types is assumed but not directly tested.

• Scaling Limit of Synthetic Data: While the paper shows no overfitting with current synthetic data scales, there might be a point where synthetic data alone cannot provide the necessary nuances or complexities present in real-world data, leading to a plateau in performance or even negative transfer if the synthetic world is too simplistic. The Qwen2.5-1.5B-Instruct's saturation might hint at this for smaller models or certain architectures.

• Complexity of Synthetic Data Generation: While presented as scalable, generating high-quality synthetic data (especially for natural language tasks like PhantomWiki) still requires sophisticated grammar design, logic programming, and potentially LLM-based generation, which can be complex to curate and manage for new reasoning types.

  Overall, this paper provides a highly valuable contribution by empirically validating the power of synthetic data for improving LLM reasoning. It opens exciting avenues for constructing more capable and efficient LLMs by decoupling the learning of reasoning skills from the availability of vast, domain-specific real-world data.