1. Bibliographic Information

• Title: GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization
• Authors: Zhouhong Gu, Xingzhou Chen, Xiaoran Shi, Tao Wang, Suhang Zheng, Tianyu Li, Hongwei Feng, Yanghua Xiao
• Affiliations: The authors are affiliated with the School of Computer Science at Fudan University and Alibaba Group. This indicates a collaboration between academia and industry, often leading to research that is both theoretically grounded and practically relevant.
• Journal/Conference: Published in the long papers track of the Association for Computational Linguistics (ACL) 2025. ACL is a premier, top-tier conference in the field of computational linguistics and Natural Language Processing (NLP), signifying that the paper has undergone a rigorous peer-review process and is considered a significant contribution.
• Publication Year: 2025 (as per the source link, indicating an upcoming publication).
• Abstract: The paper addresses the challenge of controlling Large Language Model (LLM) outputs to follow specific, predefined constraints. Existing methods, like direct data synthesis or preferential response optimization, often fail with fine-grained constraints. The authors introduce Generative Adversarial Policy Optimization (GAPO), a new framework that combines a Generative Adversarial Network (GAN) with an encoder-only reward model. This approach adversarially generates training samples of increasing difficulty, helping the model learn complex constraints progressively. The encoder-only architecture is specifically chosen to better assess the relationship between a prompt and its response. Experiments show that GAPO significantly outperforms existing methods like PPO, DPO, and KTO, especially in tasks requiring detailed constraint adherence.
• Original Source Link:
  • Official Source: https://aclanthology.org/2025.acl-long.13/
  • PDF Link: https://aclanthology.org/2025.acl-long.13.pdf
  • The paper is slated for formal publication at ACL 2025.

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: As LLMs become more powerful, the need to precisely control their outputs to follow specific rules (constraints) becomes crucial for real-world applications like legal document generation or medical reporting.
  • Existing Gaps: Current methods for instilling constraint-following behavior fall short.
    1. Direct Synthesis: Creating instruction-response pairs that follow rules teaches the model what a correct answer looks like but not why. This can lead to "hallucination" where the model generates superficially correct but factually wrong text.
    2. Preferential Response Optimization: Using pairs of "good" vs. "bad" responses for a fixed prompt (e.g., DPO, PPO) helps the model align with preferences but doesn't effectively teach it to understand changes in the constraints themselves, especially when the constraints are subtle or complex. This is because the dominant decoder-only architecture of LLMs is not well-suited to compare a generated response back to the prompt's details.
  • Fresh Angle: The paper proposes a new paradigm called Preferential Prompt Learning. Instead of comparing two responses to one prompt, this method compares one response against two slightly different prompts (one "accepted" and one "rejected"). This forces the model to learn the fine-grained meaning of constraints within the prompt. The paper introduces GAPO as a framework specifically designed to solve the optimization challenges this new paradigm creates.

• Main Contributions / Findings (What):

  • A Novel Framework (GAPO): The paper introduces Generative Adversarial Policy Optimization (GAPO), which uniquely integrates a Generative Adversarial Network (GAN) structure with Proximal Policy Optimization (PPO). This allows for automated, progressive learning where the model is trained on increasingly difficult examples generated on the fly.
  • Effective Use of Preferential Prompts: GAPO successfully leverages "preferential prompt" data by using an encoder-only reward model. Unlike decoder-only models, encoders can look at both the prompt and response simultaneously (bidirectional attention), making them far better at judging if a response correctly follows the prompt's constraints.
  • Simplified and Automated Training: GAPO automates the difficult process of creating a curriculum of training data. The generator and reward model train each other in a loop, with the reward model becoming a stricter judge as the generator gets better. This simplifies the traditional PPO pipeline, which often requires a high-quality, pre-trained reward model before starting.
  • Superior Performance: Extensive experiments show GAPO significantly outperforms established baselines (SFT, DPO, KTO, PPO) on tasks requiring strict, fine-grained constraint following. Notably, methods like DPO suffer a "catastrophic failure" on these tasks, while GAPO remains robust.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Large Language Models (LLMs): These are deep learning models with billions of parameters, trained on vast amounts of text data (e.g., GPT-4, Qwen). They can understand and generate human-like text but require fine-tuning to align with specific tasks or constraints.
  • Reinforcement Learning from Human Feedback (RLHF): A multi-stage process to align LLMs with human preferences. It typically involves: (1) Supervised Fine-Tuning (SFT) on a base model, (2) Training a "reward model" to score outputs based on human-labeled preference data, and (3) Using a reinforcement learning algorithm like PPO to fine-tune the SFT model to maximize the scores from the reward model.
  • Proximal Policy Optimization (PPO): A popular reinforcement learning algorithm used in RLHF. The LLM acts as the "policy" or "generator," which learns to produce text that gets a high score from a separate, fixed "reward model." PPO ensures the new policy doesn't stray too far from the original one, maintaining stability.
  • Direct Preference Optimization (DPO): A more recent and simpler alternative to PPO. It cleverly re-frames the RL objective so that the LLM can be trained directly on preference data (chosen vs. rejected responses) without needing a separate reward model, making the process less complex and computationally cheaper.
  • Generative Adversarial Networks (GANs): A framework involving two neural networks—a Generator and a Discriminator—that compete with each other. The Generator creates fake data (in this case, text), and the Discriminator tries to distinguish the fake data from real data. In GAPO, the LLM is the Generator, and the Reward Model acts as the Discriminator.
  • Encoder-only vs. Decoder-only Architectures:
    • Decoder-only (e.g., GPT series): These models process text sequentially from left to right. When generating a token, they can only "see" the tokens that came before it (unidirectional attention). This makes them excellent for text generation but poor at tasks requiring a holistic understanding of a full input sequence, like comparing a prompt and a response.
    • Encoder-only (e.g., BERT, Longformer): These models can see the entire input sequence at once (bidirectional attention). This makes them superior for understanding tasks, like classification or judging the relationship between two pieces of text (like a prompt and a response). GAPO leverages this strength for its reward model.
  • Preferential Response vs. Preferential Prompt: This is a key distinction introduced by the paper, illustrated in Figure 1.

    • Preferential Response: The standard approach. Given one prompt, you have a chosen response and a rejected response. This teaches the model which kind of answer is better.

    • Preferential Prompt: The paper's novel approach. Given one response, you have an accepted prompt (which the response correctly follows) and a rejected prompt (which the response violates). This teaches the model to understand the fine-grained meaning of the constraints in the prompt.

      该图像是示意图，展示了“Preferential Response”（优选回应）与“Preferential Prompt”（优选提示）两种方法的流程差异。图中通过对比两者对提示和回应的处理方式，强调优选提示结合人工注释和基于规则生成机制，减少了人工负担且能实现更细粒度的偏好区分。

• Previous Works:

  • RLHF Methods: The paper situates itself within the literature on model alignment. It acknowledges classical RLHF using PPO, which requires a separate reward model, and more recent methods like DPO, SimPO, IPO, and KTO, which streamline the process by optimizing on preferences directly. However, it points out their weaknesses: RLHF needs a lot of high-quality data, and DPO can be unstable and overfit.
  • Constraint Following Augmentation: The paper also connects to the field of controlled text generation. It reviews three main approaches:
    1. Search-based methods (e.g., Constrained Beam Search), which modify the decoding algorithm but are often slow.
    2. Score-based methods, which guide generation with differentiable functions but don't guarantee satisfaction.
    3. Model-centric approaches, which bake constraints into the model via pre-training (e.g., CTRL) or instruction-tuning. The paper argues these methods often require heavy manual engineering or specialized training.

• Differentiation:

  • vs. PPO: In standard PPO, the reward model is trained once and then held fixed. In GAPO, the reward model is part of a GAN-like loop; it continuously improves alongside the generator, becoming a progressively harder critic. This creates an automated curriculum.
  • vs. DPO/KTO/SimPO: These methods implicitly derive a reward function and are designed for decoder-only models. They struggle with Preferential Prompts because their architecture is not suited for the fine-grained comparison between a prompt and a response. GAPO's use of an explicit, powerful encoder-only reward model is its key advantage here.

4. Methodology (Core Technology & Implementation)

The core of the paper is the GAPO framework, which is detailed in Section 3. It's an adversarial learning process designed to train a generator (π_θ) to produce text that adheres to a set of constraints.

该图像是示意图，展示了GAPO框架的两个调优阶段：预热训练阶段使用已有偏好数据训练奖励模型，随后对抗训练阶段通过奖励模型的反馈进行策略优化以更新生成器，同时奖励模型结合生成器产出数据和已有偏好数据继续训练。

As shown in Figure 2, the process has two phases: a Warmup Phase and an Adversarial Training Phase.

• Principles: The central idea is to have a generator (the LLM being fine-tuned) and a reward model (the discriminator) "play a game." The generator tries to create responses that fool the reward model into thinking they are valid. The reward model trains to get better at spotting invalid responses, including those from the generator. This constant competition pushes both models to improve.

• Steps & Procedures: The following table from the paper defines the symbols used in the methodology section.

  Manual Transcription of Table 1: All definitions used in the GAPO section.

  Symbol	Definition
  T	Free-text description component
  C	Constraint set
  P	Input prompt (T , C)
  R	Generated text output
  πθ(t|c)	Generator that produces next token t given context c
  πref	Reference generator for comparison
  L(R, Ci)	Constraint satisfaction function
  D	Training dataset
  D'	Augmented dataset
  R(c, t)	Reward model evaluating token t in context c
  V π(c)	Expected future rewards given context c
  Qπ(c, t)	Expected cumulative reward for token t in context c
  ¯R	Generator-produced text output

  1. Preliminary of Constrained Generation The goal is formally defined as an expectation maximization problem. Given a prompt $P = (\tau, \mathcal{C})$, where $\tau$ is text and $\mathcal{C}$ is a set of constraints, the objective is to train a generator $\pi_\theta$ that maximizes the expected number of satisfied constraints. $$E ( \pi _ { \theta } ) = \mathbb { E } _ { R \sim \pi _ { \theta } ( P ) } \left[ \sum _ { C _ { i } \in \mathcal { C } } \mathcal { L } ( R , C _ { i } ) \right] ,$$ where the constraint satisfaction function $\mathcal{L}(R, C_i)$ is binary: $$\mathcal { L } ( R , C _ { i } ) = \left\{ \begin{array} { l l } { 1 } & { \mathrm { if } R \models C _ { i } , } \\ { 0 } & { \mathrm { o t h e r w i s e } . } \end{array} \right.$$

  • $E(\pi_\theta)$: The expected score of the generator $\pi_\theta$.
  • $R \sim \pi_\theta(P)$: A response $R$ sampled from the generator given prompt $P$.
  • $\mathcal{L}(R, C_i)$: A function that returns 1 if response $R$ satisfies constraint $C_i$, and 0 otherwise.
  • $R \models C_i$: A logical statement meaning "R satisfies C_i".

  2. Constraint-Aware Data Augmentation To create the "preferential prompt" dataset, the authors start with a dataset $\mathcal{D}$ of valid (prompt, response) pairs. For each pair $(P_i, R_i)$, they create a "rejected" prompt $P_i^{\mathrm{reject}}$ by perturbing the original constraints $\mathcal{C}_i$ in one of two ways:

  • Constraint Modification: Change an existing constraint $C_{i,j}$ so the original response $R_i$ no longer satisfies it.
  • Constraint Insertion: Add a new constraint that conflicts with the original response $R_i$. This results in an augmented dataset $\mathcal{D}'$ containing triplets of (accepted_prompt, response) and (rejected_prompt, response).

  3. Adversarial Learning Framework (Algorithm 1) The algorithm alternates between training the reward model and the generator.

  Algorithm 1: Generative Adversarial Policy Optimization (GAPO)

  • Warmup Phase (Lines 2-6): Before the adversarial loop begins, the reward model $R(c,t)$ is pre-trained (warmed up) on the initial preferential prompt dataset. It learns to distinguish between (accepted_prompt, response) pairs (labeled 1) and (rejected_prompt, response) pairs (labeled 0). It is trained using a standard Binary Cross-Entropy (BCE) loss.

  • Adversarial Training Phase (Lines 8-18): This is an iterative process.

    • Train the Reward Model (Lines 10-13): On odd-numbered steps, the reward model is updated. It is trained on a combination of data:

      1. The original "gold" preference data: (accepted_prompt, response, label=1) and (rejected_prompt, response, label=0).
      2. Data generated by the current generator: (prompt, generated_response, label=0). This teaches the reward model to identify the generator's current flaws. The reward model's loss function is: $$L _ { R } ( \theta ) = - \operatorname { \mathbb { E } } _ { ( c , t , y ) \sim \mathcal { D } ^ { \prime } } \Bigg [ y \log R ( c , t ) + ( 1 - y ) \log ( 1 - R ( c , t ) ) \Bigg ]$$
      • $L_R(\theta)$: The loss for the reward model.
      • $(c, t, y) \sim \mathcal{D}'$: A sample from the augmented dataset, where $c$ is context, $t$ is the token, and $y$ is the label (1 for good, 0 for bad).
      • $R(c,t)$: The reward model's prediction of how good token $t$ is in context $c$.

    • Train the Generator (Lines 15-16): On even-numbered steps, the generator (π_θ) is updated using the PPO algorithm. The goal is to generate text that the newly updated reward model will score highly. The generator's loss (objective) function is: $$L _ { G } ( \theta ) = \mathbb { E } _ { n } \left[ \frac { \pi _ { \theta } ( t _ { n } | c _ { n } ) } { \pi _ { \mathrm { r e f } } ( t _ { n } | c _ { n } ) } A _ { n } \right]$$

      • $L_G(\theta)$: The loss for the generator.
      • $\pi_\theta(t_n|c_n)$: The probability of generating token $t_n$ given context $c_n$ by the current policy.
      • $\pi_{\mathrm{ref}}(t_n|c_n)$: The probability from a reference model (usually the SFT model before PPO), used to prevent the policy from changing too drastically.
      • $A_n$: The Advantage Function, which measures how much better a specific action (generating token $t_n$) is compared to the average action at that state. It is defined as: $A _ { n } = Q ^ { \pi } ( c _ { n } , t _ { n } ) - V ^ { \pi } ( c _ { n } )$
      • $Q^\pi(c_n, t_n)$: The action-value function, estimating the total future reward after taking action $t_n$ in state $c_n$.
      • $V^\pi(c_n)$: The value function, estimating the average total future reward from state $c_n$. It is learned by a separate "critic" model.

5. Experimental Setup

• Datasets:

  • Product Description Dataset (PDD): A new dataset created by the authors for generating product descriptions from a list of facts (property-value pairs). The key constraints are: (1) include all facts, and (2) do not add any new facts. The dataset statistics are shown in Table 2.

    Manual Transcription of Table 2: PDD dataset statistics.

    Name	#Product #PV-Pair #Sample		#Token
    PDD-Raw	201 93,616	-	-
    PDD-Train	201 76,913	26,419	17,541,881
    PDD-Rej-Train	201 66,838	26,419	14,983,806
    PDD-Test	201 49,470	6,605	4,212,440
    PDD-Rej-Test	201 31,280	6,605	3,629,544

  • IFEval: A standard benchmark for evaluating how well LLMs follow instructions. The authors augmented the dataset with new samples generated by GPT-4 to increase its size and diversity. The statistics are in Table 3.

    Manual Transcription of Table 3: IFEval dataset statistics.

    Name	#Type	#Sample	#Token
    IFEval-Response	9	540	355,199
    IFEval-Train	9	432	143,151
    IFEval-Rej-Train	9	432	141,963
    IFEval-Test	9	108	-

• Evaluation Metrics:

  • For IFEval: The paper uses the benchmark's own automated evaluation script, which calculates prompt-level accuracy (all instructions in a prompt are followed) and instruction-level accuracy (the proportion of individual instructions followed correctly across all prompts).
  • For PDD: A multi-faceted approach is used:
    1. Reward Model Score: An automated score from a fine-tuned Longformer model, which is an encoder-only model suitable for long text.
    2. LLM-as-a-Judge: Using powerful external LLMs (GPT-4o and GPT-3.5-turbo) to evaluate the quality and constraint adherence of the generated text.
    3. Human Evaluation: Human annotators manually score the outputs based on strict criteria (completeness and accuracy).

• Baselines: The experiments compare GAPO against a comprehensive set of methods:

  • Prompt-Based Methods: These involve no model training, only different prompting strategies on the base Qwen-2.5-7B model.
    • Direct Generation: Simple prompting.
    • Chain-of-Thought (CoT): Prompting the model to "think step by step."
    • Plan-and-Solve (Plan-N-Solve): Prompting the model to first create a plan and then execute it.
  • Training-Based Methods: These involve fine-tuning the Qwen-2.5-7B model.
    • Supervised Fine-Tuning (SFT): Standard fine-tuning on correct examples.
    • DPO, KTO, SimPO, ORPO: Recent preference alignment methods that don't use an explicit reward model.
    • PPO: The classic RLHF algorithm with a fixed, pre-trained reward model.
    • GAPO (Ours): The proposed method with a co-evolving reward model.

6. Results & Analysis

• Core Results:

  • IFEval Benchmark (Table 4): GAPO achieves the highest overall score (83.9%), significantly outperforming all other methods. PPO (75.6%) and SFT (78.3%) are also strong. In contrast, DPO-like methods (DPO, SimPO, ORPO) perform very poorly, with scores around 30-33%. This suggests they struggle to learn from the complex, mixed-constraint preference data of IFEval.

    Manual Transcription of Table 4: Performance comparison across different categories on IFEval Benchmark.

    Model	Prompt	Punctuation Format Length Content Combination ChangeCase Startend Keywords Language All
    Qwen-2.5-7B	Naive Prompt	17.6	88.1	42.3	66.7	20.0	62.5	66.7	52.6	90.9	57.8
    Qwen-2.5-7B	CoT	23.5	78.6	53.8	33.3	13.3	62.5	66.7	57.9	100.0	57.8
    Qwen-2.5-7B	Plan-N-Solve	23.5	81.0	38.5	66.7	0.0	68.8	44.4	63.2	90.9	56.1
    Qwen-2.5-7B + SFT	Naive Prompt	100.0	92.9	57.7	83.3	26.7	75.0	88.9	81.6	90.9	78.3
    Qwen-2.5-7B + DPO	Naive Prompt	17.6	45.2	26.9	16.7	6.7	31.2	11.1	42.1	63.6	33.3
    Qwen-2.5-7B + KTO	Naive Prompt	11.8	71.4	38.5	50.0	6.7	50.0	44.4	76.3	100.0	54.4
    Qwen-2.5-7B + SimPO Naive Prompt		11.8	45.2	23.1	16.7	0.0	31.2	0.0	39.5	63.6	30.6
    Qwen-2.5-7B + ORPO Naive Prompt		5.9	40.5	34.6	33.3	20.0	25.0	33.3	55.3	9.1	33.9
    Qwen-2.5-7B + PPO	Naive Prompt	94.1	90.5	50.0	66.7	33.3	62.5	88.9	84.2	90.9	75.6
    Qwen-2.5-7B + GAPO Naive Prompt		100.0	95.2	57.7	83.3	46.7	75.0	100.0	92.1	100.0	83.9

  • PDD Dataset (Table 5): The performance gap is even more stark. On this dataset, which relies heavily on preferential prompts, the DPO-like methods experience a "catastrophic failure," scoring near zero across all metrics (e.g., 5.4%, 2.9%, 7.5% on GPT-4o and 0% on human evaluation). This confirms the hypothesis that decoder-only architectures are fundamentally unsuited for this type of fine-grained prompt-response validation. In contrast, the encoder-based reward model approaches, PPO and GAPO, perform exceptionally well, with GAPO leading (90.2% GPT-4o score and 89% human score).

    Manual Transcription of Table 5: Comprehensive model performance comparison on PDD dataset.

    Model	Prompt	Reward Model		LLM-as-a-Judge		Human
    		LongFormer- Base-40963k	LongFormer- Large-40963k	GPT-40	GPT3.5-turbo
    Qwen2.5-7B	Naive Prompt	61.4	52.3	75.4	73.7	45
    Qwen2.5-7B	CoT	58.4	50.5	71.5	72.6	43
    Qwen2.5-7B	Plan-N-Solve	62.8	53.7	72.5	78.1	51
    Qwen2.5-7B + SFT	Naive Prompt	70.1	59.8	82.6	80.3	60
    Qwen2.5-7B + DPO	Naive Prompt	12.5	11.3	5.4	9.6	0
    Qwen2.5-7B + KTO	Naive Prompt	64.5	57.1	72.6	74.8	49
    Qwen2.5-7B + SimPO	Naive Prompt	5.3	7.6	2.9	3.8	0
    Qwen2.5-7B + ORPO	Naive Prompt	21.4	20.8	7.5	8.2	0
    Qwen2.5-7B + PPO	Naive Prompt	89.4	88.5	89.7	86.4	81
    Qwen2.5-7B + GAPO	Naive Prompt	95.4	94.3	90.2	90.0	89

• Effectiveness of Preferential Prompt vs. Preferential Response (Table 6): This ablation study directly compares training on the two types of preference data. The results clearly show that training with Preferential Prompt (PP) data consistently yields better performance than training with Preferential Response (PR) data for both PPO and GAPO. For example, with 6.6k samples, GAPO trained on PP data achieves a 95.4% score, which is 12.5 points higher than when trained on PR data (82.9%). This validates the core hypothesis that learning from prompt variations is a more effective way to teach constraint understanding.

  Manual Transcription of Table 6: Comparative Analysis of using Preferential Response and Preferential Prompt.

  Model	Reward Model	#Training Samples	#Token	PDD Score	∆No Train		∆PR vs. PP
  No Training
  Qwen-2.5-7B			-	61.4	-		-
  No Preferential Data
  Qwen-2.5-7B + SFT	-	3,300	6,561,531	70.1	+ 8.3		-
  Training w/ Preferential Response (PR)
  Qwen-2.5-7B + PPO	Qwen-2.5-7B	2,000	4,295,575	61.8	+	0.4	6.7
  Qwen-2.5-7B + PPO	Qwen-2.5-7B	4,000	8,660,218	72.4	+ 11.0	-	2.7
  Qwen-2.5-7B + PPO	Qwen-2.5-7B	6,600	13,243,796	78.5	+ 17.1		10.9
  Qwen-2.5-7B + GAPO	Longformer-0.4B	2,000	4,295,575	63.3	+ 1.9		7.3
  Qwen-2.5-7B + GAPO	Longformer-0.4B	4,000	8,660,218	74.4	+ 13.0	-	6.9
  Qwen-2.5-7B + GAPO	Longformer-0.4B	6,600	13,243,796	82.9	+ 21.5	-	12.5
  Training w/ Preferential Prompt (PP)
  Qwen-2.5-7B + PPO	Qwen-2.5-7B	2,000	4,219,814	68.5	+ 7.1	+	6.7
  Qwen-2.5-7B + PPO	Qwen-2.5-7B	4,000	8,506,194	75.1	13.7 +	+	2.7
  Qwen-2.5-7B + PPO	Qwen-2.5-7B	6,600	12,984,601	89.4	+ 28.0	+	10.9
  Qwen-2.5-7B + GAPO	Longformer-0.4B	2,000	4,219,814	70.6	+ 9.2	+	7.3
  Qwen-2.5-7B + GAPO	Longformer-0.4B	4,000	8,506,194	81.3	+ 19.9	+	6.9
  Qwen-2.5-7B + GAPO	Longformer-0.4B	6,600	12,984,601	95.4	+ 34.0	+	12.5

• Training Efficiency Analysis (Section 5.3): The data in Table 6 also reveals that GAPO is more sample-efficient. When scaling from ~4.2M to ~13.0M tokens on PP data, GAPO's score improves by 24.8 points, whereas PPO's improves by 20.9 points. This suggests the adversarial training dynamic helps GAPO learn more effectively from each data point.

• Detailed Performance Analysis (Figure 3): This figure shows scatter plots of performance against various complexity factors. The key takeaway is that GAPO's performance is robust. The trend lines are relatively flat, indicating that its performance does not significantly degrade as prompt length, output length, or the number of constraints increases.

  该图像是一个多子图散点图，展示了GAPO在PDD和IFEval基准测试中性能与提示长度、输出长度、约束数量等变量的相关性分析。每个子图中蓝色点表示样本数据，阴影区域显示性能趋势的不确定性。图中数据基于PDD测试集的300个随机样本和IFEval测试集的全部108个样本。

• Adversarial Process Dynamics (Figure 4): This plot shows the reward model scores for generated samples over the course of adversarial training. The score starts near zero (during the warmup phase W, the generator is untrained and produces poor samples) and steadily increases. The different lines likely represent different model runs or checkpoints. The smooth, upward-trending curves show that the generator is consistently improving, and the reward model is successfully tracking this improvement. The fact that the curves plateau rather than oscillate wildly indicates that the adversarial training is stable and avoids common GAN pitfalls like mode collapse.

  该图像是一个折线图，展示了在连续对抗训练阶段中奖励模型得分的演变情况。横轴表示训练阶段，包括预热阶段（W）和交替进行的生成器与奖励模型训练阶段（A1到A15），纵轴显示奖励模型得分，曲线反映了模型得分随训练阶段推进逐步上升的趋势。

• Case Study (Figure 5): A qualitative example shows the outputs of different models for a pet backpack description.

  • The base Qwen-2.5-7B model fails on length constraints and adds irrelevant emotional content.
  • SFT and PPO are better but still struggle with precision.
  • GAPO's output is superior: it perfectly adheres to the word count, includes all factual information without adding extra facts, and correctly incorporates the specified emotion ("Pride") in a natural way.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces GAPO, a novel and effective framework for teaching LLMs to follow complex, fine-grained constraints. By combining GAN-based training with PPO and using a specialized encoder-only reward model, GAPO excels at learning from "preferential prompts." The experimental results provide strong evidence that this approach is more robust and effective than existing alignment techniques like DPO and standard PPO, especially for tasks where precise constraint understanding is critical.

• Limitations & Future Work: The authors transparently discuss two main limitations:

  1. Computational Cost: GAPO is more computationally intensive than methods like DPO because it requires training three models simultaneously (generator, reward model, and the critic for the value function in PPO). This could be a barrier to adoption for teams with limited resources.
  2. Dependency on Base Model: The framework's effectiveness relies on starting with a base model that already has a reasonable level of generation capability. If the base model is too weak, it may produce incoherent text, which in turn would prevent the reward model from learning a meaningful signal, breaking the adversarial loop.

• Personal Insights & Critique:

  • Novelty and Impact: The conceptual shift from "Preferential Response" to "Preferential Prompt" is a simple but powerful idea. It directly targets the core weakness of current methods: their inability to reason about the prompt's constraints. The use of an encoder-only reward model is a perfect architectural match for this problem. This work provides a compelling solution for a critical problem in applied NLP.
  • Methodological Rigor: The combination of GAN dynamics with the stability of PPO is a clever piece of engineering. It automates the process of curriculum learning, which is often a manual and difficult task. The results, particularly the "catastrophic failure" of DPO, are striking and provide a clear, falsifiable claim about the limitations of certain architectures for specific tasks.
  • Potential Improvements/Open Questions:
    • Could the computational cost be reduced? Perhaps a parameter-efficient fine-tuning (PEFT) approach could be applied to the generator or reward model. Alternatively, the reward model and generator updates could be less frequent.
    • How does GAPO perform on other types of constraints, such as logical reasoning or complex formatting (e.g., generating valid JSON with a nested schema)?
    • The dependency on a strong base model is a key practical point. It would be interesting to see a study on how performance degrades as the capability of the base model decreases.
  • Overall, GAPO represents a significant step forward in the quest for reliable and controllable LLMs. Its principles could be highly influential in developing next-generation alignment techniques.