1. Bibliographic Information

1.1. Title

The title of the paper is "Plug and Play Language Models: A Simple Approach to Controlled Text Generation". The central topic is introducing a novel, flexible, and simple method for controlling the attributes (like topic or sentiment) of text generated by large, pre-trained language models without requiring retraining or fine-tuning of the base model itself.

1.2. Authors

The authors of the paper are:

• Sumanth Dathathri (CMS, Caltech)

• Andrea Madotto (HKUST)

• Janice Lan (Uber AI)

• Jane Hung (Uber AI)

• Eric Frank (Uber AI)

• Piero Molino (Uber AI)

• Jason Yosinski (Uber AI)

• Rosanne Liu (Uber AI)

  Their affiliations indicate a collaboration between academic institutions (Caltech, HKUST) and an industrial research lab (Uber AI), suggesting a mix of theoretical rigor and practical application focus.

1.3. Journal/Conference

The paper was published on arXiv, a preprint server for scientific papers. Specifically, this version is $v4$ of arXiv:1912.02164. While arXiv itself is not a peer-reviewed journal or conference, papers often appear there before or concurrently with submission to reputable venues. Given the authors' affiliations with leading institutions and research labs, it's likely intended for a top-tier machine learning or natural language processing conference or journal.

1.4. Publication Year

The paper was published on 2019-12-04T18:32:15.000Z. Thus, the publication year is 2019.

1.5. Abstract

Large transformer-based language models (LMs) excel at text generation but struggle with controlling specific attributes (e.g., topic, sentiment) without costly architectural modifications or fine-tuning. This paper introduces the Plug and Play Language Model (PPLM), a simple alternative for controllable text generation. PPLM combines a pre-trained LM (without any retraining) with one or more lightweight attribute classifiers. These classifiers, which can be as simple as a user-specified bag-of-words (BoW) or a single learned layer (with significantly fewer parameters than the LM), guide text generation. The core mechanism involves sampling, where gradients from the attribute model influence the LM's hidden activations through forward and backward passes, thereby steering the generation. The authors demonstrate PPLMs' ability to control various topics and sentiment styles, with evaluations showing strong attribute alignment and fluency. The flexibility of PPLMs, allowing any combination of differentiable attribute models, opens doors for diverse applications.

1.6. Original Source Link

The original source link is: https://arxiv.org/abs/1912.02164v4. The PDF link is: https://arxiv.org/pdf/1912.02164v4.pdf. This indicates the paper is publicly available as a preprint on arXiv.

2. Executive Summary

2.1. Background & Motivation

The proliferation of large transformer-based language models (LMs) like GPT-2 has led to unprecedented capabilities in generating coherent and natural-sounding text. These models are typically trained on vast amounts of unannotated text data using a simple log-likelihood objective, learning to predict the next word in a sequence.

However, a significant challenge arises when researchers or users want to control specific attributes of the generated text. For instance, guiding an LM to generate text about a particular topic (e.g., "science" or "politics") or with a desired sentiment (e.g., "positive" or "negative") is not straightforward. Existing methods often involve:

1. Modifying the model architecture: Integrating control signals directly into the LM's structure.

2. Fine-tuning on attribute-specific data: Retraining a portion or the entirety of the large LM on datasets explicitly labeled with the desired attributes.

   Both approaches are problematic because:

• They entail significant computational cost, especially for very large LMs, making them inaccessible for many researchers or applications.

• They often lack flexibility, as each new attribute or combination of attributes may require a new fine-tuning process or architectural change. This limits the "on-the-fly" adaptation needed for diverse applications.

• The trained control codes are fixed upfront, making it difficult to adapt to new, unforeseen control requirements.

  The paper aims to address this core problem: how to achieve fine-grained, flexible control over the attributes of generated text from a pre-trained LM without incurring the significant cost and inflexibility of retraining or fine-tuning the large base model. The authors draw inspiration from Plug & Play Generative Networks (PPGN) in computer vision, which demonstrated how a base generative model can be steered by a separate discriminator. This provides the innovative idea to "plug" a smaller attribute model into a large LM.

2.2. Main Contributions / Findings

The paper's primary contributions are:

• Introduction of the Plug and Play Language Model (PPLM): A novel, simple, and highly flexible method for controllable language generation. PPLM combines a pre-trained, unconditional language model (like GPT-2) with one or more separate, simple attribute models, without requiring any modifications or fine-tuning of the large LM's parameters.
• Gradient-Based Latent Space Steering: PPLM guides text generation by performing gradient-based updates in the LM's hidden activation space (specifically, the history matrix $H_t$). Gradients from the attribute model push the LM's internal representations towards states more likely to produce text with the desired attributes. This ex post facto optimization means no retraining is needed.
• Simple and Flexible Attribute Models: The attribute models can be remarkably simple, such as:
  • Bag-of-Words (BoW) classifiers: User-specified lists of keywords.
  • Single-layer discriminators: Small neural networks (e.g., ~1K parameters) trained to classify attributes like sentiment. These are orders of magnitude smaller than the base LM.
• Demonstrated Control over Diverse Attributes: PPLM successfully demonstrates control over a range of attributes, including 7 distinct topics (e.g., Science, Politics) and sentiment (positive/negative).
• Extensive Evaluation: The method is rigorously evaluated using both automated metrics (perplexity, n-gram diversity Dist-1/2/3, external sentiment classifiers) and human annotations (attribute relevance, fluency), confirming its ability to generate attribute-aligned, fluent text.
• Competitive Performance without LM Training: PPLM's performance is shown to be on par with or even superior to strong baselines like CTRL (a 1.6B parameter conditional LM) and fine-tuned GPT-2 models, critically achieving this without any training of the base language model.
• Novel Applications: PPLM is shown to be applicable to:
  • Language detoxification: By guiding generation away from toxic content using the negative gradient of a toxicity classifier.
  • Structurally constrained story writing: Filling in story skeletons while adhering to specific attributes.
• Flexibility and Combinatorial Control: A key finding is PPLM's inherent flexibility, allowing any combination of differentiable attribute models to steer generation simultaneously, acting as "control knobs" with adjustable strengths. This opens up creative and diverse applications.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully understand PPLM, a foundational understanding of the following concepts is essential:

• Language Models (LMs): At their core, LMs are probabilistic models that predict the likelihood of a sequence of words. Given a sequence of tokens $X = \{x_0, \ldots, x_n\}$, an LM is trained to compute the unconditional probability p(X). This probability is typically broken down using the chain rule of probability: $$p ( { \boldsymbol { X } } ) = \prod _ { i = 1 } ^ { n } p ( x _ { i } | x _ { 0 } , \cdot \cdot \cdot , x _ { i - 1 } )$$ This means the probability of a sequence is the product of the probabilities of each word given all preceding words. During generation, an LM iteratively samples the next word $x_i$ based on the previously generated words $x_0, \ldots, x_{i-1}$.

• Transformer Architecture: Introduced by Vaswani et al. (2017), the Transformer is a neural network architecture that has become dominant in natural language processing. It revolutionized sequence modeling by replacing recurrent layers (like Recurrent Neural Networks (RNNs)) with attention mechanisms, specifically self-attention. This allows the model to weigh the importance of different words in the input sequence when processing each word, capturing long-range dependencies more effectively and enabling parallel computation. Large LMs like GPT-2 are built upon this architecture.

• Hidden States / Latent Representations: In neural networks, hidden states or latent representations are the internal numerical representations of the input data as it passes through the network's layers. These are dense vector embeddings that capture abstract features and meanings. In PPLM, manipulating these hidden states (specifically the history matrix $H_t$ in a Transformer's recurrent view) is the key to controlling text generation.

• Gradients and Gradient Ascent/Descent: Gradients are vectors that indicate the direction of the steepest ascent (or descent) of a function. In machine learning, gradient descent is commonly used to minimize a loss function by iteratively adjusting model parameters in the direction opposite to the gradient. Conversely, gradient ascent is used to maximize a function (e.g., log-likelihood) by moving in the direction of the gradient. PPLM uses gradient ascent to increase the log-likelihood of desired attributes.

• Softmax Function: The softmax function is often used in the output layer of a neural network for multi-class classification. It takes a vector of arbitrary real values (called logits in LMs) and transforms them into a probability distribution over multiple classes (e.g., the vocabulary words). The output values are between 0 and 1 and sum to 1. For a vector $z = [z_1, \ldots, z_K]$: $$\operatorname { S o f t m a x } ( z ) _ i = \frac { e ^ { z _ i } } { \sum _ { j = 1 } ^ { K } e ^ { z _ j } }$$ In an LM, $z$ would be the logit vector $o_{t+1}$ and $K$ would be the size of the vocabulary.

• Kullback-Leibler (KL) Divergence: Also known as relative entropy, KL divergence is a non-symmetric measure of how one probability distribution $P$ is different from a second, reference probability distribution $Q$. A KL divergence of 0 indicates that the two distributions are identical. Minimizing KL divergence between the modified and unmodified LM output distributions helps PPLM ensure fluency by keeping the generation close to the original LM's natural distribution. $$D_{KL}(P \| Q) = \sum_i P(i) \log \left(\frac{P(i)}{Q(i)}\right)$$ where P(i) and Q(i) are the probabilities of event $i$ in distributions $P$ and $Q$, respectively.

• Bag-of-Words (BoW): A simplified representation of text that treats a document as an unordered collection of words, disregarding grammar and even word order, but keeping multiplicity. It's often used for feature extraction in text classification. In PPLM, a BoW can be a simple list of keywords related to a topic.

3.2. Previous Works

The paper frames its contribution by contrasting with existing approaches to controllable text generation:

• Controlled Generation (Fine-tuning, GANs, Conditional Models):

  • Reinforcement Learning (RL) fine-tuning (Ziegler et al., 2019): This involves training LMs using human feedback or reward signals to guide generation towards desired styles. While effective, it requires significant retraining of the entire model.
  • Generative Adversarial Networks (GANs) (Yu et al., 2017): GANs use a generator-discriminator setup, where the generator learns to produce realistic data, and the discriminator tries to distinguish real from generated data. For controlled generation, the discriminator can be conditioned on attributes. However, training GANs for text is notoriously difficult and unstable.
  • Conditional Generative Models (Kikuchi et al., 2016; Ficler & Goldberg, 2017; Keskar et al., 2019): These models are explicitly trained to generate text conditioned on specific attributes $p(x|a)$. CTRL (Keskar et al., 2019) is a prime example, training a very large LM (1.6B parameters) with over 50 different control codes upfront. This approach yields high-quality results but is costly to train and inflexible to new control codes.
  • Differentiation: PPLM differs by not requiring retraining of the large LM and allowing flexible, on-the-fly assembly of the LM with any differentiable attribute controller.

• Plug & Play Generative Networks (PPGN) (Nguyen et al., 2017): This work in computer vision is a direct inspiration for PPLM. PPGN demonstrated how to combine a pre-trained, unconditional image generator p(x) with a separate discriminator (attribute model) $p(a|x)$ to sample from the conditional distribution $p(x|a) \propto p(a|x)p(x)$. They achieved this by iteratively modifying the latent representation of the generator using gradients from the discriminator. PPLM adapts this core idea to the domain of language generation.

• Noisy Channel Modeling (Shannon, 1948; Yu et al., 2016; 2019; Yee et al., 2019; Ng et al., 2019): This theory, originally for communication, suggests that a received signal $x$ can be decoded from a corrupted signal $y$ by maximizing $p(x)p(y|x)$. In sequence-to-sequence tasks, this means sampling from a forward model $p_{\text{forward}}(x|y)$ and then reranking based on $p(x)p(y|x)$. PPLM uses a similar scoring equation for samples, but without a forward proposal model for $p(x|a)$, it relies on latent space updates, mirroring PPGN.

• Weighted Decoding (Holtzman et al., 2018; Ghazvininejad et al., 2017; Baheti et al., 2018): These methods modify the decoding procedure (how the next word is chosen) to incorporate a scoring function related to the desired attribute. This can involve directly weighting the output probabilities or using beam search with a modified scoring function. However, as noted by See et al. (2019), weighted decoding can often sacrifice fluency and coherence. PPLM differentiates by making updates in the continuous latent space rather than directly manipulating discrete output probabilities, allowing for more subtle and coherent guidance.

• Text Style Transfer (Shen et al., 2017; Hu et al., 2017; Li et al., 2018; Lample et al., 2019): This related field focuses on changing the style of existing text while preserving its content. Approaches include variational auto-encoders (VAEs) for disentangled representations or n-gram replacement. PPLM's focus is on controllable generation from scratch, and it employs an offline discriminator for optimization, which has been suggested to outperform adversarial training for attribute removal (Elazar & Goldberg, 2018).

3.3. Technological Evolution

The field of natural language generation has seen a rapid evolution:

1. Early Statistical Models: N-gram models (Manning et al., 1999) formed the basis, relying on probabilities of word sequences.

2. Neural Language Models: Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs) (Bengio et al., 2003) improved context understanding but struggled with long-range dependencies and parallelization.

3. Attention Mechanisms: The advent of attention (Bahdanau et al., 2014) allowed models to selectively focus on parts of the input, greatly enhancing sequence-to-sequence tasks like machine translation.

4. Transformers: The Transformer architecture (Vaswani et al., 2017), purely based on attention, enabled massive parallelization and state-of-the-art performance, leading to very large pre-trained LMs.

5. Large Pre-trained LMs: Models like GPT (Radford et al., 2018b), BERT (Devlin et al., 2019), and GPT-2 (Radford et al., 2019) pre-trained on enormous text corpora demonstrated impressive general-purpose language understanding and generation, establishing a new paradigm of transfer learning in NLP.

   This paper's work (PPLM) fits into the post-GPT-2 era, where the challenge shifted from generating coherent text to generating coherent text with specific controls. While large LMs excel at the former, they lack inherent mechanisms for the latter without expensive retraining. PPLM offers an elegant solution that leverages the power of these pre-trained models while adding a flexible control layer, without altering their core parameters.

3.4. Differentiation Analysis

Compared to the main methods in related work, PPLM offers several core differentiations and innovations:

• No Base LM Retraining/Fine-tuning: This is PPLM's most significant advantage. Unlike CTRL (which trains a 1.6B parameter model with control codes from scratch) or GPT2-FT-RL (which fine-tunes GPT-2 using reinforcement learning), PPLM leaves the large, pre-trained language model completely untouched. This drastically reduces computational cost, time, and resource requirements, making control accessible.

• Flexibility and Plug-and-Play Nature: PPLM allows for on-the-fly combination of a general-purpose LM with any number of small, specialized, and differentiable attribute models. These attribute models can be easily swapped, combined, or created for new tasks without affecting the core LM. This contrasts with CTRL, where control codes are fixed upfront, and with fine-tuning approaches which require a new training run for each new attribute.

• Gradient-Based Latent Space Control: Instead of directly modifying output probabilities (like Weighted Decoding) or requiring explicit conditional training, PPLM steers generation by applying gradients from the attribute model directly to the LM's hidden activations (latent representations). This subtle manipulation in the continuous latent space allows for more nuanced and coherent guidance, preserving fluency better than direct output manipulation, and is inspired by the successful PPGN in computer vision.

• Simplicity of Attribute Models: PPLM demonstrates effective control with extremely simple attribute models, such as Bag-of-Words (which requires no training) or single-layer discriminators (with negligible parameters compared to the LM). This low barrier to entry for attribute model creation enhances its flexibility.

• Combinatorial Control: PPLM naturally supports combining multiple attribute models (e.g., topic and sentiment, or multiple topics) to steer generation simultaneously, offering fine-grained, multi-faceted control via "control knobs" for each attribute's strength.

  In essence, PPLM provides a cost-effective, highly flexible, and conceptually simple method to imbue powerful, off-the-shelf LMs with controllable generation capabilities, bridging the gap between strong generative power and practical, adaptable control.

4. Methodology

4.1. Principles

The core idea behind PPLM is to transform an unconditional language model p(x) into a conditional one $p(x|a)$ on the fly, where $a$ represents one or more desired attributes. This is achieved by leveraging Bayes' theorem: $p(x|a) \propto p(a|x)p(x)$ Here, p(x) is the pre-trained, unconditional language model, and $p(a|x)$ is a separate attribute model (e.g., a classifier) that estimates the likelihood of text $x$ having attribute $a$.

Instead of explicitly sampling from this combined distribution directly (which is hard for discrete text), PPLM adopts an approach inspired by Plug & Play Generative Networks (PPGN) from computer vision. It iteratively modifies the internal latent representations (hidden states) of the pre-trained language model using gradient ascent. At each generation step, the gradients from the attribute model $p(a|x)$ push the LM's hidden activations in a direction that increases the likelihood of the generated text possessing the desired attribute. To ensure the generated text remains fluent and coherent, the method also incorporates mechanisms to keep the generation close to the original language model's distribution, primarily by also maximizing p(x) and using KL divergence regularization and geometric mean fusion.

The intuition is that by gently nudging the LM's internal "thought process" (its latent states) towards a specific attribute at each step, the subsequent word generation will naturally align with that attribute, all without altering the vast parameters of the pre-trained LM.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Language Modeling with Transformers

The paper first re-summarizes the Transformer architecture (Vaswani et al., 2017) using recurrent notation for clarity, as PPLM operates on the recurrent prediction step. Given a sequence of tokens $X = \{x_0, \ldots, x_n\}$, a language model computes its unconditional probability as the product of conditional probabilities: $$p ( { \boldsymbol { X } } ) = \prod _ { i = 1 } ^ { n } p ( x _ { i } | x _ { 0 } , \cdot \cdot \cdot , x _ { i - 1 } )$$ This means that each token $x_i$ is predicted based on the preceding history $x_0, \ldots, x_{i-1}$.

In a Transformer, this history is effectively summarized in a history matrix $H_t$. This $H_t$ consists of the key-value pairs from all previous time steps, across all attention layers up to time $t$. Specifically, $H _ { t } = [ ( K _ { t } ^ { ( 1 ) } , V _ { t } ^ { ( 1 ) } ) , \cdot \cdot \cdot , ( K _ { t } ^ { ( l ) } , V _ { t } ^ { ( l ) } ) ]$, where $(K_t^{(i)}, V_t^{(i)})$ are the key-value pairs from the $i$-th layer generated at all time-steps from 0 to $t$. Efficient Transformer implementations (like Hugging Face transformers library) use this cached $H_t$ to generate the next token $x_{t+1}$ given the current token $x_t$.

The recurrent interpretation of a Transformer can be summarized as: $$o _ { t + 1 } , H _ { t + 1 } = \mathbf { L M } ( x _ { t } , H _ { t } )$$ Here, $\mathbf{LM}$ represents the pre-trained Transformer model. It takes the current token $x_t$ and the history matrix $H_t$ to produce two outputs:

• $o_{t+1}$: The logit vector for the next token, which has a dimension equal to the vocabulary size.

• $H_{t+1}$: The updated history matrix, including information from $x_t$.

  The next token $x_{t+1}$ is then sampled from the probability distribution obtained by applying a linear transformation $W$ (which maps the logit vector to a vocabulary-sized vector) and then the Softmax function to $o_{t+1}$: $$x _ { t + 1 } \sim p _ { t + 1 } = \operatorname { S o f t m a x } ( W o _ { t + 1 } )$$ This process allows for efficient language generation without recalculating previous conditional probabilities at each step.

4.2.2. Steering Generation: Ascending $\log p(a|x)$

To control the output of the language model, PPLM introduces an update mechanism at each generation step $t$. The goal is to shift the history matrix $H_t$ in a direction that increases the log-likelihood of the desired attribute $a$ under the conditional attribute model $p(a|x)$.

A delta $\Delta H_t$ is initialized to zero at each time step. This $\Delta H_t$ represents the modification to the history matrix. The attribute model, which predicts $p(a|x)$, is re-expressed as $p(a | H_t + \Delta H_t)$, as its prediction depends on the internal representations. Gradient-based updates are then applied to $\Delta H_t$ to maximize this attribute likelihood: $$\Delta H _ { t } \gets \Delta H _ { t } + \alpha \frac { \nabla _ { \Delta H _ { t } } \log p ( a | H _ { t } + \Delta H _ { t } ) } { \| \nabla _ { \Delta H _ { t } } \log p ( a | H _ { t } + \Delta H _ { t } ) \| ^ { \gamma } }$$ Let's break down this formula:

• $\Delta H_t$: The current update vector being adjusted.

• $\alpha$: The step size, a hyperparameter that controls how aggressively the gradient pushes the history matrix towards the desired attribute. A larger $\alpha$ means stronger control.

• $\nabla_{\Delta H_t} \log p(a | H_t + \Delta H_t)$: The gradient of the log-likelihood of the attribute $a$ with respect to $\Delta H_t$. This gradient indicates the direction in which to adjust $\Delta H_t$ to maximize $p(a|x)$.

• $\| \nabla_{\Delta H_t} \log p(a | H_t + \Delta H_t) \|$: The Euclidean norm (magnitude) of the gradient vector.

• $\gamma$: A scaling coefficient for the normalization term. If $\gamma=1$, it normalizes the gradient to unit length, making the step size $\alpha$ independent of the gradient magnitude. If $\gamma=0$, there is no normalization. The paper mentions adaptive normalization in Section S11.3 for BoW models.

  This update can be repeated $m$ times in a loop at each time step $t$; in practice, the authors use 3 to 10 iterations.

After computing $\Delta H_t$, a forward pass through the LM is performed with the updated history matrix $\widetilde{H}_t = H_t + \Delta H_t$ to obtain the perturbed logits $\widetilde{o}_{t+1}$: $$\widetilde { o } _ { t + 1 } , H _ { t + 1 } = \mathrm { L M } ( x _ { t } , \widetilde { H } _ { t } )$$ These perturbed logits $\widetilde{o}_{t+1}$ are then used to generate a new probability distribution $\widetilde{p}_{t+1}$ over the vocabulary.

The following figure (Figure 1 from the original paper) illustrates this process:

该图像是示意图，展示了Plug and Play语言模型的三个阶段。第一步通过语言模型进行前向传播，以计算所需属性的可能性。第二步利用属性模型的梯度对语言模型的内部潜在表示进行更新。第三步生成新的词汇分布，并从中采样下一个 token。逐步更新潜在表示的过程使生成的文本朝向期望的属性过渡，图中以红色虚线区域表示可选择更新的最近潜在表示窗口。

Figure 1 shows three main steps:

1. Step 1 (Forward Pass for Attribute Likelihood): The current token $x_t$ and history $H_t$ are fed into the LM. An attribute model then computes $p(a|x)$ based on the LM's internal representations or output.

2. Step 2 (Backward Pass for Latent Updates): Gradients from the attribute model's log-likelihood with respect to $H_t$ (or rather, $\Delta H_t$) are computed. These gradients are used to update $\Delta H_t$ using the formula above, resulting in a modified latent representation $\widetilde{H}_t$. The dotted-red region indicates that for efficiency, only a window of recent latents might be modified.

3. Step 3 (New Distribution and Sampling): The current token $x_t$ and the modified history $\widetilde{H}_t$ are fed back into the LM to generate an updated distribution $\widetilde{p}_{t+1}$ over the vocabulary. The next token $x_{t+1}$ is then sampled from this new distribution.

   This process is repeated at each time step, leading to a gradual transition towards the desired attribute in the generated text.

4.2.3. Ensuring Fluency: Ascending $\log p(x)$

Simply pushing the LM towards an attribute can lead to unrealistic, adversarial, or degenerate text, as the generation might move into low probability regions of the original language model. To maintain fluency and coherence, PPLM employs two mechanisms that ensure the generated text remains aligned with the unconditional language model p(x):

4.2.3.1. Kullback-Leibler (KL) Divergence Minimization

In addition to ascending $\log p(a|x)$, PPLM also implicitly adds a term to the objective that minimizes the KL divergence between the output distribution of the modified language model $(\widetilde{p}_{t+1})$ and the unmodified language model $(p_{t+1})$. This is typically achieved by adding a term proportional to $\log p(x)$ to the gradient objective before computing the overall gradient. The full objective being maximized would then be a combination of $\log p(a|x)$ and $\log p(x)$. In practice, a hyperparameter $\lambda_{KL}$ is used to control the strength of this regularization term. The paper states that setting $\lambda_{KL}=0.01$ generally works well, indicating that a small pull towards the original LM distribution is sufficient.

4.2.3.2. Post-norm Geometric Mean Fusion

Beyond affecting the past via $\Delta H_t$, PPLM also performs post-norm fusion (similar to Stahlberg et al., 2018) at the very last step, when choosing the next token. This doesn't directly affect $\Delta H_t$ but directly combines the modified output distribution $\widetilde{p}_{t+1}$ with the unmodified output distribution $p_{t+1}$ to form the final sampling distribution. The next token $x_{t+1}$ is sampled from a distribution created by taking the geometric mean of $\widetilde{p}_{t+1}$ and $p_{t+1}$: $$x _ { t + 1 } \sim \frac { 1 } { \beta } \left( \widetilde { p } _ { t + 1 } ^ { \gamma _ { g m } } p _ { t + 1 } ^ { 1 - \gamma _ { g m } } \right)$$ Let's break down this formula:

• $\widetilde{p}_{t+1}$: The probability distribution over the vocabulary obtained from the modified logits (after latent space updates).

• $p_{t+1}$: The probability distribution over the vocabulary obtained from the unmodified logits (if no latent space updates were applied).

• $\gamma_{gm}$: A hyperparameter for the geometric mean fusion. It controls the blending strength.

  • As $\gamma_{gm} \to 1$, the sampling distribution converges to the one from the updated LM ($\widetilde{p}_{t+1}$), giving full weight to the attribute control.
  • As $\gamma_{gm} \to 0$, it converges to the unconditional LM distribution ($p_{t+1}$), effectively disabling attribute control.
  • The paper finds values in the range 0.8-0.95 work well, indicating a strong preference for the modified distribution while still being influenced by the original.

• $\beta$: A normalizing factor that ensures the resulting distribution sums to 1, making it a valid probability distribution.

  The following figure (Figure 2 from the original paper) provides an oversimplified view of why both $\log p(a|x)$ and $\log p(x)$ are needed:

  该图像是示意图，展示了最大化 $\log p(a | x)$ 和 $\log p(x)$ 的步骤关系。图中用黑点表示待考虑的句子，先沿着最大化 $\log p(a | x)$ 的方向移动，再沿着最大化 $\log p(x)$ 的方向前进。箭头指示了在连续隐藏表示空间 $H$ 中的移动，而非离散的字节对空间 $x$，强调了每步仅在 $H$ 空间中进行调整。

Figure 2 depicts a point representing a sentence in a conceptual space. It's first pushed towards maximizing $\log p(a|x)$ (attribute), and then towards maximizing $\log p(x)$ (fluency). In practice, PPLM combines these objectives in a single step, operating in the continuous space of hidden representations $H$ rather than the discrete word space.

4.2.4. Sampling and Ranking

The attribute model $p(a|x)$ in PPLM serves two crucial functions:

1. Scoring: It provides a score (the log-likelihood of the desired attribute) that can be used to rank multiple generated samples.

2. Gradient Ascent Direction: It provides the gradient used to update the latent space (history matrix).

   These functionalities can be combined. One strategy is to generate $r$ samples (e.g., using different random seeds or slight variations in the process) and then rank them based on their log-likelihood score for the desired attribute to choose the best one.

Additionally, to combat the problem of repetitive, low-quality text (a known issue in language generation, Holtzman et al., 2018; 2019), PPLM computes Dist-1, Dist-2, and Dist-3 scores (measures of n-gram diversity) for each generated passage. Samples with a mean Dist score below a certain threshold $\tau$ are discarded. This ensures that selected samples are not only attribute-aligned but also diverse and fluent.

4.2.5. BoW Attribute Models

The simplest type of attribute model used in PPLM is based on a Bag of Words (BoW). Given a pre-defined set of keywords $\{w_1, \ldots, w_k\}$ that characterize a topic, and the language model's output distribution $p_{t+1}$ at time $t+1$, the log-likelihood of the attribute $a$ (i.e., the topic represented by the BoW) is defined as: $$\log p ( a | x ) = \log \Big ( \sum _ { i } ^ { k } p _ { t + 1 } [ w _ { i } ] \Big )$$ Here:

• $k$: The total number of words in the Bag of Words for the specific attribute.

• $p_{t+1}[w_i]$: The probability that the word $w_i$ from the BoW will be the next token, as predicted by the language model's current output distribution $p_{t+1}$.

  This objective encourages the LM to generate words from the specified BoW. An interesting observation is that optimizing for words directly in the BoW also implicitly increases the probability of generating other topically related words that are not explicitly in the BoW.

4.2.6. Discriminator Attribute Models

When an attribute is too complex to be defined by a simple BoW (e.g., sentiment, style, toxicity), a more sophisticated discriminator attribute model can be used. These discriminators are typically small neural networks trained on datasets with input sentences $x$ and corresponding labels $y_x$.

In PPLM, the discriminators consist of a single-layer classifier. For an input sequence $x$ of length $t$, the LM produces output embeddings $o_{:t}^x$. The discriminator $f$ is trained on the mean of these embeddings across time, denoted as $\bar{o}_t^x$. This means the discriminator learns to classify the attribute based on the aggregate internal representation of the text. The number of parameters in such a single-layer classifier is negligible compared to the large LM itself (e.g., embedding-dimension $\times$ number of attributes + number of attributes).

While the discriminator's loss is a function of the entire sequence, PPLM adopts a greedy approach during generation. At each step, it optimizes for a higher probability of the sequence having a specific attribute by considering changes only to the next token to be generated. The objective is to maximize: $$\log p ( a | x ) = \log f ( o _ { : t + 1 } , o _ { t + 2 } )$$ Here:

• $f$: The discriminator function.

• $o_{:t+1}$: The LM embeddings for the sequence up to the current time step $t+1$.

• $o_{t+2}$: The LM embedding for the hypothetical next token $x_{t+1}$ (since $o_{t+2}$ is a function of $x_{t+1}$).

  Crucially, $x_{t+1}$ is sampled from $\operatorname{Softmax}(W \widetilde{o}_{t+1})$, which itself depends on $\Delta H_t$. A hard sample of $x_{t+1}$ would limit diversity and potentially lead to language degeneration. Instead, PPLM uses the distribution $\widetilde{p}_{t+1}$ (the output probabilities for the next token) to obtain a (biased) estimate of the next token's embedding, and then uses this to update $\Delta H_t$. This "softer optimization" approach aims to shift the distribution $\widetilde{p}_{t+1}$ towards one that, in expectation, has a higher likelihood of having the desired attribute $a$.

4.2.7. Early Stopping of Latent Updates (Section S11.1)

Degeneration, characterized by the occurrence of repetitive words, is a common issue in language generation, especially when strong control signals are applied. In PPLM-BoW, if the step size $\alpha$ is too large, the model might degenerate by repeatedly generating keywords from the Bag of Words.

To mitigate this, PPLM can employ an early stopping mechanism for latent updates. Instead of performing updates at every generation step for the entire sequence, the latent updates can be stopped after a certain number of time steps (e.g., 20 time steps). This allows the initial guidance to steer the topic, but then lets the LM continue generation without continuous, aggressive nudging, which helps maintain fluency and prevents unwanted repetition.

4.2.8. Finite Horizon Update (Section S11.2)

For computational efficiency and potentially better fluency, PPLM can modify only a finite window of the history matrix $H_t$, rather than the entire history from time 0 to $t$. At each time step $t$, only the key-value pairs corresponding to the most recent $w$ tokens, i.e., $H_t[t-w:t]$, are modified. This means each $H_i$ is modified at most $w$ times.

The paper found that setting the window size $w=5$ produces more fluent passages for BoW-based control. For neural attribute models (discriminators), updating the entire latent history was found to be effective.

4.2.9. Adaptive Gradient Normalization (Section S11.3)

For Bag-of-Words attribute models, the goal is often to ensure that a word from the bag appears at least once in the generated passage, not necessarily at every single time step. To account for this nuance, PPLM uses an adaptive normalization scheme for the gradient term in Equation 3, particularly for BoW models.

Instead of normalizing the gradient directly by its current norm $\| \nabla_{\Delta H_t} \log p(a | H_t + \Delta H_t) \|$, it normalizes by the maximum gradient norm observed over time. Formally, the normalization constant at time step $t$ becomes: $$\operatorname* { m a x } _ { i = 0 \ldots t } \| \nabla _ { H ^ { ( i ) } } \mathcal { L } \big ( o _ { i + 1 } \big ) \|$$ This means that earlier in the generation, when attribute words are less likely to appear, the updates are smaller, allowing for a more gradual and less aggressive steering. This adaptive approach implies making smaller updates when it is less likely for a BoW word to appear, leading to a more natural integration of the attribute.

4.3. Full Architectural Flow

The overall flow of PPLM at each generation step can be summarized as follows:

1. Input: The current token $x_t$ and the history matrix $H_t$ from the previous step.
2. Unmodified LM Forward Pass:
   • Feed $x_t$ and $H_t$ into the pre-trained $\mathbf{LM}$ to get the unmodified logits $o_{t+1}$ and unmodified history $H_{t+1}$.
   • Compute the unmodified probability distribution $p_{t+1} = \operatorname{Softmax}(W o_{t+1})$.
3. Latent Representation Update (Steering):
   • Initialize $\Delta H_t = 0$.
   • Loop $m$ times (e.g., 3-10 iterations):
     • Compute the log-likelihood of the desired attribute $a$ given the hypothetically modified history $H_t + \Delta H_t$, i.e., $\log p(a | H_t + \Delta H_t)$. The form of $p(a|x)$ depends on whether it's a BoW or discriminator model.
     • Compute the gradient of this log-likelihood with respect to $\Delta H_t$.
     • Update $\Delta H_t$ using the gradient ascent formula (Equation 3), potentially incorporating KL divergence minimization and adaptive normalization.
   • Obtain the final modified history matrix $\widetilde{H}_t = H_t + \Delta H_t$. (Optionally, only update a finite window of $H_t$).
4. Modified LM Forward Pass:
   • Feed $x_t$ and $\widetilde{H}_t$ into the pre-trained $\mathbf{LM}$ to get the modified logits $\widetilde{o}_{t+1}$.
   • Compute the modified probability distribution $\widetilde{p}_{t+1} = \operatorname{Softmax}(W \widetilde{o}_{t+1})$.
5. Post-norm Geometric Mean Fusion:
   • Combine $p_{t+1}$ and $\widetilde{p}_{t+1}$ using the geometric mean fusion formula (Equation 4) to get the final sampling distribution.
6. Sampling the Next Token:
   • Sample the next token $x_{t+1}$ from the final sampling distribution (e.g., using top-k sampling).
7. Ranking and Filtering (Post-generation):
   • After generating $r$ full sequences for a given prefix, compute the attribute likelihood for each sequence using $p(a|x)$.
   • Compute n-gram diversity scores (Dist-1/2/3).
   • Rank sequences by attribute likelihood and filter out those with low diversity (below threshold $\tau$).
   • Select the best ranked and filtered sample.

5. Experimental Setup

5.1. Datasets

The paper utilizes various datasets for training attribute models and for evaluating the PPLM's performance across different control tasks.

• Bag-of-Words (BoW) Attribute Models:

  • Source: The word lists were manually curated from $www.enchantedlearning.com/wordlist$.
  • Characteristics: These are simple, user-defined lists of keywords for specific topics. No training data is needed for these attribute models, as they directly score based on word presence.
  • Topics: The paper defined seven distinct topics using these BoW lists. The complete word lists are provided in Section S17 of the supplementary information.
    • SCIENCE: astronomy, atom, biology, cell, chemical, chemistry, climate, control, data, electricity, element, energy, evolution, experiment, fact, flask, fossil, funnel, genetics, gravity, hypothesis, lab, laboratory, laws, mass, matter, measure, microscope, mineral, molecule, motion, observe, organism, particle, phase, physics, research, scale, science, scientist, telescope, temperature, theory, tissue, variable, volume, weather, weigh
    • FANTASY/MAGIC: beast, Cerberus, demon, dragon, fairy, Frankenstein, ghost, Godzilla, giant, horror, hydra, imp, monster, mummy, ogre, orc, savage, spirit, sprite, titan, troll, undead, unicorn, vampire, witch, zombie
    • SPACE: planet, galaxy, space, universe, orbit, spacecraft, earth, moon, comet, star, astronaut, aerospace, asteroid, spaceship, starship, galactic, satellite, meteor
    • POLITICS: affirm, appropriation, aristocracy, authoritarian, authority, authorization, brief, capitalism, communism, constitution, conservatism, court, deficit, diplomacy, direct, democracy, equality, exports, fascism, federation, government, ideology, imports, initiative, legislature, legitimacy, liberalism, liberty, majority, order, political, culture, politics, power, primary, property, ratification, recall, referendum, republic, socialism, state, subsidy, tariff, imports, tax, totalitarian
    • MILITARY: academy, advance, aircraft, ally, ammo, ammunition, armor, arms, army, arrow, arsenal, artillery attack, attention, ballistic barracks, base, battalion, battery, battle, battlefield bomb, bombard, bombardment, brig, brigade, bullet, camouflage, camp, cannon, captain, capture, carrier, casualty, catapult, cavalry, colonel, combat, command, commander, commission, company, conflict, conquest, convoy, corps, covert, crew, decode, defeat, defend, defense, destroyer, division, draft, encode, enemy, engage, enlist, evacuate, explosive, fight, fire, fleet, force, formation, fort, front, garrison, general, grenade, grunt, guerrilla, gun, headquarters, helmet, honor, hospital, infantry, injury, intelligence, invade, invasion, jet, kill, leave, lieutenant, major, maneuver, marines, MIA, mid, military, mine, missile, mortar, navy, neutral, offense, officer, ordinance, parachute, peace, plane, platoon, private, radar, rank, recruit, regiment, rescue, reserves, retreat, ribbon, sabotage, sailor, salute, section, sergeant, service, shell, shoot, shot, siege, sniper, soldier, spear, specialist, squad, squadron, staff, submarine, surrender, tactical, tactics, tank, torpedo, troops, truce, uniform, unit, veteran, volley, war, warfare, warrior, weapon, win, wound
    • RELIGION: Absolute, Affect, Aid, Angel, Anthem, Apostle, Archangel, Archbishop, Balance, Ban, Belief, Benefit, Bible, Bishop, Bless, Blessing, Bliss, Bond, Bow, Buddhism, Canon, Cantor, Cathedral, Celestial, Chapel, Charity, Choice, Christianity, Church, Comfort, Community, Conflict, Connection, Conquest, Conservative, Control, Conversion, Convert, Core, Counsel, Courage, Covenant, Creative, Creator, Creed, Cross, Crusade, Darkness, Decision, Deity, Destiny, Devil, Disciple, Discipline, Discussion, Divine, Divinity, Doctrine, Duty, Effect, Elder, Energy, Essence, Eternal, Ethics, Event, Evidence, Exile, Exodus, Faith, Family, Fate, Father, Favor, Fundamental, Gift, Glory, God, Gospel, Grace, Growth, Guru, Habit, Hallow, Halo, Happiness, Harmony, Healing, Heaven, Hebrew, Holy, Honor, Hope, Host, Humane, Immortal, Influence, Insight, Instruction, Issue, Jesuit, Jesus, Joy, Judaism, Judgment, Justice, Karma, Keen, Keystone, Kingdom, Latin, Life, Light, Love, Loving, Marriage, Meaning, Mercy, Messiah, Minister, Miracle, Mission, Mortal, Mosque, Movement, Music, Mystery, Nature, Nun, Official, Oracle, Order, Organ, Orthodox, Outlook, Pacific, Pagan, Parish, Participation, Pastor, Patriarch, Peace, Perception, Personal, Perspective, Petition, Pilgrim, Politics, Power, Practice, Prayer, Prelude, Presence, Priest, Principle, Privacy, Prophet, Protection, Purpose, Query, Quest, Question, Quiet, Radiant, Radical, Rally, Rebirth, Redemption, Refuge, Relationship, Relative, Religion, Religious, Revelation, Ritual, Role, Sacrament, Sacred, Sacrifice, Sage, Saint, Salvation, Sanctuary, Savior, Scripture, Scriptures, Sect, Security, Sense, Serious, Serve, Service, Sharia, Shepherd, Shrine, Silence, Sin, Society, Soul, Source, Spirit, Spiritual, Split, Statue, Sunday, Support, Supreme, Teaching, Temple, Tests, Text, Torah, Tradition, Traditional, Trust, Unique, Unity, Unknown, Value, Vanity, Virtue, Vision, Voice, Voices, Watch, Weight, Whole, Wisdom, Wonder, Yang, Yin, Zeal
    • COMPUTERS: algorithm, analog, app, application, array, backup, bandwidth, binary, bit, bite, blog, blogger, bookmark, boot, broadband, browser, buffer, bug, bus, byte, cache, caps, captcha, CD, client, command, compile, compress, computer, configure, cookie, copy, CPU, dashboard, data, database, debug, delete, desktop, development, digital, disk, document, domain, dot, download, drag, dynamic, email, encrypt, encryption, enter, FAQ, fle, firewall, firmware, flaming, flash, folder, font, format, frame, graphics, hack, hacker, hardware, home, host, html, icon, inbox, integer, interface, Internet, IP, iteration, Java, joystick, kernel, key, keyboard, keyword, laptop, link, Linux, logic, login, lurking, Macintosh, macro, malware, media, memory, mirror, modem, monitor, motherboard, mouse, multimedia, net, network, node, offline, online, OS, option, output, page, password, paste, path, piracy, pirate, platform, podcast, portal, print, printer, privacy, process, program, programmer, protocol, RAM, reboot, resolution, restore, ROM, root, router, runtime, save, scan, scanner, screen, screenshot, script, scroll, security, server, shell, shift, snapshot, software, spam, spreadsheet, storage, surf, syntax, table, tag, template, thread, toolbar, trash, undo, Unix, upload, URL, user, UI, username, utility, version, virtual, virus, web, website, widget, wiki, window, Windows, wireless, worm, XML, Zip
    • LEGAL: affidavit, allegation, appeal, appearance, argument, arrest, assault, attorney, bail, bankrupt, bankruptcy, bar, bench, warrant, bond, booking, capital, crime, case, chambers, claim, complainant, complaint, confess, confession, constitution, constitutional, contract, counsel, court, custody, damages, decree, defendant, defense, deposition, discovery, equity, estate, ethics, evidence, examination, family, law, felony, fle, fraud, grievance, guardian, guilty, hearing, immunity, incarceration, incompetent, indictment, injunction, innocent, instructions, jail, judge, judiciary, jurisdiction, jury, justice, law, lawsuit, lawyer, legal, legislation, liable, litigation, manslaughter, mediation, minor, misdemeanor, moot, murder, negligence, oath, objection, opinion, order, ordinance, pardon, parole, party, perjury, petition, plaintiff, plea, precedent, prison, probation, prosecute, prosecutor, proxy, record, redress, resolution, reverse, revoke, robbery, rules, sentence, settlement, sheriff, sidebar, standing, statute, stay, subpoena, summons, suspect, testimony, thief, trial, trustee, venue, verdict, waiver, warrant, will, witness, writ, zoning

• Discriminator Attribute Models:

  • Sentiment Discriminator:
    • Dataset: SST-5 dataset (Socher et al., 2013). This dataset consists of movie reviews labeled with five sentiment classes (very negative, negative, neutral, positive, very positive). The PPLM discriminator simplifies this to POSITIVE and NEGATIVE sentiment.
    • Purpose: To train a simple single-layer classifier that predicts sentiment from the LM's mean latent representations.
    • Effectiveness: The paper notes that even though the discriminator was trained on movie review data, it effectively controlled sentiment for prefixes not necessarily associated with movie reviews (e.g., "The painting", "The potato").
  • Toxicity Classifier:
    • Dataset: Toxic Comment Classification Challenge (Jigsaw) dataset. This dataset contains comments from Wikipedia's talk page edits that have been labeled by human raters for different types of toxicity (toxic, severe toxic, obscene, threat, insult, identity hate).
    • Purpose: To train a single-layer classifier that detects toxicity. This classifier is then used with PPLM to steer generation away from toxic content (detoxification) by following the negative gradient.

• Prefixes for Generation:

  • PPLM-BoW Evaluation: 20 prefixes were chosen randomly from www2.eit.ac.nz/library/ls_guides_sentencestarters.html.

    • "In summary", "This essay discusses", "Views on", "The connection", "Foundational to this is", "To review,", "In brief,", "An illustration of", "Furthermore,", "The central theme", "To conclude,", "The key aspect", "Prior to this", "Emphasised are", "To summarise", "The relationship", "More importantly,", "It has been shown", "The issue focused on", "In this essay".

  • PPLM-Discrim Evaluation: 15 prefixes were used.

    • "Once upon a time", "The book", "The chicken", "The city", "The country", "The horse", "The lake", "The last time", "The movie", "The painting", "The pizza", "The potato", "The president of the country", "The road", "The year is 1910."

  • Odd Combinations (S9): Specific, often illogical, combinations of prefixes and topics (e.g., prefix "The potato" with topic "RELIGION") were also tested to demonstrate robustness.

  • Toxicity Evaluation (S13): Natural prompts ("Blacks", "Asians", "Jews") and adversarial triggers (generated by Wallace et al., 2019 codebase) were used.

    These datasets and prefixes were chosen to represent diverse linguistic tasks and domains, ensuring a thorough validation of PPLM's ability to control various attributes and generalize beyond specific training domains.

5.2. Evaluation Metrics

The paper uses a combination of automated and human evaluation methods to assess two key properties:

1. Attribute Alignment: Whether the generated text successfully possesses the desired attribute (topic or sentiment).
2. Text Quality (Fluency & Diversity): Whether the quality of the text (coherence, naturalness, non-repetitiveness) is maintained despite attribute control.

5.2.1. Automated Evaluation Metrics

• Perplexity (PPL):

  1. Conceptual Definition: Perplexity is a common intrinsic evaluation metric for language models. It quantifies how well a probability model predicts a sample. Lower perplexity generally indicates a better model, meaning the model assigns higher probabilities to the observed test data, thus predicting it more "accurately" or "confidently." In the context of text generation, it's used as an indirect measure of fluency and naturalness by evaluating how surprised an external LM is by the generated text.
  2. Mathematical Formula: For a sequence of tokens $X = (x_1, x_2, \ldots, x_N)$, the perplexity is defined as: $$\mathrm{PPL}(X) = \exp \left( - \frac{1}{N} \sum_{i=1}^{N} \log P(x_i | x_1, \ldots, x_{i-1}) \right)$$
  3. Symbol Explanation:
     • $X$: The generated text sequence.
     • $N$: The total number of tokens in the sequence $X$.
     • $P(x_i | x_1, \ldots, x_{i-1})$: The probability of the $i$-th token $x_i$ given the preceding tokens $x_1, \ldots, x_{i-1}$, as predicted by an external (different) language model.
  • Note: The paper uses a GPT LM (Radford et al., 2018a) to compute perplexity, which is different from the GPT-2 LM (Radford et al., 2019) used for generation. This ensures an unbiased external assessment.

• Distinct N-grams (Dist-1, Dist-2, Dist-3):

  1. Conceptual Definition: These metrics measure the diversity and non-repetitiveness of generated text. They calculate the proportion of unique n-grams (sequences of n words) among all n-grams in the generated text. Higher Dist-N scores indicate more diverse text, with less repetition.
  2. Mathematical Formula: For Dist-N, where $N \in \{1, 2, 3\}$: $$\mathrm{Dist-N} = \frac{\text{Number of unique N-grams}}{\text{Total number of N-grams}}$$
  3. Symbol Explanation:
     • Number of unique N-grams: The count of distinct sequences of $N$ tokens found in the generated text.
     • Total number of N-grams: The total count of all sequences of $N$ tokens in the generated text.
  • Note: These scores are measured across all samples generated for a given attribute control task (e.g., all samples generated for the "Science" topic), indicating overall diversity of the model's output for that attribute.

• External Sentiment Classifiers:

  1. Conceptual Definition: For evaluating sentiment control, PPLM uses an external sentiment classifier (independent of the one used as an attribute model) to determine the sentiment (e.g., positive or negative) of the generated text. This provides an objective measure of how well the generated text aligns with the target sentiment.
  2. Mathematical Formula: (Not explicitly provided in the paper, but implied by the task) For a text $x$ and a target sentiment $s_{target}$: $$\text{Sentiment Accuracy} = \frac{\text{Number of texts classified as } s_{target}}{\text{Total number of texts}}$$
  3. Symbol Explanation: Sentiment Accuracy is simply the fraction of generated samples that an independent classifier correctly identifies as having the desired target sentiment.
  • Note: The paper mentions using an external sentiment classifier trained on IMDB movie reviews (Maas et al., 2011), which is a different dataset from the SST-5 dataset used to train the PPLM's internal sentiment attribute model. This further ensures independence in evaluation.

5.2.2. Human Evaluation Metrics

Human annotators were used to assess aspects that are difficult for automated metrics, primarily fluency and attribute relevance.

• Fluency:

  1. Conceptual Definition: Human annotators are asked to rate the fluency (grammatical correctness, coherence, naturalness) of each individual generated sample. This directly captures the human perception of text quality.
  2. Rating Scale: A 1-5 scale, where 1 means "not fluent at all" and 5 means "very fluent."
  3. Process: Each sample is evaluated by three individuals, and the average of these annotations is used. The generation method and order of samples are hidden to avoid bias.

• Attribute Relevance (A/B Testing):

  1. Conceptual Definition: For assessing attribute relevance (e.g., topic relevance, sentiment strength), human annotators perform A/B testing. They are presented with pairs of generated texts (from different PPLM variants or baselines) and asked to rank which one better exhibits the desired attribute.
  2. Ranking Options: Annotators can choose which text is "more" relevant, or select "neither" or "both" if the texts are equally good/bad.
  3. Process: All combinatorial pairs of the four ablation study variants (B, BR, BC, BCR) are tested. Each pair is evaluated by three individuals, and majority-voting is used to determine the outcome. The generation method and order of samples are hidden.

5.3. Baselines

The paper compares PPLM against several baseline approaches to demonstrate its effectiveness and efficiency:

5.3.1. Ablation Study Variants

To understand the contribution of different components of PPLM, an ablation study was conducted with four variants based on the GPT-2 345M model as the base LM:

• B (Baseline): The unchanged, unconditional GPT-2 LM. Text is generated by sampling once from this model without any attribute control or reranking.
• BR (Baseline + Reranking): Similar to $B$, but $r$ samples are generated from the unconditional GPT-2 LM. The best sample is then chosen based on its log-likelihood score for the desired attribute and filtered based on its Dist score (to avoid repetitiveness). This evaluates the effect of post-generation selection.
• BC (Baseline + Control): Text is generated by applying the PPLM method: updating the latent representations ($\widetilde{H}_t$) using gradients from the attribute model at each step, and then sampling once. This evaluates the core gradient-based steering mechanism.
• BCR (Baseline + Control + Reranking): Combines the BC approach with reranking. Latent representations are updated, $r$ samples are generated, and the best sample is chosen based on log-likelihood score after filtering for Dist scores. This is the full PPLM approach.

5.3.2. External Baselines

• CTRL (Keskar et al., 2019):

  • Description: A large conditional Transformer Language Model (1.6 billion parameters) explicitly trained to generate text conditioned on over 50 different control codes (e.g., "Politics:", "Reviews Rating: 5.0").
  • Mechanism: It directly learns $p(x|a)$ by conditioning the model on specific tokens representing attributes during training.
  • Comparison Point: This is a strong baseline representing the state-of-the-art in explicitly trained conditional LMs. The paper matches CTRL control codes to PPLM attributes (e.g., "LEGAL Text:" for PPLM-BoW LEGAL topic; "Reviews Rating: 5.0" for PPLM-Discrim POSITIVE).

• GPT2-FT-RL (Ziegler et al., 2019):

  • Description: A GPT-2 LM that has been fine-tuned using Reinforcement Learning (RL) based on human preferences. This approach trains a reward model from human feedback and then fine-tunes the LM to maximize this reward.
  • Mechanism: While PPLM aims for on-the-fly control without LM training, GPT2-FT-RL represents a method where the LM is explicitly adapted for desired styles (e.g., positivity) through extensive fine-tuning.
  • Comparison Point: This baseline is specifically evaluated for POSITIVE sentiment control, as it was fine-tuned for positivity.

• WD (Weighted Decoding) (Ghazvininejad et al., 2017):

  • Description: A simple baseline that directly integrates conditioning into the decoding procedure. Instead of modifying latent states, it modifies the probabilities of the next token based on the attribute.
  • Mechanism for BoW (Topic Control): It modifies the softmax output distribution $\tilde{p}_{t+1}(w_i)$ by adding a scaled term if $w_i$ is in the BoW: $$\tilde { p } _ { t + 1 } ( w _ { i } ) = p _ { t + 1 } ( w _ { i } ) + \tau \mathbb { 1 } _ { \mathrm { B o W } } ( w _ { i } ) p _ { t + 1 } ( w _ { i } )$$ where $\tau$ is a scaling factor and $\mathbb{1}_{\mathrm{BoW}}(w_i)$ is an indicator function.
  • Mechanism for Discriminator (Sentiment Control): It samples from a distribution $\tilde{p}_{t+1}(w_i)$ proportional to the attribute model's probability p(a=\hat{a} | x_{0:t}, w_i) multiplied by the LM's probability $p_{t+1}(w_i)$: $$\tilde { p } _ { t + 1 } ( w _ { i } ) \propto p ( a = \hat { a } | x _ { 0 : t } , w _ { i } ) p _ { t + 1 } ( w _ { i } )$$
  • Comparison Point: This baseline directly shows the limitations of controlling output distributions without altering the underlying latent representations.

5.4. Hyperparameters

The paper lists the specific hyperparameters used for different tasks in Table S18. These parameters control the strength of attribute influence, fluency preservation, and optimization process.

The following are the results from [Table S18] of the original paper:

Key hyperparameters include:

• $m$: Number of gradient update steps per token generation.

• $λkl$: Weight for the KL divergence term to maintain fluency.

• $α$: Step size for gradient ascent in latent space (strength of attribute control).

• $γ$: Scaling coefficient for gradient normalization.

• $γgm$: Weight for geometric mean fusion to balance modified and unmodified distributions.

• $r$: Number of samples generated for reranking.

• $τ$: Threshold for Dist score to filter out repetitive samples.

• Sampling Strategy: For all reported results, top-k sampling with $k=10$ is used to draw from the softmax distribution over the vocabulary. This prevents language degeneration that can occur with greedy decoding or pure beam search.

6. Results & Analysis

The experimental results consistently demonstrate PPLM's effectiveness in controlling text generation attributes while largely preserving fluency. The ablation studies highlight the significant impact of latent space manipulation, and comparisons to strong baselines confirm PPLM's competitive performance without the need for LM retraining.

6.1. Core Results Analysis

6.1.1. BoW Attribute Models (Topic Control)

The paper evaluated PPLM's ability to control text generation towards 7 different topics using Bag-of-Words (BoW) attribute models. The results show PPLM can effectively steer the LM towards desired topics.

The following are the results from [Table 4] of the original paper:

Analysis of Ablation Study (Table 4):

• Topic Relevance: The BCR (full PPLM) and BC (latent update only) variants show significantly higher topic relevance (51.7% and 46.9% respectively) compared to the baseline B (11.1%) and BR (reranking only, 15.8%). This highlights that gradient-based latent updates are the primary driver of attribute control, far more effective than just reranking existing samples.
• Fluency: BCR maintains a respectable fluency score of 3.52, close to the baseline B (3.60) and BR (3.68), indicating minimal degradation. BC shows a slight drop (3.39), suggesting that reranking (R) helps recover some fluency.
• Diversity: Dist-1, Dist-2, and Dist-3 scores are similar across all four ablation approaches, indicating that PPLM does not significantly compromise the diversity of generated text.
• Perplexity: BC and BCR show slightly higher perplexity (43.62 and 44.04) than $B$ (39.85) and BR (38.39), which is expected as steering the model away from its natural distribution might increase its "surprise" to an external LM. However, the increase is modest.

Comparison with Baselines (Table 4):

• PPLM-BCR (51.7% / 56.0%) performs comparably to CTRL (50.0%) in topic relevance and often outperforms WD (35.7%).
• PPLM-BCR maintains fluency scores comparable to CTRL and WD.
• CTRL shows lower perplexity (24.48) which is expected as it's a large model explicitly trained for conditional generation. However, PPLM achieves comparable attribute control without this extensive training.

Qualitative Examples (Table 3): The following are the results from [Table 3] of the original paper:

Table 3 shows effective topic control (e.g., [Military], [Space], [Science]) from a single prefix "The issue focused". It also illustrates that optimizing for specific BoW words (highlighted brightly) leads to the generation of related topical words not in the BoW (softly highlighted), suggesting a deeper conceptual steering.

Variance Across Topics (Table S8): The following are the results from [Table S8] of the original paper:

Table S8 provides detailed per-topic results. It shows that some topics (e.g., Religion, Politics, Science, Computers) are easier to control, achieving high attribute relevance (e.g., Politics BCR at 69.4%, Science BCR at 61.7%), while others (Military, Space) are more challenging. Even for challenging topics, BCR generally significantly outperforms $B$ and BR. The human evaluation for topic relevance is further visualized in Figure S3.

该图像是图表，展示了不同主题的相关性，分别为基线（B）、基线加重新排序（BR）、梯度（BC）和梯度加重新排序（BCR）的比较。可以看到，采取PPLM梯度步骤在不同主题上的相关性上有显著差别，特别是在政治和科学等主题中表现突出。

Figure S3 clearly shows the substantial gain in topic relevance from applying the PPLM gradient step ($B$ to BC) and the additional benefit of reranking (BC to BCR).

6.1.2. Discriminator Attribute Models (Sentiment Control)

PPLM was also evaluated for sentiment control using a single-layer discriminator trained on the SST-5 dataset.

The following are the results from [Table 6] of the original paper:

Analysis of Ablation Study (Table 6):

• Sentiment Accuracy: BCR achieves very high sentiment accuracy (73.7% human, 78.8% external classifier), a significant leap from $B$ (19.3% human). Notably, for sentiment control, reranking (BR at 41.5%) already provides a considerable gain, but combining it with latent manipulation (BCR) further boosts accuracy substantially.
• Fluency: BCR maintains fluency (3.29 human) that is comparable to $B$ (3.54) and BR (3.65). BC shows a more noticeable drop in fluency (2.79), suggesting that reranking is particularly important for maintaining fluency when using discriminator models.
• Diversity: Dist-1/2/3 scores remain generally consistent, although BC sees a slight drop in diversity, which is recovered by BCR.
• Perplexity: Similar to topic control, perplexity for BCR (46.6) is slightly higher than $B$ (42.1).

Comparison with Baselines (Table 6):

• PPLM-BCR (73.7% / 70.0%) performs comparably to CTRL (76.7%) in sentiment accuracy based on human evaluation.
• Crucially, PPLM-BR, BC, and BCR all significantly outperform GPT2-FT-RL (13.3% human accuracy), a GPT-2 model specifically fine-tuned for positivity. This is a strong indicator of PPLM's efficiency, achieving better attribute control without training the base LM.
• PPLM-BCR also significantly outperforms WD (18.9% human accuracy).

Qualitative Examples (Table 5): The following are the results from [Table 5] of the original paper:

Table 5 shows triplets of generated samples: uncontrolled, POSITIVE, and NEGATIVE. It demonstrates clear sentiment shifts (e.g., "delicious wonderfully moist" vs. "chickenpox epidemic may be over but the flu is about to get worse") while maintaining local coherence, even with diverse prefixes like "The chicken" and "The country."

6.1.3. Language Detoxification

The paper demonstrates PPLM's potential for language detoxification. By plugging in a toxicity classifier (trained on Jigsaw Toxic Comment Classification Challenge) as the attribute model and following its negative gradient, PPLM can steer generation away from toxic content.

The following are the results from [Table S23] of the original paper:

• For natural prompts (e.g., "Asians", "Jews", "Blacks"), the toxicity percentages from GPT-2 are initially 10%, 12%, and 8%. With PPLM detoxification, these percentages are reduced to 6%, 4%, and 10% respectively, showing a general trend of reduction, though not always statistically significant (P-values > 0.05).
• For adversarial triggers (carefully crafted prompts designed to elicit toxic output), GPT-2's toxicity percentage averages 63.6%. PPLM drastically reduces this to an average of 4.6%, with high statistical significance (P-value of 1.63E-86), demonstrating strong detoxification capabilities against targeted attacks. This highlights PPLM's practical value in mitigating harmful LM biases.

6.1.4. Controlled Story Writing

PPLM was applied to assistive story writing by filling in the blanks between predefined story skeletons (e.g., "Once upon a time... Every day... But, one day... Because of that... Until, finally... And, ever since then..."). This demonstrates PPLM's ability to adhere to structural constraints while generating attribute-controlled content. Examples in Tables S20 and S21 show how PPLM can produce coherent stories under topics like [Fantasy], [Religion], or [Politics] within a given narrative structure.

The following are the results from [Table S20] of the original paper:

6.1.5. Fine-Grained Control with PPLM-BoW

Table S17 demonstrates the effect of tuning the step size $\alpha$ (the strength parameter). Increasing $\alpha$ leads to a stronger influence of the BoW on the generated text, making it more topical.

The following are the results from [Table S17] of the original paper:

For [Science] topic with prefix "The potato", as $\alpha$ increases from 0.001 to 0.05, the text becomes increasingly focused on scientific concepts, but at very high $\alpha$ (0.1), it starts to show degeneration (e.g., repeating "experiment"). This highlights the importance of tuning hyperparameters for optimal balance.

6.1.6. Combining Multiple Controllers

PPLM's flexibility allows combining multiple attribute models, even of different types (e.g., BoW and discriminator). Table S22 illustrates this by controlling generation towards a mixed topic (e.g., WINTER, POLITICS, KITCHEN) while simultaneously maintaining a POSITIVE sentiment. This combinatorial capability enables more nuanced and creative control.

The following are the results from [Table S22] of the original paper:

6.2. Data Presentation (Tables)

The following are the results from [Table 1] of the original paper:

The following are the results from [Table 2] of the original paper:

The following are the results from [Table 3] of the original paper:

The following are the results from [Table 4] of the original paper:

The following are the results from [Table 5] of the original paper:

The following are the results from [Table 6] of the original paper:

The following are the results from [Table S7] of the original paper:

The following are the results from [Table S8] of the original paper:

The following are the results from [Table S9] of the original paper:

The following are the results from [Table S10] of the original paper:

The following are the results from [Table S11] of the original paper:

The following are the results from [Table S12] of the original paper:

The following are the results from [Table S13] of the original paper:

The following are the results from [Table S14] of the original paper:

The following are the results from [Table S15] of the original paper:

The following are the results from [Table S16] of the original paper:

The following are the results from [Table S17] of the original paper:

The following are the results from [Table S18] of the original paper:

The following are the results from [Table S19] of the original paper:

The following are the results from [Table S20] of the original paper:

The following are the results from [Table S21] of the original paper:

The following are the results from [Table S22] of the original paper:

The following are the results from [Table S23] of the original paper:

The following are the results from [Table S24] of the original paper:

The following are the results from [Table S25] of the original paper:

The following are the results from [Table S26] of the original paper:

The following are the results from [Table S27] of the original paper:

6.3. Ablation Studies / Parameter Analysis

The ablation study, detailed in Tables 4 and 6, clearly validates the design choices of PPLM:

• Impact of Latent Space Manipulation (BC vs. $B$ / BR): In both BoW (topic) and Discriminator (sentiment) tasks, BC (gradient-based control) shows a dramatic increase in attribute relevance compared to $B$ (no control) or BR (reranking only). This confirms that directly manipulating the LM's latent representations is the most effective mechanism for controlling attributes. For topic control, BC's relevance jumped from 11.1% ($B$) to 46.9%. For sentiment, from 19.3% ($B$) to 39.6%.
• Contribution of Reranking (BCR vs. BC / BR vs. $B$):
  • Reranking alone (BR) provides some improvement over baseline ($B$), but it is limited (e.g., 15.8% for topic, 41.5% for sentiment).
  • When combined with latent manipulation (BCR), reranking consistently further improves attribute relevance (e.g., from 46.9% to 51.7% for topic, 39.6% to 73.7% for sentiment) and helps maintain fluency. This suggests that latent manipulation steers the generation into a better region of the LM's output space, from which reranking can then pick the optimal sample.
• Fluency Trade-offs: While BC can sometimes lead to a slight drop in fluency (e.g., 3.39 for BoW, 2.79 for Discrim), the full PPLM-BCR approach largely mitigates this, achieving fluency scores close to or sometimes even better than the baseline. This indicates that the KL divergence regularization and geometric mean fusion effectively balance attribute control with coherence.
• Parameter Sensitivity ($α$): The fine-grained control demonstrated in Table S17 highlights that the step size $\alpha$ is a crucial "control knob." Increasing $\alpha$ leads to stronger attribute alignment, but excessively high values can cause text degeneration (e.g., repetition of keywords). This indicates the need for careful tuning of hyperparameters for specific tasks and desired levels of control.
• Early Stopping (S11.1): Table S19 shows that early stopping of latent updates after a certain number of steps can prevent the model from degenerating into repetitive text, especially when $α$ is set higher. This is a practical heuristic to balance control and fluency.

6.4. Fluency Distribution

Figures S5 and S6 illustrate the distribution of human-evaluated fluency scores for PPLM-BoW and PPLM-Discrim, respectively.

该图像是一个直方图，展示了通过 PPLM-BoW 方法生成的文本在四种方法下的流畅性得分分布。不同方法对应的得分分布相似，表明其流畅性评分在1到5的范围内的分布情况。

该图像是柱状图，展示了基于 PPLM-Discrim 的流畅度评分的分布。图中显示了四种方法（baseline, gradient, baseline+rereanking, gradient+rereanking）的评分分布相似，均衡地覆盖了各个评分区间。虚线表示评分的均值。

Both histograms show that the fluency scores across all four ablation methods (B, BR, BC, BCR) are similarly distributed. This visually corroborates the quantitative results that PPLM (especially BCR) can achieve significant attribute control without severely degrading the overall fluency or naturalness of the generated text, as perceived by human annotators. The distributions are generally skewed towards higher fluency scores, indicating good overall text quality.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces the Plug and Play Language Model (PPLM), a novel and highly effective method for controllable text generation. PPLM addresses the significant challenge of guiding large, pre-trained language models (LMs) to generate text with specific attributes (e.g., topic, sentiment) without the prohibitive costs and inflexibility of retraining or fine-tuning the base LM.

The core innovation lies in its gradient-based sampling mechanism, which combines a fixed, pre-trained LM with one or more lightweight, easily configurable attribute classifiers. These classifiers, ranging from simple Bag-of-Words lists to small single-layer discriminators, guide text generation by iteratively pushing the LM's hidden activations (latent representations) in a desired direction. Mechanisms like KL divergence regularization and geometric mean fusion are employed to ensure the generated text remains fluent and coherent.

Extensive evaluations, both automated and human-annotated, confirm PPLM's ability to produce text that is strongly aligned with desired attributes while maintaining high fluency and diversity. PPLM's performance is shown to be competitive with, and often superior to, existing conditional LMs (like CTRL) and fine-tuned GPT-2 models, despite PPLM not requiring any training of the base LM. Its flexibility allows for combinatorial control of multiple attributes and novel applications such as language detoxification and constrained story writing.

7.2. Limitations & Future Work

The authors discuss several aspects related to limitations and future work:

• Ethical Considerations (Section S6): The paper acknowledges the dual-use nature of controllable LMs. While PPLM can be used for language detoxification, the same mechanism could potentially be exploited for generating more toxic language. This highlights an inherent ethical challenge with general-purpose machine learning technologies, emphasizing the need for responsible development and deployment.
• Hyperparameter Tuning: While PPLM is flexible, achieving optimal control often requires tuning hyperparameters like step size $\alpha$, KL divergence weight $\lambda_{KL}$, and geometric mean fusion weight $\gamma_{gm}$. The paper provides recommended ranges, but task-specific tuning remains a practical consideration.
• Degeneration at High Control Strength: As observed with very high $\alpha$ values (Table S17), aggressive steering can still lead to text degeneration (e.g., repetition). Mitigation strategies like early stopping of latent updates help but indicate a trade-off inherent in strong control.
• Computational Cost of Latent Updates: While not retraining the LM, performing multiple gradient updates ($m$ iterations) at each token generation step does add computational overhead during inference compared to standard unconditional sampling. However, this is still orders of magnitude cheaper than full LM retraining.
• Attribute Model Simplicity: The paper successfully demonstrates control with very simple attribute models. Future work could explore more complex or expressive attribute models and their impact on control fidelity and fluency.
• Beyond Examples: The authors explicitly state that the flexibility of PPLM allows for diverse and creative applications beyond the examples given, suggesting that exploring these new use cases is a direction for future research.

7.3. Personal Insights & Critique

The PPLM paper presents an elegant and impactful solution to controllable text generation. My personal insights and critiques are as follows:

• Elegance of Latent Space Steering: The most striking aspect is the conceptual elegance of steering text generation through gradient-based manipulation of latent representations. Instead of trying to force discrete outputs or retrain massive models, PPLM subtly nudges the "thought process" of the LM. This allows the inherent fluency and generative capacity of the pre-trained model to shine through, merely guided by external signals. This approach, inspired by PPGN, clearly translates well to NLP.
• True "Plug and Play" Value: The ability to "plug in" any differentiable attribute model is a significant practical advantage. This democratizes controlled generation, moving it from requiring multi-billion-parameter training to the much simpler task of training a small classifier or even curating a word list. This dramatically lowers the barrier to entry for researchers and developers wanting to experiment with new control types.
• Efficiency and Sustainability: In an era where large language models are computationally expensive to train and fine-tune, PPLM offers a far more efficient and sustainable pathway to customization. This is particularly relevant as models continue to grow in size.
• Balancing Act is Key: The paper effectively highlights the delicate balance between attribute control and fluency. The use of KL divergence and geometric mean fusion is critical for preventing text degeneration and maintaining the quality inherent in the base LM. The hyperparameters become vital "control knobs" for navigating this trade-off.
• Potential for Broader Applications: Beyond the demonstrated topic and sentiment control, the framework's generality (any differentiable attribute model) opens doors for controlling highly abstract text properties, like formality, humor, complexity, or even domain-specific constraints, with minimal effort. This could be applied to fields like creative writing assistance, tailored content generation for different audiences, or even automated scientific abstract writing.
• Critique - Implicit Attribute Learning: While BoW models are explicit, discriminator-based attribute models implicitly learn the attribute. The LM might leverage spurious correlations in the discriminator's training data. This is a general challenge with discriminator-guided generation and not unique to PPLM, but it's a factor to consider for robust, unbiased control.
• Critique - Interpretability of Latent Updates: While effective, the exact impact of gradient updates in high-dimensional latent space on the generated text is not always transparent. Understanding why specific words are generated or how complex attribute interactions play out could be an area for future research in interpretability.
• Unverified Assumptions / Future Work Ideas:

  • Dynamic Attribute Models: Could the attribute models themselves be dynamically updated or learned during generation based on user feedback, forming a more interactive and adaptive control system?

  • More Sophisticated Fluency Metrics: While perplexity and n-gram diversity are standard, more advanced measures of semantic coherence or stylistic consistency could provide deeper insights into fluency preservation.

  • Cross-Lingual Control: Exploring PPLM's applicability to multilingual models or cross-lingual attribute control (e.g., generate German text with English sentiment classifier).

  • Robustness to Adversarial Attribute Models: Investigating the robustness of PPLM against intentionally misleading or poorly trained attribute models.

    In conclusion, PPLM provides a powerful, flexible, and resource-efficient paradigm for controllable text generation, significantly advancing the usability and applicability of large pre-trained language models across diverse tasks. Its core idea is a testament to the power of latent space manipulation and the elegance of "plug-and-play" modularity in modern AI systems.