Bibliographic Information

• Title: Flow Matching for Generative Modeling
• Authors: Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matt Le
• Journal/Conference: Published on arXiv, a preprint server, and associated with Meta AI (FAIR) and Weizmann Institute of Science. While not a peer-reviewed journal or conference proceeding directly, arXiv preprints are widely circulated and cited in the machine learning community, and the affiliations indicate top-tier research institutions.
• Publication Year: 2022
• Abstract: The paper introduces a novel generative modeling paradigm called Flow Matching (FM) for training Continuous Normalizing Flows (CNFs) at an unprecedented scale. FM is a simulation-free approach that trains CNFs by regressing the vector fields of fixed conditional probability paths. This method is compatible with a broad family of Gaussian probability paths, which includes existing diffusion paths as special cases. When applied to diffusion paths, FM offers a more robust and stable alternative for training diffusion models. Crucially, Flow Matching enables the use of other, non-diffusion probability paths, with a particular emphasis on Optimal Transport (OT) displacement interpolation. OT paths are shown to be more efficient than diffusion paths, leading to faster training and sampling, and improved generalization. Experimental results on ImageNet demonstrate that training CNFs with Flow Matching consistently achieves better performance in terms of both likelihood and sample quality compared to alternative diffusion-based methods, while enabling fast and reliable sample generation using off-the-shelf numerical ODE solvers.
• Original Source Link: https://arxiv.org/abs/2210.02747
• PDF Link: https://arxiv.org/pdf/2210.02747v2.pdf

Executive Summary

Background & Motivation (Why)

The paper addresses two key challenges in the field of generative modeling, particularly concerning Continuous Normalizing Flows (CNFs) and diffusion models:

1. Scalability of CNFs: CNFs are a powerful class of generative models capable of modeling arbitrary probability paths. However, their training typically involves expensive numerical Ordinary Differential Equation (ODE) simulations (e.g., maximum likelihood training), which makes them difficult to scale to high-dimensional data like images. Existing simulation-free methods for CNFs often suffer from intractable integrals or biased gradients.

2. Limitations of Diffusion Models: Recent advancements in generative modeling have largely been driven by diffusion-based models, which offer scalable and relatively stable training. However, diffusion processes impose restrictions on the sampling probability paths, often leading to very long training times and requiring specialized techniques for efficient sampling. Their underlying stochastic differential equations (SDEs) implicitly define probability paths, which can be suboptimal for training and sampling efficiency.

   The paper's novel approach, Flow Matching (FM), aims to overcome these limitations by providing an efficient, simulation-free method for training CNFs. It directly works with probability paths, offering greater flexibility and control than diffusion-based methods.

Main Contributions / Findings (What)

The primary contributions of this work are:

• Introduction of Flow Matching (FM): A novel, simulation-free objective for training CNFs. FM works by regressing a neural network's vector field ($v_t$) onto a target vector field ($u_t$) that generates a desired probability path ($p_t$). This objective is simple, intuitive, and allows for scalable CNF training beyond the constraints of diffusion processes.
• Conditional Flow Matching (CFM): A practical formulation of FM that allows for efficient, unbiased training. By constructing target probability paths and vector fields using per-example (conditional) formulations, CFM enables sampling unbiased estimates of the loss without requiring explicit knowledge of intractable marginal probability paths or vector fields. It is proven that CFM and FM have identical gradients.
• General Family of Gaussian Probability Paths: Flow Matching is compatible with a broad class of Gaussian conditional probability paths, which subsumes existing diffusion paths (e.g., Variance Exploding (VE) and Variance Preserving (VP) paths) as special instances. This demonstrates FM's generality and provides a more robust and stable alternative for training models that use diffusion-like paths.
• Optimal Transport (OT) Conditional Paths: The paper introduces a particularly effective instance of non-diffusion probability paths based on Optimal Transport (OT) displacement interpolation. These OT paths are simpler (forming straight-line trajectories), leading to faster training, faster sample generation, and better generalization compared to diffusion paths.
• State-of-the-Art Performance: Empirically, models trained with Flow Matching using OT paths achieve consistently better performance on ImageNet (in terms of likelihood and sample quality, measured by NLL/BPD and FID) than alternative diffusion-based methods. They also offer better trade-offs between computational cost (fewer Number of Function Evaluations (NFEs)) and sample quality, enabling fast and reliable sample generation with standard ODE solvers.
• Improved Sampling Efficiency: OT paths result in more efficient sampling, requiring fewer NFEs for comparable numerical error and sample quality, and generating coherent images earlier in the sampling process compared to diffusion paths.

Prerequisite Knowledge & Related Work

To fully grasp the contributions of "Flow Matching for Generative Modeling," a foundational understanding of several key concepts in deep learning and generative models is essential.

Foundational Concepts

• Generative Models: These are a class of machine learning models that learn the underlying distribution of a dataset to generate new data samples that resemble the training data. Examples include images, text, or audio.
• Probability Density Function (PDF) / Probability Density Path: A PDF, denoted p(x), describes the likelihood of a random variable $x$ taking on a given value. In this paper, a probability density path, $p_t(x)$, represents a sequence of PDFs that evolve over time $t \in [0, 1]$, smoothly transforming one distribution (e.g., simple noise at $t=0$) into another (e.g., the complex data distribution at $t=1$).
• Vector Field: A vector field, denoted v(x), assigns a vector to each point $x$ in a space. In the context of CNFs, a time-dependent vector field $v_t(x)$ describes the instantaneous velocity at which data points move through the probability space as time $t$ progresses.
• Ordinary Differential Equation (ODE): An ODE is an equation that relates a function to its derivatives. In CNFs, the evolution of data points from a simple distribution to a complex one is modeled as an ODE where the rate of change of a data point $\phi_t(x)$ is given by the vector field $v_t(\phi_t(x))$. $$\frac{d}{dt} \phi_t(x) = v_t(\phi_t(x)) \\ \phi_0(x) = x$$ Here, $\phi_t(x)$ represents the position of an initial point $x$ at time $t$ as it flows through the vector field $v_t$.
• Continuous Normalizing Flows (CNFs): Introduced by Chen et al. (2018), CNFs are generative models that transform a simple base distribution (e.g., a standard Gaussian) into a complex target data distribution using a continuous, invertible transformation (a "flow"). This transformation is defined by an ODE where the vector field is parameterized by a neural network. CNFs allow for exact likelihood computation and flexible model architectures.
• Push-forward Equation / Change of Variables Formula: This fundamental concept in probability theory relates the PDF of a random variable $X$ to the PDF of a transformed random variable $Y = f(X)$. For CNFs, it describes how the initial probability distribution $p_0$ is transformed into $p_t$ by the flow $\phi_t$. $p_t = [\phi_t]_* p_0$ where the push-forward operator is defined by $$[\phi_t]_* p_0 (x) = p_0(\phi_t^{-1}(x)) \operatorname{det}\left[\frac{\partial \phi_t^{-1}}{\partial x}(x)\right]$$ Here, $p_0(\phi_t^{-1}(x))$ is the density of the initial point that maps to $x$ at time $t$, and the determinant of the Jacobian of the inverse transformation, $\operatorname{det}[\frac{\partial \phi_t^{-1}}{\partial x}(x)]$, accounts for the change in volume (scaling) during the transformation.
• Continuity Equation: A partial differential equation (PDE) from fluid dynamics that describes the conservation of probability mass. In CNFs, it links the time evolution of a probability density $p_t(x)$ to the divergence of the probability current (density multiplied by velocity field). $$\frac{d}{dt} p_t(x) + \mathrm{div}(p_t(x) v_t(x)) = 0$$ This equation ensures that the vector field $v_t(x)$ correctly generates the probability path $p_t(x)$. The divergence operator div is defined as $\sum_{i=1}^d \frac{\partial}{\partial x^i}$.
• Diffusion Models: A class of generative models (e.g., Ho et al. (2020), Song et al. (2020b)) that learn to reverse a gradual diffusion process. They start by corrupting data with Gaussian noise over many steps until it becomes pure noise, then learn to reverse this process, typically modeled as a stochastic differential equation (SDE), to generate data from noise.
• Denoising Score Matching: A training objective (Vincent, 2011) used by many diffusion models. It involves training a neural network to predict the noise added to a noisy data sample. This implicitly learns the score function ($\nabla \log p(x)$) of the data distribution, which is crucial for reversing the diffusion process.
• Optimal Transport (OT): A mathematical theory (McCann, 1997; Villani, 2009) concerned with finding the most efficient way to transform one probability distribution into another. The "optimal" part usually refers to minimizing some cost function (e.g., squared Euclidean distance for Wasserstein-2 distance). OT displacement interpolation refers to the geodesic path between two distributions in the space of probability measures.

Previous Works

• Original CNFs (Chen et al., 2018): These models were groundbreaking for their continuous nature and exact likelihood computation but faced scalability issues due to the computational cost of solving ODEs for training (both forward and backward propagation).
• Regularization for CNFs: Several works attempted to make CNF training more tractable by regularizing the ODE (e.g., Dupont et al., 2019; Finlay et al., 2020), but these efforts did not fundamentally change the sequential nature of ODE simulations.
• Simulation-Free CNF Training: Prior attempts at simulation-free CNF training, such as those by Rozen et al. (2021) and Ben-Hamu et al. (2022), either involved intractable integrals or suffered from biased gradients, preventing their widespread adoption and scalability. Flow Matching addresses these limitations by providing an unbiased, scalable simulation-free approach.
• Score-Based Generative Models / Diffusion Models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019; Song et al., 2020b): These models gained prominence due to their scalability and high-quality sample generation. They train a neural network to estimate the score function (gradient of the log-probability density) of noisy data distributions. While powerful, they are constrained by the specific diffusion processes chosen, which can lead to inefficient sampling or less direct control over the probability path. Flow Matching draws inspiration from the conditional objective of denoising score matching but generalizes it to directly match vector fields.
• Improvements in Diffusion Models: Works like loss-rescaling (Song et al., 2021), classifier guidance (Dhariwal & Nichol, 2021), and learning noise schedules (Nichol & Dhariwal, 2021; Kingma et al., 2021) have enhanced diffusion models, but often within the confines of Gaussian conditional paths defined by simple diffusion processes.

Technological Evolution

The field has evolved from GANs (Goodfellow et al., 2014) and early Normalizing Flows (Rezende & Mohamed, 2015) to CNFs for continuous transformations, and then to Diffusion Models for their stability and scalability. While CNFs offered theoretical elegance and exact likelihoods, their computational cost hindered their application to complex, high-dimensional data. Diffusion models offered a practical workaround, enabling impressive image generation but with specific constraints on the underlying stochastic processes and often slow sampling. This paper represents a convergence: it aims to imbue CNFs with the scalability of diffusion models while retaining the flexibility to define more general and efficient probability paths, effectively breaking the "diffusion barrier" for scalable CNF training.

Differentiation

• From CNF Maximum Likelihood: Flow Matching is simulation-free, unlike traditional CNF training that requires expensive ODE simulations for both forward and backward passes. This makes FM significantly more scalable.
• From Prior Simulation-Free CNF Methods: FM offers unbiased gradients and scales effectively to high dimensions, addressing the limitations (intractable integrals, biased gradients) of earlier simulation-free approaches.
• From Denoising Score Matching (Diffusion Models): While inspired by denoising score matching's conditional objective, FM directly regresses vector fields rather than score functions. This allows FM to generalize beyond diffusion processes and directly specify desired probability paths. It also offers a more robust and stable training alternative even for diffusion-like paths.
• Path Generality: FM explicitly allows for the use of arbitrary conditional probability paths, rather than being implicitly constrained by the dynamics of an SDE. This is epitomized by the introduction of Optimal Transport paths, which are shown to be more efficient than diffusion paths. The paper moves away from reasoning about SDEs and directly defines the desired "flow."

Methodology (Core Technology & Implementation Details)

The core idea of Flow Matching (FM) is to train a Continuous Normalizing Flow (CNF) by directly regressing its neural network-parameterized vector field $v_t(x; θ)$ onto a target vector field $u_t(x)$ that generates a desired probability path $p_t(x)$. This sidesteps the computational challenges of traditional CNF training while offering greater flexibility than diffusion models.

Principles

The underlying principle of Flow Matching is to match the instantaneous velocities of a learned flow to a predefined "ideal" flow. This ideal flow transforms a simple prior distribution ($p_0$) into the complex data distribution ($q$) through a probability path $p_t(x)$. By minimizing the squared difference between the learned vector field and this target vector field, the CNF learns to generate the desired data distribution efficiently.

The key insight is that while the overall (marginal) target probability path $p_t(x)$ and its corresponding vector field $u_t(x)$ might be intractable to compute directly (as they depend on the unknown data distribution $q(x_1)$), they can be constructed from simpler conditional probability paths $p_t(x|x_1)$ and conditional vector fields $u_t(x|x_1)$. These conditional components are defined for each individual data sample $x_1$ and are tractable.

Steps & Procedures

1. Define the Flow Matching Objective: The general Flow Matching (FM) objective aims to match the learned CNF vector field $v_t(x)$ (parameterized by $\theta$) to a target vector field $u_t(x)$ that generates the target probability path $p_t(x)$: $$\mathcal{L}_{\mathtt{FM}}(\theta) = \mathbb{E}_{t, p_t(x)} \|v_t(x) - u_t(x)\|^2$$

   • $\mathcal{L}_{\mathtt{FM}}(\theta)$: The Flow Matching loss function, which depends on the learnable parameters $\theta$ of the CNF's neural network.

   • $\mathbb{E}_{t, p_t(x)}$: Expectation taken over time $t$ (uniformly sampled from [0, 1]) and samples $x$ drawn from the probability path $p_t(x)$.

   • $v_t(x)$: The vector field learned by the CNF neural network at time $t$ and position $x$.

   • $u_t(x)$: The target vector field that defines the desired probability path $p_t(x)$.

   • $\|\cdot\|^2$: The squared Euclidean (L2) norm, measuring the difference between the learned and target vector fields.

     This objective is initially intractable because $p_t(x)$ and $u_t(x)$ are unknown and depend on the data distribution $q(x_1)$.

2. Constructing Target $p_t(x)$ and $u_t(x)$ from Conditional Paths: To make the objective tractable, the paper proposes constructing the marginal probability path $p_t(x)$ and its vector field $u_t(x)$ by averaging over conditional paths:

   • Conditional Probability Path: For each data sample $x_1 \sim q(x_1)$, a conditional probability path $p_t(x|x_1)$ is defined such that it starts as a simple distribution (e.g., standard Gaussian $p(x) = \mathcal{N}(x|0, I)$) at $t=0$ and becomes concentrated around $x_1$ (e.g., $\mathcal{N}(x|x_1, \sigma_{\min}^2 I)$) at $t=1$.
   • Marginal Probability Path: The overall target probability path $p_t(x)$ is obtained by marginalizing these conditional paths over the data distribution $q(x_1)$: $$p_t(x) = \int p_t(x|x_1) q(x_1) dx_1$$ At $t=1$, $p_1(x) \approx q(x)$, thus bridging noise to data.
   • Marginal Vector Field: Similarly, a marginal vector field $u_t(x)$ is derived by a weighted average of conditional vector fields $u_t(x|x_1)$: $$u_t(x) = \int u_t(x|x_1) \frac{p_t(x|x_1) q(x_1)}{p_t(x)} dx_1$$ Theorem 1 formally proves that this marginal vector field $u_t(x)$ indeed generates the marginal probability path $p_t(x)$.

3. Introduce Conditional Flow Matching (CFM) Objective: Despite the theoretical elegance, $p_t(x)$ and $u_t(x)$ still involve intractable integrals. The key innovation is Conditional Flow Matching (CFM), which is a modified training objective: $$\mathcal{L}_{\mathtt{CFM}}(\theta) = \mathbb{E}_{t, q(x_1), p_t(x|x_1)} \big|\big| v_t(x) - u_t(x|x_1) \big|\big|^2$$

   • $\mathbb{E}_{t, q(x_1), p_t(x|x_1)}$: Expectation taken over time $t$, data sample $x_1 \sim q(x_1)$, and sample $x$ drawn from the conditional probability path $p_t(x|x_1)$.

   • $u_t(x|x_1)$: The conditional target vector field that generates $p_t(x|x_1)$.

     This CFM objective is tractable because:

   • We can sample $x_1$ from the training data.

   • We can easily sample $x$ from $p_t(x|x_1)$ and compute $u_t(x|x_1)$ since these are defined for a single $x_1$ and $t$. Theorem 2 proves that, in expectation, CFM and FM have identical gradients. This means optimizing CFM effectively optimizes the overall FM objective, learning the marginal data distribution without ever explicitly computing it.

4. Define General Gaussian Conditional Probability Paths: The paper specifies a general family of Gaussian conditional probability paths: $$p_t(x|x_1) = \mathcal{N}(x \mid \mu_t(x_1), \sigma_t(x_1)^2 I)$$

   • $\mathcal{N}(\cdot \mid \mu, \Sigma)$: A Gaussian distribution with mean $\mu$ and covariance $\Sigma$. Here, $\Sigma = \sigma_t(x_1)^2 I$ means an isotropic Gaussian (spherical covariance).

   • $\mu_t(x_1)$: The time-dependent mean, dependent on the data sample $x_1$.

   • $\sigma_t(x_1)$: The time-dependent scalar standard deviation (std), also dependent on $x_1$.

     These paths are set to satisfy boundary conditions:

   • At $t=0$: $\mu_0(x_1) = 0$, $\sigma_0(x_1) = 1$, so $p_0(x|x_1) = \mathcal{N}(x|0, I)$ (standard Gaussian noise).

   • At $t=1$: $\mu_1(x_1) = x_1$, $\sigma_1(x_1) = \sigma_{\min}$ (a sufficiently small value), so $p_1(x|x_1)$ is a concentrated Gaussian around $x_1$.

     The corresponding flow map $\psi_t(x)$ that transforms a standard Gaussian noise $x_0 \sim \mathcal{N}(0, I)$ to a sample $x \sim p_t(x|x_1)$ is given by: $$\psi_t(x_0) = \sigma_t(x_1) x_0 + \mu_t(x_1)$$ Theorem 3 then derives the unique conditional vector field $u_t(x|x_1)$ that generates this Gaussian path: $$u_t(x|x_1) = \frac{\sigma_t'(x_1)}{\sigma_t(x_1)} \left(x - \mu_t(x_1)\right) + \mu_t'(x_1)$$

   • $\sigma_t'(x_1)$ and $\mu_t'(x_1)$ denote the derivatives with respect to time $t$.

5. Special Instances of Conditional Probability Paths:

   • Example I: Diffusion Conditional VFs: The FM framework can recover paths used in diffusion models. For instance, the reversed (noise to data) Variance Exploding (VE) path and Variance Preserving (VP) path lead to specific choices for $\mu_t(x_1)$ and $\sigma_t(x_1)$.

     • VE Path: $\mu_t(x_1) = x_1$, $\sigma_t(x_1) = \sigma_{1-t}$. Plugging these into Theorem 3 yields $u_t(x|x_1) = - \frac{\sigma_{1-t}'}{\sigma_{1-t}} (x - x_1)$.
     • VP Path: $\mu_t(x_1) = \alpha_{1-t} x_1$, $\sigma_t(x_1) = \sqrt{1 - \alpha_{1-t}^2}$, where $\alpha_t = e^{-\frac{1}{2}T(t)}$. Plugging these into Theorem 3 yields $u_t(x|x_1) = \frac{\alpha_{1-t}'}{1-\alpha_{1-t}^2} (\alpha_{1-t} x - x_1)$. These derived vector fields coincide with those previously used in deterministic probability flows for diffusion models (Song et al., 2020b). Using FM with these paths provides a more stable training alternative to score matching.

   • Example II: Optimal Transport (OT) Conditional VFs: A novel and more efficient choice is to define $\mu_t(x_1)$ and $\sigma_t(x_1)$ to change linearly in time: $$\mu_t(x_1) = t x_1 \\ \sigma_t(x_1) = 1 - (1 - \sigma_{\min}) t$$ This results in a conditional vector field: $$u_t(x|x_1) = \frac{x_1 - (1 - \sigma_{\min}) x}{1 - (1 - \sigma_{\min}) t}$$ This OT path is characterized by straight-line trajectories (Figure 3) and a constant direction in time for its vector field (Figure 2), making it simpler to model. The flow $\psi_t(x_0)$ for this path is the Optimal Transport displacement map between the two Gaussians: $$\psi_t(x_0) = (1 - (1 - \sigma_{\min}) t) x_0 + t x_1$$ The CFM loss for OT paths becomes: $$\mathcal{L}_{\mathrm{cFM}}(\theta) = \mathbb{E}_{t, q(x_1), p(x_0)} \Big\| v_t\big(\psi_t(x_0)\big) - \Big(x_1 - (1 - \sigma_{\min}) x_0 \Big) \Big\|^2$$ This particular form simplifies the regression target.

6. Sampling and Likelihood Evaluation: Once the CNF vector field $v_t(x)$ is trained, sampling is achieved by solving the ODE $\frac{d}{dt} \phi_t(x) = v_t(\phi_t(x))$ from $t=0$ (noise) to $t=1$ (data), starting with $x_0 \sim \mathcal{N}(0, I)$. Likelihood estimation (NLL/BPD) requires solving a coupled ODE that tracks both the flow trajectory and the change in log-density, using techniques like the Hutchinson trace estimator for the divergence term (Appendix C).

Mathematical Formulas & Key Details

• Continuous Normalizing Flow (CNF) Definition (Equation 1): $$\frac{d}{dt} \phi_t(x) = v_t(\phi_t(x)) \\ \phi_0(x) = x$$

  • $\phi_t(x)$: The position of a particle at time $t$, starting from $x$ at $t=0$. It represents the flow map.
  • $v_t(x)$: The time-dependent vector field, typically parameterized by a neural network, that defines the velocity of particles at position $x$ and time $t$.

• Flow Matching (FM) Objective (Equation 5): $$\mathcal{L}_{\mathtt{FM}}(\theta) = \mathbb{E}_{t, p_t(x)} \|v_t(x) - u_t(x)\|^2$$

  • $\mathcal{L}_{\mathtt{FM}}(\theta)$: The loss function to minimize, dependent on the neural network parameters $\theta$.
  • $\mathbb{E}_{t, p_t(x)}$: Expectation over time $t \sim \mathcal{U}[0,1]$ (uniform distribution) and data points $x$ sampled from the marginal probability path $p_t(x)$.
  • $v_t(x)$: The neural network's predicted vector field.
  • $u_t(x)$: The ground-truth marginal vector field.

• Marginal Probability Path (Equation 6): $$p_t(x) = \int p_t(x|x_1) q(x_1) dx_1$$

  • $p_t(x)$: The marginal (overall) probability density at time $t$.
  • $p_t(x|x_1)$: The conditional probability density at time $t$, given a specific data sample $x_1$.
  • $q(x_1)$: The unknown data distribution.

• Marginal Vector Field (Equation 8): $$u_t(x) = \int u_t(x|x_1) \frac{p_t(x|x_1) q(x_1)}{p_t(x)} dx_1$$

  • $u_t(x)$: The marginal (overall) vector field.
  • $u_t(x|x_1)$: The conditional vector field that generates $p_t(x|x_1)$.
  • $\frac{p_t(x|x_1) q(x_1)}{p_t(x)}$: This term is equivalent to $q(x_1|x,t)$, the posterior probability of $x_1$ given $x$ at time $t$. It acts as a weighting factor.

• Conditional Flow Matching (CFM) Objective (Equation 9): $$\mathcal{L}_{\mathtt{CFM}}(\theta) = \mathbb{E}_{t, q(x_1), p_t(x|x_1)} \big|\big| v_t(x) - u_t(x|x_1) \big|\big|^2$$

  • $\mathbb{E}_{t, q(x_1), p_t(x|x_1)}$: Expectation over time $t \sim \mathcal{U}[0,1]$, data sample $x_1 \sim q(x_1)$, and sample $x \sim p_t(x|x_1)$.
  • $v_t(x)$: The neural network's predicted vector field.
  • $u_t(x|x_1)$: The ground-truth conditional vector field, which is tractable to compute.

• Gaussian Conditional Probability Path (Equation 10): $$p_t(x|x_1) = \mathcal{N}(x \mid \mu_t(x_1), \sigma_t(x_1)^2 I)$$

  • $\mathcal{N}(\cdot \mid \mu, \Sigma)$: Gaussian probability density function.
  • $\mu_t(x_1)$: Time-dependent mean, conditioned on $x_1$.
  • $\sigma_t(x_1)$: Time-dependent standard deviation, conditioned on $x_1$.
  • $I$: Identity matrix, implying an isotropic (spherical) Gaussian.

• Conditional Flow Map $\psi_t(x_0)$ (Equation 11): $$\psi_t(x_0) = \sigma_t(x_1) x_0 + \mu_t(x_1)$$

  • $\psi_t(x_0)$: The point at time $t$ resulting from transforming an initial noise sample $x_0 \sim \mathcal{N}(0,I)$.
  • $x_0$: A sample from the standard Gaussian noise distribution.

• Conditional Vector Field $u_t(x|x_1)$ (Equation 15, from Theorem 3): $$u_t(x|x_1) = \frac{\sigma_t'(x_1)}{\sigma_t(x_1)} \left(x - \mu_t(x_1)\right) + \mu_t'(x_1)$$

  • $\sigma_t'(x_1)$: Derivative of $\sigma_t(x_1)$ with respect to time $t$.
  • $\mu_t'(x_1)$: Derivative of $\mu_t(x_1)$ with respect to time $t$.

• Optimal Transport (OT) Path Parameters (Equation 20): $$\mu_t(x_1) = t x_1 \\ \sigma_t(x_1) = 1 - (1 - \sigma_{\min}) t$$

  • $t$: Time parameter, $t \in [0,1]$.
  • $x_1$: Data sample.
  • $\sigma_{\min}$: A small positive constant, the final standard deviation at $t=1$.

• OT Conditional Vector Field (Equation 21): $$u_t(x|x_1) = \frac{x_1 - (1 - \sigma_{\min}) x}{1 - (1 - \sigma_{\min}) t}$$

• Continuity Equation (Equation 26): $$\frac{d}{dt} p_t(x) + \mathrm{div}(p_t(x) v_t(x)) = 0$$

  • $\mathrm{div}(p_t(x) v_t(x))$: The divergence of the probability current $p_t(x) v_t(x)$. This equation describes the conservation of probability mass over time.

• Log-Probability ODE for CNF (Equation 27): $$\frac{d}{dt} \log p_t(\phi_t(x)) + \mathrm{div}(v_t(\phi_t(x))) = 0$$

  • This equation relates the change in log-probability density along a flow trajectory to the divergence of the vector field. It is integrated to compute likelihoods.

• Bits-Per-Dimension (BPD) (Equation 38): $$\mathrm{BPD} = \mathbb{E}_{x_1} \left[ - \frac{\log_2 p_1(x_1)}{d} \right] = \mathbb{E}_{x_1} \left[ - \frac{\log p_1(x_1)}{d \log 2} \right]$$

  • $\mathrm{BPD}$: A common metric for generative models, representing the average number of bits required to encode each dimension of a data point. Lower BPD indicates a better model.
  • $\log_2 p_1(x_1)$: Log-likelihood of a data point $x_1$ under the learned model's distribution at $t=1$.
  • $d$: Dimensionality of the data.

Experimental Setup

The experiments aim to validate Flow Matching and its Optimal Transport (OT) path formulation, comparing it against established diffusion-based generative models.

Datasets

The evaluation was performed on standard image datasets:

• CIFAR-10: A dataset of 60,000 $32 \times 32$ color images in 10 classes, with 6,000 images per class. It is a commonly used benchmark for image classification and generation tasks due to its relatively small size and moderate complexity.
• ImageNet: A large-scale hierarchical image database. The paper uses downsampled versions at resolutions of $32 \times 32$, $64 \times 64$, and $128 \times 128$ pixels.
  • ImageNet 32x32 / 64x64: These downsampled versions (e.g., from Chrabaszcz et al., 2017) are used to allow for more rapid experimentation while still representing a significant challenge compared to CIFAR-10 due to their greater diversity and number of classes.
  • ImageNet 128x128: A more challenging high-resolution benchmark, used to demonstrate the scalability and performance of the proposed method on complex image generation.
• Data Preprocessing: For images, standard practice includes center cropping and resizing to the appropriate dimension. For $32 \times 32$ and $64 \times 64$ resolutions, the same preprocessing as Chrabaszcz et al. (2017) is used. Pixel values are often transformed (e.g., from [0, 255] to $[-1, 1]$) before being fed into the neural network.

Evaluation Metrics

For every metric cited, a conceptual definition, mathematical formula (if applicable), and symbol explanation are provided.

• Negative Log-Likelihood (NLL) / Bits-Per-Dimension (BPD):

  • Conceptual Definition: NLL measures how well a generative model predicts the probability of observed data. A lower NLL indicates that the model assigns a higher probability to the real data, suggesting a better fit to the true data distribution. BPD is a normalized version of NLL often used for images, representing the average number of bits needed to encode each pixel (or dimension) of an image. Lower BPD values are better.
  • Mathematical Formula: $$\mathrm{BPD} = \mathbb{E}_{x_1} \left[ - \frac{\log_2 p_1(x_1)}{d} \right]$$
  • Symbol Explanation:
    • $\mathrm{BPD}$: Bits-Per-Dimension.
    • $\mathbb{E}_{x_1}[\dots]$: Expectation over data samples $x_1$ from the real data distribution.
    • $\log_2 p_1(x_1)$: The base-2 logarithm of the model's assigned probability density to the data sample $x_1$ at the final time $t=1$.
    • $d$: The total number of dimensions in the data sample (e.g., for a $32 \times 32$ RGB image, $d = 32 \times 32 \times 3 = 3072$).
    • In practice, the paper uses the natural logarithm $\ln p_1(x_1)$ and converts to base-2 by dividing by $\ln 2$, as shown in the paper's Appendix C, equation 38.
  • Implementation Detail: NLL is computed using an importance-weighted estimate, and dequantization (adding uniform noise to discrete pixel values to treat them as continuous) is applied as standard. dopri5 ODE solver with absolute and relative tolerances of 1e-5 is used.

• Frechet Inception Distance (FID):

  • Conceptual Definition: FID is a widely used metric to assess the quality of images generated by generative models. It measures the similarity between the feature distributions of real and generated images. Lower FID scores indicate higher quality and diversity of generated images, implying that they are closer to the real data distribution.
  • Mathematical Formula: The FID score is calculated by computing the Fréchet distance between two Gaussian distributions, one representing the features of real images and the other representing the features of generated images. These features are extracted from an intermediate layer of a pre-trained Inception-v3 model. $$\mathrm{FID} = \|\mu_r - \mu_g\|^2 + \mathrm{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$$
  • Symbol Explanation:
    • $\mu_r$: The mean feature vector of real images.
    • $\mu_g$: The mean feature vector of generated images.
    • $\|\cdot\|^2$: The squared Euclidean distance.
    • $\mathrm{Tr}(\cdot)$: The trace of a matrix.
    • $\Sigma_r$: The covariance matrix of features of real images.
    • $\Sigma_g$: The covariance matrix of features of generated images.
    • $(\Sigma_r \Sigma_g)^{1/2}$: Matrix square root.

• Number of Function Evaluations (NFE):

  • Conceptual Definition: NFE refers to the number of times the neural network (which parameterizes the vector field) must be evaluated by the ODE solver to compute a trajectory (e.g., to generate a sample or evaluate likelihood). Lower NFE indicates faster sampling or inference.
  • Importance: This is a crucial metric for CNFs and diffusion models, as ODE solver steps dominate computational cost during sampling and likelihood estimation.

• PSNR (Peak Signal-to-Noise Ratio):

  • Conceptual Definition: PSNR is used to measure the quality of reconstruction of lossy compression codecs, and in this context, image super-resolution. It is a quantitative measure of image quality where higher values generally indicate better quality (less distortion) between two images.
  • Mathematical Formula: $$\mathrm{PSNR} = 10 \cdot \log_{10} \left( \frac{\mathrm{MAX}_I^2}{\mathrm{MSE}} \right)$$
  • Symbol Explanation:
    • $\mathrm{MAX}_I$: The maximum possible pixel value of the image (e.g., 255 for 8-bit grayscale or RGB images).
    • $\mathrm{MSE}$: Mean Squared Error between the original and the reconstructed image. $$\mathrm{MSE} = \frac{1}{mn} \sum_{i=0}^{m-1} \sum_{j=0}^{n-1} [I(i,j) - K(i,j)]^2$$
      • $I$: Original image.
      • $K$: Reconstructed (upsampled) image.
      • m, n: Dimensions of the image.

• SSIM (Structural Similarity Index Measure):

  • Conceptual Definition: SSIM is a perception-based model that considers image degradation as perceived change in structural information, while also incorporating luminance and contrast masking. It is a full reference metric, meaning it measures the quality of an image with reference to a distortion-free original image. Values range from -1 to 1, with 1 indicating perfect similarity.
  • Mathematical Formula: $$\mathrm{SSIM}(x,y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$$
  • Symbol Explanation:
    • $x$: Original image window.
    • $y$: Reconstructed image window.
    • $\mu_x$: Average of $x$.
    • $\mu_y$: Average of $y$.
    • $\sigma_x^2$: Variance of $x$.
    • $\sigma_y^2$: Variance of $y$.
    • $\sigma_{xy}$: Covariance of $x$ and $y$.
    • $c_1 = (K_1 L)^2$, $c_2 = (K_2 L)^2$: Two variables to stabilize the division with weak denominators; $L$ is the dynamic range of the pixel values (e.g., $2^{bits\_per\_pixel}-1$), and $K_1, K_2$ are small constants (e.g., 0.01, 0.03).

Baselines

The paper compares Flow Matching against several prominent diffusion-based generative models:

• DDPM (Denoising Diffusion Probabilistic Models): Ho et al. (2020). This is a foundational diffusion model that uses a simple noise-prediction loss.
• Score Matching (SM): Song et al. (2020b). This baseline refers to models trained using the standard score matching objective, often applied to diffusion processes to estimate the score function.
• ScoreFlow (SF): Song et al. (2021). An improved score-based model that uses a loss-rescaling technique and is motivated by an NLL upper bound.
• GAN-based Models (for ImageNet 128x128 FID comparison):
  • MGAN (Hoang et al., 2018)
  • PacGAN2 (Lin et al., 2018)
  • Logo-GAN-AE (Sage et al., 2018)
  • Self-cond. GAN (Lui et al., 2019)
  • Uncond. BigGAN (Lui et al., 2019)
  • PGMGAN (Armandpour et al., 2021) These are included to benchmark FM-OT against non-diffusion state-of-the-art generative adversarial networks (GANs) on high-resolution image synthesis.

Model Architecture and Hyperparameters: For a fair comparison, all models (including FM with Diffusion and FM with OT) are trained using the same U-Net architecture (from Dhariwal & Nichol, 2021), identical hyperparameters, and similar numbers of training iterations. This ensures that observed performance differences are due to the training objective and probability path choice, rather than architectural advantages.

Results & Analysis

The experimental results consistently demonstrate the superior performance and efficiency of Flow Matching, particularly when combined with Optimal Transport (OT) probability paths, across various image generation tasks.

Core Results

The main comparisons are presented in Table 1, evaluating models on CIFAR-10 and ImageNet at different resolutions for likelihood (NLL/BPD), sample quality (FID), and sampling efficiency (NFE).

• Quantitative Performance (Table 1 Transcription): The following table summarizes the performance of different models on CIFAR-10 and ImageNet datasets.

  Model	NLL↓	FID↓	NFE↓	Model	NLL↓	FID↓
  Ablations				MGAN (Hoang et al., 2018)		58.9
  DDPM	3.12	7.48	274	PacGAN2 (Lin et al., 2018)		57.5
  Score Matching	3.16	19.94	242	Logo-GAN-AE (Sage et al., 2018)		50.9
  ScoreFlow	3.09	20.78	428	Self-cond. GAN (Lui et al., 2019)		41.7
  Ours				Uncond. BigGAN (Lui et al., 2019)		25.3
  FM w/ Diffusion	3.10	8.06	183	PGMGAN (Armandpour et al., 2021)		21.7
  FM w/ OT	2.99	6.35	142	FM w/ OT	2.90	20.9

  • NLL (Negative Log-Likelihood) / BPD: Lower values are better. $FM w/ OT$ consistently achieves the lowest NLL across all resolutions (2.99 on CIFAR-10, 3.53 on ImageNet 32x32, 3.31 on ImageNet 64x64, 2.90 on ImageNet 128x128). This indicates that models trained with $FM w/ OT$ learn the data distribution more accurately than diffusion-based methods.
  • FID (Frechet Inception Distance): Lower values are better. $FM w/ OT$ achieves the best FID scores across all resolutions (6.35 on CIFAR-10, 5.02 on ImageNet 32x32, 14.45 on ImageNet 64x64, 20.9 on ImageNet 128x128). This demonstrates superior sample quality compared to all baselines. On ImageNet 128x128, $FM w/ OT$ achieves state-of-the-art FID among unconditional models.
  • NFE (Number of Function Evaluations): Lower values are better for faster sampling. $FM w/ OT$ requires significantly fewer NFEs for sampling (142 on CIFAR-10, 122 on ImageNet 32x32, 138 on ImageNet 64x64), making it the most computationally efficient method at inference time.

• ImageNet 128x128 Sample Quality: For ImageNet $128 \times 128$, $FM w/ OT$ achieves an FID of 20.9, outperforming several strong GAN baselines (e.g., Uncond. BigGAN at 25.3, PGMGAN at 21.7). This highlights Flow Matching's capability to generate high-fidelity images at larger resolutions. Non-curated samples are shown in Figure 1, Figure 11, Figure 12, and Figure 13.

Ablations / Parameter Sensitivity

• Impact of Flow Matching vs. Score Matching (with Diffusion paths):

  • Comparing FM w/ Diffusion to DDPM, Score Matching, and ScoreFlow, FM w/ Diffusion generally offers competitive or better NLL and FID, while significantly reducing NFE. For example, on ImageNet 32x32, FM w/ Diffusion achieves 3.54 NLL, 6.37 FID, and 193 NFE, outperforming DDPM (3.54 NLL, 6.99 FID, 262 NFE) and Score Matching (3.56 NLL, 5.68 FID, 178 NFE - lower FID but worse NLL and higher NFE than $FM w/ OT$). This suggests that Flow Matching provides a more robust and stable training alternative even when using diffusion-style paths.

• Impact of Optimal Transport (OT) Paths:

  • Comparing $FM w/ OT$ to FM w/ Diffusion, the OT path consistently yields better performance across all metrics (NLL, FID, NFE). This is a strong empirical validation of the OT path design.
  • Faster Training: Figure 5 shows that FM-OT achieves lower FID scores faster during training on ImageNet $64 \times 64$ compared to FM-Diffusion and other baselines. This indicates faster convergence. The paper also notes that FM (with a larger model) requires significantly fewer training iterations (500k) for ImageNet-128 compared to Dhariwal & Nichol (2021) (4.36m), suggesting greater training efficiency in terms of gradient updates/image throughput.
  • Sampling Efficiency and Trajectory Characteristics:
    • Figure 6 (Sample paths) qualitatively illustrates that the OT path model begins to form recognizable images much earlier in the generation process, with noise being removed roughly linearly. In contrast, diffusion paths often retain significant noise until the very end, sometimes "overshooting" the final sample.
    • Figure 4 (left) shows that FM with OT paths introduces the checkerboard pattern (in a 2D synthetic dataset) much earlier and results in more stable training trajectories than FM with diffusion or Score Matching.
    • Figure 7 (right) quantifies this: $FM w/ OT$ maintains better FID scores at very low NFE values, indicating superior sample quality for a given computational budget. For instance, on ImageNet $32 \times 32$, $FM w/ OT$ achieves decent FID at much lower NFE than other models.
    • Figure 7 (left) also shows that $FM w/ OT$ models produce better numerical error (closer to high NFE solutions) with fewer NFEs, requiring approximately 60% fewer NFEs to reach the same error threshold as diffusion models. This implies OT paths are "easier" for ODE solvers to integrate accurately.
    • Figure 10 demonstrates that the NFE required for sampling remains constant during training for Flow Matching models, unlike score matching where it can fluctuate significantly. This ensures predictable sampling costs.

• Conditional Sampling: Table 2 demonstrates Flow Matching's applicability to conditional generation, specifically image super-resolution (upsampling from $64 \times 64$ to $256 \times 256$).

  Model	FID↓	IS↑	PSNR↑	SSIM↑
  Reference	1.9	240.8
  Regression	15.2	121.1	27.9	0.801
  SR3 (Saharia et al, 2022)	5.2	180.1	26.4	0.762
  FM / OT	3.4	200.8	24.7	0.747

  • $FM / OT$ achieves a FID of 3.4 and IS of 200.8, significantly outperforming SR3 (FID 5.2, IS 180.1), a strong diffusion-based super-resolution model. While its PSNR and SSIM are slightly lower than SR3, the paper argues that FID and IS are better indicators of generation quality. This suggests FM-OT can generate more perceptually realistic upsampled images. Examples are shown in Figures 14 and 15.

• Negative Log-Likelihood (NLL) with varying K (Table 4 Transcription): This table presents Negative Log-Likelihood values (in bits per dimension) on the test set for different values of $K$ using uniform dequantization, demonstrating the stability and robustness of the likelihood estimates.

  Model	K=1	K=20	K=50	K=1	K=5	K=15	K=1	K=5	K=10
  Ablation
  DDPM	3.24	3.14	3.12	3.62	3.57	3.54	3.36	3.33	3.32
  Score Matching	3.28	3.18	3.16	3.65	3.59	3.57	3.43	3.41	3.40
  ScoreFlow	3.21	3.11	3.09	3.63	3.57	3.55	3.39	3.37	3.36
  Ours
  FM / Diffusion	3.23	3.13	3.10	3.64	3.58	3.56	3.37	3.34	3.33
  FM W/ OT	3.11	3.01	2.99	3.62	3.56	3.53	3.35	3.33	3.31

  • As $K$ (number of uniform samples for dequantization) increases, the NLL values generally stabilize and slightly decrease, which is expected for a more accurate estimate. $FM W/ OT$ consistently shows the lowest NLL values across different $K$ values, reinforcing its superior likelihood modeling capabilities.

Visual Analysis of Vector Fields and Trajectories

• Figure 2 visually compares the OT path's conditional vector field to the diffusion path's conditional score function. It highlights that the OT vector field has a more constant direction over time, suggesting it is simpler for a neural network to learn and fit. This simplicity likely contributes to faster training and better generalization.
• Figure 3 shows Diffusion and OT trajectories. OT trajectories appear as straight lines, representing direct movement from noise to data. Diffusion trajectories, in contrast, are more curved and can "overshoot" the target, requiring backtracking, which implies less efficient sampling.

Overall Robustness and Stability

The results, including the faster training convergence (Figure 5) and constant sampling cost during training (Figure 10), point to Flow Matching being a more robust and stable training framework for generative CNFs compared to existing score-based methods. This stability contributes to its ability to scale to larger and more complex datasets.

Conclusion & Personal Thoughts

Conclusion Summary

The paper successfully introduces Flow Matching (FM), a groundbreaking simulation-free framework for training Continuous Normalizing Flows (CNFs) that significantly enhances their scalability and performance in generative modeling. By reformulating the problem to match vector fields of fixed conditional probability paths, FM overcomes the computational bottlenecks of traditional CNF training and the inherent limitations of diffusion models.

Key achievements include:

• Scalable and Stable CNF Training: FM, particularly its Conditional Flow Matching (CFM) objective, provides an efficient and unbiased training mechanism for CNFs that scales effectively to high-dimensional data, unlike previous CNF methods.

• Generalization Beyond Diffusion: FM is compatible with a general family of Gaussian probability paths, encompassing and improving upon existing diffusion paths. Crucially, it introduces Optimal Transport (OT) displacement interpolation paths, which are empirically shown to be more efficient.

• Superior Performance: Models trained with FM and OT paths consistently achieve state-of-the-art results on ImageNet in terms of sample quality (FID), likelihood (NLL/BPD), and significantly faster sampling (fewer NFEs) compared to alternative diffusion-based methods.

• Efficient Sampling: The OT paths lead to simpler, straight-line trajectories, resulting in faster and more reliable sample generation using off-the-shelf ODE solvers, and faster convergence during training.

• Conditional Generation: The framework is also demonstrated to be effective for conditional generation tasks, such as image super-resolution, outperforming strong baselines.

  In essence, Flow Matching offers a powerful paradigm shift, allowing researchers to directly specify and learn optimized probability paths, thereby pushing the boundaries of what CNFs can achieve in generative modeling.

Limitations & Future Work

The authors implicitly acknowledge a few areas for further exploration and improvement:

• Exploring More General Probability Paths: The paper focuses on a general family of Gaussian conditional paths, including isotropic Gaussians. Future work could investigate non-isotropic Gaussians or even more general kernel-based distributions to define conditional paths, potentially leading to even more expressive and efficient models.
• Theoretical Guarantees for OT Paths: While empirical results strongly favor OT paths, the paper notes that the marginal VF for OT conditional paths is not necessarily an Optimal Transport solution itself. Further theoretical work could explore the properties of these marginal VFs and potential benefits or limitations.
• Energy Consumption: The paper briefly mentions the increasing energy demand for training large deep learning models (Amodei et al., 2018; Thompson et al., 2020) and highlights that Flow Matching's ability to train with fewer gradient updates/image throughput contributes to energy savings. While this is a positive, the absolute energy footprint of training models on large datasets like ImageNet remains a challenge for the field.

Personal Insights & Critique

• Elegance of CFM: The derivation of Conditional Flow Matching (CFM) as an unbiased estimator for Flow Matching (FM) is highly elegant. It provides a clever way to bypass the intractability of marginalizing over the data distribution, drawing inspiration from Denoising Score Matching but generalizing it to vector fields. This theoretical foundation is robust and crucial for its practical success.
• Direct Path Specification: The framework's ability to explicitly define and work with arbitrary probability paths, rather than being constrained by SDE formulations, opens up a new avenue for generative model design. This shifts the focus from implicit stochastic processes to direct trajectory engineering, offering greater control and potential for optimization.
• The Power of OT Paths: The empirical success of Optimal Transport paths is a standout finding. The intuition that straight-line trajectories and a constant-direction vector field are "simpler" for neural networks to learn is well-supported by the performance gains. This suggests that designing physically or geometrically motivated paths (like OT) could be a powerful general strategy in generative modeling. It simplifies the learning problem for the neural network.
• Bridging CNFs and Diffusion Models: This work effectively bridges the gap between CNFs and diffusion models. It demonstrates that the core ideas behind diffusion (transforming noise to data) can be implemented within a scalable CNF framework, leveraging the best of both worlds: CNF's flexibility in path design and diffusion's training stability.
• Untested Assumptions and Open Questions:
  • What are the failure modes of Flow Matching? Does it struggle with certain data distributions or high-frequency details more than others?
  • While the U-Net architecture is powerful, FM's performance might further benefit from architectures specifically designed for vector field regression or ODE dynamics.
  • The choice of $\sigma_{\min}$ for OT paths is mentioned as "sufficiently small." What is the sensitivity of the model to this parameter, and are there optimal strategies for its selection?
  • The paper focuses on image generation. How well does Flow Matching transfer to other data modalities (e.g., audio, text, time series) where CNFs have shown promise?
• Impact on Generative AI: Flow Matching represents a significant step forward for generative modeling. By making CNFs scalable and providing a flexible framework for path design, it could lead to new generations of generative models with improved sample quality, faster inference, and more interpretable generation processes. The emphasis on faster training and sampling is also vital for the environmental sustainability of large-scale AI research.