1. Bibliographic Information

1.1. Title

Disentangling Style and Content in Anime Illustrations

1.2. Authors

• Sitao Xiang (University of Southern California)
• Hao Li (University of Southern California, Pinscreen, USC Institute for Creative Technologies)

1.3. Journal/Conference

The paper was published on arXiv, a preprint server. While arXiv hosts preprints, many papers published there are later accepted to reputable conferences or journals. Hao Li is a well-known researcher in computer graphics and vision. The mention of affiliations like the University of Southern California and the USC Institute for Creative Technologies indicates a strong academic and research background.

1.4. Publication Year

2019

1.5. Abstract

This paper addresses the challenges in AI-generated artwork, specifically the difficulty of creating high-quality stylized content while preserving high-level semantics and separating fine-grained styles from various artists. The authors propose a novel Generative Adversarial Disentanglement Network (GADN) designed to disentangle two complementary factors of variation, even when only one factor is labeled. In the context of anime illustrations, this model fully decomposes images into style and content. The training process is challenging because, given a style, various content data can exist, but not vice versa. Their approach is divided into two stages: first, encoding an input image into a style-independent content representation, and second, using a dual-conditional generator. The authors demonstrate the model's ability to generate high-fidelity anime portraits with a fixed content and a wide array of styles from over a thousand artists, and vice versa, all within a single end-to-end network. The paper highlights applications in style transfer and claims superior output compared to the current state-of-the-art.

1.6. Original Source Link

https://arxiv.org/abs/1905.10742 (Publication Status: Preprint on arXiv) PDF Link: https://arxiv.org/pdf/1905.10742v3.pdf

2. Executive Summary

2.1. Background & Motivation

The core problem the paper addresses is the struggle of existing AI-generated artwork methods to produce high-quality stylized content while maintaining semantic integrity and effectively separating fine-grained artistic styles.

This problem is crucial in the field of computer graphics and generative AI because style transfer and stylized content generation are highly sought-after capabilities for creative applications, digital art, and content creation. Current methods face several limitations:

• Neural Style Transfer (NST) methods [5]: While groundbreaking, they primarily rely on matching neural network features (e.g., Gram matrices) which mainly capture texture statistics. This often fails to capture high-level semantic style elements like character proportions, facial feature shapes, or overall artistic interpretation. They can also alter the content of the input image in undesirable ways.

• Image-to-Image Translation (I2I) methods [11, 27]: These can learn domain-specific styles but typically require a separate network for each pair of domains (e.g., photorealistic to Van Gogh). This approach does not scale well to a large number of styles or artists. Methods like StarGAN [2] attempt to handle multiple domains with one network but often lack an explicit content space, hindering true style-content disentanglement.

• Disentangled Representation Learning: While methods like DC-IGN [15] aim for disentanglement, they often demand highly structured data (e.g., images with same content but different style, and vice-versa), which is rarely available for style transfer. Unsupervised methods like InfoGAN [1] can discover disentangled factors but cannot explicitly enforce the meaning of these factors (e.g., ensuring one factor is style and another is content).

  The paper's entry point is to formulate style transfer as a specific instance of a general problem: training a generative network where two complementary factors of variation can be fully disentangled and independently controlled, given that only one factor is labeled in the dataset. For style transfer, this means having labeled style (e.g., artist identity) but unlabeled content.

2.2. Main Contributions / Findings

The paper proposes a novel Generative Adversarial Disentanglement Network (GADN) with the following primary contributions:

• Novel Two-Stage Disentanglement Framework: A robust method to disentangle style and content when only style is labeled, overcoming limitations of prior approaches that struggled with unconstrained encoder output distributions or blurry outputs.

• Stage 1: Style-Independent Content Encoder: A unique design where an encoder learns a style-independent content representation. This stage employs an adversarial classifier that attempts to classify the generator's output (combining the input content with a different artist's style) rather than the encoder's direct output, coupled with KL-divergence losses to constrain latent distributions. This addresses the issue of encoder output instability found in previous methods.

• Stage 2: Dual-Conditional Generator with Adversarial Classifier and Content Loss: A GAN-based generator that takes both content and style codes as input. It incorporates a discriminator, an auxiliary classifier (C2) that is adversarial (trained to classify generated samples as not belonging to the conditioned style), and an explicit content reconstruction loss to ensure content preservation. The adversarial nature of C2 is a key innovation for learning comprehensive style features.

• High-Fidelity Stylized Generation: Demonstrated the ability to generate high-fidelity anime portraits, faithfully capturing style-specific elements like facial feature shapes, color saturation, highlights, and contours, while preserving content.

• Independent Control: Achieved independent control over style and content, enabling generation of fixed content with various styles (from over a thousand artists) and vice-versa, using a single end-to-end network.

• Superior Style Transfer: Showed superior style transfer results compared to state-of-the-art baselines like Neural Style Transfer and StarGAN, particularly in capturing high-level artistic semantics.

• Generality Demonstration: Applied the method to the NIST handwritten digit dataset to disentangle writer identity from digit class (and vice-versa), proving the generality of the framework beyond anime.

• Ablation Studies: Provided detailed ablation studies to justify the design choices, including the placement of the adversarial classifier and the necessity of the explicit content loss.

  The key conclusion is that true semantic-level artwork synthesis with disentangled style and content is achievable through their two-stage framework, significantly improving upon existing style transfer methods by modeling high-level artistic semantics and visual quality.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a foundational grasp of the following concepts is essential:

• Deep Learning: A subfield of machine learning that uses artificial neural networks with multiple layers (deep neural networks) to learn representations of data with multiple levels of abstraction. These networks learn directly from data, often eliminating the need for manual feature engineering.
• Generative Models: A class of statistical models that learn the distribution of data in a dataset and can then generate new samples that resemble the training data. This paper focuses on generative models for images.
• Generative Adversarial Networks (GANs): A framework for training generative models, introduced by Ian Goodfellow et al. A GAN consists of two neural networks: a Generator (G) and a Discriminator (D), which are trained simultaneously in a zero-sum game.
  • Generator (G): Learns to generate realistic data samples (e.g., images) from random noise. Its goal is to fool the discriminator.
  • Discriminator (D): Learns to distinguish between real data samples (from the training set) and fake data samples (generated by $G$). Its goal is not to be fooled by the generator.
  • Adversarial Training: $G$ and $D$ are trained iteratively. $G$ tries to maximize the probability of $D$ making a mistake (classifying fake as real), while $D$ tries to minimize this probability. This adversarial process drives both networks to improve, eventually leading $G$ to produce highly realistic data.
• Autoencoders (AEs): A type of neural network used for unsupervised learning of efficient data codings (representations). An Autoencoder aims to learn a compressed representation (encoding) of input data in an unsupervised manner. It consists of two parts:
  • Encoder: Maps the input data to a lower-dimensional latent space representation.
  • Decoder: Reconstructs the input data from the latent space representation. The goal is to minimize the reconstruction error between the input and the output.
• Variational Autoencoders (VAEs): A type of generative model that builds upon Autoencoders by introducing a probabilistic approach to the latent space. Instead of mapping inputs directly to a fixed latent vector, the encoder maps them to parameters (mean and variance) of a probability distribution (typically a Gaussian distribution). The decoder then samples from this distribution to reconstruct the input. This probabilistic formulation enables VAEs to generate new, diverse samples by sampling from the learned latent distribution. A key component of VAEs is the Kullback-Leibler (KL) divergence loss, which regularizes the latent distribution to be close to a prior distribution (e.g., a standard normal distribution), encouraging a continuous and well-structured latent space.
• Disentangled Representations: In machine learning, a disentangled representation is one where individual dimensions or subsets of dimensions in a latent space correspond to distinct, independent factors of variation in the data. For example, in images of faces, one dimension might control head pose, another might control expression, and another might control identity, all independently. This allows for fine-grained control over generation and better interpretability.
• Convolutional Neural Networks (CNNs): A class of deep neural networks primarily used for analyzing visual imagery. CNNs use convolutional layers to automatically and adaptively learn spatial hierarchies of features from input images. They are highly effective for tasks like image classification, object detection, and image generation. The paper also mentions residue blocks [7], which are building blocks within CNNs that help train very deep networks by allowing information to bypass some layers, mitigating the vanishing gradient problem.
• Image-to-Image Translation: The task of transforming an image from one domain to another, such as converting a grayscale image to color, a satellite image to a map, or a photograph to a painting. GANs have been particularly successful in this area.
• Kullback-Leibler (KL) Divergence: A measure of how one probability distribution $P$ is different from a second, reference probability distribution $Q$. It quantifies the amount of information lost when $Q$ is used to approximate $P$. In VAEs, it's used to constrain the learned latent distribution to be close to a simple prior distribution (e.g., standard normal). The formula for KL divergence from $Q$ to $P$ for continuous distributions is: $ D_{KL}(P || Q) = \int_{-\infty}^{\infty} p(x) \log\left(\frac{p(x)}{q(x)}\right) dx $ Where:
  • $P$ and $Q$ are probability distributions.
  • p(x) and q(x) are the probability density functions of $P$ and $Q$ respectively.

3.2. Previous Works

The paper extensively discusses prior research in neural style transfer, image-to-image translation, and disentangled representation learning.

3.2.1. Neural Style Transfer (NST)

• Gatys et al. [5]: Introduced the groundbreaking idea of decomposing an image into content and style using a pre-trained deep neural network (e.g., VGG).
  • Method: They optimize an output image by minimizing a content loss (matching features of the content image at higher layers) and a style loss (matching Gram matrices of features of a style image at lower layers). Gram matrices capture texture statistics.
  • Content Loss: Measures the squared Euclidean distance between the feature representations of the content image and the generated image.
  • Style Loss: Measures the squared Frobenius norm of the difference between the Gram matrices of the style image and the generated image. A Gram matrix $G_l \in \mathbb{R}^{N_l \times N_l}$ for a layer $l$ with $N_l$ filters and feature map $F_l \in \mathbb{R}^{N_l \times M_l}$ (where $M_l$ is the flattened spatial dimensions) is calculated as $G_l = F_l F_l^T$.
  • Limitation: As acknowledged by the authors, this method primarily transfers texture statistics and often fails to capture high-level semantic style or to preserve content faithfully, sometimes altering colors or shapes that are part of the original content.
• Luan et al. [18] and Liao et al. [16]: Extensions that use masks or dense correspondences to improve spatial control over style transfer.
• Huang et al. [9]: Represents style using affine transformation parameters of instance normalization layers, moving slightly beyond pure texture features.
• Critique in paper: The authors argue that style is domain-dependent and that texture statistics alone are insufficient. They believe style transfer should be seen as an image-to-image translation problem.

3.2.2. Image-to-Image Translation (I2I)

• Isola et al. [11] (pix2pix): Introduced a general framework for image-to-image translation using conditional GANs.
  • Method: Requires paired training data (input image and corresponding output image). The generator learns to map an input image to an output image in the target domain, and the discriminator learns to distinguish real target images from generated ones.
  • Limitation: The need for paired data is a significant constraint.
• Zhu et al. [27] (CycleGAN) and Yi et al. [26] (DualGAN): Removed the need for supervised (paired) training data.
  • Method: Use cycle consistency loss to enable training with unpaired datasets. For example, in CycleGAN, mapping an image from domain A to B, and then back from B to A, should ideally reconstruct the original image from A.
  • Application: Demonstrated impressive results, e.g., translating photorealistic images to styles of famous painters like Van Gogh.
  • Limitation: Still typically requires a different network for each pair of domains, which doesn't scale to many styles.
• Liu et al. [17]: Proposed training an encoder and generator for each domain, encoding into a shared code space and generating from it.
• Choi et al. [2] (StarGAN): A unified GAN for multi-domain image-to-image translation.
  • Method: Allows a single generator to translate images among multiple domains by conditioning on domain labels.
  • Limitation: The paper notes that StarGAN lacks an explicit content space, meaning it doesn't truly disentangle style and content in the way the proposed method aims to. Its conditioning is primarily on domain labels, not separable style and content codes.
• Critique in paper: The authors aim to go a step further than StarGAN by having one set of networks for many styles, treating content and style as two different factors of variation within a single large domain, with explicit disentanglement.

3.2.3. Disentangled Representation Learning

• Kulkarni et al. [15] (DC-IGN): Achieved clean disentanglement of factors of variation.
  • Limitation: Requires very well-structured data (batches with same content different style, and same style different content), which is impractical for style transfer.
• Chen et al. [1] (InfoGAN): An unsupervised method that can discover disentangled factors of variation from unorganized data.
  • Method: Modifies the GAN objective to maximize the mutual information between a small subset of the latent variables and the observations.
  • Limitation: Being unsupervised, there's no way to explicitly enforce the meaning of the disentangled factors (e.g., ensuring one factor is explicitly style and another is content).
• Mathieu et al. [20]: An example where the setting is similar to the authors' problem, with only one factor labeled.
• Chou et al. [3]: A related technique in audio processing (voice conversion) that shares a similar structure to the proposed approach, particularly its two-stage design. The authors specifically reference their first stage as being similar to [3].
• Critique in paper: The authors position their problem between DC-IGN (too structured data needed) and InfoGAN (no explicit control over factor meaning). They want to enforce the meaning of style and content but with only one factor (style via artist label) controlled in the training data.

3.3. Technological Evolution

The field has evolved from texture-matching Neural Style Transfer (Gatys et al.) to image-to-image translation methods that learn domain mappings (pix2pix, CycleGAN) and then to multi-domain translation (StarGAN). Concurrently, generative models have moved towards disentangled representations to gain finer control over generation. This paper sits at the intersection of these trends, attempting to achieve scalable multi-style transfer with explicit style and content disentanglement, overcoming the limitations of prior disentanglement methods by working with partially labeled data. It combines adversarial training, VAE-like regularization, and explicit content consistency to address the specific challenges of anime style decomposition.

3.4. Differentiation Analysis

Compared to the main methods in related work, the core innovations and differences of this paper's approach are:

• Addressing High-Level Semantic Style: Unlike Neural Style Transfer that focuses on texture statistics, this method aims to disentangle and control high-level semantic style elements (e.g., eye shape, facial proportions, shading techniques), which are crucial for artistic expressions like anime.
• Scalability to Many Styles: In contrast to image-to-image translation methods (like CycleGAN) that require a separate network per domain pair, or even StarGAN which handles multiple domains but without clear content disentanglement, this approach uses a single network to manage over a thousand artist-specific styles and truly disentangle style and content into distinct latent codes.
• Robust Disentanglement with Partial Labels: The paper tackles the challenging scenario where only style (via artist labels) is explicitly available, while content is unlabeled. Previous disentanglement methods often required more structured data or lacked control over the meaning of disentangled factors.
• Novel Stage 1 Design: The use of an adversarial classifier that sees the generator's output (combining input content with a different artist's style) rather than the encoder's direct output, combined with KL-divergence regularization, prevents the encoder from encoding style information while maintaining content and addresses the instability issues observed in prior approaches (e.g., similar to [3]).
• Adversarial Stage 2 Classifier: A unique aspect where the Stage 2 classifier is trained to classify generated samples as "not by" the conditioned artist, forcing the generator to learn comprehensive style features beyond what's minimally required for basic classification.
• Explicit Content Consistency Loss: The introduction of an explicit content reconstruction loss (L_cont) in Stage 2 ensures that the generator preserves the content of the input image, which was found to be necessary in their ablation studies, especially when the content code is fixed-size rather than fully convolutional.

4. Methodology

The proposed method, the Generative Adversarial Disentanglement Network (GADN), is designed to disentangle two complementary factors of variation (specifically style and content in images) when only one of them is labeled. The method is divided into two distinct stages. The overall training procedure is visualized in Figure 1.

4.1. Principles

The core idea is to learn separate latent representations for style and content that are truly independent. This is achieved through an adversarial training setup where an encoder maps an image to a content code that is style-independent, and a generator synthesizes images based on both a content code and a style code. The style code is learned from artist labels. The training is challenging because given a style (an artist), many content variations exist, but it's hard to find the same content across different styles. The two-stage approach progressively refines the disentanglement.

4.1.1. Per-pixel L2 Distance

The paper defines a specific per-pixel L2 distance (not its square) for reconstruction loss. For two 3-channel images $X, Y \in \mathbb { R } ^ { \hat { h } \times w \times 3 }$, this distance is given by: $$| | X - Y | | = \frac { 1 } { h w } \sum _ { i = 1 } ^ { h } \sum _ { j = 1 } ^ { w } | | X _ { i j } - Y _ { i j } | | _ { 2 }$$ Where:

• $| | X - Y | |$: The defined per-pixel L2 distance between images $X$ and $Y$.
• $h$: Height of the image.
• $w$: Width of the image.
• $X_{ij}$: The pixel at row $i$ and column $j$ in image $X$.
• $Y_{ij}$: The pixel at row $i$ and column $j$ in image $Y$.
• $| | \cdot | | _ { 2 }$: The standard Euclidean (L2) norm for the 3-channel (RGB) pixel vector. This metric calculates the average Euclidean distance between corresponding pixel vectors across the entire image.

4.2. Core Methodology In-depth (Layer by Layer)

The method consists of two stages:

4.2.1. Stage 1: Style Independent Content Encoding

The goal of Stage 1 is to train an encoder $E(\cdot)$ that encodes as much information as possible about the content of an image, but no information about its style. The decoder (which is later referred to as generator $G(\cdot, \cdot)$) reconstructs the image using this content code and a style code.

Initial (Less Successful) Approach: The authors first considered a simpler approach, inspired by [3], involving an encoder $E(\cdot)$ and a decoder $G(\cdot)$.

1. Reconstruction Loss: Minimize the reconstruction error between the input image $x$ and its reconstruction G(E(x)). $$\mathcal { L } _ { \mathrm { r e c } } = \underset { x \sim p ( x ) } { \mathbb { E } } [ | | x - G ( E ( x ) ) | | ]$$ Where:
   • $\mathcal { L } _ { \mathrm { r e c } }$: The reconstruction loss.
   • p(x): The distribution of training samples.
   • E(x): The content code produced by the encoder for image $x$.
   • $G(\cdot)$: The decoder (generator) network.
   • $| | \cdot | |$: The per-pixel L2 distance defined above. The objective for $E$ and $G$ is to minimize this loss: $ \underset { E , G } { \operatorname* { m i n } } \ \mathcal { L } _ { \mathrm { r e c } } $
2. Adversarial Classifier for Style: To prevent the encoder from embedding style information into E(x), an adversarial classifier $C(\cdot)$ is introduced. $C(\cdot)$ tries to classify the encoder's output E(x) by artist, while the encoder $E(\cdot)$ tries to maximize the classifier's loss (i.e., fool the classifier). $$\mathcal { L } _ { C } = \underset { x , a \sim p ( x , a ) } { \mathbb { E } } [ \mathrm { N L L } ( C ( E ( x ) ) , a ) ]$$ Where:

   • $\mathcal { L } _ { C }$: The classifier's loss.

   • p(x, a): The joint distribution of images $x$ and their corresponding artist labels $a$.

   • $NLL(\cdot, \cdot)$: Negative Log-Likelihood, commonly used for classification tasks. For a predicted probability distribution $\mathbf{y}$ and true class $i$, $NLL(\mathbf{y}, i) = -\log(y_i)$.

   • C(E(x)): The output of the classifier given the content code. The classifier $C$ aims to minimize this loss: $\underset { C } { \mathrm { m i n } } \ \mathcal { L } _ { C }$. The encoder $E$ aims to maximize this loss (or minimize its negative): $ \underset { E , G } { \mathrm { m i n } } \ \mathcal { L } _ { \mathrm { r e c } } - \lambda \mathcal { L } _ { C } $ Where:

   • $\lambda$: A weight factor balancing reconstruction and style disentanglement.

     Problem with Initial Approach: This approach suffered from a conflict: the generator needs style information to reconstruct the input, but the encoder must not encode style. The authors found this setup did not adequately prevent $E(\cdot)$ from encoding style information, potentially due to the unconstrained nature of the latent code space allowing the encoder-decoder to "trick" the classifier by transforming the code distribution (discussed in Appendix C.1).

Proposed Stage 1 Method (Refined): To address the limitations, the authors propose several key changes:

1. Style Code S(a): Instead of the encoder providing all necessary information, a separate style function $S(\cdot)$ is introduced. S(a) maps an artist label $a$ to a distinct style vector (or style code). This function is not an encoding network; it simply retrieves a learned vector associated with each artist. The generator $G(\cdot, \cdot)$ now takes both the content code E(x) and the style code S(a) as input. The reconstruction loss is updated to: $$\mathcal { L } _ { \mathrm { r e c } } = \underset { x , a \sim p ( x , a ) } { \mathbb { E } } [ | | x - G ( E ( x ) , S ( a ) ) | | ]$$ Where:

   • S(a): The style code for artist $a$.

2. Classifier Input Modification: Crucially, the adversarial classifier $C(\cdot)$ (later denoted $C_1(\cdot)$) no longer classifies E(x). Instead, it tries to classify the generator's output $G ( E ( x ) , S ( a ^ { \prime } ) )$, where $a^{\prime}$ is a different artist from the actual author $a$ of image $x$. This is a critical insight: the classifier C1 should enforce that the generated image (with the input content and a foreign style) does not resemble the original artist's style. $$\mathcal { L } _ { C } = \underset { x , a \sim p ( x , a ) } { \mathbb { E } } [ \mathrm { N L L } ( C ( G ( E ( x ) , S ( a ^ { \prime } ) ) ) , a ) ]$$ Where:

   • $a^{\prime}$: A style label different from the true artist $a$ of image $x$, sampled independently. The classifier $C_1$ minimizes this loss: $\underset { C } { \mathrm { m i n } } \ \mathcal { L } _ { C }$.

3. KL-Divergence Regularization: To further constrain the latent space and prevent the encoder from "tricking" the classifier by constantly transforming the latent distribution, KL-divergence losses are introduced, akin to Variational Autoencoders (VAEs). The outputs of E(x) and S(a) are treated as parameters of multivariate normal distributions, and these losses force their distributions to be close to a standard normal distribution $\mathcal{N}(0, I)$.

   • Encoder KL Loss: $$\mathcal { L } _ { E \mathrm { - K L } } = \underset { x \sim p ( x ) } { \mathbb { E } } [ D _ { \mathrm { K L } } ( E ( x ) | | \mathcal { N } ( 0 , I ) ) ]$$
   • Style Function KL Loss: $$\mathcal { L } _ { S \mathrm { - K L } } = \underset { a \sim p ( a ) } { \mathbb { E } } [ D _ { \mathrm { K L } } ( S ( a ) | | \mathcal { N } ( 0 , I ) ) ]$$ Where:
   • $D_{KL}(\cdot || \cdot)$: The Kullback-Leibler divergence.
   • $\mathcal{N}(0, I)$: A standard multivariate normal distribution (mean zero, identity covariance matrix). These losses encourage the content and style codes to occupy a well-behaved, continuous latent space, making them more suitable for sampling and interpolation.

Overall Stage 1 Optimization Objective: The combined objective for the encoder $E$, generator $G$, and style function $S$ is to minimize the reconstruction error while simultaneously maximizing the classifier's loss (for disentanglement) and regularizing the latent codes: $$\underset { E , G , S } { \mathrm { m i n } } \ \mathcal { L } _ { \mathrm { r e c } } - \lambda _ { C } \mathcal { L _ { C } } + \lambda _ { E \mathrm { - K L } } \mathcal { L } _ { E \mathrm { - K L } } + \lambda _ { S \mathrm { - K L } } \mathcal { L } _ { S \mathrm { - K L } }$$ Where:

• $\lambda_C, \lambda_{E-KL}, \lambda_{S-KL}$: Hyperparameters balancing the different loss terms.
• Note: In this formulation, L_rec is implicitly L_rec' from the paper's equation, as it is the reconstruction loss of the input image $x$ using its correct style S(a).

4.2.2. Stage 2: Dual-Conditional Generator

Stage 1 provides a well-behaved content encoder $E(\cdot)$ and style function $S(\cdot)$, but the autoencoder structure often produces blurry outputs lacking fine texture information. Stage 2 builds upon this by training a dual-conditional generator $G(\cdot, \cdot)$ and a discriminator $D(\cdot)$ within a Generative Adversarial Network (GAN) framework, conditioned on both content and style. The encoder $E(\cdot)$ and style function $S(\cdot)$ from Stage 1 are fixed during Stage 2.

Components:

1. Discriminator $D$: A network that tries to distinguish between real images (from the training dataset) and fake images (generated by $G$).

   • Instead of binary cross-entropy, the paper follows LSGAN [19] and uses least squares loss for stability and better gradient behavior.
   • Discriminator Loss (Real Samples): $$\mathcal { L } _ { D \mathrm { - r eal } } = \underset { x \sim p ( x ) } { \mathbb { E } } [ ( D ( x ) - 1 ) ^ { 2 } ]$$ This term encourages the discriminator to output 1 for real images.
   • Discriminator Loss (Fake Samples): $$\mathcal { L } _ { D \mathrm { - f ake } } = \underset { x ^ { \prime } \sim p ( x ) } { \mathbb { E } } [ ( D ( G ( E ( x ) , S ( a ^ { \prime } ) ) ) + 1 ) ^ { 2 } ]$$ This term encourages the discriminator to output -1 for fake images (generated with content E(x) and any style S(a')). The discriminator $D$ minimizes the sum of these two losses: $\underset { D } { \mathrm { m i n } } \ \mathcal { L } _ { D \mathrm { - r eal } } + \mathcal { L } _ { D \mathrm { - f ake } }$.

2. Generator $G$ (Adversarial Loss): The generator tries to fool the discriminator. $$\mathcal { L } _ { D \mathrm { - a d v } } = \underset { x \sim p ( x ) } { \mathbb { E } } [ D ( G ( E ( x ) , S ( a ^ { \prime } ) ) ) ^ { 2 } ]$$ This term encourages the generator to output images that cause the discriminator to output 0 (or close to 0), effectively classifying them as "real".

3. Auxiliary Classifier C2: This is a separate classifier (distinct from $C_1$ in Stage 1). It has two roles:

   • Classify Real Samples: Trained to classify real training images to their correct artists. $$\mathcal { L } _ { C _ { 2 } \mathrm { - r eal } } = \underset { x , a \sim p ( x , a ) } { \mathbb { E } } [ \mathrm { N L L } ( C _ { 2 } ( x ) , a ) ]$$ This term ensures C2 learns to recognize real styles.
   • Adversarial Classification on Generated Samples: This is a key difference from previous conditional GANs. Instead of classifying generated samples as their conditioned style (as in AC-GAN [22]) or being uncertain [24], C2 is trained to explicitly classify generated images as "not" belonging to the conditioned style $a^{\prime}$. For this, they define "negative log-unlikelihood" (NLU): $${ \mathrm { N L U } } ( \mathbf { y } , i ) = - \log ( 1 - y _ { i } )$$ Where:
     • $\mathbf{y}$: The output probability vector from the classifier C2.
     • $i$: The index of the target class (here, the conditioned artist $a'$). This means the classifier is pushed to output a low probability for the target class $i$. The C2 loss on generated samples is: $$\mathcal { L } _ { C _ { 2 } \mathrm { { r ake } } } = \underset { x \sim p ( x ) } { \mathbb { E } } [ \mathrm { N L U } ( C _ { 2 } ( G ( E ( x ) , S ( a ^ { \prime } ) ) ) , a ^ { \prime } ) ]$$ The classifier $C_2$ minimizes the sum of these two losses: $\underset { C _ { 2 } } { \mathrm { m i n } } \ \mathcal { L } _ { C _ { 2 } \mathrm { - r eal } } + \mathcal { L } _ { C _ { 2 } \mathrm { - f ake } }$.
   • Generator Adversarial Loss (for C2): The generator $G$ tries to generate samples that do get classified as the artist $a^{\prime}$ it is conditioned on. $$\mathcal { L } _ { C _ { 2 } \mathrm { - } \mathrm { a d v } } = \underset { x \sim p ( x ) } { \mathbb { E } } [ \mathrm { N L L } ( C _ { 2 } ( G ( E ( x ) , S ( a ^ { \prime } ) ) ) , a ^ { \prime } ) ]$$ This term encourages the generator to produce images that are convincing enough to be classified by C2 as belonging to style $a'$.

4. Content Consistency Loss: To explicitly enforce the preservation of content, the generated image G(E(x), S(a')) is fed back through the fixed encoder $E$. The output content code from this (E(G(E(x), S(a'))))) should be close to the original content code E(x). $$\mathcal { L } _ { \mathrm { c o n t } } = \underset { x \sim p ( x ) } { \mathbb { E } } \left[ | | E ( G ( E ( x ) , S ( a ^ { \prime } ) ) ) - E ( x ) | | _ { 2 } ^ { 2 } \right]$$ Where:

   • $| | \cdot | | _ { 2 } ^ { 2 }$: The squared Euclidean distance between the content codes.

     Overall Stage 2 Optimization Objectives:

• Discriminator $D$: $$\underset { D } { \mathrm { m i n } } \ \mathcal { L } _ { D \mathrm { - r eal } } + \mathcal { L } _ { D \mathrm { - f ake } }$$
• Classifier $C_2$: $$\underset { C _ { 2 } } { \mathrm { m i n } } \ \mathcal { L } _ { C _ { 2 } \mathrm { - r eal } } + \mathcal { L } _ { C _ { 2 } \mathrm { - f ake } }$$
• Generator $G$ and Style Function $S$: $$\underset { G , S } { \mathrm { m i n } } \ \lambda _ { D } \mathcal { L } _ { D \mathrm { - a d v } } + \lambda _ { C _ { 2 } } \mathcal { L } _ { C _ { 2 } \mathrm { - a d v } } + \lambda _ { \mathrm { c o n t } } \mathcal { L } _ { \mathrm { c o n t } } + \lambda _ { S \mathrm { - K L } } \mathcal { L } _ { S \mathrm { - K L } }$$ Where:

  • $\lambda_D, \lambda_{C2}, \lambda_{cont}, \lambda_{S-KL}$: Hyperparameters balancing the various loss terms.

  • The KL-divergence loss for $S$ ($L_S-KL$) is included again to maintain regularization of the style codes. The KL-divergence loss for $E$ ($L_E-KL$) is not included here because $E$ is fixed in Stage 2.

    Pre-training $C_2$: It is recommended to pre-train $C_2(\cdot)$ on real samples only ($L_C2-real$) prior to Stage 2 to ensure it is effective at classifying real styles from the start.

The overall training procedure is summarized in Figure 1, illustrating the networks and loss terms involved in both stages.

The following figure (Figure 1) from the original paper shows the training procedure:

该图像是论文中的示意图，展示了提出的方法的训练流程，分为两个阶段。图中用方框表示网络，用圆角矩形表示损失项，蓝色部分对应$E$、$S$和$G$，红色部分对应$D$和$C$，黑色部分为二者共有。图中包含的公式有损失函数$L_{rec}$、$L_{C1}$、$L_{D-real}$、$L_{C2-real}$、$L_{D-fake}$、$L_{D-adv}$、$L_{C2-fake}$、$L_{C2-adv}$和$L_{cont}$。

4.2.3. Network Architecture

All networks (encoder $E$, generator $G$, discriminators $D$, classifiers $C_1$, $C_2$) are built using residue blocks [7].

• Residue Block: Unlike common residue blocks with two convolution layers, this implementation uses only one convolution per block, increasing the number of blocks instead. ReLU activations are applied on the residue branch before being added to the shortcut branch.
• Common Part: A base structure is used across most networks, with specific input/output layers added for each. The common part consists of a sequence of blocks.
  • $C$: Stride 1 convolution.
  • SC: Stride 2 convolution.
  • $F$: Fully connected layer. The table below (Table 1 from the original paper) describes the common part of the network architecture.

The following are the results from Table 1 of the original paper:

Where:

• Layer Channel (first row): Describes the type of layer. - indicates the input, SC means Stride 2 Convolution, $F$ means Fully Connected.

• Size (second row): Represents the number of output channels for convolutional layers, or feature size for fully connected layers, as the sequence progresses from left to right for $encoder/discriminator/classifier$.

• Size (third row): Represents the spatial size of the output feature map, as the sequence progresses from left to right for $encoder/discriminator/classifier$. For the generator, the sequence runs from right to left, meaning it starts with the $F$ layer and progressively upsamples.

  Specifics for Each Network:

• Classifiers C1 and C2: Output layer size corresponds to the number of artists (1,139 in their main experiment).

• Discriminator $D$: Output layer size is 1 (for real/fake discrimination).

• Encoder $E$: Outputs two parallel layers, each with 256 features, representing the mean and standard deviation of the content code distribution. This implies a 256-dimensional content code.

• Generator $G$: The input layer sums the content code dimensions (256) and style code dimensions (256, for a total of 512). This means the style codes S(a) are 256-dimensional vectors.

5. Experimental Setup

5.1. Datasets

5.1.1. Anime Illustrations

• Source: Images obtained from Danbooru, an image board for anime-style art.
• Processing:
  1. Selected images with exactly one artist tag.
  2. Used AnimeFace 2009 [21], an open-source tool, to detect faces.
  3. Each detected face was rotated to an upright position, cropped, and scaled to $256 \times 256$ pixels.
• Scale and Characteristics: The final training set comprised 106,814 images from 1,139 artists. Artists were included only if they had at least 50 images. This dataset is suitable for validating the method's ability to disentangle style and content across a large variety of fine-grained artistic styles, as artist identity serves as a proxy for style. The focus on anime portraits provides a consistent subject matter, allowing the model to learn stylistic variations more effectively.
• Example Data Sample: The paper itself contains many examples of anime portraits generated by the model (e.g., Figures 2, 3, 4, 10, 12, 13, 14), which visually demonstrate the type of data used. These images feature distinct anime art styles with varying facial features, shading, and overall aesthetic.

5.1.2. NIST Handwritten Digit Dataset

• Source: The recently released full NIST handwritten digit dataset [25], which is a superset of the MNIST dataset.
• Scale and Characteristics: A total of 402,953 images, with metadata including the identity of the writer (3,579 different writers). Images are $28 \times 28$ pixels.
• Purpose: Used to demonstrate the generality of the method beyond anime, by disentangling digit class and writer identity.
  • Experiment 1: Disentangling writer identity ($\mathcal{W}$) from digit class and residual variations ($\mathcal{D} + \mathcal{R}$) when only the writer label is known.
  • Experiment 2: Disentangling digit class ($\mathcal{D}$) from writer identity and residual variations ($\mathcal{W} + \mathcal{R}$) when only the digit label is known.
• Example Data Sample: Figure 8 in the paper shows examples of handwritten digits, illustrating how the same digit written by the same person can still exhibit variations.

5.2. Evaluation Metrics

The paper uses a combination of qualitative (visual inspection) and quantitative metrics. For quantitative evaluation, especially concerning the encoder's output distribution in the NIST experiments, several metrics are employed.

5.2.1. Visual Quality and Disentanglement (Qualitative)

• Assessed by human experts (implicitly, as the authors make claims about fidelity to style) comparing generated images to real ones. This evaluates how well the model captures style-specific shapes, appearances of facial features (eyes, mouth, chin, hair), overall color saturation, and contrast.
• Style Transfer quality is evaluated by comparing generated images with baselines.

5.2.2. Classification Accuracy on Generated Samples

• Conceptual Definition: Measures how often an independently trained classifier correctly identifies the style (artist) of a generated image. The idea is that if a generated image truly embodies a specific style, a classifier should be able to recognize it. However, the paper critically discusses the limitations of this metric, particularly when the classifier is not adversarial (Appendix C.2).
• Mathematical Formula: Let $N$ be the total number of generated samples, and $I(\cdot)$ be the indicator function. Let $C_2$ be the classifier, and a'_{true} be the true artist label the image was conditioned on, and $C_2(G(E(x), S(a'_{true})))$ be the predicted artist. $ \text{Accuracy} = \frac{1}{N} \sum_{k=1}^N I(\text{argmax}(C_2(G(E(x_k), S(a'{true,k})))) = a'{true,k}) $ Where:
  • $N$: Total number of generated samples evaluated.
  • $G(E(x_k), S(a'_{true,k}))$: The $k$-th generated image, conditioned on content $E(x_k)$ and true style a'_{true,k}.
  • $C_2(\cdot)$: The classifier network.
  • $\text{argmax}(\cdot)$: Returns the index of the class with the highest predicted probability.
  • $I(\cdot)$: Indicator function, which is 1 if its argument is true, and 0 otherwise.

5.2.3. Mean Euclidean Distance to Class Center (Quantitative for Encoder Output)

• Conceptual Definition: This metric assesses how tightly samples belonging to the same class (e.g., same digit, same writer) are clustered in the encoder's latent space. A smaller average distance indicates better clustering for that class. Conversely, if a feature is successfully disentangled and removed from the encoder's output, then samples grouped by that feature should not be tightly clustered.
• Mathematical Formula: For a given class $c$ (e.g., a specific digit or writer), let $K_c$ be the number of samples belonging to that class, and $z_k$ be the latent code (encoder output) for sample $k$. Let $\mu_c = \frac{1}{K_c} \sum_{k \in \text{class } c} z_k$ be the centroid of class $c$ in the latent space. The average Euclidean distance for class $c$ is: $ \text{AvgDist}c = \frac{1}{K_c} \sum{k \in \text{class } c} ||z_k - \mu_c||2 $ The overall mean Euclidean distance is then the average of $\text{AvgDist}_c$ across all classes, or a weighted average. The paper typically reports the mean over all samples, effectively: $ \text{Mean Euclidean Distance} = \frac{1}{N} \sum{c=1}^{N_{classes}} \sum_{k \in \text{class } c} ||z_k - \mu_c||_2 $ Where:
  • $N$: Total number of samples in the dataset.
  • $N_{classes}$: Number of classes (e.g., number of digits, number of writers).
  • $z_k$: The latent code (output of $E(\cdot)$) for sample $k$.
  • $\mu_c$: The centroid of class $c$ in the latent space.
  • $||\cdot||_2$: Euclidean (L2) norm.

5.2.4. Naive Bayesian Classifier Performance (Quantitative for Encoder Output)

• Conceptual Definition: This evaluates how well class information (e.g., writer or digit) can be predicted solely from the encoder's output code. A "naive Bayesian classifier" assumes independence between features, and each class is modeled by a simple probability distribution (e.g., axis-aligned multivariate normal distribution). If a feature (e.g., writer identity) is successfully purged from the encoder's output, then a classifier trained on this output should perform poorly for that feature. Conversely, if a feature (e.g., digit class) is successfully encoded, the classifier should perform well.
• Metrics reported:
  • Average probability given to the correct class: The average predicted probability that the classifier assigns to the true class of each sample. Higher is better for features that should be encoded.
  • Average rank of the correct class: If the classifier outputs probabilities for all classes, this is the average rank (1 being highest probability) of the true class among all classes. Lower is better.
  • Top-1 accuracy: The standard classification accuracy: percentage of samples where the true class is the one with the highest predicted probability. Higher is better.

5.2.5. Nested Dropout [23]

• Conceptual Definition: A technique used to order the dimensions of a latent code by their importance for reconstruction. This allows for visualizing high-dimensional data in lower dimensions (e.g., 2D) by selecting the most important dimensions, without needing complex dimensionality reduction techniques like PCA that might be sensitive to feature scaling or hard to apply consistently across different training stages.

5.3. Baselines

The paper compares its style transfer results with two prominent methods:

• Neural Style Transfer (NST) [5]: The foundational method that transfers texture statistics using Gram matrices of neural network features.

• StarGAN [2]: A multi-domain image-to-image translation GAN that uses a single network to translate images across multiple target domains by conditioning on domain labels.

  These baselines are representative because they cover different aspects of style transfer: NST represents feature-matching approaches, while StarGAN represents GAN-based image-to-image translation methods capable of handling multiple styles. Comparing against these allows the authors to highlight their improvements in capturing high-level semantic style and scaling to many fine-grained styles.

5.4. Training Hyperparameters

The following are the results from Table 2 of the original paper:

Where:

• $\lambda_{C1}$: Weight for the Stage 1 classifier loss.

• $\lambda_{E-KL}$: Weight for the Stage 1 encoder KL-divergence loss.

• $\lambda_{S-KL}$: Weight for the style function KL-divergence loss (used in both stages).

• $\lambda_D$: Weight for the Stage 2 discriminator adversarial loss.

• $\lambda_{C2}$: Weight for the Stage 2 classifier adversarial loss.

• $\lambda_{cont}$: Weight for the Stage 2 content consistency loss.

  The following are the results from Table 2 of the original paper:

  Stage	Learning rate		Algorithm	Batch	Time
  	S	Others
  1	0.005	5 × 10−5	Adam	8	400k
  C2 pre-train	-	10-4	Adam	16	200k
  2	0.01	2 × 10−5	RMSprop	8	400k

Where:

• S: Refers to the style function $S(\cdot)$ learning rate.

• Others: Refers to the learning rate for all other networks ($E, G, D, C_1, C_2$).

• Algorithm: The optimizer used. Adam for Stage 1 and C2 pre-train, RMSprop for Stage 2 (chosen for GAN stability).

• Batch: Batch size.

• Time: Number of training iterations.

  NIST Dataset Hyperparameters (Table 3): The paper also provides adjusted hyperparameters for the NIST dataset experiments, specifically for disentangling writer identity ($\mathcal{W}$) vs. digit class and residual variations ($\mathcal{D} + \mathcal{R}$), and vice versa.

The following are the results from Table 3 of the original paper:

This table shows that hyperparameters were adjusted for the different NIST disentanglement tasks, indicating fine-tuning for optimal performance in each specific scenario. For example, $λC2$ and $λcont$ values differ, as do the training times for C2 pre-train.

6. Results & Analysis

6.1. Core Results Analysis

The paper demonstrates the effectiveness of its Generative Adversarial Disentanglement Network through qualitative and quantitative results, primarily focusing on anime illustrations and NIST handwritten digits.

6.1.1. Disentangled Representation of Style and Content (Anime)

The core goal of the paper is to show that the style code and content code can independently control their respective aspects of a generated image.

• Fixed Style, Varying Content: Figure 2 illustrates this by showing images generated by fixing the style from a specific artist (e.g., Sayori or Swordsouls) and varying the content. The generated images consistently exhibit the chosen artist's style while showing diverse facial features and poses, demonstrating successful style consistency.

• Fixed Content, Varying Style: Figure 3 shows the opposite: images generated from a single content code combined with an assortment of style codes (from both training artists and randomly sampled from the style distribution). This effectively demonstrates the model's ability to render the same content in a wide variety of styles, faithfully capturing style-specific shapes, appearances, and aspect ratios of facial features (eyes, mouth, chin, hair, blushes, highlights, contours, color saturation, contrast).

  The following figure (Figure 2) from the original paper shows images generated by fixing the style in each group of two rows and varying the content:

  该图像是论文中的插图，展示了通过固定风格并变化内容生成的多组动漫人物头像。图中包括两种不同风格，上下两组分别为Sayori和Swordsouls风格，最左列为训练集中原始风格图像。

The following figure (Figure 3) from the original paper shows images generated from a single content code and an assortment of styles:

该图像是论文中展示的插图，展示了通过固定内容编码与多样化风格编码生成的多张动漫人物头像，包含训练集中艺术家风格和随机风格样本，体现风格与内容的有效解耦。

6.1.2. Style Transfer

The model's ability to disentangle style and content naturally lends itself to style transfer applications. The paper compares its method to Neural Style Transfer [5] and StarGAN [2].

• Neural Style Transfer: Primarily transfers color and texture statistics, often failing to capture high-level semantic style and altering the original content. For example, it might change hair color or facial structure inappropriately.

• StarGAN: Manages to transfer overall color usage and some prominent features like eye size. However, it fails to capture intricate style elements and semantic stylistic changes in facial features.

• Proposed Method: Figure 4 (in the original paper) clearly shows that the proposed method transfers the style of the target artist much more faithfully, capturing semantic style aspects like eye shape, facial contours, shading techniques, and highlights, while largely preserving the content of the source image. This validates the claim of superior output in terms of high-level artistic semantics and visual quality.

  The following figure (Figure 4) from the original paper shows style transfer results comparing the proposed method with StarGAN and Neural Style Transfer:

  该图像是论文中展示的动漫头像风格与内容解耦结果示意图，展示了固定内容下不同艺术家风格的多样化生成效果，验证了模型在风格迁移和风格分离上的能力。

6.1.3. Generality on NIST Handwritten Digit Dataset

To show the method's generality, experiments were conducted on the NIST dataset.

• Disentangling Writer Identity ($\mathcal{W}$) from Digit Class + Residual ($\mathcal{D} + \mathcal{R}$):

  • Figure 5 shows generated samples where each column represents a fixed writer style and each row represents a distinct digit class (with some residual variation). This demonstrates successful disentanglement, where the writer's handwriting style is consistently applied across different digits.

  • Figure 6a (proposed method's E(x) output) shows that digits form 10 clearly distinct clusters in a 2D latent space, even though the digit label was not used during training for disentanglement. This indicates that writer information (the labeled factor to be purged) was successfully removed, reducing intra-digit variation and allowing digit class to emerge as a dominant factor in the content code. In contrast, a vanilla VAE (Figure 6b) shows more confused clusters for several digits.

    The following figure (Figure 5) from the original paper shows generated samples when disentangling $\mathcal { W }$ from $\mathcal { D } + \mathcal { R }$:

    该图像是图5，展示了在将风格 W 从内容与剩余因素 D + R 中解耦时生成的样本，体现了模型对风格和内容的分离能力。

The following figure (Figure 6) from the original paper shows comparison of output distribution between stage 1 encoder and vanilla VAE:

该图像是图表，展示了图6中阶段1编码器与传统VAE编码器输出分布的对比，左图为阶段1编码器输出，右图为VAE编码器输出，可见前者分布更为稠密且结构清晰。

• Disentangling Digit Class ($\mathcal{D}$) from Writer Identity + Residual ($\mathcal{W} + \mathcal{R}$):

  • Figure 7 shows generated samples where each row represents a fixed digit class and each column represents a distinct writer style (with residual variation). The variation within rows (different writer styles) is more dramatic than in Figure 5, which is expected because here the unlabeled factor ($\mathcal{W} + \mathcal{R}$) is responsible for style and is more complex.

  • Figure 9 shows that the distributions of E(x) for each individual digit are very similar, indicating that the digit class (the labeled feature to be purged from E(x)) was indeed successfully removed from the encoder's output.

    The following figure (Figure 7) from the original paper shows generated samples when disentangling $\mathcal { D }$ from $\mathcal { W } + \mathcal { R }$:

    该图像是手写数字风格迁移的示意插图，展示了通过解耦不同风格因子 $\\mathcal { D }$ 与其他因子 $\\mathcal { W } + \\mathcal { R }$ 后生成的样本。图中纵向排列的数字类别保持一致，横向展示了不同风格的变化样例。

The following figure (Figure 9) from the original paper shows distribution of each digit from stage 1 encoder for $\mathcal { D }$ vS. $\mathcal { W } + \mathcal { R }$:

该图像是一个图表，展示了论文中阶段1编码器对不同类别的数字分布，每个子图用不同颜色区分，呈现出各类别在二维空间上的聚集情况，体现了内容编码的分离效果。

6.1.4. Quantitative Analysis of Disentangling on NIST Dataset

The paper quantifies the effectiveness of the Stage 1 encoder ($E(\cdot)$) by analyzing its output space. Nested Dropout [23] was used to project high-dimensional codes to lower dimensions (2 or 8) for analysis.

• Mean Euclidean Distance to Class Center (Table 4): This metric measures how tightly samples of a class cluster in the latent space.

  • For $E_{\mathcal{W}}$ (encoder trained to disentangle writer): Samples cluster tightly by digit (red numbers are small) but not by writer (blue numbers are similar to black "whole dataset" numbers). This means digit information is preserved in a clustered way, while writer information is diffused.

  • For $E_{\mathcal{D}}$ (encoder trained to disentangle digit): Samples cluster more tightly by writer (blue numbers are smaller) but not by digit (red numbers are similar to black "whole dataset" numbers). This means writer information is preserved, while digit information is diffused.

  • $E_V$ (vanilla VAE): Shows less distinct clustering, confirming the benefit of the disentanglement objective.

    The following are the results from Table 4 of the original paper:

    Encoder	By writer	By digit	Whole dataset
    Ew	1.2487	0.2788	1.2505
    ED	0.7929	1.2558	1.2597
    Ev	1.2185	0.4672	1.2475
    (a) First 2 dimensions
    Encoder	By writer	By digit	Whole dataset
    Ew	2.6757	2.0670	2.6957
    ED	2.4020	2.6699	2.7409
    Ev	2.6377	1.7629	2.7363

• Naive Bayesian Classifier Performance (Tables 5, 6, 7): These tables show how well writer or digit labels can be classified from the encoder's output.

  • Average probability to correct class (Table 5): For $E_{\mathcal{W}}$, it assigns very low probability to writer but high to digit. For $E_{\mathcal{D}}$, it's the opposite. This confirms successful removal of the labeled factor while retaining the unlabeled one.

  • Average rank of correct class (Table 6): Reflects the same trend; for $E_{\mathcal{W}}$, digit has a low rank (good), writer has a high rank (bad).

  • Top-1 accuracy (Table 7): For $E_{\mathcal{W}}$ (2 dimensions), digit classification accuracy is 94%, while writer is near 0%. This is strong evidence of disentanglement.

    The following are the results from Table 5 of the original paper:

    Encoder	By writer	By digit
    Ew	0.000293	0.9001
    ED	0.001441	0.1038
    Ev	0.000337	0.6179

The following are the results from Table 5 of the original paper:

(a) First 2 dimensions (b) First 8 dimensions

The following are the results from Table 6 of the original paper:

(a) First 2 dimensions

The following are the results from Table 6 of the original paper:

(b) First 8 dimensions

The following are the results from Table 7 of the original paper:

(a) First 2 dimensions

The following are the results from Table 7 of the original paper:

(b) First 8 dimensions

6.2. Ablation Studies / Parameter Analysis

The ablation studies provided in Appendix C justify key design choices.

6.2.1. Effect of Stage 1 Classifier Input

The paper argues that the instability of the encoder's output distribution was a major problem in simpler Stage 1 designs (where the classifier $C_1$ directly classifies E(x)).

• MLP Classifier: Figure 10 (Columns 3 & 4) shows that if an MLP classifier directly classifies E(x), the style-neutral reconstruction ($G(E(x), \mathbf{0})$) still retains significant style information (e.g., eye size, facial contour, blush amount) from the input image, indicating poor disentanglement.
• Proposed Classifier: In contrast, Figure 10 (Columns 5 & 6) shows that the proposed method, where $C_1$ classifies G(E(x), S(a')) (generator's output with different style), yields much better style-neutral reconstructions, effectively removing style information. This is achieved without compromising the quality of reconstructions with the correct style.
• Stability of Encoder Distribution: Figure 11 illustrates the latent space distributions of encoders over training iterations on the NIST dataset.

  • Vanilla VAE (Figure 11a): The distribution is stable but clusters are not well separated.

  • MLP Classifier (Figure 11b): The distribution is unstable, with digit clusters fluctuating wildly, supporting the conjecture that the encoder can "trick" the classifier by constantly transforming its output distribution.

  • Proposed Classifier (Figure 11c): The distribution is much more stable, with digit clusters remaining consistent and well-separated, validating the effectiveness of using the generator's output as C1's input and KL-divergence regularization.

    The following figure (Figure 10) from the original paper shows comparison of stage 1 image reconstruction with correct style and zero style, using different methods:

    该图像是图10，展示了使用不同方法在第一阶段基于正确风格和零风格进行图像重构的对比。第一列为原始数据图像，后续列分别为VAE重建、多层感知机分类器和作者分类器在正确风格及零风格条件下的重构效果。

The following figure (Figure 11) from the original paper shows change of output distribution of different encoders:

该图像是图表，展示了不同编码器输出分布的变化，包括(a) VAE编码器，(b) 结合MLP分类器的第一阶段编码器，以及(c) 结合论文提出的一阶段分类器的编码器，反映了编码特征的聚类效果差异。

6.2.2. Adversarial Stage 2 Classifier

The paper proposes an adversarial C2 that explicitly classifies generated images as "not by" the conditioned artist (NLU loss), rather than classifying them as the conditioned artist.

• Impact on Generated Quality: Figure 12 shows samples generated when C2 is not adversarial. Visually, these samples appear less faithful to the target style compared to the main results (Figure 2). The authors suggest that an adversarial C2 forces the generator to learn more comprehensive style features, as it needs to avoid any recognizable style artifacts that C2 might pick up.

  The following figure (Figure 12) from the original paper shows images generated from fixed style and different contents, when stage 2 classifier is not adversarial:

  该图像是论文中展示的插图，展示了在第二阶段分类器非对抗训练情况下，固定风格与不同内容生成的多张动漫人物头像，表现了风格一致而内容多样的生成效果。

• Classification Accuracy as Quality Measure (Table 8): The paper critically analyzes the use of classification accuracy on generated samples.

  • A classifier trained only on real samples (non-adversarial C2) might not capture all style aspects. Thus, a high accuracy by such a classifier on generated samples might not imply truly successful style transfer.

  • The results in Table 8 show a huge discrepancy: a non-adversarial C2 can classify samples from a generator (even one trained with adversarial C2) with high accuracy (86.65% or 88.59%). However, an adversarial C2 (which learns to find subtle differences) classifies the same generated samples with very low accuracy (14.37% or 1.85%).

  • This highlights that a high accuracy score from a non-adversarial classifier can be misleading and does not guarantee high-fidelity style transfer.

    The following are the results from Table 8 of the original paper:

    	Adversarial C2	Non-adversarial C2
    G trained with adversarial C2	14.37%	86.65%
    G trained with non-adversarial C2	1.85%	88.59%

6.2.3. Explicit Conditioning on Content in Stage 2 Generator

The paper investigates the necessity of the content consistency loss (L_cont) in Stage 2.

• Without L_cont ($λ_cont = 0$):

  • Fixed Style, Varying Content (Figure 13): The content code largely loses its ability to control the content of the generated image. While some variation exists, the generated characters often look very similar in content despite different content codes.
  • Fixed Content, Varying Style (Figure 14): The style code begins to control both style and content. The characters maintain a similar head pose (a small residual effect of the content code), but facial features and overall body shape change drastically with style, showing that content is not preserved.

• Conclusion: This ablation study demonstrates that an explicit content consistency loss is crucial for preventing partial mode collapse where the content code loses its meaning or the style code takes over content control. The ability to control content is initially present from Stage 1 but can be lost in Stage 2 without this explicit constraint.

  The following figure (Figure 13) from the original paper shows images generated from fixed style and different contents, when explicit condition on content is removed:

  该图像是论文中的插图，展示了在移除显式内容条件后，以固定风格生成的多种不同内容的动漫人像。

The following figure (Figure 14) from the original paper shows images generated from fixed content and different styles, when explicit condition on content is removed:

该图像是论文中的插图，展示了图14中在移除对内容的显式条件后，固定内容但应用不同风格生成的多张动漫头像图像，体现了风格与内容的解耦能力。

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper successfully introduces a novel Generative Adversarial Disentanglement Network (GADN) that enables true semantic-level artwork synthesis by effectively disentangling style and content in images. The proposed two-stage framework first trains a style-independent content encoder and then employs a dual-conditional GAN for synthesis. This approach allows for the generation of high-fidelity anime portraits where content can be fixed while style varies across thousands of artists, and vice versa. The method showcases significant improvements in modeling high-level artistic semantics and visual quality compared to existing neural style transfer and image-to-image translation techniques. The generality of the approach was further validated on the NIST handwritten digit dataset, demonstrating its applicability whenever two factors of variation need to be disentangled and only one is labeled.

7.2. Limitations & Future Work

The authors acknowledge several limitations and suggest future directions:

• Dataset Specificity: The primary experiments focused on anime illustrations and portraits. The method's effectiveness on a wider range of artistic styles or subjects (e.g., landscapes, full-body characters) has not been extensively tested, mainly due to the scarcity of large, labeled datasets for diverse artworks.
• Inconsistency in Small Features: Minor inconsistencies were observed in small features like eye colors and facial expressions. This is attributed to the per-pixel reconstruction loss in Stage 1, which might de-prioritize small features (as their contribution to overall pixel error is minor) and the use of a fixed-size code rather than a fully convolutional architecture.
• Data Tagging for Small Features: For features like eye color and facial expressions, additional tags (if available from Danbooru) could be incorporated to condition the encoder and generator, potentially improving consistency.
• Loss Functions Aligned with Human Perception: Future work could explore loss functions that are more aligned with humans' perception of visual importance to better preserve fine details.
• Fully Convolutional Architectures with Spatial Transformation: The authors hypothesize that a fully convolutional architecture (which preserves richer content information) might require some form of non-rigid spatial transformation to effectively capture changes in facial feature shapes as part of style, which is a promising direction.
• Extension to Broader Styles and Subjects: Future work aims to extend the method to styles beyond anime, and to model entire character bodies or even entire scenes.

7.3. Personal Insights & Critique

This paper presents a highly rigorous and well-justified approach to style and content disentanglement, particularly excelling in the challenging domain of anime art.

• Innovation in Adversarial Training: The key insight regarding the adversarial classifier's input in Stage 1 (C1 classifying G(E(x), S(a'))) and the novel adversarial C2 with negative log-unlikelihood in Stage 2 are particularly innovative. These choices directly address the issues of latent space instability and the need for comprehensive style feature learning, which are common pitfalls in GAN-based disentanglement. The detailed ablation studies clearly validate these design decisions.
• Explicit Content Preservation: The content consistency loss (L_cont) is a practical and effective solution to ensure content preservation, especially given the choice of a fixed-size content code which might otherwise struggle against the powerful style guidance.
• Strong Justification for "Style" Definition: The paper's explicit discussion on the definition of "style" (more than just texture statistics, domain-dependent, different ways of presenting the same subject) is commendable. It provides a solid theoretical foundation for why their semantic-level approach is necessary and distinguishes it from many prior works. Their critique of StyleGAN and similar works for using "style" on purely photographic datasets without clear justification is insightful.
• Generality Demonstrated: The NIST experiments are crucial for demonstrating the broader applicability of the framework beyond the specific anime domain. It suggests that this methodology could be adapted to various other tasks where disentangling labeled and unlabeled factors of variation is desired.
• Potential Areas for Improvement/Future Research:

  • Perceptual Losses: While per-pixel L2 loss was used, integrating perceptual losses (e.g., based on VGG features) for content reconstruction might help preserve fine details and mitigate issues with small features, aligning better with human perception.

  • Hierarchical Latent Codes: The use of a fixed-size content code is a known limitation. Exploring hierarchical or multi-scale latent representations (similar to progressive GANs or StyleGAN) could potentially improve detail preservation and offer more nuanced control, possibly addressing the small feature inconsistency.

  • User Study: Given the subjective nature of "style," a comprehensive user study involving artists or domain experts could further validate the qualitative superiority claims.

  • Semantic Segmentation Integration: For style transfer, explicitly using semantic segmentation masks (as mentioned in related work like [18]) could provide even finer-grained control over which parts of the content are modified by style (e.g., ensuring eyes remain eye-like but take on a specific artistic rendition).

    Overall, this paper makes a significant contribution to generative AI and style transfer by providing a robust and well-reasoned framework for style-content disentanglement in a challenging, real-world artistic domain. Its rigorous methodology and detailed ablation studies make it a valuable resource for researchers in the field.