1. Bibliographic Information

1.1. Title

Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

1.2. Authors

Xiaojun Wu, Dixiang Zhang, Ruyi Gan, Junyu Lu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, Yan Song. The authors are affiliated with the International Digital Economy Academy, South China University of Technology, and the University of Science and Technology of China. Their research backgrounds appear to be in areas related to artificial intelligence, natural language processing, computer vision, and multimodal learning, particularly with a focus on large models and their applications.

1.3. Journal/Conference

The paper is published on arXiv, a preprint server for electronic preprints of scientific papers. As arXiv is a preprint server, it means the paper has not yet undergone or completed peer review for formal publication in a journal or conference. However, arXiv is widely recognized and used in the research community for rapid dissemination of findings, making it influential, especially in fast-moving fields like AI.

1.4. Publication Year

2024 (Published at UTC: 2024-01-26T07:17:50.000Z)

1.5. Abstract

The paper introduces Taiyi-Diffusion-XL, a novel bilingual (Chinese and English) text-to-image (T2I) model designed to bridge the gap in open-source models supporting Chinese language. The core methodology involves extending the capabilities of CLIP and Stable-Diffusion-XL through a process called bilingual continuous pre-training. This includes efficiently expanding CLIP's vocabulary by integrating frequently used Chinese characters into its tokenizer and embedding layers, alongside an absolute position encoding expansion. Furthermore, the model enriches text prompts using a large vision-language model (LVLM) to generate better image captions and achieve higher visual quality. Empirical results demonstrate that the developed CLIP model excels in bilingual image-text retrieval, and Taiyi-Diffusion-XL outperforms previous models in bilingual image generation. The research contributes to the development and open-sourcing of this model, marking a significant advancement in image generation, particularly for Chinese language applications, and fostering broader language support in multimodal research.

1.6. Original Source Link

https://arxiv.org/abs/2401.14688 The paper is available as a preprint on arXiv. PDF Link: http://arxiv.org/pdf/2401.14688v3

2. Executive Summary

2.1. Background & Motivation

The field of text-to-image (T2I) generation has seen rapid advancements with powerful diffusion models. However, a significant limitation persists: most prominent open-source T2I models predominantly support English, leaving a notable gap for bilingual or specifically Chinese language support. This creates a challenge for users and researchers in Chinese-speaking regions, as relying on translation software to convert Chinese prompts to English can lead to a loss of meaning, cultural nuances, and emotional context, ultimately resulting in suboptimal image generation. While some prior works like Taiyi-Diffusion and Alt-Diffusion have attempted to adapt T2I models for Chinese, they often achieve this by replacing the original English text encoders with Chinese-specific ones, which inadvertently leads to a loss of the model's original English understanding capabilities.

The core problem the paper aims to solve is the lack of a high-quality, open-source T2I model that robustly supports both Chinese and English without sacrificing the capabilities of either language. The motivation stems from the need for more inclusive and effective tools for diverse linguistic contexts, ensuring that the richness of language is preserved in the image generation process. The paper's innovative idea is to extend existing powerful English-centric models like CLIP and Stable-Diffusion-XL to become truly bilingual through continuous pre-training, efficient vocabulary expansion, and the strategic use of large vision-language models for prompt enrichment, rather than replacing core components.

2.2. Main Contributions / Findings

The paper makes several significant contributions to the field of bilingual text-to-image generation:

• Efficient Algorithms for Bilingual Expansion: The authors developed and applied efficient algorithms for expanding the vocabulary and position encoding within CLIP's tokenizer and embedding layers. This enables the model to effectively process both Chinese and English, facilitating more accurate and culturally sensitive image generation without discarding original language capabilities.

• Enrichment of Text Prompts by Large Vision-Language Models (LVLMs): The research introduces an approach to enhance text prompts by leveraging LVLMs. This technique generates richer and more accurate image captions, which in turn leads to higher visual quality and better alignment between generated images and complex textual descriptions. This innovative use of LVLMs significantly improves the model's ability to interpret and visualize intricate text.

• Creation and Open-Sourcing of Bilingual Models: The paper culminates in the development and open-sourcing of the Taiyi-Diffusion-XL model, along with an enhanced CLIP model (Our-CLIP). These models represent a substantial advancement in the research and application of bilingual text-to-image generation, providing a powerful tool for the community.

  The key conclusions and findings of the paper are:

• The developed Our-CLIP model demonstrates superior performance in bilingual image-text retrieval tasks, significantly outperforming previous CLIP and AltCLIP models on both English and Chinese datasets (Flickr30K-CN, MSCOCO-CN).

• Taiyi-Diffusion-XL exhibits advanced bilingual image generation capabilities, surpassing previous open-source models like Alt-Diffusion, Pai-Diffusion, and Taiyi-v0.1 in terms of CLIP Sim, Inception Score (IS), and Fréchet Inception Distance (FID) on both English (COCO) and Chinese (COCO-CN) datasets.

• Qualitative human evaluation indicates that Taiyi-Diffusion-XL achieves a photographic style comparable to Midjourney and significantly outperforms other open-source bilingual models, though a gap with commercial models like DALL-E 3 and Midjourney exists, primarily attributed to differences in training data scale and quality. These findings collectively address the need for more diverse language support in multimodal research, making high-quality text-to-image generation accessible for Chinese language applications while maintaining strong English capabilities.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand the Taiyi-Diffusion-XL paper, a foundational grasp of several key concepts in deep learning and multimodal AI is necessary.

• Text-to-Image (T2I) Generation: This is the task of generating an image from a natural language description (text prompt). Recent advancements allow for the creation of photorealistic and stylistically diverse images based on user input.
• Diffusion Models: A class of generative models that learn to reverse a diffusion process. They start with a noisy image and progressively denoise it over several steps to generate a clear image.
  • Diffusion Process (Forward Process): Gradually adds Gaussian noise to an image until it becomes pure noise.
  • Reverse Process (Denoising Process): Learns to predict and remove noise at each step, transforming pure noise back into a coherent image.
• Latent Diffusion Models (LDMs): A specific type of diffusion model (e.g., Stable Diffusion) that performs the diffusion process in a lower-dimensional latent space rather than directly in the pixel space. This significantly reduces computational cost and memory requirements while maintaining high image quality. An autoencoder (composed of an encoder and a decoder) is used to map images to and from this latent space.
• CLIP (Contrastive Language-Image Pre-training): A neural network trained on a vast dataset of image-text pairs to learn highly effective visual representations from natural language. CLIP consists of two main components:
  • Image Encoder: Maps an image into a high-dimensional embedding space.
  • Text Encoder: Maps a text description into the same high-dimensional embedding space. The training objective is to maximize the similarity (e.g., cosine similarity) between correct image-text pairs and minimize it for incorrect pairs. This allows CLIP to perform zero-shot tasks like image-text retrieval (finding images for a given text or vice versa) and image classification.
• Stable-Diffusion-XL (SD-XL): An advanced version of Stable Diffusion models, known for generating high-resolution images with improved aesthetic quality and better prompt adherence. SD-XL utilizes a more complex architecture and training strategy compared to earlier versions, often incorporating multiple text encoders and operating at higher resolutions.
• UNet: A convolutional neural network architecture originally developed for biomedical image segmentation. It has an encoder-decoder structure with "skip connections" that directly connect corresponding layers in the encoder and decoder. In diffusion models, UNet is commonly used as the noise predictor ($\epsilon_{\theta}$), tasked with predicting the noise component in the latent space at each denoising step.
• Tokenizer: A component of natural language processing (NLP) models that breaks down raw text into smaller units called tokens. These tokens can be words, subwords, or characters. For Chinese, character-based tokenization or subword tokenization (e.g., BPE or WordPiece) is common due to the lack of explicit word boundaries.
• Embedding Layers: In NLP models, after tokenization, tokens are converted into numerical vectors called embeddings. Embedding layers map discrete tokens to continuous vector representations that capture semantic meaning.
• Position Encoding: In transformer-based models (which CLIP's text encoder is), position encoding is added to token embeddings to provide information about the relative or absolute position of tokens in a sequence. This is crucial because transformers process tokens in parallel without inherent sequential understanding. Absolute position encoding assigns a unique positional vector to each position.
• Large Vision-Language Models (LVLMs): Models that integrate capabilities from both large language models (LLMs) and computer vision models. They can understand and generate content across modalities, performing tasks like visual question answering, image captioning, and multimodal reasoning. The paper uses an LVLM to enrich text prompts by generating more detailed and accurate image descriptions.

3.2. Previous Works

The paper contextualizes its contributions by discussing the landscape of text-to-image (T2I) generation and bilingual models.

• Early Generative Models:
  • Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017): Consist of a generator and a discriminator network that compete, leading to the generator creating increasingly realistic data.
  • Variational Autoencoders (VAEs) (Kingma & Welling, 2013): Learn a probabilistic mapping from data to a latent space and back, allowing for data generation.
  • Flow-based models (Rezende & Mohamed, 2015): Explicitly model the probability density function of data, enabling exact likelihood computation and efficient sampling.
  • Autoregressive models (Ramesh et al., 2021; Ding et al., 2021; 2022): Generate data sequentially, pixel by pixel or token by token. DALL-E (Ramesh et al., 2021) is an early example.
• Advancements in Diffusion Models:
  • The paper highlights the rise of diffusion models as leading technology (Vincent, 2011; Ho et al., 2020; Song et al., 2020; Cao et al., 2022).
  • DALL-E 2 (Ramesh et al., 2022): Utilizes a hierarchical diffusion model with CLIP latents.
  • Imagen (Saharia et al., 2022) and Deepfloyd-IF (Shonenkov et al., 2023): Demonstrate photorealistic image generation with deep language understanding using diffusion models.
  • Latent Diffusion Model (LDM) (Rombach et al., 2022): The foundation for Stable Diffusion variants (SD-v1.5, SD-v2.1, SD-XL), which perform diffusion in a latent space, integrating CLIP text models for textual feature extraction to reduce computational overhead.
• Text-to-Image Models in Bilingual Context:
  • Initial Chinese CLIP variants: Taiyi-CLIP (Zhang et al., 2022), Chinese-CLIP (Yang et al., 2022), and Alt-CLIP (Chen et al., 2022) focused on replacing CLIP's text encoder with a Chinese-specific one and pre-training on Chinese datasets for image-text matching.
  • Chinese Diffusion Models: Following the CLIP adaptation, Taiyi-Diffusion (Zhang et al., 2022), Alt-Diffusion (Ye et al., 2023), and Pai-Diffusion (Wang et al., 2023) replaced the text encoder in Stable Diffusion and continued training on Chinese image-text datasets to generate images from Chinese prompts.
  • Limitation of previous Chinese T2I models: A crucial drawback of these models is that replacing the original CLIP text encoder often leads to the loss of the model's English language capabilities, making them Chinese-centric rather than truly bilingual.
• Text-Image Datasets:
  • Traditional datasets: COCO (Lin et al., 2014) and Flickr (Young et al., 2014) for English, and COCO-CN (Li et al., 2019) and Flickr-CN (Li et al., 2016) for Chinese, provide foundational data but are limited in scale (below one million entries).
  • Web-crawled datasets: Large-scale datasets like Laion (Schuhmann et al., 2021) (English, up to billions of pairs) and Wukong (Gu et al., 2022) (Chinese, 100 million pairs) are critical for training modern diffusion models.

3.3. Technological Evolution

The evolution of text-to-image generation has moved from early generative models (GANs, VAEs, Autoregressive models) towards the highly effective diffusion models. Initially, these models were largely English-centric. The next wave of innovation focused on adapting these powerful models for other languages, particularly Chinese, due to its distinct linguistic characteristics. Early attempts at bilingual support often involved replacing English-specific components with language-specific alternatives, which, while effective for the target language, compromised the model's original capabilities.

This paper's work, Taiyi-Diffusion-XL, fits within this technological timeline as a refinement step. Instead of replacement, it focuses on extension and continuous pre-training, aiming to achieve true bilingualism where both English and Chinese capabilities are maintained and enhanced. The integration of Large Vision-Language Models (LVLMs) for prompt enrichment also signifies a step towards more sophisticated understanding and utilization of textual input in multimodal generation. This represents an evolution from merely translating text or swapping encoders to deeply integrating multilingual understanding and leveraging advanced LLMs for better input conditioning.

3.4. Differentiation Analysis

Compared to the main methods in related work, Taiyi-Diffusion-XL presents several core differences and innovations:

• Bilingual Capability Preservation vs. Replacement:

  • Previous Chinese T2I models (e.g., Taiyi-Diffusion, Alt-Diffusion, Pai-Diffusion): These models typically replace the English CLIP text encoder with a Chinese-specific encoder, leading to a loss of English language understanding.
  • Taiyi-Diffusion-XL's Innovation: It extends the capabilities of the original English-centric CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach involves integrating Chinese characters and expanding CLIP's vocabulary and position encoding, rather than replacing the encoder. This ensures that the model retains its strong English capabilities while gaining robust Chinese support.

• Prompt Enrichment with LVLMs:

  • Standard T2I models: Rely directly on user-provided text prompts, which can sometimes be brief or ambiguous.
  • Taiyi-Diffusion-XL's Innovation: It leverages Large Vision-Language Models (LVLMs) to enrich text prompts. The LVLM generates more detailed and accurate image captions from initial prompts and even imperfect web-crawled captions. This leads to a richer understanding of the desired image content, resulting in higher visual quality and better alignment with complex descriptions. This is a novel way to improve input conditioning for T2I models.

• Open-Source Stable-Diffusion-XL Basis:

  • The paper leverages the powerful Stable-Diffusion-XL as its backbone, building upon its advanced architecture and training methodologies. This ensures Taiyi-Diffusion-XL starts with a strong foundation for high-resolution image generation, unlike some earlier Chinese models that might have been based on older Stable Diffusion versions or custom architectures.

    In essence, Taiyi-Diffusion-XL innovates by moving beyond mere language adaptation through component replacement to a more sophisticated strategy of bilingual augmentation and AI-driven prompt enhancement, positioning it as a leading open-source solution for truly bilingual text-to-image generation.

4. Methodology

The methodology for Taiyi-Diffusion-XL involves a systematic approach to adapt and enhance existing state-of-the-art models for bilingual text-to-image generation. It can be broadly divided into Dataset Preparation, CLIP Training for bilingual understanding, and Taiyi-XL Training for image generation.

4.1. Principles

The core idea behind the Taiyi-Diffusion-XL method is to achieve robust bilingual (Chinese and English) text-to-image generation by extending the capabilities of existing powerful models rather than replacing their core components. The theoretical basis and intuition are rooted in the effectiveness of continuous pre-training for language adaptation and the benefits of rich, contextually relevant prompts for generative models. The key principles include:

1. Bilingual Continuous Pre-training: Starting with well-established English-centric models (CLIP and Stable-Diffusion-XL), the goal is to expand their linguistic understanding to include Chinese without forgetting their original English knowledge. This is more efficient and effective than training from scratch or replacing components.
2. Efficient Vocabulary Expansion: For CLIP's text encoder, directly integrating frequently used Chinese characters into its existing tokenizer and embedding layers, combined with absolute position encoding expansion, allows for efficient processing of mixed-language inputs. This avoids the pitfalls of language-specific encoders that might lose cross-lingual alignment.
3. Prompt Enrichment via Large Vision-Language Models (LVLMs): Recognizing that the quality of generated images heavily depends on the clarity and detail of text prompts, the method employs LVLMs to refine and enrich user-provided or web-crawled captions. This leverages the advanced understanding capabilities of LVLMs to produce high-quality, detailed descriptions that guide the image generation process more effectively.
4. Latent Diffusion for Efficiency: Maintaining the latent diffusion model paradigm ensures computational efficiency by performing the denoising process in a compressed latent space, making the training and inference practical for high-resolution image generation.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Dataset Preparation

The first crucial step in the methodology is curating a high-quality dataset of image-text pairs, denoted as (X, Y), where $X$ is an image and $Y$ is its descriptive text.

• Emphasis on Comprehensive Descriptions: Unlike traditional datasets that might rely on simple tags, Taiyi-Diffusion-XL prioritizes comprehensive descriptions that capture materials, styles, colors, and spatial layouts of the images.
• Addressing Web-Crawled Data Limitations: Web-crawled resources often contain noisy, irrelevant, or inaccurate tags. To overcome this, the paper employs vision-language large models (LVLMs) to generate synthetic captions.
• LVLM for Caption Generation: Specifically, the Lyrics model (Lu et al., 2023b;a) is utilized. This model inherits the language capabilities of bilingual large language models (Gan et al., 2023) and expands the visual capabilities of LLMs.
  • Inputs to Lyrics: The Lyrics model takes three inputs: the image ($X$), the web crawl caption (potentially imperfect), and instructions for generating the description.
  • Instructions:
    • For Chinese: "请详细描述图片内容" (Please describe the image content in detail).
    • For English: "Write a detailed description of the given image."
  • Output of Lyrics: The Lyrics model generates new, accurate descriptive text ($Y'$) by extracting features from the images and distilling useful information from the imperfect web-crawled captions. This process effectively enhances the richness, relevance, and detail of the descriptions.
• Final Dataset: The generated high-quality text $Y'$ is then combined with the original images $X$ to form enhanced image-text pairs (X, Y'), which are subsequently used for training the Taiyi-XL model.

4.2.2. CLIP Training

The foundation of Taiyi-Diffusion-XL's bilingual understanding lies in its enhanced CLIP model.

• Starting Point: The process begins with a pre-trained English-only CLIP model (Radford et al., 2021). This provides a strong initial understanding of English image-text relationships.
• Bilingual Adaptation and Continuous Pre-training: The CLIP model undergoes an extension process to accommodate bilingual adaptation and the requirements of high-quality image-text data. This is achieved through bilingual continuous pre-training.
  • Stage 1: Large-scale Bilingual Training: In the first stage, the CLIP model is continuously pre-trained on a large-scale bilingual dataset. This includes prominent datasets like Laion (Schuhmann et al., 2021) (primarily English) and Wukong (Gu et al., 2022) (primarily Chinese). This stage also involves crucial data cleaning and quality enhancement to ensure effective learning from diverse sources.
    • Loss Function: A contrastive loss function is employed. This function aims to bring the embeddings of matching image-text pairs closer in the latent space while pushing non-matching pairs further apart.
    • Training Approach: A distributed, memory-efficient training approach (Chen et al., 2023) is used to handle the massive datasets involved.
  • Stage 2: Enriched Dataset Training: The second stage continues training on the enriched dataset (the one prepared using LVLMs as described in Section 4.2.1). This stage focuses on leveraging the diverse perspectives and detailed information captured in these high-quality image-text pairs to further refine the CLIP model's understanding.
• Vocabulary and Position Encoding Expansion (Implicit in CLIP Training): While not explicitly detailed as a separate step here, the abstract mentions "efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion." This expansion is a key part of how CLIP adapts to bilingual contexts during its continuous pre-training. It ensures that the text encoder can properly tokenize and embed Chinese characters and understand their positional relationships within a sentence.

4.2.3. Taiyi-XL Training

The Taiyi-XL training process is the core of the text-to-image generation methodology, building upon the bilingual CLIP model.

4.2.3.1. Initialization and Training

• Model Initialization: The Taiyi-XL model, denoted as $\mathcal{G}_{\theta}$, is initialized with several key components:
  • A noise predictor $\epsilon_{\theta}$: This is typically implemented as a time-conditional UNet (Ronneberger et al., 2015), responsible for predicting the noise component at each step of the diffusion process.
  • A CLIP text encoder $\tau_{\theta}$: This is the bilingual CLIP text encoder trained in Section 4.2.2, which maps textual descriptions into a latent representation.
  • A latent encoder $\mathcal{E}$: Part of the Variational Autoencoder (VAE) (Kingma & Welling, 2013), which compresses an input image into a lower-dimensional latent representation.
  • A dataset $\mathcal{D}$: Comprising image-text pairs $(x_i, y_i)$, where $x_i$ is an image and $y_i$ is its corresponding textual descriptor.
• Training Phase: The model is trained at mixed resolutions of $512 \times 512$ and $1024 \times 1024$. The objective is to minimize a loss function $L$ that guides the image denoising process. The latent representations ($z_t$) are obtained from $\mathcal{E}(x)$ during training.
• Loss Function: The loss function $L$ is defined as: $ L(\theta):=\mathbb{E}{\mathcal{E}(x), y, \epsilon \sim \mathcal{N}(0,1), t}\left[\left|\epsilon-\epsilon{\theta}\left(z_{t}, t, \tau_{\theta}(y)\right)\right|_{2}^{2}\right] $
  • Explanation of Symbols:
    • $\theta$: Represents all the trainable parameters of the Taiyi-XL model.
    • $\mathbb{E}[\cdot]$: Denotes the expectation (average) over various components.
    • $\mathcal{E}(x)$: The latent representation of an image $x$, obtained by passing the image through the latent encoder $\mathcal{E}$.
    • $y$: The textual descriptor (prompt) corresponding to the image $x$.
    • $\epsilon \sim \mathcal{N}(0,1)$: Pure Gaussian noise sampled from a standard normal distribution. This is the target noise that the model aims to predict.
    • $t$: The current timestep in the diffusion process, typically ranging from 1 to $T$.
    • $\epsilon_{\theta}(z_t, t, \tau_{\theta}(y))$: The noise predictor (UNet) with parameters $\theta$. It takes the noisy latent representation $z_t$, the current timestep $t$, and the text embedding $\tau_{\theta}(y)$ (obtained from the CLIP text encoder) as input, and predicts the noise component present in $z_t$.
    • $\|\cdot\|_{2}^{2}$: Represents the squared $L_2$ norm, which calculates the squared Euclidean distance between the actual noise $\epsilon$ and the predicted noise $\epsilon_{\theta}(\cdot)$. The goal is to minimize this difference.
  • Function: This loss function is a common objective in denoising diffusion probabilistic models. It trains the noise predictor $\epsilon_{\theta}$ to accurately predict the noise $\epsilon$ that was added to a clean latent $z_0$ at a given timestep $t$, conditioned on the text prompt.
• Model Implementation: The noise predictor $\epsilon_{\theta}(z_t, t)$ is implemented as a time-conditional UNet (Ronneberger et al., 2015).
• Latent Representations: The latent representations $z_t$ are efficiently derived from $\mathcal{E}(x)$ during training. After the generation process, these latent representations are decoded back to the image space using a VAE decoder.
• Text Encoder Optimization: The text encoder $\tau_{\theta}$, parameterized as a transformer model (part of CLIP), is optimized jointly with the noise predictor $\epsilon_{\theta}$ as per Equation 1. This ensures that the text encoder provides relevant conditioning information for the image generation task.
• Optimization: Model parameters $\theta$ are iteratively updated using gradient descent to minimize the loss function $L(\theta, e)$: $ \theta_{e+1}=\theta_{e}-\eta \cdot \nabla_{\theta} L\left(\theta_{e}, e\right) $
  • Explanation of Symbols:
    • $\theta_e$: The model parameters at epoch or iteration $e$.
    • $\theta_{e+1}$: The updated model parameters for the next epoch or iteration.
    • $\eta$: Represents the learning rate, which controls the step size of parameter updates.
    • $\nabla_{\theta} L(\theta_e, e)$: The gradient of the loss function $L$ with respect to the model parameters $\theta$ at epoch $e$. This indicates the direction of steepest ascent of the loss. By subtracting it, parameters are adjusted in the direction that minimizes the loss.

4.2.3.2. Text-to-Image Generation

Once the Taiyi-XL model is trained, the text-to-image generation process involves:

• Feature Extraction: The trained bilingual text encoder $\tau_{\theta}$ is used to extract textual features from the user's input text prompt $y$. This results in the text embedding $\tau_{\theta}(y)$.
• Latent Diffusion Process: These extracted textual features $\tau_{\theta}(y)$ are then integrated into the latent diffusion process. This integration is crucial for guiding the image generation towards the content described in the prompt.
• Iterative Denoising: The generation starts from the last timestep $T$ with pure noise (a randomly sampled latent vector). The model iteratively denoises this input, step by step, gradually removing noise and adding coherent image features.
• Convergence to Clean Image: The process continues until it converges to $x_0$, which represents the clean, generated image, as described by: $ x_{t-1}=x_{t}-\epsilon_{\theta}\left(x_{t}, t, \tau_{\theta}(y)\right), \quad \lim {t \rightarrow 0} x{t}=x_{0} $
  • Explanation of Symbols:
    • $x_t$: The noisy latent representation at timestep $t$.
    • $x_{t-1}$: The denoised latent representation for the previous timestep t-1.
    • $\epsilon_{\theta}(x_t, t, \tau_{\theta}(y))$: The noise predictor (UNet) predicts the noise component in $x_t$, conditioned on the timestep $t$ and the text embedding $\tau_{\theta}(y)$.
    • $x_t - \epsilon_{\theta}(\dots)$: This operation subtracts the predicted noise from the current noisy latent, effectively denoising it.
    • $\lim_{t \rightarrow 0} x_t = x_0$: As the timestep $t$ approaches 0 (meaning the process moves from pure noise to a clean image), the latent representation $x_t$ converges to the final clean image $x_0$.
  • Function: This equation describes the reverse diffusion (denoising) process, where the model iteratively refines the latent representation guided by the predicted noise and the text conditioning, ultimately producing a coherent image. This iterative process enhances computational efficiency compared to pixel-space diffusion models.

5. Experimental Setup

5.1. Datasets

The experiments for Taiyi-Diffusion-XL utilized a combination of large-scale web-crawled datasets and established benchmark datasets for evaluation.

• Training Datasets:

  • Laion (Schuhmann et al., 2021): A massive dataset primarily consisting of English image-text pairs, often containing hundreds of millions or billions of entries. It's a widely used resource for training CLIP and latent diffusion models.
  • Wukong (Gu et al., 2022): A large-scale Chinese cross-modal pre-training benchmark dataset, primarily containing Chinese image-text pairs, with sizes up to 100 million entries.
  • Enriched Dataset (Internal): An internal dataset curated by the authors, where vision-language large models (specifically Lyrics) were used to generate high-quality, comprehensive descriptions for images, distilling information from web-crawled captions and image features. This dataset plays a crucial role in CLIP and Taiyi-XL training, emphasizing detailed captions over simple tags.

• Evaluation Datasets:

  • Flickr30K (Young et al., 2014): A standard English image captioning dataset containing 30,000 images, each paired with five human-annotated captions.

  • MSCOCO (Lin et al., 2014): Microsoft Common Objects in Context, a large-scale object detection, segmentation, and captioning dataset for English. It contains hundreds of thousands of images with multiple captions per image.

  • Flickr30K-CN (Li et al., 2016): The Chinese version of Flickr30K, providing Chinese captions for the same set of images, used to evaluate Chinese image-text retrieval.

  • MSCOCO-CN (Li et al., 2019): The Chinese version of MSCOCO, offering Chinese captions, used for evaluating Chinese image-text retrieval and image generation.

    These datasets were chosen to cover both large-scale pre-training (Laion, Wukong, internal enriched dataset) and rigorous evaluation on established benchmarks in both English and Chinese (Flickr30K, MSCOCO, and their Chinese counterparts). This allows for a comprehensive validation of the model's performance across different languages and tasks.

5.2. Evaluation Metrics

The evaluation framework encompassed both machine and human evaluations. For machine evaluation, several quantitative metrics were employed to assess different aspects of the model's performance.

5.2.1. CLIP Performance Evaluation (Image-to-Text Retrieval and Text-to-Image Retrieval)

These metrics assess the ability of the CLIP model to correctly match images with their corresponding text descriptions, and vice-versa, in a zero-shot setting.

• Recall@K (R@K): This metric measures the proportion of queries for which the correct item (image or text) is found among the top $K$ retrieved results. R@1 specifically refers to whether the correct item is the top-1 retrieved result.
  • Conceptual Definition: Recall@K evaluates the retrieval accuracy. For $Image -> Text$ retrieval, it asks: "If I query with an image, is its correct caption among the top K retrieved captions?" For $Text -> Image$ retrieval, it asks: "If I query with a text, is its correct image among the top K retrieved images?" A higher R@K indicates better retrieval performance.
  • Mathematical Formula (for a single query and a set of candidates): $ \text{Recall@K} = \begin{cases} 1 & \text{if the correct item is in the top K retrieved candidates} \ 0 & \text{otherwise} \end{cases} $ The final Recall@K reported is typically the average over all queries in the test set.
  • Symbol Explanation:
    • $K$: An integer representing the number of top retrieved results considered (e.g., 1, 5, 10).
    • correct item: The ground truth corresponding image or text for a given query.
    • top K retrieved candidates: The K items (images or texts) that have the highest similarity scores with the query, according to the model.

5.2.2. CLIP Similarity (CLIP Sim)

• Conceptual Definition: CLIP Similarity measures the semantic alignment between a generated image and its guiding text description. It quantifies how well the generated image visually represents the concepts expressed in the text prompt. A higher CLIP Sim score indicates better text-image consistency.
• Mathematical Formula: $ \text{CLIP Sim} = \frac{1}{N} \sum_{i=1}^{N} \text{cosine_similarity}(\text{CLIP_Image_Encoder}(G(y_i)), \text{CLIP_Text_Encoder}(y_i)) $
  • Symbol Explanation:
    • $N$: The total number of generated image-text pairs being evaluated.
    • $i$: Index for each pair.
    • $G(y_i)$: The image generated by the text-to-image model (e.g., Taiyi-XL) from the text prompt $y_i$.
    • $\text{CLIP\_Image\_Encoder}(\cdot)$: The image encoder component of the CLIP model.
    • $\text{CLIP\_Text\_Encoder}(\cdot)$: The text encoder component of the CLIP model.
    • $\text{cosine\_similarity}(v_1, v_2) = \frac{v_1 \cdot v_2}{\|v_1\| \|v_2\|}$: Measures the cosine of the angle between two vectors $v_1$ and $v_2$. A value closer to 1 indicates higher similarity.
  • Function: This metric uses the CLIP model's learned multimodal embedding space to evaluate the semantic coherence between a generated image and its prompt.

5.2.3. Inception Score (IS)

• Conceptual Definition: The Inception Score (IS) is used to evaluate the quality and diversity of generated images. A higher IS suggests that the generated images are both "sharper" (high quality, recognizable objects) and "more diverse" (cover a wide range of categories). It does not directly evaluate text-image alignment.
• Mathematical Formula: $ \text{IS} = \exp\left( \mathbb{E}x \left[ D{KL}(p(y|x) || p(y)) \right] \right) $
  • Symbol Explanation:
    • $\exp(\cdot)$: Exponential function.
    • $\mathbb{E}_x[\cdot]$: Expectation over generated images $x$.
    • $D_{KL}(p(y|x) || p(y))$: The Kullback-Leibler (KL) divergence between two probability distributions.
    • $p(y|x)$: The conditional class probability distribution for a generated image $x$, obtained by feeding $x$ into a pre-trained Inception v3 network and taking its softmax output. This represents how likely an image contains a recognizable object.
    • p(y): The marginal class probability distribution, which is the average of $p(y|x)$ over all generated images. This represents the diversity of objects in the generated images.
  • Function: IS rewards images that have easily recognizable objects (low entropy $p(y|x)$) and a wide variety of these objects across the generated set (high entropy p(y)).

5.2.4. Fréchet Inception Distance (FID)

• Conceptual Definition: The Fréchet Inception Distance (FID) measures the "distance" between the distribution of generated images and the distribution of real images. It is a popular metric for assessing the realism and diversity of generated samples. A lower FID score indicates that the generated images are more similar to real images in terms of visual quality and statistical properties.
• Mathematical Formula: $ \text{FID} = |\mu_1 - \mu_2|_2^2 + \text{Tr}(\Sigma_1 + \Sigma_2 - 2(\Sigma_1 \Sigma_2)^{1/2}) $
  • Symbol Explanation:
    • $\mu_1$: The mean activation vector of the real images in a specific layer (e.g., last pooling layer) of a pre-trained Inception v3 network.
    • $\mu_2$: The mean activation vector of the generated images in the same layer of the Inception v3 network.
    • $\|\cdot\|_2^2$: The squared Euclidean distance.
    • $\text{Tr}(\cdot)$: The trace of a matrix (sum of diagonal elements).
    • $\Sigma_1$: The covariance matrix of the real images' activations.
    • $\Sigma_2$: The covariance matrix of the generated images' activations.
  • Function: FID compares the first (mean) and second (covariance) moments of the feature distributions of real and generated images. It is considered more robust than IS in capturing both image quality and diversity.

5.2.5. Human Preference Evaluation

• Conceptual Definition: This involves human raters assessing the quality, aesthetic appeal, and prompt adherence of generated images. Due to inherent subjectivity, the paper primarily employs a case analysis approach.
• Function: Instead of quantitative scoring, human evaluation focuses on qualitative examination, highlighting unique attributes, performance nuances, and distinct characteristics of image generation outcomes produced by different models. This helps in discerning subtle differences in style, detail, and prompt following that quantitative metrics might miss.

5.3. Baselines

For comparative analysis, Taiyi-XL was benchmarked against a range of established and state-of-the-art models, including both commercial and open-source solutions:

• Commercial/Closed-Source Models:

  • Midjourney (https://www.midjourney.com/): A highly regarded text-to-image generative AI program known for its high aesthetic quality and artistic output.
  • DALL-E 3 (Betker et al., 2023, https://openai.com/dall-e-3): OpenAI's latest iteration of the DALL-E series, recognized for its exceptional prompt-following capabilities and image quality.

• Open-Source Stable Diffusion Variants:

  • SD-XL (Podell et al., 2023): Stable Diffusion XL, the foundation model upon which Taiyi-XL is built. It's a powerful version of Stable Diffusion designed for high-resolution synthesis.
  • SD-v1.5 (Rombach et al., 2022): An earlier, widely used version of Stable Diffusion.

• Open-Source Chinese/Bilingual T2I Models:

  • Taiyi-v0.1 (Wang et al., 2022): The authors' previous work on Chinese text-to-image generation.
  • Alt-Diffusion (Ye et al., 2023): A multilingual text-to-image diffusion model.
  • Pai-Diffusion (Wang et al., 2023): Another Chinese diffusion model developed for text-to-image synthesis.

• CLIP Models (for retrieval evaluation):

  • CLIP (Radford et al., 2021): The original English-only CLIP model.

  • AltCLIP (Chen et al., 2022): A CLIP variant designed for extended language capabilities, including Chinese.

  • Our-CLIP: The CLIP model developed in this paper, which serves as the text encoder for Taiyi-XL.

    These baselines were chosen to represent the spectrum of current T2I capabilities, from general-purpose (SD-XL), to cutting-edge commercial (DALL-E 3, Midjourney), to specific Chinese/bilingual open-source models, allowing for a comprehensive evaluation of Taiyi-XL's advancements, particularly in bilingual generation and fidelity to textual prompts.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate the superior performance of Taiyi-XL and its underlying Our-CLIP model, especially in bilingual contexts.

6.1.1. CLIP Model Evaluation

The paper first evaluates the CLIP model, which is the backbone for text encoding in Taiyi-XL. The results, presented in Table 1, highlight the Our-CLIP model's exemplary performance in zero-shot image-text retrieval tasks across both English and Chinese datasets.

The following are the results from Table 1 of the original paper:

Analysis:

• English Datasets (Flickr30K, MSCOCO): Our-CLIP consistently outperforms the original CLIP and AltCLIP across all R@K metrics for both $Image -> Text$ and $Text -> Image$ retrieval. For instance, on $MSCOCO Text -> Image R@1$, Our-CLIP achieves 49.2%, significantly higher than CLIP's 36.5% and AltCLIP's 42.9%. This indicates that the continuous pre-training and enhancements applied to CLIP not only maintained but improved its English comprehension.

• Chinese Datasets (Flickr30K-CN, MSCOCO-CN): The original CLIP model performs very poorly on Chinese datasets, with R@1 scores often in the single digits or even 0%, highlighting its English-centric nature. AltCLIP shows a substantial improvement, demonstrating its multilingual adaptation. However, Our-CLIP further surpasses AltCLIP on Chinese tasks. Notably, for $Text -> Image R@1$ on Flickr30K-CN, Our-CLIP achieves 88.1% compared to AltCLIP's 84.8%. On $MSCOCO-CN Text -> Image R@1$, Our-CLIP reaches 69.7%, exceeding AltCLIP's 62.8%.

  This superior performance of Our-CLIP in bilingual contexts is critical as it forms the text encoder for Taiyi-XL. A robust CLIP model ensures a more nuanced understanding of user prompts in both languages, directly leading to better-aligned and higher-quality image generation.

6.1.2. Diffusion Model Evaluation

The next set of results focuses on the image generation capabilities of Taiyi-XL compared to other diffusion models using metrics like CLIP Sim, FID, and IS.

The following are the results from Table 2 of the original paper:

Analysis:

• English Dataset (COCO): Taiyi-XL demonstrates superior performance across all three metrics. It achieves the highest CLIP Sim (0.254), indicating strong semantic alignment between generated images and English text prompts. It also has the lowest FID (22.543), meaning its generated images are statistically closer to real images (better realism and diversity), and the highest IS (35.465), signifying high quality and diversity of generated content. This indicates that Taiyi-XL not only preserves but enhances the English generation capabilities compared to SD-XL and other baselines.

• Chinese Dataset (COCO-CN): Similar to the English results, Taiyi-XL leads all other models on the Chinese dataset. It boasts the highest CLIP Sim (0.225), the lowest FID (67.675), and the highest IS (22.965). The improvement over other Chinese-specific models like Taiyi-v0.1, Alt-Diffusion, and Pai-Diffusion is evident across all metrics. This confirms Taiyi-XL's robust bilingual capabilities, making it a leading solution for high-fidelity image generation from both English and Chinese prompts.

  Overall, these quantitative results strongly validate the effectiveness of Taiyi-XL's approach, particularly its ability to excel in both English and Chinese image generation tasks simultaneously.

6.1.3. Human Preference Evaluation

Qualitative assessments provide further insights into the model's performance nuances. Figure 3 illustrates Chinese text-to-image generation, and Figure 4 shows English text-to-image generation comparisons.

The following figure (Figure 3 from the original paper) shows the comparison of Different Models in Chinese Text-to-Image Generation Performance:

该图像是一个图表，展示了Taiyi-v0.1、Alt-Diffusion、Pai-Diffusion、DALL·E 3与Taiyi-XL五种模型，针对三条中文提示生成图像的对比效果，体现了各模型在细节和画质上的差异。

The following figure (Figure 4 from the original paper) shows the comparison of Different Models in English Text-to-Image Generation Performance:

Analysis:

• Improvements of XL versions: Both figures show that XL versions of models (SDXL, Taiyi-XL) generally produce significantly better images than their 1.5 counterparts (SD-v1.5, Alt-Diffusion). This indicates the benefits of larger model parameters, improved algorithms, and advanced training methodologies.
• Prompt Following: DALL-E 3 is noted for its exceptional prompt-following capability, setting a high benchmark for aligning images precisely with textual descriptions. It occasionally produces overly vivid colors.
• Aesthetic and Style: Taiyi-XL is characterized by a photographic style and is described as closely paralleling the performance of Midjourney in aesthetic appeal, particularly in English generation (Figure 4).
• Bilingual Support: A key distinction and strength of Taiyi-XL is its enhanced support for bilingual (Chinese and English) text-to-image generation. This is crucial for diverse linguistic contexts and is a major differentiator compared to models that are either English-only or lose English capabilities when adapted to Chinese. As seen in Figure 3, Taiyi-XL generates visually coherent and prompt-aligned images from Chinese text, outperforming Taiyi-v0.1, Alt-Diffusion, and Pai-Diffusion.
• Gap with Commercial Models: While Taiyi-XL significantly surpasses current bilingual open-source models, the authors acknowledge a gap with commercial models like DALL-E 3 and Midjourney. This disparity is primarily attributed to differences in the quantity, quality, and diversity of the image-text data used for training. Taiyi-XL was trained exclusively on copyright-compliant image-text data, highlighting the ongoing challenges related to data access and copyright in AIGC development.

6.1.4. Impact of Latent Consistency Models (LCM)

The paper also briefly discusses the impact of using Latent Consistency Models (LCM) to accelerate the image generation process, as visualized in Figure 5.

The following figure (Figure 5 from the original paper) shows Taiyi-XL generation examples with Latent Consistency Model:

Analysis:

• Inference Steps vs. Image Quality: A notable observation from LCM tests is the correlation between the reduction in inference steps and a consequent decline in image quality.
  • Single Step Generation: When constrained to a single generation step, the resulting images predominantly exhibit only basic outlines and lack finer details. For example, in Figure 5, the "astronaut riding a white horse" generated in 1 step is barely recognizable.
  • Eight Step Generation: Extending the generation process to 8 steps ensures a considerably higher quality of the generated images, preserving satisfactory levels of detail and overall image fidelity. The images generated with 8 steps in Figure 5 are significantly more detailed and visually appealing.
• Balancing Speed and Quality: This finding suggests that while LCM can effectively speed up generation, a balance must be struck. A minimum number of steps (e.g., eight) is crucial for maintaining acceptable image quality.

6.2. Ablation Studies / Parameter Analysis

The paper does not explicitly detail ablation studies on components of Taiyi-XL (e.g., the effect of LVLM-enriched prompts vs. original prompts, or specific aspects of the vocabulary expansion). However, the comparison of Our-CLIP with CLIP and AltCLIP (Table 1) implicitly serves as an ablation for the effectiveness of the bilingual continuous pre-training and vocabulary expansion strategies on the CLIP text encoder. The superior performance of Our-CLIP suggests these modifications are highly effective.

The analysis of Latent Consistency Models (LCM) (Figure 5) can be seen as a parameter analysis regarding the number of sampling steps. It shows that 8 steps are significantly better than 4 steps, and 4 steps are better than 1 step, directly illustrating the impact of this hyper-parameter on image quality for accelerated generation.

7. Conclusion & Reflections

7.1. Conclusion Summary

This research successfully introduces Taiyi-Diffusion-XL, a significant advancement in bilingual (Chinese and English) text-to-image generation. By implementing a novel bilingual continuous pre-training approach that efficiently expands CLIP's vocabulary and position encoding, the authors have developed an Our-CLIP model that excels in bilingual image-text retrieval. This enhanced CLIP model then serves as the foundation for Taiyi-Diffusion-XL, which further benefits from Large Vision-Language Models (LVLMs) to enrich text prompts, leading to higher visual quality and better semantic alignment in generated images. Empirical results quantitatively demonstrate Taiyi-Diffusion-XL's superior performance over previous open-source models across both English and Chinese datasets, while qualitative evaluations highlight its aesthetic quality comparable to commercial tools like Midjourney for specific styles. The project also contributes by open-sourcing the model, fostering future research and collaboration in diverse linguistic contexts within multimodal AI.

7.2. Limitations & Future Work

The authors acknowledge a primary limitation:

• Data Quantity, Quality, and Diversity: The performance gap between Taiyi-XL and state-of-the-art commercial models like DALL-E 3 and Midjourney is primarily attributed to differences in the quantity, quality, and diversity of the image-text data used for training. Taiyi-XL was trained exclusively on copyright-compliant data, which, while ethically sound, might limit the scale and breadth compared to datasets potentially available to commercial entities.

  Based on these limitations and the paper's overall direction, potential future research directions could include:

• Data Curation Strategies: Exploring novel methods for curating larger, higher-quality, and more diverse copyright-compliant bilingual datasets to further close the gap with commercial models. This could involve more advanced LVLM-driven synthetic data generation or improved filtering techniques for public data.

• Advanced LVLM Integration: Further integrating LVLMs for more complex prompt understanding, style transfer, or even interactive image generation, moving beyond mere caption enrichment.

• Cross-Cultural Nuances: Enhancing the model's understanding of cross-cultural nuances and less common linguistic expressions to generate images that are not only grammatically correct but also culturally sensitive and relevant.

• Efficiency for Low-Resource Languages: Applying similar bilingual expansion techniques to other low-resource languages to broaden the scope of multilingual text-to-image generation.

• Ethical AI and Bias Mitigation: Investigating and mitigating potential biases that might arise from training data in bilingual contexts, ensuring fair and inclusive image generation.

7.3. Personal Insights & Critique

Taiyi-Diffusion-XL represents a commendable and practical step forward in making advanced text-to-image generation truly accessible and effective for bilingual users, particularly those working with Chinese. The core innovation of extending existing powerful models through continuous pre-training and careful vocabulary expansion, rather than component replacement, is a robust and resource-efficient strategy. This preserves valuable pre-trained knowledge, which is crucial for achieving high performance in multiple languages. The integration of Large Vision-Language Models (LVLMs) for prompt enrichment is also a particularly insightful move. It addresses a common bottleneck in T2I models – the quality of the input prompt – by automatically enhancing it, thereby potentially improving user experience and generation quality without requiring users to craft overly elaborate prompts.

A potential area for deeper exploration, not explicitly detailed in the paper, is the specific technical implementation of the vocabulary and position encoding expansion. While mentioned in the abstract, a more detailed breakdown of how Chinese characters were integrated into CLIP's tokenizer (e.g., character-level, subword-level, or a hybrid approach) and how the absolute position encoding was adapted would be beneficial for other researchers looking to replicate or extend this work to other languages.

The acknowledged limitation regarding training data quantity, quality, and diversity is a common challenge for open-source models competing with well-funded commercial counterparts. This highlights an unverified assumption in the open-source community: that models can quickly catch up with commercial ones solely through architectural improvements. Often, the sheer scale and proprietary nature of commercial training datasets provide an almost insurmountable advantage. Taiyi-Diffusion-XL's focus on copyright-compliant data is highly responsible but underscores the broader ethical and practical dilemma in AI development regarding data access.

The methods and conclusions of this paper could be transferred to other domains requiring multilingual multimodal understanding. For instance, similar techniques could be used to build bilingual video generation models, or to improve multilingual visual question-answering systems. The LVLM-driven prompt enrichment could also inspire better data annotation strategies for multimodal datasets. Overall, Taiyi-Diffusion-XL is not just an improvement for Chinese T2I; it offers a generalizable blueprint for building robust, genuinely bilingual multimodal AI systems.