1. Bibliographic Information

1.1. Title

The title of the paper is "DiT4SR: Taming Diffusion Transformer for Real-World Image Super-Resolution". This title clearly indicates the central topic, which is the adaptation of Diffusion Transformer (DiT) models for the specific task of Real-World Image Super-Resolution (Real-ISR).

1.2. Authors

The authors are:

• Zheng-Peng Duan (1, 2)

• Jiawei Zhang (2)

• Xin Jin (1)

• Ziheng Zhang (1)

• Zheng Xiong (2)

• Dongqing Zou (2, 3)

• Jimmy S. Ren (2, 4)

• Chunle Guo (1)

• Chongyi Li (1)

  Affiliations:

1. vCIP CS, Nankai University

2. SensTime Research

3. pBVR

4. Hong Kong Metropolitan University

   The diverse affiliations, including universities and research labs (SensTime Research), suggest a collaboration between academic and industrial research, which is common in cutting-edge AI research. Jimmy S. Ren is a known figure in computer vision research.

1.3. Journal/Conference

The paper is published at (UTC): 2025-03-30T20:27:22.000Z. While a specific conference or journal is not explicitly mentioned in the provided text as the final publication venue (e.g., CVPR, NeurIPS), the arXiv preprint suggests it is undergoing peer review or has been submitted to a major venue. Given the publication date in 2025, it is likely intended for a top-tier computer vision or machine learning conference. The quality of research in this domain often appears in venues like CVPR, ICCV, ECCV, or NeurIPS, which are highly reputable and influential in the field.

1.4. Publication Year

The paper was published on March 30, 2025.

1.5. Abstract

The paper addresses the Real-World Image Super-Resolution (Real-ISR) problem, leveraging large-scale pre-trained diffusion models due to their rich generative priors. It focuses on Diffusion Transformer (DiT) models, which have shown superior performance over traditional UNet-based architectures in image generation. The core question investigated is whether advanced DiT-based diffusion models can be adopted for Real-ISR.

To answer this, the authors propose DiT4SR, a pioneering work in taming large-scale DiT models for Real-ISR. Unlike methods such as ControlNet that directly inject low-resolution (LR) image embeddings, DiT4SR integrates LR embeddings directly into the original attention mechanism of DiT. This allows for bidirectional information flow between the LR latent and the generated latent, enabling the LR stream to evolve and provide progressively refined guidance aligned with the diffusion process. Additionally, a cross-stream convolution layer is introduced to inject LR guidance into the generated latent, compensating for DiT's limited ability to capture local information. These designs enable DiT4SR to achieve superior performance in Real-ISR, as validated by extensive experiments.

1.6. Original Source Link

The original source link is: https://arxiv.org/abs/2503.23580 The PDF link is: https://arxiv.org/pdf/2503.23580v2.pdf This indicates the paper is available as a preprint on arXiv, a common platform for sharing research before formal peer review and publication. The v2.pdf suggests it's an updated version of the preprint.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is Real-World Image Super-Resolution (Real-ISR). This task involves recovering a high-resolution (HR) image from its low-resolution (LR) counterpart, which has undergone various complex degradations (e.g., compression, blur, noise).

This problem is important because Real-ISR is highly ill-posed; a single LR image can correspond to multiple possible HR images. Beyond simply removing degradations, it requires generating perceptually realistic details to enhance visual quality, which demands significant prior knowledge from the model. Traditional Super-Resolution (SR) methods often struggle with the complex, real-world degradations and the generation of realistic, high-fidelity details.

Prior research has turned to large-scale pre-trained text-to-image (T2I) diffusion models like Stable Diffusion (SD) due to their rich generative priors learned from vast datasets of high-quality images. These models, often based on UNet architectures, leverage techniques like ControlNet to guide the generative process with LR images.

However, recent advancements have shown that Diffusion Transformers (DiT), such as those used in SD3 and Flux, offer overwhelmingly superior performance in image generation compared to UNet-based architectures. This raises a critical question: Can the advanced DiT-based diffusion model be effectively adopted for Real-ISR? The challenge lies in efficiently and robustly integrating LR information into the DiT architecture, especially considering DiT's global attention mechanism and potential limitations in capturing local details. An intuitive ControlNet-like approach might not fully leverage DiT's unique characteristics.

2.2. Main Contributions / Findings

The paper's primary contributions are:

1. Pioneering DiT for Real-ISR with Integrated LR Stream: DiT4SR is presented as one of the first works to adapt large-scale DiT models for Real-ISR. Instead of merely copying blocks (like ControlNet), DiT4SR integrates the LR stream directly into the original DiT block's attention mechanism. This design enables bidirectional information interaction between the LR guidance and the diffusion process, allowing the LR stream to evolve and provide more refined, context-aware guidance throughout the diffusion process.

2. Cross-Stream Convolution for Local Information: The authors introduce a convolutional layer to inject LR guidance into the Noise Stream's MLP (Multi-Layer Perceptron) processing. This mechanism specifically compensates for DiT's inherent limitation in capturing local information effectively due to its global attention mechanism, which is crucial for restoring fine details in SR tasks.

   The key findings are that these simple yet effective designs endow the DiT model with superior performance in Real-ISR. Extensive experiments demonstrate DiT4SR's capability to produce high-quality restoration results, often outperforming state-of-the-art methods on real-world datasets, particularly in clarity, detail abundance, and the processing of fine structures. A user study further confirms its superiority in image realism and fidelity.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand DiT4SR, several foundational concepts are crucial:

• Super-Resolution (SR): The task of enhancing the resolution of an image, typically by generating a high-resolution (HR) image from a low-resolution (LR) input.
• Real-World Image Super-Resolution (Real-ISR): A more challenging variant of SR where LR images are degraded by complex, often unknown, real-world factors (e.g., camera noise, compression artifacts, blur) rather than just simple downsampling. This task requires not only removing degradations but also hallucinating realistic details.
• Generative Models: A class of machine learning models designed to generate new data samples that resemble the training data. For Real-ISR, they are used to generate plausible HR details.
• Diffusion Models: A type of generative model that works by learning to reverse a gradual diffusion (noise addition) process. During training, noise is progressively added to data, and the model learns to denoise it. During inference, the model starts with random noise and iteratively denoises it to generate a new data sample.
• Latent Diffusion Models (LDM) / Stable Diffusion (SD): A variant of diffusion models that operates in a lower-dimensional latent space rather than directly on pixel space. This makes them computationally more efficient. Stable Diffusion is a prominent example of an LDM capable of high-quality image generation from text prompts.
• UNet Architecture: A convolutional neural network architecture, originally designed for biomedical image segmentation, characterized by its U-shaped structure. It features an encoder (downsampling path) that captures context and a decoder (upsampling path) that enables precise localization, with skip connections between corresponding encoder and decoder layers to preserve fine-grained information. UNets are widely used as the backbone for many diffusion models.
• Transformers: A neural network architecture that relies heavily on the self-attention mechanism to weigh the importance of different parts of the input data. Transformers have revolutionized Natural Language Processing (NLP) and are increasingly popular in computer vision tasks.
• Self-Attention Mechanism: A core component of transformers. It allows a model to weigh the importance of different elements in a sequence when processing a single element. The fundamental formula for Scaled Dot-Product Attention is: $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ Where:
  • $Q$ (Query), $K$ (Key), $V$ (Value) are matrices derived from the input embeddings.
  • $Q K^T$ calculates the dot product between queries and keys, determining attention scores.
  • $\sqrt{d_k}$ is a scaling factor, typically the square root of the dimension of the key vectors, used to prevent large dot products from pushing the softmax function into regions with tiny gradients.
  • softmax normalizes the scores to produce attention weights.
  • $V$ is then weighted by these attention weights.
• Diffusion Transformer (DiT): A transformer-based architecture used as the backbone for diffusion models, replacing the traditional UNet. DiTs process image patches as sequences of tokens, leveraging the global receptive field and scaling properties of transformers for improved generative performance.
• Multimodal Diffusion Transformers (MM-DiT): An extension of DiT that handles multiple modalities (e.g., text and image) by processing them in separate "streams" (e.g., Text Stream, Noise Stream) but allowing them to interact bidirectionally through joint attention mechanisms.
• ControlNet: An neural network architecture that adds conditional control to large pre-trained diffusion models (typically UNet-based). It works by creating a trainable copy of the diffusion model's UNet encoder, which takes an additional condition (e.g., an LR image, edge map, human pose) as input. The outputs of this conditional encoder are then injected into the original UNet's skip connections to guide the generation process. It usually employs a one-way information flow.
• Variational Autoencoder (VAE) encoder/decoder: VAEs are generative models used to learn a latent space representation of data. The encoder maps high-dimensional input (e.g., an image) to a lower-dimensional latent representation, while the decoder reconstructs the original input from the latent representation. In LDMs, the VAE encoder is used to transform images into a latent space for diffusion, and the VAE decoder reconstructs images from the denoised latent.

3.2. Previous Works

The paper contextualizes its contribution by discussing prior work in Image Super-Resolution (ISR) and Diffusion Models for Real-ISR.

• Traditional ISR Methods:
  • Convolutional Networks (CNNs): Early deep learning methods like SRCNN [18], EDSR [32], RCAN [72], RAN [71] achieved significant progress but struggled with Real-ISR's complex degradations.
  • Transformers: More recent ISR methods like IPT [10], HAT [13], SwinIR [29], DAT [12] leveraged transformer architectures. SwinIR is explicitly used as a baseline.
• GAN-based Real-ISR: To address the ill-posedness and generate realistic details, Generative Adversarial Networks (GANs) were proposed, such as ESRGAN [52], GLEAN [6], DGAN [9], EDAN [31]. While generating perceptually realistic details, these often suffer from training instability and unnatural visual artifacts [8, 30, 60]. Real-ESRGAN [53] and BSRGAN [67] further explored complex degradation models. Real-ESRGAN is used as a baseline.
• Diffusion-based Real-ISR (UNet-centric): With the success of Stable Diffusion (SD) [41], many methods built upon its UNet architecture, leveraging its generative priors.
  • ControlNet-like Approaches: StableSR [51] and DiffBIR [33] inject LR information as conditions via ControlNet [68] or similar mechanisms. SeeSR [58] and PASD [63] further incorporate high-level semantic information. ResShift [65], OSEDiff [57], and SUPIR [64] are other examples. SUPIR specifically investigates scaling effects.
  • Other methods focus on efficiency, such as OSD [35], PatchScaler [34], Distillation-free one-step diffusion [27], SINSr [54], Addsr [61].
• Diffusion Transformer (DiT) for Generation: DiT [36] itself revolutionized generative modeling.
  • PixArt-alpha [11], SD3 [20], and Flux [3] are prominent examples of large-scale T2I models that leverage Multimodal Diffusion Transformers (MM-DiTs) to integrate text and image modalities. These models achieve state-of-the-art performance due to their ability for full interaction between modalities. SD3 is the direct foundation for DiT4SR.
• Diffusion Transformer (DiT) for SR/Restoration:
  • DiT-SR [14] trains DiT-based SR models from scratch.
  • DreamClear [2] proposes a DiT-based image restoration model but still adopts ControlNet to inject LR information, which the authors argue prevents it from fully leveraging DiT's advantages. DreamClear is used as a baseline.
  • Two concurrent methods, TSD-SR [19] and One diffusion step [28], explore one-step DiT-based SR models using flow trajectory concepts.

3.3. Technological Evolution

The evolution of image super-resolution has progressed from traditional signal processing methods to deep learning. Initially dominated by Convolutional Neural Networks (CNNs), the field saw a shift towards Generative Adversarial Networks (GANs) to tackle the perceptual quality aspect of Real-ISR. More recently, diffusion models, particularly Latent Diffusion Models (LDMs) like Stable Diffusion (SD), have emerged as powerful generative priors for Real-ISR, largely due to their ability to produce highly realistic and diverse results.

The architectural backbone of these diffusion models has also evolved. Initially, UNet architectures were standard. However, Transformers, with their global attention mechanisms and scalability, have proven superior in general image generation tasks, leading to the development of Diffusion Transformers (DiT). These DiT models, especially Multimodal Diffusion Transformers (MM-DiTs) like SD3, have pushed the boundaries of generative performance.

This paper's work (DiT4SR) represents a crucial next step in this evolution. It aims to bridge the gap between the advanced DiT architecture (proven in generation) and the Real-ISR task, moving beyond UNet-based SD and even DiT-based models that still rely on ControlNet-like external conditioning. By directly integrating LR information within the DiT's attention mechanism and addressing DiT's local information limitations, DiT4SR seeks to unlock the full potential of DiT for Real-ISR.

3.4. Differentiation Analysis

DiT4SR differentiates itself from previous methods primarily in how it integrates low-resolution (LR) information into the Diffusion Transformer (DiT) architecture for Real-World Image Super-Resolution (Real-ISR).

• Compared to UNet-based Diffusion Models (e.g., StableSR, DiffBIR, SeeSR, SUPIR): These methods build upon UNet-based Stable Diffusion and typically use ControlNet or similar mechanisms. DiT4SR fundamentally differs by using a DiT backbone, which has shown overwhelming performance over UNet in image generation due to its global attention and scalability.

• Compared to ControlNet-like Approaches (e.g., SD3-ControlNet, DreamClear): This is the most direct point of differentiation.

  • ControlNet (and SD3-ControlNet): Duplicates several blocks of the main network, processes LR information in these separate blocks, and then injects the LR embedding into the Noise Stream (or UNet's skip connections) via trainable linear layers. This creates a one-directional information flow. The LR stream does not evolve or bidirectionally interact with the generated latent.
  • DreamClear: Although DreamClear uses a DiT-based backbone for image restoration, it still employs the ControlNet paradigm for LR information injection.
  • DiT4SR's Innovation: DiT4SR completely abandons the ControlNet-like approach.
    1. It integrates the LR Stream directly into the original DiT block's attention mechanism. This enables bidirectional information interaction between the LR latent and the generated latent (Noise Stream), allowing the LR Stream to continuously adapt and evolve alongside the diffusion process. This creates progressively refined guidance.
    2. It introduces a cross-stream convolution layer to inject LR guidance between the MLP layers of the LR Stream and Noise Stream. This addresses DiT's limited ability to capture local information, which is crucial for fine details in SR. ControlNet primarily focuses on global structural guidance, not local detail enhancement through a convolutional injection like DiT4SR.

• Compared to other DiT-based SR models (e.g., DiT-SR, concurrent one-step methods): While DiT-SR trains DiT models from scratch for SR, DiT4SR focuses on taming large-scale pre-trained DiT models (specifically SD3) by designing a control mechanism. The concurrent one-step methods focus on flow trajectory distillation for efficiency, which is a different aspect from DiT4SR's control mechanism design.

  In essence, DiT4SR's core innovation lies in its tailored approach to leverage DiT's architectural strengths by enabling rich, bidirectional interaction of LR guidance and addressing DiT's local information shortcomings through a novel convolutional injection, rather than simply adapting existing conditioning mechanisms.

4. Methodology

4.1. Principles

The core idea behind DiT4SR is to effectively adapt a large-scale pre-trained Diffusion Transformer (DiT) model, specifically SD3, for the task of Real-World Image Super-Resolution (Real-ISR). The theoretical basis and intuition are rooted in two main observations:

1. Leveraging DiT's Generative Power: DiT-based models (like SD3) have demonstrated overwhelming performance in image generation due to their Multimodal Diffusion Transformers (MM-DiTs), which facilitate rich bidirectional information flow between different modalities (e.g., text and image) via attention. The principle is to extend this multi-modal interaction to include low-resolution (LR) image information as another "stream" that can actively guide the generation of high-resolution (HR) images.

2. Addressing DiT's Limitations for SR: While DiT excels at global context and generation, its attention mechanism operates globally and may lack the ability to sufficiently capture local information, which is paramount for restoring fine details in SR tasks. The principle is to complement the global attention-based guidance with a mechanism that specifically introduces local LR guidance.

   Therefore, DiT4SR aims to integrate LR information directly and interactively into the DiT's core attention mechanism (enabling bidirectional flow and evolution of guidance) and to provide a targeted local information injection through a convolutional layer. This combined approach allows the DiT to leverage its generative priors while being precisely controlled by LR input to produce high-fidelity Real-ISR results.

4.2. Core Methodology In-depth (Layer by Layer)

DiT4SR is built upon the DiT-architectured SD3. The overall process involves a diffusion process in the latent space, guided by LR information and text prompts.

4.2.1. Overall Architecture

The DiT4SR architecture is similar to SD3 but with key modifications to integrate LR information.

As shown in Figure 3(a), the process starts with a noisy latent $\mathbf{Z}$ and an LR image $\mathbf{I}_{LR}$.

• Input Processing:

  1. The noisy latent $\mathbf{Z} \in \mathbb{R}^{H \times W \times C}$ (where H, W, C are height, width, and channels) is first flattened into a patch sequence of length $K = \frac{H}{2} \cdot \frac{W}{2}$. This sequence is then projected into a $D$-dimensional space using a linear layer to obtain the noisy image token $\mathbf{X} \in \mathbb{R}^{K \times D}$. Position embedding is added to $\mathbf{X}$ to retain spatial information.

  2. The LR image $\mathbf{I}_{LR}$ is encoded into a latent space using a pre-trained VAE encoder to obtain the LR latent. Since LR latent and noisy latent are both visual representations, they undergo the same processing: flattening into a patch sequence, linear projection to $D$-dimensions, and addition of the same position embedding to form the LR image token $\mathbf{L} \in \mathbb{R}^{K \times D}$.

  3. A text caption describing $\mathbf{I}_{LR}$ is encoded by three pre-trained text models (CLIP-L [39], CLIP-G [15], and T5 XXL [40]). Pooled representations from CLIP models are combined with the timestep $t$ to modulate DiT's internal features. All three text representations are combined to construct a text token $\mathbf{C} \in \mathbb{R}^{M \times D}$ of length $M$.

     These three tokens ($\mathbf{X}$, $\mathbf{L}$, $\mathbf{C}$) represent the Noise Stream, LR Stream, and Text Stream, respectively. The MM-DiT block of SD3 is modified into an MM-DiT-Control block to handle these three streams. After passing through $N$ MM-DiT-Control blocks and an unpatch operation, the Noise Stream outputs the denoised latent for the current timestep $t$. This diffusion process is repeated for $T$ steps, and the final clean latent $\mathbf{Z}_0$ is then decoded by the VAE decoder to obtain the desired HR result $\mathbf{R}$.

The following figure (Figure 3 from the original paper) shows the overall architecture:

该图像是论文中DiT4SR方法的结构示意图，包括(a)整体框架和(b)单个MM-DiT-Control模块的详细结构，展示了多流融合Transformer在扩散模型中的应用及LR信息的注入机制。

This design contrasts with SD3-ControlNet (shown in Figure 2(a)), where ControlNet processes the LR Stream in additional DiT blocks and injects LR embedding into the Noise Stream via trainable linear layers, establishing only a one-way information flow. In contrast, DiT4SR (shown in Figure 2(b)) integrates the LR Stream directly into the original DiT blocks, enabling bidirectional information flow.

The following figure (Figure 2 from the original paper) shows the network structure comparison:

该图像是论文中图2的示意图，比较了SD3-ControlNet与DiT4SR的网络结构。图中红色箭头表示信息流向，DiT4SR实现了LR流与噪声流的双向信息交互，而SD3-ControlNet仅为单向流动，限制了交互。

4.2.2. LR Integration in Attention

This is the first key modification within each MM-DiT-Control block. Instead of separate ControlNet blocks, the LR Stream is integrated into the joint attention mechanism of the DiT.

The input to the joint attention mechanism comprises the noisy image token $\mathbf{X}$, the LR image token $\mathbf{L}$, and the text token $\mathbf{C}$. For each stream, separate linear projections are used to compute the Query ($\mathbf{Q}$), Key ($\mathbf{K}$), and Value ($\mathbf{V}$) matrices.

The input $Q$, $K$, $V$ for the joint attention are formulated as: $ \begin{array} { r } { \mathbf { Q } = P _ { \mathbf { Q } } ^ { \mathbf { X } } ( \mathbf { X } ) \circledast P _ { \mathbf { Q } } ^ { \mathbf { L } } ( \mathbf { L } ) \circledast P _ { \mathbf { Q } } ^ { \mathbf { C } } ( \mathbf { C } ) , } \ { \mathbf { K } = P _ { \mathbf { K } } ^ { \mathbf { X } } ( \mathbf { X } ) \circledast P _ { \mathbf { K } } ^ { \mathbf { L } } ( \mathbf { L } ) \circledast P _ { \mathbf { K } } ^ { \mathbf { C } } ( \mathbf { C } ) , } \ { \mathbf { V } = P _ { \mathbf { V } } ^ { \mathbf { X } } ( \mathbf { X } ) \circledast P _ { \mathbf { V } } ^ { \mathbf { L } } ( \mathbf { L } ) \circledast P _ { \mathbf { V } } ^ { \mathbf { C } } ( \mathbf { C } ) , } \end{array} $ Where:

• $\mathbf{X} \in \mathbb{R}^{K \times D}$: The noisy image token (from the Noise Stream).

• $\mathbf{L} \in \mathbb{R}^{K \times D}$: The LR image token (from the LR Stream).

• $\mathbf{C} \in \mathbb{R}^{M \times D}$: The text token (from the Text Stream).

• $P_{\mathbf{Q}}^{\mathbf{X}}, P_{\mathbf{K}}^{\mathbf{X}}, P_{\mathbf{V}}^{\mathbf{X}}$: Pre-trained fixed linear projections for the noisy image token $\mathbf{X}$.

• $P_{\mathbf{Q}}^{\mathbf{C}}, P_{\mathbf{K}}^{\mathbf{C}}, P_{\mathbf{V}}^{\mathbf{C}}$: Pre-trained fixed linear projections for the text token $\mathbf{C}$.

• $P_{\mathbf{Q}}^{\mathbf{L}}, P_{\mathbf{K}}^{\mathbf{L}}, P_{\mathbf{V}}^{\mathbf{L}}$: Newly created trainable linear projections specifically for the LR image token $\mathbf{L}$. Their weights are initialized to zeros to ensure that at the beginning of training, the LR Stream has minimal impact, which then progressively increases as the model learns.

• $\circledast$: This symbol denotes an operation that concatenates (or stacks) the outputs of the linear projections for $\mathbf{X}$, $\mathbf{L}$, and $\mathbf{C}$ along a sequence dimension. This creates a combined sequence of queries, keys, and values from all three streams, allowing them to interact through a single attention mechanism.

  The joint attention in the MM-DiT-Control block is then calculated using the standard Scaled Dot-Product Attention formula: $ \mathrm { A t t e n t i o n } ( \mathbf { Q } , \mathbf { K } , \mathbf { V } ) = \underbrace { \mathrm { s o f t m a x } ( \frac { \mathbf { Q } \mathbf { K } ^ { T } } { \sqrt { d } } ) } _ { \mathrm { a t t e n t i o n } \operatorname* { m a p } } \mathbf { V } $ Where:

• $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$: The concatenated query, key, and value matrices from all three streams.

• $d$: The dimension of the keys (and queries), used as a scaling factor.

• softmax: The softmax function applied to the scaled dot products to obtain attention weights.

  This formulation allows for comprehensive interaction between all three streams. The paper visualizes the attention maps between noisy image token $\mathbf{X}$ and LR image token $\mathbf{L}$ in Figure 4(a), showing clear diagonal activations in both self-attention (e.g., $\mathbf{X} \to \mathbf{X}$, $\mathbf{L} \to \mathbf{L}$) and cross-attention regions (e.g., $\mathbf{X} \to \mathbf{L}$, $\mathbf{L} \to \mathbf{X}$), indicating strong bidirectional information interaction. This interaction is crucial because it allows the Noise Stream to be guided by LR information, and simultaneously, the LR Stream to adapt and provide context-aware guidance based on the evolving state of the Noise Stream.

The following figure (Figure 4 from the original paper) visualizes attention maps:

LR Residual: The paper notes that the information interaction between $\mathbf{L}$ and $\mathbf{X}$ can decay through successive attention blocks (as shown in Figure 4(b) without LR Residual). This happens because the LR Stream might experience undesired disruptions as it evolves with the Noise Stream. To maintain the consistency of LR guidance in deeper transformer blocks, an additional shortcut (residual connection) is introduced. This LR Residual directly transfers the input LR information to the output of the joint attention mechanism within the LR Stream. This mechanism ensures that the LR guidance is effectively preserved and exerts a consistent influence throughout the diffusion process.

4.2.3. LR Injection between MLP

This is the second key modification, addressing DiT's limitation in capturing local information. The joint attention mechanism operates at a global level, relying primarily on position embeddings for spatial information. For Super-Resolution tasks, accurately restoring fine details requires strong local information. Relying solely on global attention is insufficient, as demonstrated by results in Figure 5(b) which show challenges in restoring text and maintaining fidelity.

To overcome this, DiT4SR introduces an LR Injection mechanism between the MLP (Multi-Layer Perceptron) layers of the LR Stream and the Noise Stream.

• Within both the MLP for the LR Stream and the Noise Stream, the hidden state dimensions are first expanded by a factor of 4 and then projected back to the original size using two linear projections.
• Let these intermediate features be $\bar{\phi(\mathbf{X})} \in \mathbb{R}^{K \times 4D}$ for the Noise Stream and $\eta(\mathbf{L}) \in \mathbb{R}^{K \times 4D}$ for the LR Stream.
• The LR information from $\eta(\mathbf{L})$ is injected into $\bar{\phi(\mathbf{X})}$ using a 3x3 depth-wise convolution layer.

  1. First, $\eta(\mathbf{L})$ is reshaped from its token form ($\mathbb{R}^{K \times 4D}$) to an image-like form $\eta(\mathbf{L})' \in \mathbb{R}^{\frac{H}{2} \times \frac{W}{2} \times 4D}$.

  2. This reshaped LR feature is then passed through a 3x3 depth-wise convolution layer. The weights of this convolution layer are also initialized to zeros, allowing its influence to grow gradually during training.

  3. After the convolution, the output is reshaped back to the image token form and added to $\bar{\phi(\mathbf{X})}$.

     This 3x3 depth-wise convolution layer serves two critical purposes:

1. Strengthening LR guidance: It provides an additional channel for LR information to influence the Noise Stream.

2. Capturing Local Information: Unlike global attention, convolution inherently excels at capturing local spatial patterns. By using a convolutional layer, DiT4SR effectively compensates for DiT's limited local information-capturing ability. This is crucial for restoring fine structures like text, as shown in Figure 5(d), which performs significantly better than using a simple linear layer (Figure 5(c)).

   The following figure (Figure 5 from the original paper) illustrates the effect of LR injection:

   该图像是图5的示意图，展示了使用不同方法注入低分辨率（LR）信息对超分辨率恢复的效果对比。(a)为LR输入，(b)未注入LR信息，(c)通过线性层注入LR信息，(d)为DiT4SR方法，用卷积层替代线性层，显著提升了局部细节恢复效果。

5. Experimental Setup

5.1. Datasets

Training Datasets

The authors use a combination of several high-quality image datasets for training, augmented with self-captured images:

• DIV2K [1]: A widely used dataset for super-resolution and image restoration tasks.

• DIV8K [22]: An extension of DIV2K with 8K resolution images.

• Flickr2K [46]: Another common dataset for image restoration, consisting of a large number of diverse images.

• FFHQ (first 10K face images) [24]: A high-quality dataset of human faces, useful for training models that need to generate realistic facial details.

• 1K self-captured high-resolution images: Added to fully exploit the potential of the method and scale up the training dataset, indicating a focus on real-world applicability.

  To create LR-HR training pairs, the degradation pipeline from Real-ESRGAN [53] is utilized, using the same parameter configuration as SeeSR [58]. This ensures the training data mimics real-world degradations. The resolutions are set to $128 \times 128$ for LR images and $512 \times 512$ for HR images, implying a $\times 4$ scaling factor.

Evaluation Datasets

DiT4SR is evaluated on four widely used real-world datasets specifically designed for Real-ISR tasks, all with a scaling factor of $\times 4$.

• DrealSR [56]: Consists of 93 images. Center-cropping is adopted, with LR images at $128 \times 128$ resolution.

• RealSR [5]: Consists of 100 images. Center-cropping is adopted, with LR images at $128 \times 128$ resolution.

• RealLR200 [58]: Proposed in SeeSR, it comprises 200 images of significantly different resolutions and lacks corresponding ground-truth (GT) images.

• RealLQ250 [2]: Established by DreamClear, it consists of 200 images with a fixed resolution of $256 \times 256$ and also lacks corresponding GT images.

  The inclusion of datasets without GT images (RealLR200, RealLQ250) is crucial for Real-ISR as real-world scenarios often lack perfect GTs, making evaluation metrics that do not require GTs (non-reference metrics) particularly relevant.

5.2. Evaluation Metrics

The paper employs a suite of perceptual and non-reference image quality assessment (IQA) metrics, acknowledging that traditional full-reference metrics like PSNR and SSIM [55] may not accurately reflect visual quality for generative tasks [4, 23, 64]. The metrics used are:

1. LPIPS (Learned Perceptual Image Patch Similarity) [69]:

   • Conceptual Definition: LPIPS measures the perceptual distance between two images by comparing their deep features extracted from a pre-trained CNN (e.g., AlexNet, VGG). It aims to align better with human perception of image similarity than traditional pixel-wise metrics like PSNR or SSIM. A lower LPIPS score indicates higher perceptual similarity (better fidelity).
   • Mathematical Formula: $ \text{LPIPS}(x, x_0) = \sum_{l} \frac{1}{H_l W_l} |w_l \odot (\phi_l(x) - \phi_l(x_0))|_2^2 $
   • Symbol Explanation:
     • $x$: The generated image.
     • $x_0$: The reference (ground truth) image.
     • $\phi_l(\cdot)$: Feature stack from layer $l$ of a pre-trained CNN (e.g., AlexNet).
     • $w_l$: A learnable scalar weight for channel $l$.
     • $H_l, W_l$: Height and width of the feature map at layer $l$.
     • $\odot$: Element-wise product.
     • $\|\cdot\|_2^2$: Squared $L_2$ norm.
     • The sum is taken over different layers $l$.

2. MUSIQ (Multi-scale Image Quality Transformer) [25]:

   • Conceptual Definition: MUSIQ is a non-reference image quality assessment (NR-IQA) metric that uses a transformer architecture to predict image quality scores. It operates on multiple scales of the input image and aggregates features using attention, designed to correlate well with human quality judgments for images with various distortions. A higher MUSIQ score indicates better image quality.
   • Mathematical Formula: (The specific full mathematical formula for MUSIQ's internal transformer computations is complex and not typically provided in a single equation. Instead, its score is a scalar output from its trained model.) $ \text{MUSIQ Score} = f_{\text{MUSIQ}}(\text{Image}) $
   • Symbol Explanation:
     • $f_{\text{MUSIQ}}$: The trained MUSIQ transformer model.
     • Image: The input image for which quality is to be assessed.

3. MANIQA (Multi-dimension Attention Network for No-Reference Image Quality Assessment) [62]:

   • Conceptual Definition: MANIQA is another non-reference IQA metric that leverages a multi-dimension attention network to predict image quality. It extracts features from different dimensions (e.g., channel, spatial, temporal if applicable) and uses attention mechanisms to weigh their importance, aiming for high correlation with human perception. A higher MANIQA score indicates better image quality.
   • Mathematical Formula: (Similar to MUSIQ, MANIQA is a complex neural network. Its output is a scalar quality score.) $ \text{MANIQA Score} = f_{\text{MANIQA}}(\text{Image}) $
   • Symbol Explanation:
     • $f_{\text{MANIQA}}$: The trained MANIQA network.
     • Image: The input image.

4. ClipIQA (Exploring CLIP for Assessing the Look and Feel of Images) [50]:

   • Conceptual Definition: ClipIQA is a non-reference IQA metric that utilizes the CLIP (Contrastive Language-Image Pre-training) model's ability to understand image content and relate it to textual concepts. It typically involves comparing an image against quality-related text prompts to infer its quality, or using CLIP's image encoder features to predict quality. A higher ClipIQA score indicates better image quality.
   • Mathematical Formula: (Like MUSIQ and MANIQA, ClipIQA is model-based.) $ \text{ClipIQA Score} = f_{\text{ClipIQA}}(\text{Image}) $
   • Symbol Explanation:
     • $f_{\text{ClipIQA}}$: The model derived from CLIP for quality assessment.
     • Image: The input image.

5. LIQE (Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective) [70]:

   • Conceptual Definition: LIQE is a blind (non-reference) IQA method that estimates image quality by considering vision-language correspondence. It often uses multitask learning to predict various quality attributes that collectively determine the overall quality score, aiming for strong correlation with human judgment across different distortion types. A higher LIQE score indicates better image quality.
   • Mathematical Formula: (Similar to the other NR-IQA metrics, LIQE is a neural network output.) $ \text{LIQE Score} = f_{\text{LIQE}}(\text{Image}) $
   • Symbol Explanation:
     • $f_{\text{LIQE}}$: The trained LIQE network.
     • Image: The input image.

5.3. Baselines

The paper compares DiT4SR against a comprehensive set of state-of-the-art Real-ISR methods, categorized by their underlying architecture:

1. GAN-based Methods:

   • Real-ESRGAN [53]: A highly influential method for real-world blind super-resolution using pure synthetic data for training.
   • SwinIR [29]: A transformer-based image restoration model, often used for SR, demonstrating strong performance.

2. Diffusion-based Methods with UNet Architecture: These methods are built upon Stable Diffusion (SD) models, which typically employ a UNet backbone.

   • ResShift [65]: An efficient diffusion model for image super-resolution using residual shifting.
   • StableSR [51]: Exploits diffusion prior for real-world image super-resolution.
   • SeeSR [58]: A semantics-aware real-world image super-resolution method.
   • DiffBIR [33]: A method for blind image restoration using generative diffusion prior.
   • OSEDiff [57]: An one-step effective diffusion network for real-world image super-resolution.
   • SUPIR [64]: Investigates model scaling for photorealistic image restoration in the wild, often built on SDXL.

3. Diffusion-based Methods with DiT Architecture: These methods specifically use a Diffusion Transformer backbone.

   • DreamClear [2]: A DiT-based image restoration model that, despite using DiT, still adopts ControlNet for LR information injection.

   • SD3-ControlNet: This is a custom baseline initialized with SD3.5-medium parameters and trained under the same settings as DiT4SR. It represents a direct application of ControlNet to SD3, serving as a crucial comparison to highlight the benefits of DiT4SR's integrated approach versus a ControlNet-like paradigm for DiT.

     These baselines cover a wide range of architectures and approaches within the Real-ISR field, allowing for a robust evaluation of DiT4SR's performance.

6. Results & Analysis

6.1. Core Results Analysis

The paper presents quantitative and qualitative comparisons, along with a user study, to demonstrate the effectiveness of DiT4SR.

Quantitative Comparisons (Table 1): The following are the results from Table 1 of the original paper:

• DrealSR and RealSR: For these datasets, DiT4SR achieves competitive results. While SeeSR and DiffBIR sometimes show slightly better LPIPS (lower is better), DiT4SR performs well on MUSIQ, MANIQA, and ClipIQA. For instance, on DrealSR, DiT4SR's MUSIQ (64.950) is strong, close to DiffBIR (65.665) and SeeSR (65.077). On RealSR, DiT4SR's MANIQA (0.661) is the highest, and MUSIQ (68.073) is very competitive.

• RealLR200 and RealLQ250: These are crucial real-world datasets that lack GT images, making non-reference metrics especially important. DiT4SR exhibits overwhelming performance on these datasets, achieving the top performance across all non-reference metrics (MUSIQ, MANIQA, ClipIQA, LIQE). For example, on RealLQ250, DiT4SR achieves MUSIQ 71.832, MANIQA 0.632, ClipIQA 0.578, and LIQE 4.356, consistently outperforming all other methods by a significant margin.

  These quantitative results highlight DiT4SR's superior capability in producing high-quality restoration results, especially for challenging real-world scenarios where GT images are unavailable. This success can be attributed to its effective leverage of DiT's generative capabilities through its novel LR integration and local information injection mechanisms.

Qualitative Comparisons (Figure 6): The following figure (Figure 6 from the original paper) shows qualitative comparisons:

该图像是多个真实世界低分辨率图像与不同超分辨率方法重建结果的对比示意图。图中包含DiT4SR和其他五种算法的细节放大区域，展示了DiT4SR在细节恢复上的优势。

Figure 6 visually demonstrates DiT4SR's advantages.

• Clarity and Detail: In the first two rows, DiT4SR generates results with better clarity and more abundant details, even when encountering severe blurring degradations. This implies that the model effectively utilizes the generative capabilities of SD3 to synthesize plausible high-frequency information.
• Fine Structures: The last two rows show DiT4SR excelling at processing fine structures, such as architectural details and text. Notably, SD3-ControlNet, which also uses SD3 as a backbone but with a ControlNet-like mechanism, fails to handle these aspects as effectively. This observation underscores the superiority of DiT4SR's bidirectional information interaction and cross-stream convolutional injection over the one-way control offered by ControlNet. The more comprehensive information interaction allows DiT4SR to better leverage LR information for high-fidelity restoration.

User Study (Table 2): To further validate perceptual quality, a user study was conducted with 80 volunteers. Participants compared DiT4SR's results against four latest methods (SeeSR, DiffBIR, SUPIR, DreamClear) on randomly selected LR images. They answered two questions: (1) Which result has higher image realism? (2) Which has better fidelity?

The following are the results from Table 2 of the original paper:

The results demonstrate that DiT4SR consistently outperforms all compared methods in both image realism and fidelity, with winning rates ranging from 72.7% to 83.6% for realism and 64.5% to 79.5% for fidelity. This strong preference from human evaluators further confirms DiT4SR's ability to generate perceptually superior results.

6.2. Ablation Studies / Parameter Analysis

The paper conducts an ablation study on RealLQ250 using MUSIQ and MANIQA metrics to evaluate the effectiveness of each proposed component. All variants are trained under the same settings as the full model.

The following are the results from Table 3 of the original paper:

The following figure (Figure 7 from the original paper) shows visual comparison for the ablation study:

该图像是论文中图7的消融实验对比示意图，展示了不同变体对图像细节恢复的影响。变体A、B、C分别去除LR Integration、LR Residual和LR Injection；变体D用线性层替代了卷积层。完整模型效果最佳。

Effectiveness of LR Integration

• Variant A (× LR Integration, $√ LR Residual$, Conv LR Injection): This variant removes the LR Stream from the attention computation. As shown in Table 3, both MUSIQ (66.963) and MANIQA (0.574) scores significantly decline compared to the FULL model (MUSIQ 71.832, MANIQA 0.632). Figure 7(b) visually confirms that without bidirectional information interaction between the LR Stream and the generated latent, degradations cannot be effectively removed. This indicates that relying solely on LR injection between MLP layers is insufficient. The bidirectional interaction allows the LR guidance to adapt and refine itself, which is crucial for addressing complex degradations.

Effectiveness of LR Residual

• Variant B ($√ LR Integration$, × LR Residual, Conv LR Injection): This variant removes the LR Residual connection. Table 3 shows a decline in performance for both MUSIQ (70.887) and MANIQA (0.614) compared to the FULL model. Visually, Figure 7(c) reveals noticeable artifacts in the results, degrading image fidelity. This confirms the role of LR Residual in stabilizing the LR Stream's evolution and preserving LR guidance consistency in deeper DiT blocks, preventing undesired disruptions and leading to higher fidelity results.

Effectiveness of LR Injection

• Variant C ($√ LR Integration$, $√ LR Residual$, × LR Injection): This variant removes the LR Injection between MLP layers. Table 3 shows a slight decline in metrics (MUSIQ 71.202, MANIQA 0.610). While LR integration in attention alone can produce passable results, Figure 7(d) shows noticeable content distortions, especially in fine details like the eye region. This confirms that global attention is not sufficient for SR tasks that require local information for accurate detail restoration.

• Variant D ($√ LR Integration$, $√ LR Residual$, Linear LR Injection): This variant replaces the 3x3 depth-wise convolution layer with a linear layer for LR injection. Although Table 3 shows similar performance in metrics (MUSIQ 71.607, MANIQA 0.621) compared to the FULL model, Figure 7(e) clearly indicates that artifacts and distortions have not been alleviated. This visual evidence is crucial, demonstrating that the convolutional layer's ability to capture precise local information is essential for enhancing fidelity, especially for fine structures, a capability not fully reflected by the non-reference metric data. The 3x3 depth-wise convolution layer effectively compensates for DiT's limited local information-capturing ability.

  In summary, the ablation studies rigorously validate the contribution of each proposed component: LR Integration in Attention is fundamental for effective bidirectional guidance, LR Residual ensures guidance consistency across layers, and Convolutional LR Injection is critical for capturing local information and enhancing fine detail restoration.

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper successfully introduces DiT4SR, a novel approach that effectively "tames" large-scale Diffusion Transformer (DiT) models for Real-World Image Super-Resolution (Real-ISR). DiT4SR is identified as one of the pioneering works in this specific direction, moving beyond UNet-based diffusion models and ControlNet-like conditioning mechanisms.

The core contributions lie in two key design choices:

1. Integrated LR Stream in Attention: Instead of external injection, DiT4SR integrates low-resolution (LR) embeddings directly into the original attention mechanism of DiT. This enables a powerful bidirectional information flow between the LR latent and the generated latent, allowing the LR stream to dynamically evolve and provide progressively refined and context-aware guidance throughout the diffusion process. The introduction of an LR Residual further ensures the consistency of this guidance in deeper layers.

2. Cross-Stream Convolutional Injection: A convolutional layer is introduced to inject LR-guided information into the generated latent (specifically, between the MLP layers of the LR and Noise Streams). This design is crucial for enhancing LR guidance and, more importantly, compensating for DiT's inherent limitation in capturing local information, which is vital for restoring fine details in SR tasks.

   Extensive experiments, including quantitative evaluations on multiple real-world datasets and a comprehensive user study, consistently demonstrate DiT4SR's superior performance in terms of image quality, realism, and fidelity compared to state-of-the-art Real-ISR methods, including those based on UNet and DiT with ControlNet.

7.2. Limitations & Future Work

The paper explicitly states that its work highlights the potential of leveraging DiT for high-quality image restoration and paves the way for future research in this direction. However, it does not explicitly list specific limitations of DiT4SR or detailed future work directions beyond this general statement.

Based on the nature of the model and general challenges in DiT-based models, potential implicit limitations and future work could include:

• Computational Cost: Large-scale DiT models are computationally intensive. While DiT4SR is built on a pre-trained SD3, the training and inference costs for fine-tuning and running such a model might still be substantial compared to simpler UNet-based or GAN-based approaches. Future work could explore efficiency improvements, such as distillation or optimized inference strategies.
• Generalizability to Other Restoration Tasks: While shown effective for Real-ISR, it's not explicitly tested for other image restoration tasks (e.g., denoising, deblurring, inpainting) where DiT could also be applied. Future work could investigate its adaptability and potential for unified image restoration.
• Adaptive Guidance Strength: The current LR integration involves trainable linear projections and a convolutional layer initialized to zeros, gradually increasing their impact. Future work might explore more dynamic or adaptive weighting mechanisms for the LR stream's influence across different diffusion steps or image regions.
• Understanding Bidirectional Interaction: While the paper demonstrates the effectiveness of bidirectional interaction, a deeper theoretical understanding or more fine-grained control over how the LR Stream's evolution is influenced by the Noise Stream could lead to further improvements.

7.3. Personal Insights & Critique

DiT4SR offers a highly impactful contribution by demonstrating that Diffusion Transformers, previously celebrated for general image generation, can be effectively specialized for Real-World Image Super-Resolution. The paper's core insight—that DiT's architectural strengths for multimodal interaction need a more profound integration than simple ControlNet-like conditioning—is particularly insightful.

The method's ability to enable bidirectional information flow for LR guidance is a significant advancement. Unlike ControlNet, where the control signal is often "static" or only conditionally integrated, DiT4SR allows the LR stream itself to evolve and become more "aware" of the diffusion process and the noisy latent. This dynamic guidance is likely a major factor in its superior performance, especially on complex real-world degradations where the LR information needs to be interpreted contextually.

Furthermore, the explicit introduction of a cross-stream convolution layer to address DiT's local information weakness is a clever and effective design choice. It acknowledges that while transformers excel at global relationships, convolutions remain powerful for local pattern recognition, and integrating both synergistically can yield superior results in pixel-level tasks like SR. The ablation study visually confirming the necessity of this convolutional layer, even when metrics might seem similar to a linear layer, underscores the importance of qualitative assessment in perceptual tasks.

A potential area for critique or further exploration could be the interpretability of the bidirectional attention. While effective, understanding precisely how the LR stream adapts and what information it prioritizes at different diffusion steps could open avenues for more targeted control or optimization. Also, the computational efficiency of DiT4SR relative to other SR methods, especially during inference, might be a practical consideration for real-world deployment, though this is often a trade-off for higher quality in DiT-based models.

The implications of DiT4SR extend beyond Real-ISR. Its principles of deeply integrating conditional information into DiT's attention and supplementing global attention with local convolutional processing could be transferable to other conditional generation or image restoration tasks. For instance, in tasks like image editing or inpainting where precise local control is needed alongside global coherence, similar bidirectional stream integration and convolutional injection could be explored. The work sets a strong precedent for how to effectively adapt powerful foundation models to specific, challenging downstream tasks without merely treating them as black boxes.