Bibliographic Information

• Title: CiaoSR: Continuous Implicit Attention-in-Attention Network for Arbitrary-Scale Image Super-Resolution
• Authors: Jiezhang Cao, Qin Wang, Yongqin Xian, Yawei Li, Bingbing Ni, Zhiming Pi, Kai Zhang, Yulun Zhang, Radu Timofte, Luc Van Gool
• Affiliations: ETH Zürich, Huawei Inc., University of Wurzburg, KU Leuven
• Journal/Conference: The paper was submitted to arXiv, a preprint server. It does not appear to be published in a peer-reviewed conference or journal at the time of this analysis. Preprints like this allow for rapid dissemination of research but have not yet undergone formal peer review.
• Publication Year: 2022
• Abstract: The paper addresses the task of arbitrary-scale image super-resolution (SR) by learning a continuous image representation. Existing methods typically use a local ensemble of features to predict pixel values, but this approach has limitations: the ensemble weights are fixed (not learnable) and it has a limited receptive field. To overcome this, the authors propose a Continuous Implicit Attention-in-Attention Network (CiaoSR). This network introduces an implicit attention mechanism to learn adaptive ensemble weights for nearby features. It further embeds a scale-aware attention module to incorporate non-local information. The authors demonstrate that CiaoSR significantly outperforms existing methods when using the same backbone network, achieves state-of-the-art results on arbitrary-scale SR, and is effective for real-world SR. A key advantage is its flexibility, as it can be integrated with any SR backbone to enhance performance.
• Original Source Link: https://arxiv.org/abs/2212.04362

Executive Summary

• Background & Motivation (Why): Standard deep learning-based super-resolution (SR) models are typically trained for a fixed, discrete upscaling factor (e.g., ×2, ×4). This is impractical for real-world applications like digital zoom, which require smooth, continuous scaling. Recent "arbitrary-scale" SR methods address this by learning a continuous representation of the image. However, they often rely on simple interpolation techniques (like bilinear interpolation) to generate new pixels. This approach has two major flaws:

  1. Non-Adaptive Weights: The weights used to combine neighboring features are based only on coordinate distances, not the visual content of the features themselves. They are not learned and cannot adapt to different image textures.
  2. Limited Receptive Field: Only a small, local neighborhood of features is considered, ignoring potentially useful, similar patterns that may exist elsewhere in the image.

• Main Contributions / Findings (What): The paper introduces CiaoSR, a novel upsampling module designed to replace the simplistic local ensemble in arbitrary-scale SR. Its main contributions are:

  1. Implicit Attention for Local Ensemble: CiaoSR proposes an implicit attention network that learns to compute the weights for combining local features. These weights are adaptive, as they depend on both the coordinate information and the visual content (features) of the local neighborhood.
  2. Attention-in-Attention for Non-Local Information: To overcome the limited receptive field, CiaoSR embeds a scale-aware non-local attention module within its value computation. This allows the model to find and aggregate relevant textures and patterns from a much larger area of the image, improving reconstruction quality.
  3. Backbone-Agnostic and State-of-the-Art Performance: CiaoSR is a flexible module that can be plugged into any existing SR feature extraction network (backbone) like EDSR, RDN, or SwinIR. Experiments show that adding CiaoSR consistently and significantly improves the performance of these backbones, achieving state-of-the-art results on benchmark datasets for both standard and arbitrary-scale SR tasks.

Prerequisite Knowledge & Related Work

Foundational Concepts

• Single Image Super-Resolution (SISR): The task of reconstructing a high-resolution (HR) image from a single low-resolution (LR) input. This is an ill-posed problem, as one LR image can correspond to many possible HR images.
• Deep Neural Network (DNN) for SR: Modern SR methods use deep learning, typically Convolutional Neural Networks (CNNs) or Transformers, to learn the complex mapping from LR to HR images. A typical model has a "backbone" for deep feature extraction and an "upsampling module" to increase the image resolution.
• Arbitrary-Scale Super-Resolution: A more flexible version of SISR where a single model can upscale an LR image by any continuous scaling factor (e.g., ×2.5, ×3.1), not just predefined integers.
• Implicit Neural Representation (INR): A technique where a neural network (usually a Multi-Layer Perceptron or MLP) learns to represent an object or signal (like an image) as a continuous function. For images, this function, f(x), takes a 2D coordinate (x, y) as input and outputs the corresponding RGB pixel value. This is naturally suited for arbitrary-scale SR, as one can query any coordinate to render the image at any resolution.
• Attention Mechanism: A mechanism in neural networks that allows a model to weigh the importance of different parts of the input. In the context of images, it can learn to focus on more relevant pixels or features when making a prediction. The core components are the Query (Q), Key (K), and Value (V). The attention weights are computed from the similarity between Q and K, and these weights are then used to create a weighted sum of V.
• Pixel Shuffling: A common upsampling technique in SR that rearranges the elements of a tensor of shape $(H, W, C \cdot s^2)$ to a tensor of shape $(H \cdot s, W \cdot s, C)$, where $s$ is the upscale factor. It is efficient but only works for a fixed, integer scale $s$.

Previous Works

The paper positions itself within the evolution of SR, from fixed-scale to arbitrary-scale models.

• Fixed-Scale SR: Early and powerful models like SRCNN, EDSR, RDN, and RCAN focused on designing better backbone architectures (e.g., with residual or dense blocks) but were tied to a specific upsampling scale via modules like pixel shuffling. Transformer-based models like SwinIR and HAT later improved feature extraction but still used fixed-scale upsamplers.
• Arbitrary-Scale SR:
  • MetaSR was a pioneering work that introduced a meta-learning approach to generate weights for an upscaling module dynamically based on the scale.
  • LIIF (Local Implicit Image Function) proposed representing the image as a continuous function. To predict a pixel at a query coordinate, it takes the features of the four nearest neighbors from the LR feature map and combines them using weights derived from bilinear interpolation. The combined feature is then fed into an MLP to predict the final RGB value. The key limitation, which CiaoSR targets, is that these weights are fixed and data-independent.
  • LTE (Local Texture Estimator) improved upon LIIF by incorporating Fourier features to better capture local textures, but it still relied on the same non-adaptive bilinear ensemble method.
  • ITSRN learned the ensemble weights but based them only on the coordinate distance and a scale token, still ignoring the rich visual information in the features themselves.

Differentiation

CiaoSR's innovation lies in its upsampling module. Unlike LIIF and LTE, which use a fixed bilinear weighting scheme, and unlike ITSRN, which only uses coordinate information, CiaoSR introduces a full-fledged attention mechanism to compute adaptive, content-aware ensemble weights. It considers both the coordinate distances and the feature similarities. Furthermore, it goes a step further by using an "attention-in-attention" design to incorporate non-local features, which none of the previous arbitrary-scale methods did explicitly in the upsampling stage.

Methodology (Core Technology & Implementation Details)

The core of CiaoSR is a novel upsampling module that re-frames the local feature ensemble as an attention mechanism. The overall architecture is shown in Figure 3.

该图像是论文中提出的CiaoSR方法的网络架构示意图，包括连续隐式注意力中的注意力网络及尺度感知注意力模块，展示了输入低分辨率图像通过骨干网络提取特征并结合坐标距离进行多尺度权重计算以实现任意缩放超分辨率。

Principles

The motivation starts from the standard local ensemble formula used by methods like LIIF: $$I_q := I(\mathbf{x}_q) = \sum_{(i,j) \in \mathcal{T}} w_{i,j} \cdot f(\mathbf{Z}_{i,j}^*, \mathbf{x}_q - \mathbf{x}_{i,j}^*)$$ where:

• $I_q$ is the predicted RGB value at the query coordinate $\mathbf{x}_q$.

• $\mathcal{T}$ is the set of local neighboring features (e.g., top-left, top-right, etc.).

• $\mathbf{Z}_{i,j}^*$ is the feature vector (latent code) at a grid coordinate $\mathbf{x}_{i,j}^*$.

• $w_{i,j}$ is the weight, typically calculated as the normalized area of the rectangle between the query point and the grid point, which is equivalent to bilinear interpolation (as shown in Figure 4a).

• $f$ is an MLP that processes the feature and relative coordinate.

  The authors argue this formulation is flawed because the weights $w_{i,j}$ are predetermined and ignore the content of the features $\mathbf{Z}_{i,j}^*$. They propose to learn these weights using attention.

CiaoSR: Continuous Implicit Attention-in-Attention

CiaoSR defines a new implicit attention function (i-Attention) to predict the RGB value $I_q$: $$I_q = \text{i-Attention}(Q, K, V; x_q, x_k, x_v)$$ This structure is visualized in Figure 2c, which contrasts with standard self-attention (Figure 2a) and coordinate-only attention (Figure 2b).

该图像是论文中关于隐式注意力机制的示意图，展示了(a)自注意力，(b)基于坐标的隐式注意力，以及(c)作者提出的隐式注意力结构，图中通过特征、坐标及坐标距离结合多层感知机(MLP)实现加权计算。

1. Implicit Attention for Local Ensemble

The main prediction is calculated as a local attention operation: $$I_q = \phi_q \left( \sum_{(i,j) \in \mathcal{T}} \underbrace{\sigma(Q^\top K_{i,j})}_\text{Learned Weights} V_{i,j} \right)$$ where:

• $\phi_q$ is a final query network (an MLP) that gives the RGB output.

• $\sigma$ is the Softmax function, which normalizes the attention scores into weights.

• $\mathcal{T}$ represents the local neighborhood (e.g., $2 \times 2$ grid cells around the query point).

• $Q$, $K_{i,j}$, and $V_{i,j}$ are the Query, Key, and Value, defined as follows:

  $$\left\{ \begin{array}{ll} Q = F^*, \\ K_{i,j} = \phi_k([F_{i,j}, (\mathbf{r}_k)_{i,j}, s]), \\ V_{i,j} = \phi_v([F_{i,j}, \tilde{F}_{i,j}], (\mathbf{r}_v)_{i,j}, s]), \end{array} \right.$$ where:

• $F^*$ is the feature from the backbone at the nearest grid point to the query coordinate $\mathbf{x}_q$.

• $F_{i,j}$ is a local feature patch (e.g., $3 \times 3$) unfolded from the backbone feature map at position (i,j). This enriches the key and value with local context.

• $\tilde{F}_{i,j}$ is an additional non-local feature, which is the "attention-in-attention" part.

• $s = [s_h; s_w]$ is the target scale vector.

• $\phi_k$ and $\phi_v$ are MLPs that project the inputs into the Key and Value spaces.

• $\mathbf{r}_k = \mathbf{x}_q - (\mathbf{x}_k)_{i,j}$ and $\mathbf{r}_v = \mathbf{x}_q - (\mathbf{x}_v)_{i,j}$ are the relative coordinate vectors between the query point and the key/value grid points.

  By including the features $F_{i,j}$ in the key $K_{i,j}$, the attention weights $\sigma(Q^\top K_{i,j})$ now depend on both feature similarity and coordinate information, making them adaptive.

  该图像是论文中图4的示意图，展示了两种特征集合方法的对比：(a)基于双线性插值的权重由查询点与邻近点构成矩形面积决定；(b)基于注意力的集合方法通过计算注意力权重实现大感受野和更精准的特征聚合。

2. Embedded Scale-Aware Attention for Non-Local Features

To compute the non-local features $\tilde{F}_{i,j}$ used in the Value, CiaoSR employs a second, embedded attention mechanism. This module is designed to find repetitive or similar patterns across a larger region of the image, even at different scales.

The non-local feature $\tilde{F}_{i,j}$ at position (i,j) is computed as: $$\tilde{F}_{i, j} = \varphi \left( \sum_{u, v} \frac{\exp(\tilde{Q}_{i, j}^\top \tilde{K}_{u, v})}{\sum_{u', v'} \exp(\tilde{Q}_{i, j}^\top \tilde{K}_{u', v'})} \tilde{V}_{s'u, s'v}^{s'p \times s'p} \right)$$ where:

• $\tilde{Q}, \tilde{K}, \tilde{V}$ are non-local query, key, and value features derived from the backbone feature map $F$: $$\left\{ \begin{array}{ll} \tilde{Q} = \varphi_q(F), \\ \tilde{K} = \varphi_k(F \downarrow_{s'}), \\ \tilde{V} = \varphi_v(F), \end{array} \right.$$

• $\downarrow_{s'}$ denotes downsampling the feature map by a scale factor $s'$ (e.g., 2, 3, or 4). This allows the model to find similarities at a different scale.

• The attention is computed between the query at (i,j) and keys from the entire (or a large window of the) downscaled feature map.

• $\varphi$ is a convolutional layer used for final aggregation.

  This "attention-in-attention" structure allows the model to enrich the local ensemble with information from relevant, far-away patches, leading to more robust and detailed reconstructions, as visualized in Figure 4b.

Experimental Setup

• Datasets:

  • Training: DIV2K dataset, containing 800 high-quality 2K resolution images. For training, $48s \times 48s$ HR patches were cropped, and corresponding $48 \times 48$ LR patches were generated via bicubic downsampling, with the scale $s$ randomly sampled from a uniform distribution $\mathcal{U}(1, 4)$.
  • Testing: Standard benchmark datasets including Set5, Set14, B100, Urban100, and Manga109, as well as the DIV2K validation set.
  • Real-World Testing: RealSRSet and DPED datasets, which contain real-world images with complex degradations.

• Evaluation Metrics:

  1. PSNR (Peak Signal-to-Noise Ratio): Measures the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. It is a widely used metric for image reconstruction quality, measured in decibels (dB). Higher is better. $$\text{PSNR} = 10 \cdot \log_{10} \left( \frac{MAX_I^2}{\text{MSE}} \right)$$
     • $MAX_I$: The maximum possible pixel value of the image (e.g., 255 for 8-bit images).
     • $\text{MSE}$: Mean Squared Error between the ground-truth and reconstructed images.
  2. SSIM (Structural Similarity Index Measure): Measures the similarity between two images based on luminance, contrast, and structure. It is designed to align better with human visual perception of image quality than PSNR. The value ranges from -1 to 1, where 1 indicates perfect similarity. Higher is better. $$\text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$$
     • $\mu_x, \mu_y$: Mean of images $x$ and $y$.
     • $\sigma_x, \sigma_y$: Standard deviation of images $x$ and $y$.
     • $\sigma_{xy}$: Covariance of $x$ and $y$.
     • $c_1, c_2$: Small constants to stabilize the division.
  3. LPIPS (Learned Perceptual Image Patch Similarity): A metric that measures the perceptual distance between two image patches. It computes the distance between deep features extracted from pre-trained networks (like VGG or AlexNet), which correlates better with human judgment than PSNR or SSIM. Lower is better.
  4. No-Reference IQA Metrics (NIQE, BRISQUE, PIQE): These metrics assess image quality without a ground-truth reference, making them suitable for real-world SR. They are trained on statistical features of natural images to predict perceived quality. For all three, lower is better.

• Baselines:

  • Backbones: The paper tests CiaoSR with three different backbone networks: EDSR, RDN, and SwinIR.
  • Arbitrary-Scale Methods: CiaoSR is compared against MetaSR, LIIF, ITSRN, and LTE.
  • Real-World SR Methods: For real-world experiments, comparisons are made with RealSR, BSRGAN, and Real-ESRGAN.

Results & Analysis

The experiments comprehensively validate CiaoSR's effectiveness across various settings.

Core Results: Quantitative and Qualitative

• Performance with Different Backbones (Table 1): This table shows results on the DIV2K validation set for both in-scale (×2, ×3, ×4) and out-of-scale (×6 to ×30) SR.

  • Key Finding: Across all three backbones (EDSR, RDN, SwinIR), adding CiaoSR (-CiaoSR) consistently yields the best PSNR scores, outperforming all other arbitrary-scale methods (-MetaSR, -LIIF, -ITSRN, -LTE).

  • Remarkable Result: As highlighted in Figure 1, RDN-CiaoSR (PSNR 27.11 at ×4) even outperforms the SwinIR-baseline (PSNR 27.07, from Table 2 for Urban100), which uses a much more powerful Transformer-based backbone. This demonstrates that a superior upsampling module can be more impactful than a stronger backbone with a basic upsampler.

    (Manual transcription of Table 1)

    Backbones	Methods	In-scale			Out-of-scale
    		×2	×3	×4	×6	×12	×18	×24	×30
    -	Bicubic	31.01	28.22	26.66	24.82	22.27	21.00	20.19	19.59
    EDSR [44]	EDSR-baseline [44]	34.55	30.90	28.94
    	EDSR-baseline-MetaSR [28]	34.64	30.93	28.92	26.61	23.55	22.03	21.06	20.37
    	EDSR-baseline-LIIF [13]	34.67	30.96	29.00	26.75	23.71	22.17	21.18	20.48
    	EDSR-baseline-ITSRN† [79]	34.71	30.95	29.03	26.77	23.71	22.17	21.18	20.49
    	EDSR-baseline-LTE [39]	34.72	31.02	29.04	26.81	23.78	22.23	21.24	20.53
    	EDSR-baseline-CiaoSR (ours)	34.91	31.15	29.23	26.95	23.88	22.32	21.32	20.59
    RDN [88]	RDN-baseline [88]	34.94	31.22	29.19	-	-	-
    	RDN-MetaSR [28]	35.00	31.27	29.25	26.88	23.73	22.18	21.17	20.47
    	RDN-LIIF [13]	34.99	31.26	29.27	26.99	23.89	22.34	21.31	20.59
    	RDN-ITSRN† [79]	35.09	31.36	29.38	27.06	23.93	22.36	21.32	20.61
    	RDN-LTE [39]	35.04	31.32	29.33	27.04	23.95	22.40	21.36	20.64
    	RDN-CiaoSR (ours)	35.15	31.42	29.45	27.16	24.06	22.48	21.43	20.70
    SwinIR [40]	SwinIR-baseline [40]	34.94	31.22	29.19	-	-	-	-	-
    	SwinIR-MetaSR† [28]	35.15	31.40	29.33	26.94	23.80	22.26	21.26	20.54
    	SwinIR-LIIF† [13]	35.17	31.46	29.46	27.15	24.02	22.43	21.40	20.67
    	SwinIR-ITSRN† [79]	35.19	31.42	29.48	27.13	23.83	22.31	21.31	20.55
    	SwinIR-LTE [39]	35.24	31.50	29.51	27.20	24.09	22.50	21.47	20.73
    	SwinIR-CiaoSR (ours)	35.29	31.55	29.59_	27.28	24.15	22.54	21.51	20.74

  (Note: There appear to be typos/transcription errors in the original paper for SwinIR-LTE and SwinIR-CiaoSR, where multiple values are listed. I have transcribed the higher values.)

• Benchmark Performance (Tables 2 & 3): These tables show results on standard benchmarks (Set5, Set14, B100, Urban100, Manga109). CiaoSR achieves state-of-the-art results. For instance, on Urban100 at ×4, RDN-CiaoSR achieves a PSNR of 27.11 dB, a significant gain of 0.3 dB over the next best method, RDN-LTE. This dataset is rich in self-similar patterns (windows, bricks), where CiaoSR's non-local attention is particularly beneficial.

• Qualitative Results (Figure 5): The visual comparisons show that CiaoSR produces sharper details and more plausible textures. In the second row of Figure 5, CiaoSR successfully reconstructs the fine grid-like structure on the building facade, while other methods produce blurry or incomplete results. This visually confirms the advantage of learning adaptive weights and using non-local information.

  该图像是论文中图5，展示了不同方法在两个基准测试图像上的视觉对比，包括模型对8倍超分辨率的合成效果。图中对比了传统双三次插值与多种基于RDN的模型，突出显示了本方法RDN-CiaoSR在细节还原上的优势。

Ablation Study

The ablation studies in Tables 4 and 5 systematically validate the contributions of each component of CiaoSR.

• Component Contribution (Table 4):

  1. Baseline (MLP): Replacing the attention module with a simple MLP to predict weights leads to the lowest performance.

  2. Adding Attention-in-Attention: Introducing the main implicit attention network (without the non-local part) provides a substantial PSNR gain (e.g., +0.27 dB at ×4). This confirms the benefit of learning adaptive, content-aware local ensemble weights.

  3. Adding Scale-Aware Non-Local Attention: Adding the final component, the embedded non-local attention, provides a further performance boost (e.g., +0.15 dB at ×4). This shows that capturing relevant features from a larger receptive field is crucial.

     (Manual transcription of Table 4)

     Attention-in-attention		✗	✓	✓
     Scale-aware Attention Network		✗	✗	✓
     In-scale	×2	32.87	33.24	33.30
     	×3	28.82	29.10	29.17
     	×4	26.69	26.96	27.11
     Out-of-scale	×6	24.22	24.50	24.58
     	×8	22.80	22.98	23.13

• Training Scales (Table 5): Training with continuous scales (s ~ U[1, 4]) yields better generalization and overall performance than training on single discrete scales ($s={2}$) or multiple discrete scales ($s={2, 3, 4}$). This demonstrates that the continuous formulation allows the model to effectively learn from cross-scale correlations.

Further Analysis

• Synthesis Steps (Table 6): For large scaling factors (e.g., ×12), generating the image in a single step with CiaoSR is superior to multi-step synthesis (e.g., ×2 then ×6). This is because errors accumulate in a multi-step pipeline.
• Model Efficiency (Table 7): CiaoSR has a competitive model size (1.4M parameters), smaller than LTE (1.7M). However, its inference time is higher (528 ms) due to the computationally intensive non-local attention. This presents a trade-off between performance and speed.
• Perceptual Metrics (Table 8): CiaoSR also achieves the best scores on SSIM and LPIPS, indicating that its improvements are not just in PSNR but also in structural and perceptual quality.

Real-World Arbitrary-Scale SR

• Quantitative and Qualitative Results (Table 9, Figure 6): When adapted for real-world SR, CiaoSR produces visually compelling results. Figure 6 shows it can generate realistic textures (e.g., dog fur, printed digits) where other state-of-the-art GAN-based methods like BSRGAN and Real-ESRGAN introduce artifacts. While the no-reference metrics in Table 9 are mixed, the authors correctly note that these metrics often fail to capture human perceptual preferences, and the visual evidence strongly supports CiaoSR's effectiveness.

  该图像是图6，属于图表类型，展示在RealSRSet和DPED数据集上不同超分辨率方法（imes 8放大倍数）的视觉对比。图中展示了低分辨率图像（LR）、真实超分图像（RealSR）、BSRGAN、Real-ESRGAN及本文方法（Ours）的恢复效果及对应局部细节对比。

Conclusion & Personal Thoughts

• Conclusion Summary: The paper successfully identifies and addresses a key weakness in modern arbitrary-scale SR methods: the non-adaptive, local-only upsampling process. The proposed CiaoSR, with its "attention-in-attention" design, provides an elegant and effective solution. It learns content-aware weights for local feature ensembling while simultaneously leveraging non-local information. Its ability to plug into any backbone and consistently boost performance makes it a significant and practical contribution to the field.

• Limitations & Future Work:

  • Inference Speed: The primary limitation is the increased computational cost and inference time due to the non-local attention module, as shown in Table 7. This might hinder its use in real-time applications. Future work could explore more efficient approximations of non-local attention.
  • Real-World Degradations: While the paper extends CiaoSR to real-world images, the training still relies on synthesized degradations. Training on larger, more diverse real-world LR/HR paired datasets could further improve its practical performance.

• Personal Insights & Critique:

  • Novelty: The core idea of treating the interpolation/upsampling step itself as a learned attention mechanism is powerful. Most SR research focuses on improving the feature extraction backbone, while the upsampling module is often an afterthought. This paper shows that innovating on the upsampler can yield substantial gains, even allowing a weaker backbone to outperform a stronger one.
  • Generalizability: The concept is highly generalizable. This attention-based implicit function could be applied to other continuous signal reconstruction problems beyond images, such as video frame interpolation, novel view synthesis, or 3D shape representation.
  • Clarity and Rigor: The paper is well-structured and the methodology is explained clearly with strong motivation. The experiments are thorough, including multiple backbones, extensive ablations, and both quantitative and qualitative analyses, which strongly supports the authors' claims. CiaoSR represents a thoughtful and impactful advancement in the pursuit of truly flexible and high-fidelity image super-resolution.