1. Bibliographic Information

• Title: S³Mamba: Arbitrary-Scale Super-Resolution via Scaleable State Space Model
• Authors: Peizhe Xia, Long Peng, Xin Di, Renjing Pei, Yang Wang, Yang Cao, Zheng-Jun Zha.
• Affiliations: The authors are affiliated with the University of Science and Technology of China and Huawei Noah's Ark Lab.
• Journal/Conference: This paper is a preprint available on arXiv. As of its publication date, it has not yet been published in a peer-reviewed conference or journal. arXiv is a common platform for researchers to share their work early.
• Publication Year: 2024 (Published on arXiv on November 16, 2024).
• Abstract: The paper tackles Arbitrary-Scale Super-Resolution (ASSR), which aims to use a single model to upscale images to any desired resolution. Current methods, particularly those based on Implicit Neural Representations (INR), often rely on CNNs or Transformers, which suffer from high computational complexity and limited ability to model long-range dependencies. To address this, the authors propose S³Mamba, a novel method built upon a Scalable State Space Model (SSSM). The SSSM modulates its internal matrices based on the desired scale factor, enabling scalable and continuous image representation with linear computational complexity. Additionally, a scale-aware self-attention mechanism is introduced to improve the model's perception of global features at different scales. The authors claim that extensive experiments show their method achieves state-of-the-art (SOTA) performance and better generalization on both synthetic and real-world datasets.
• Original Source Link:
  • arXiv Page: https://arxiv.org/abs/2411.11906
  • PDF Link: https://arxiv.org/pdf/2411.11906v1.pdf
  • Publication Status: Preprint.

2. Executive Summary

Background & Motivation (Why)

The core problem is the inflexibility of traditional image super-resolution (SR) models, which are trained for and restricted to fixed integer scale factors (e.g., ×2, ×4). Arbitrary-Scale Super-Resolution (ASSR) aims to overcome this by creating a single model that can upscale an image to any continuous scale factor (e.g., ×2.7, ×5.1).

A popular and effective approach for ASSR is using Implicit Neural Representations (INR), which learn a continuous function that maps pixel coordinates to their corresponding color values. However, existing INR-based methods face significant challenges:

1. MLP-based methods (like LIIF) have a limited receptive field. They process each coordinate independently, failing to capture the global context and long-range dependencies within the image, which can lead to artifacts and less detailed reconstructions.

2. Transformer-based methods (like CiaoSR) can model global context effectively using self-attention, but this comes at the cost of quadratic computational complexity ($O(N^2)$), making them slow and resource-intensive, especially for high-resolution images.

   This creates a gap: a need for an efficient model that can capture global context with linear complexity while being adaptable to arbitrary scales.

Main Contributions / Findings (What)

The paper introduces S³Mamba to fill this gap. Its main contributions are:

1. Pioneering Use of SSM in ASSR: The paper is the first to introduce State Space Models (SSMs), specifically the Mamba architecture, to the task of arbitrary-scale super-resolution.
2. A Novel Scalable State Space Model (SSSM): This is the core technical innovation. The SSSM is designed to be scale-aware. It dynamically modulates its internal parameters—the state transition matrix ($A$) and sampling matrix ($B$)—based on the target scale and coordinate information. This allows the model to create a consistent continuous representation of an image across different scales, all while maintaining the linear computational complexity ($O(N)$) of SSMs.
3. A Scale-Aware Self-Attention Mechanism: Built upon the SSSM, this mechanism generates a global attention map that is conditioned on the scale and coordinates. This further enhances the network's ability to focus on relevant features for the specific target resolution, improving reconstruction quality.
4. State-of-the-Art Performance: The combined S³Mamba framework is shown to achieve superior or competitive performance on the synthetic DIV2K benchmark and demonstrates strong generalization and artifact reduction on the real-world COZ benchmark, outperforming previous methods in both quantitative metrics and visual quality.

3. Prerequisite Knowledge & Related Work

Foundational Concepts

• Image Super-Resolution (SR): The task of generating a high-resolution (HR) image from a low-resolution (LR) input. Traditional methods train a separate model for each integer scale factor (e.g., a ×2 model, a ×4 model).

• Arbitrary-Scale Super-Resolution (ASSR): A more advanced form of SR where a single model can upscale an LR image by any non-integer or large integer scale factor. This is more practical for real-world applications where a specific output resolution is required.

• Implicit Neural Representations (INR): A technique where a neural network, typically a Multi-Layer Perceptron (MLP), learns a continuous function $f(\text{coord}) \rightarrow \text{value}$. For images, this means mapping a 2D coordinate (x, y) to its RGB color. By querying this function at any coordinate in a grid, an image of any resolution can be generated. As shown in Figure 1(b), INR aims to model the continuous signal of the real world from discrete LR pixels.

  Figure 1 shows: (a) the real-world imaging process where a continuous scene is discretized into a digital image; (b) the ideal goal of ASSR using INR to reconstruct a continuous representation space; (c) the limitation of MLP-based methods which have a limited receptive field; (d) the proposed S³Mamba method which uses SSSM to model the continuous space more effectively.

• State Space Models (SSM): A class of models originating from control theory used to model sequences. An SSM maps an input sequence x(t) to an output sequence y(t) through a hidden state vector h(t). The core idea is that the state at the current timestep is an evolution of the state from the previous timestep plus an influence from the current input. Recent advancements like Mamba have made SSMs highly effective and efficient for deep learning, as they can capture long-range dependencies with linear complexity, unlike Transformers.

Previous Works

The paper positions itself within the evolution of ASSR methods:

1. Meta-Learning Approaches: MetaSR was an early method that used a "meta-upscale module" to generate weights for an upscaling network dynamically based on the scale factor.
2. INR-based MLP Methods: LIIF was a landmark paper that introduced INR to ASSR. It uses an MLP to predict the RGB value at a given continuous coordinate based on features from the four nearest pixels in the LR feature map. However, its point-wise nature limits its ability to see global context. LTE and LINF tried to improve on this by incorporating Fourier (frequency domain) information to better represent textures.
3. INR-based Transformer Methods: To overcome the local limitations of MLPs, CiaoSR and CLIT incorporated Transformers. Transformers use a self-attention mechanism to weigh the importance of all input features for each output, allowing them to capture global context. This significantly improved performance but introduced quadratic computational complexity, making them inefficient for practical use.
4. Real-World ASSR: COZ introduced a benchmark dataset specifically for real-world ASSR, featuring images captured with continuous optical zoom, providing a more realistic testbed than synthetically downscaled images.

Differentiation

S³Mamba differentiates itself from prior work in a critical way:

• vs. MLP-based methods (LIIF): It overcomes the limited receptive field by using an SSM, which can theoretically model dependencies across the entire input sequence (image scan line).
• vs. Transformer-based methods (CiaoSR): It achieves global context modeling with linear computational complexity, making it much more efficient and scalable than the quadratically complex Transformer models.
• vs. standard SSMs (Mamba): It introduces the Scalable State Space Model (SSSM), which makes the model explicitly aware of the target super-resolution scale. Standard SSMs are not designed for this, as their dynamics are only input-dependent. The SSSM modulates its internal state transitions based on the scale, which is the key to creating a consistent continuous representation.

4. Methodology (Core Technology & Implementation Details)

The core of the paper is the proposed S³Mamba framework, which is built on the novel Scalable State Space Model (SSSM).

Figure 2 illustrates: (a) The overall S³Mamba architecture, which takes an LR image and generates an HR image at an arbitrary scale. (b) The SSSM Block, which processes input features. (c) The core SSSM mechanism, showing how scale and coordinates are used to modulate the discretization step size $Δ$ and the input matrix $B$.

Principles: From SSM to Scalable SSM (SSSM)

First, the paper reviews the standard State Space Model (SSM). A continuous SSM is defined by a linear Ordinary Differential Equation (ODE): $$\begin{array} { r } { \dot { h } ( t ) = A h ( t ) + B x ( t ) , } \\ { y ( t ) = C h ( t ) + D x ( t ) . } \end{array}$$

• x(t): The input signal (e.g., a pixel feature).

• $h(t) \in \mathbb{R}^N$: The latent state vector of size $N$.

• y(t): The output signal.

• $A \in \mathbb{R}^{N \times N}$: The state transition matrix, which governs the internal dynamics of the system.

• $B \in \mathbb{R}^{N \times 1}$: The input matrix, which controls how the input affects the state.

• $C \in \mathbb{R}^{1 \times N}$: The output matrix, which controls how the state is projected to the output.

• $D \in \mathbb{R}$: A direct feed-through term.

  For digital processing, this continuous system is discretized. A common method is the zero-order hold, which transforms the continuous parameters (A, B) into discrete parameters ($\bar{A}, \bar{B}$) using a sampling step size $\Delta$: $$\begin{array} { l } { \overline { { A } } = \exp ( \Delta A ) , } \\ { \overline { { B } } = ( \Delta A ) ^ { - 1 } ( \exp ( \Delta A ) - I ) \Delta B , } \end{array}$$ This leads to the discrete recurrence relation: $$\begin{array} { l } { { h _ { k } = \overline { { { A } } } h _ { k - 1 } + \overline { { { B } } } x _ { k } , } } \\ { { y _ { k } = C h _ { k } + D x _ { k } , } } \end{array}$$

• $h_k$: The state at discrete step $k$.

• $x_k$: The input at step $k$.

• $y_k$: The output at step $k$.

  The key insight of the paper is that in standard SSMs used in vision (like Mamba), the step size $\Delta$ is learned from the input data $x_k$ but is unaware of the image scale. This is a problem for ASSR, as the "physical distance" and correlation between adjacent pixels change with the scale factor. To solve this, the authors propose the Scalable State Space Model (SSSM).

The SSSM makes the model scale-aware by modulating both $\Delta$ and $B$:

1. Modulating the Step Size $\Delta$: $$\begin{array} { c } { \Delta _ { x _ { k } } = \omega ( x _ { k } ) , } \\ { \Delta _ { x _ { k } } ^ { s c a l e } = \sigma ( s c a l e , c o o r d _ { x _ { k } } ) , } \\ { \Delta _ { x _ { k } } ^ { ' } = \Delta _ { x _ { k } } \cdot \Delta _ { x _ { k } } ^ { s c a l e } . } \end{array}$$

   • $\omega$ and $\sigma$ are small MLPs.
   • $\Delta_{x_k}$ is the standard input-dependent step size.
   • $\Delta_{x_k}^{scale}$ is a new scale modulation factor generated from the target scale and the current coord (coordinate).
   • $\Delta'_{x_k}$ is the final, scale-aware step size. This allows the model to adjust how much influence the previous state has on the current state based on the magnification level.

2. Modulating the Input Matrix $B$: The matrix $B$ controls how the input $x_k$ influences the state $h_k$. The authors apply the same modulation logic to $B$ to make the input mapping scale-aware. $$\begin{array} { c } { B _ { x _ { k } } , C _ { x _ { k } } , \Delta _ { x _ { k } } = \omega ( x _ { k } ) , } \\ { B _ { x _ { k } } ^ { s c a l e } , { \Delta _ { x _ { k } } ^ { s c a l e } } = \sigma ( s c a l e , c o o r d _ { x _ { k } } ) , } \\ { { \Delta _ { x _ { k } } ^ { ' } = \Delta _ { x _ { k } } \cdot \Delta _ { x _ { k } } ^ { s c a l e } , } } \\ { B _ { x _ { k } } ^ { ' } = B _ { x _ { k } } \cdot B _ { x _ { k } } ^ { s c a l e } . } \end{array}$$ The new scale-aware parameters $\Delta'_{x_k}$ and $B'_{x_k}$ are then used in the discretization formula to get $\bar{A}'_{x_k}$ and $\bar{B}'_{x_k}$, which are finally used in the discrete state update. This ensures the entire state evolution process is conditioned on the target scale.

Steps & Procedures: The S³Mamba Architecture

As shown in Figure 2(a), the full S³Mamba architecture works as follows:

1. Feature Extraction: An LR image is passed through a backbone feature extractor (e.g., EDSR, RDN) to get a feature map $F_{LR}$.
2. Local and Global Feature Fusion:
   • Local features $F_{LR}^{local}$ are extracted using an Unfold operation, which gathers features from a local neighborhood (similar to LIIF).
   • Global features $F_{LR}^{global}$ are extracted by passing $F_{LR}$ through the proposed SSSM.
   • These features are concatenated: $F_{fusion} = \text{concat}(F_{LR}^{local}, F_{LR}^{global})$.
3. Scale-Aware Self-Attention and Rendering: This is the decoding stage. $$\begin{array} { r l } & { \alpha _ { w e i g h t } = S S S M ( c o o r d _ { H R } , s c a l e ) , } \\ & { \quad F _ { H R } ^ { \prime } = S S S M ( \alpha _ { w e i g h t } \cdot F _ { H R } ) , } \\ & { \quad R G B _ { H R } = S S S M ( F _ { H R } ^ { \prime } ) . } \end{array}$$

   • First, an SSSM takes the target HR coordinates ($coord_{HR}$) and the scale as input to generate a global attention map $\alpha_{weight}$. This map represents the importance of different spatial locations for the given scale.

   • This attention map is multiplied with the HR features ($F_{HR}$, derived from $F_{fusion}$). The result is processed by another SSSM.

   • Finally, another SSSM block processes the refined features to produce the final output RGB values ($RGB_{HR}$).

     This design ensures that both the feature encoding and the final rendering are deeply integrated with scale information, allowing for high-quality, continuous super-resolution.

5. Experimental Setup

• Datasets:

  • DIV2K: A standard high-quality dataset used for synthetic SR tasks. It contains 800 2K-resolution images for training. LR images are generated by bicubic downsampling with a random scale factor $s$ from a uniform distribution $U(1, 4)$.
  • COZ (Continuous Optical Zoom): A real-world benchmark dataset for ASSR. It consists of 153 training images and 37 test images captured with a real camera's optical zoom, providing a more realistic and challenging scenario with complex, non-ideal degradations.

• Evaluation Metrics:

  1. PSNR (Peak Signal-to-Noise Ratio): Measures the ratio between the maximum possible power of a signal and the power of corrupting noise. It is a logarithmic metric measured in decibels (dB), where higher values indicate better reconstruction quality.

     • Conceptual Definition: It quantifies the pixel-wise difference between the ground truth and the reconstructed image. It is sensitive to large errors but may not perfectly align with human visual perception.
     • Mathematical Formula: $$\text{PSNR} = 20 \cdot \log_{10}(\text{MAX}_I) - 10 \cdot \log_{10}(\text{MSE})$$
     • Symbol Explanation:
       • $\text{MAX}_I$: The maximum possible pixel value of the image (e.g., 255 for an 8-bit grayscale image).
       • $\text{MSE}$: The Mean Squared Error between the ground truth image $I$ and the reconstructed image $K$, calculated as $\frac{1}{mn}\sum_{i=0}^{m-1}\sum_{j=0}^{n-1} [I(i,j) - K(i,j)]^2$ for images of size $m \times n$.

  2. SSIM (Structural Similarity Index): A metric that measures the perceptual similarity between two images. It is considered to be more aligned with human vision than PSNR.

     • Conceptual Definition: SSIM compares three aspects of the images: luminance, contrast, and structure. It produces a value between -1 and 1, where 1 indicates perfect similarity.
     • Mathematical Formula: $$\text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$$
     • Symbol Explanation:
       • $\mu_x, \mu_y$: The mean of images $x$ and $y$.
       • $\sigma_x^2, \sigma_y^2$: The variance of images $x$ and $y$.
       • $\sigma_{xy}$: The covariance of $x$ and $y$.
       • $c_1, c_2$: Small constants to stabilize the division.

• Baselines: The proposed S³Mamba is compared against eight state-of-the-art ASSR models: MetaSR, LIIF, LTE, LINF, SRNO, LIT, CiaoSR, and LMI. The experiments use two different SR backbones, EDSR and RDN, to show the general applicability of the proposed method.

6. Results & Analysis

Core Results

1. Results on the Real-World COZ Dataset: The following is a transcription of Table 1 from the paper.

Manually transcribed from the paper since no image was provided.

• Analysis: On the RDN backbone, the proposed method (Ours) consistently outperforms all other methods across all scales, in both PSNR and SSIM. For example, at ×3.5, it achieves 25.92/0.760, surpassing the next best LMI (25.86/0.753). The strong SSIM scores suggest that S³Mamba is particularly effective at reconstructing structural details, which is crucial for visual quality in real-world images. The visual results in Figure 3 confirm this, showing that the proposed method produces cleaner textures and fewer artifacts compared to others.

  Figure 3 shows that for ×3 SR on the real-world COZ dataset, S³Mamba (Ours) reconstructs the text and grid patterns more clearly and with fewer artifacts than other methods like CiaoSR and LIIF.

  2. Results on the Synthetic DIV2K Dataset: The following is a transcription of Table 2 from the paper.

Manually transcribed from the paper since no image was provided.

• Analysis: On the DIV2K dataset, S³Mamba achieves SOTA performance in most scenarios, especially for smaller to medium scales (e.g., ×2, ×4, ×6) and large out-of-scale factors (e.g., ×12, ×18). For instance, with the RDN backbone, it scores the highest PSNR at ×2, ×4, ×6, ×12, and ×18. It is highly competitive with CiaoSR but, as noted in the supplementary material, achieves this with only half the computational complexity. The visual comparison in Figure 4 demonstrates that S³Mamba reconstructs fine textures more faithfully than other methods, including CiaoSR, which tends to produce some artifacts.

  Figure 4 shows a close-up of a ×4 super-resolved image. The result from "Ours" is visually closest to the Ground Truth (GT), correctly rendering the grid pattern on the building, whereas other methods blur or distort it.

Ablation Study

The ablation studies validate the effectiveness of the core components of S³Mamba.

1. Effectiveness of SSSM: Table 3 compares the performance of the INR module using a standard MLP, a standard SSM, and the proposed SSSM.

Manually transcribed from the paper since no image was provided.

• Analysis: The results clearly show a performance hierarchy: SSSM > SSM > MLP.
  • Replacing MLP with a standard SSM boosts PSNR (e.g., from 34.78 to 34.85 at ×2), confirming that the global modeling capability of SSMs is superior to the point-wise MLP approach.
  • Using the proposed SSSM instead of a standard SSM provides another significant boost (from 34.85 to 34.91 at ×2). This directly demonstrates the importance of making the SSM scale-aware through the modulation of $\Delta$ and $B$.

2. Effectiveness of GFE and SFAtt: Table 4 analyzes the contribution of the Global Feature Extraction (GFE) and Scale-aware Self-Attention (SFAtt) modules within the S³Mamba architecture.

Manually transcribed from the paper since no image was provided.

• Analysis: Both modules contribute positively to the final performance.
  • Adding SFAtt (scale-aware self-attention) to the baseline improves PSNR by 0.07 dB at ×2, showing that the scale-conditioned attention mechanism helps the network adapt its rendering.
  • Adding GFE (global feature extraction via SSSM) provides a larger boost of 0.14 dB (34.85 vs. 34.71), highlighting the crucial role of global information in constructing a robust continuous representation.
  • Combining both (GFE + SFAtt) yields the best results, showing that their contributions are complementary.

7. Conclusion & Personal Thoughts

Conclusion Summary

The paper introduces S³Mamba, a novel and efficient architecture for arbitrary-scale super-resolution. The core innovation is the Scalable State Space Model (SSSM), which for the first time adapts the internal dynamics of an SSM to the target super-resolution scale. By modulating the step size and input matrix during discretization, the SSSM achieves a consistent, continuous image representation with only linear computational complexity. Complemented by a scale-aware self-attention mechanism, the full S³Mamba framework effectively captures both global context and scale-specific details. Experimental results confirm that the method achieves state-of-the-art performance on synthetic and real-world benchmarks, outperforming previous MLP and Transformer-based approaches in both accuracy and efficiency.

Limitations & Future Work

The paper presents a very strong case, but some areas could be explored further:

• Authors' Stated Limitations: The authors do not explicitly state limitations in the main body of the paper, focusing on the method's strengths. The conclusion suggests it "paves a new way," implying future work will build upon this foundation.
• Potential Limitations:
  • Extreme "Out-of-Scale" Performance: While the model performs well up to ×30, its performance degradation at extreme scales could be further analyzed. The gap between S³Mamba and CiaoSR narrows at the very largest scales (e.g., ×30 on RDN), suggesting there might be a limit to the current approach's generalization.
  • Complexity of SSSM: While computationally linear, the SSSM introduces additional MLPs and modulations. A detailed analysis of parameter counts and real-world inference speed (ms/image) compared to CiaoSR would be valuable, beyond just computational complexity.
  • Scan Direction: The paper doesn't detail the scanning mechanism (e.g., four-directional scanning like in Visual Mamba). This implementation detail can have a significant impact on performance for non-sequential 2D data like images.

Personal Insights & Critique

• A Powerful and Transferable Idea: The core concept of modulating SSM parameters ($\Delta$ and $B$) with external conditioning information (scale, coordinates) is highly innovative and elegant. This mechanism is not limited to super-resolution; it could be a general-purpose tool for any conditional sequence modeling task where the input-output relationship depends on external context (e.g., conditional audio generation, style-conditioned text generation, or even video frame prediction conditioned on motion vectors).
• The Right Tool for the Job: This work is a perfect example of the ongoing trend of finding efficient alternatives to the Transformer. By identifying the core limitation of Transformers (quadratic complexity) and finding an architecture (SSM) that remedies it while preserving the key advantage (long-range dependency modeling), the authors provide a practical and powerful solution.
• Solid Experimental Validation: The paper's strength is its rigorous evaluation. Testing on both synthetic (DIV2K) and real-world (COZ) data, along with comprehensive ablation studies, provides convincing evidence for the method's effectiveness. The superior performance on the COZ dataset is particularly noteworthy, as it proves the model's robustness to real-world image degradations.
• Critique: While the quantitative gains over the previous SOTA (CiaoSR) are sometimes modest (e.g., 0.02 dB PSNR at ×2 on EDSR), the claimed halving of computational complexity is a massive practical advantage. This efficiency-for-performance trade-off is a huge win and should perhaps have been emphasized even more as a primary contribution. The S³Mamba name is also clever, succinctly capturing its key components: Scalable State Space Mamba. Overall, this paper makes a significant and timely contribution to the field of image restoration.