Restoring images distorted by atmospheric turbulence is a ubiquitous problem in long-range imaging applications. While existing deep-learning-based methods have demonstrated promising results in specific testing conditions, they suffer from three limitations: (1) lack of generalization capability from synthetic training data to real turbulence data; (2) failure to scale, hence causing memory and speed challenges when extending the idea to a large number of frames; (3) lack of a fast and accurate simulator to generate data for training neural networks. In this paper, we introduce the turbulence mitigation transformer (TMT) that explicitly addresses these issues. TMT brings three contributions: Firstly, TMT explicitly uses turbulence physics by decoupling the turbulence degradation and introducing a multi-scale loss for removing distortion, thus improving effectiveness. Secondly, TMT presents a new attention module along the temporal axis to extract extra features efficiently, thus improving memory and speed. Thirdly, TMT introduces a new simulator based on the Fourier sampler, temporal correlation, and flexible kernel size, thus improving our capability to synthesize better training data. TMT outperforms state-of-the-art video restoration models, especially in generalizing from synthetic to real turbulence data. Code, videos, and datasets are available at \href{https://xg416.github.io/TMT}{https://xg416.github.io/TMT}.

1. Bibliographic Information

• Title: Imaging through the Atmosphere using Turbulence Mitigation Transformer
• Authors: Xingguang Zhang, Zhiyuan Mao, Nicholas Chimitt, Stanley H. Chan. The authors are affiliated with the School of Electrical and Computer Engineering at Purdue University, with one author at Samsung Research America. Their work is supported by IARPA and the NSF, indicating a focus on high-impact, government-relevant research areas like surveillance and long-range imaging.
• Journal/Conference: The paper is available as a preprint on arXiv. The formatting and "Student Member, IEEE" designation suggest it was prepared for submission to a high-impact IEEE journal or conference.
• Publication Year: The second version of the preprint was submitted in July 2022.
• Abstract: The abstract introduces the problem of restoring images distorted by atmospheric turbulence. It identifies three key limitations in existing deep learning methods: poor generalization from synthetic to real data, scalability issues (memory and speed) with multiple frames, and the lack of a fast, accurate data simulator. The paper proposes the Turbulence Mitigation Transformer (TMT) to address these gaps. TMT's contributions are threefold: (1) a physics-informed model that decouples turbulence into distortion (tilt) and blur, using a multi-scale loss for better effectiveness; (2) a novel temporal attention module that improves memory and speed efficiency; and (3) a new, fast, and accurate turbulence simulator. The authors claim that TMT outperforms state-of-the-art models, particularly in generalizing to real-world data.
• Original Source Link: https://arxiv.org/abs/2207.06465v2 (Preprint)

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: Long-range imaging is often severely degraded by atmospheric turbulence, which causes spatially varying blur and geometric warping. This distortion compromises computer vision tasks like detection and recognition.
  • Identified Gaps: The paper argues that prior deep learning solutions are flawed. They fail to generalize because the synthetic data they are trained on doesn't accurately model real turbulence. Furthermore, methods that use multiple frames (which is beneficial due to the "lucky effect") are crippled by high memory usage and slow processing speeds, limiting the number of frames they can use. Finally, creating large-scale, high-fidelity training data is a bottleneck because existing simulators are either too slow or too inaccurate.
  • Innovation: The paper introduces a holistic solution that tackles all three problems simultaneously. It proposes a new neural network architecture (TMT) specifically designed for turbulence physics, an efficient attention mechanism to handle long video sequences, and a powerful simulator to generate high-quality training data.

• Main Contributions / Findings (What):

  1. A Physics-Informed Restoration Model (TMT): Instead of treating turbulence as a generic video restoration problem, TMT explicitly models the physics. It uses a two-stage approach to first remove geometric distortions (tilt) and then tackle the blur. This is supervised by a multi-scale loss, which improves training stability and generalization.
  2. An Efficient Temporal Attention Module (TCJA): To overcome the memory bottleneck of standard transformers, TMT introduces the Temporal-Channel Joint Attention (TCJA) module. This module computes self-attention along the temporal and channel dimensions for each pixel, which is far more efficient than full spatio-temporal attention, allowing the model to process more frames and capture long-range temporal dynamics.
  3. An Improved Turbulence Simulator and Dataset: The paper enhances an existing physics-based simulator by incorporating temporal correlation, enabling flexible blur kernel sizes, and using a fast Fourier-based sampler for dense-field simulation. Using this simulator, the authors created and released the TMT dataset, the largest and most comprehensive dataset for turbulence mitigation research to date.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Atmospheric Turbulence: Random fluctuations in the atmosphere's refractive index bend light rays as they travel, causing images captured over long distances to appear blurry and warped. The distortion is dynamic and varies across the image.
  • Tilt and Blur (Zernike Decomposition): In optics, complex wavefront distortions (aberrations) can be broken down into a set of standard polynomial shapes called Zernike polynomials. The lowest-order terms correspond to simple tilt (pixel shifting or warping), while higher-order terms correspond to more complex aberrations that manifest as blur. This paper's two-stage model is directly inspired by this physical decomposition.
  • Anisoplanatic Turbulence: This term means the turbulence effect is spatially varying. For a large scene, the distortion in one part of the image is different from another. This makes simple deconvolution (which assumes a single blur kernel for the whole image) ineffective.
  • Lucky Effect: In a sequence of short-exposure images, some frames (or regions within frames) will, by chance, be less distorted than others. Multi-frame restoration methods leverage this by fusing information from many frames to reconstruct a clearer image.
  • Vision Transformer (ViT): A type of neural network architecture that uses a mechanism called self-attention. Unlike Convolutional Neural Networks (CNNs) that use fixed kernels, self-attention allows the model to weigh the importance of different parts of the input dynamically. This adaptivity makes transformers well-suited for handling spatially varying distortions like turbulence.

• Previous Works:

  • Classical Methods: Early approaches like lucky selection ([13]), complex wavelets ([4]), and optimization-based techniques ([7]) were pioneering but limited by heuristic assumptions and couldn't model the full complexity of turbulence.
  • Single-Frame Deep Learning: Models like [9]-[11] attempted restoration from a single image. However, they often rely on strong priors (e.g., knowing the object is a face) and struggle to generalize to arbitrary scenes.
  • Multi-Frame Deep Learning: The most relevant prior work is TSRWGAN [12], which also uses multiple frames. However, the authors state its generalization is limited, and its architecture is not as scalable.
  • General Video Restoration: State-of-the-art models like VRT [38] and $BasicVSR++$ [39] are powerful but are not specifically designed for turbulence. They often use computationally expensive attention mechanisms (like local windows in VRT) and lack the physics-informed structure of TMT.

• Differentiation: TMT distinguishes itself from previous work in three key ways:

  1. Physics-Informed Architecture: Unlike generic video restorers (VRT, $BasicVSR++$), TMT's two-stage (tilt-then-blur) structure directly mirrors the physical degradation process.
  2. Efficiency and Scalability: Unlike VRT, which uses computationally heavy local spatial attention, TMT's TCJA module operates on the temporal-channel axis, drastically reducing memory and computation, thus allowing for more input frames.
  3. Data-Centric Approach: Unlike TSRWGAN [12] and others that use less accurate simulators, TMT is co-developed with a superior simulator. The paper demonstrates that this improved synthetic data is key to achieving good generalization on real-world images.

4. Methodology (Core Technology & Implementation)

The paper's methodology can be broken down into the restoration model (TMT) and the data generation simulator.

The Turbulence Mitigation Transformer (TMT) Model

The TMT model follows a two-stage pipeline, as shown in the figure below. First, it corrects geometric warping in the Tilt-Removal Module, and then it removes blur and residual distortion in the Blur-Removal Module.

该图像是论文中所示的算法流程示意图，展示了一个两阶段大气湍流图像恢复方法。流程涵盖湍流模拟（包括倾斜随机场和模糊），及基于深度卷积和Transformer模块的倾斜去除与模糊去除步骤，最终实现从多帧湍流图像恢复清晰图像。

• Physics-Based Forward Model: The degradation is modeled as a composition of a tilt operator $\mathcal{T}$ and a blur operator $\mathcal{B}$: $$I ( \mathbf { x } , t ) = [ \mathcal{B} \circ \mathcal{T} ] ( J ( \mathbf { x } , t ) )$$

  • $J(\mathbf{x}, t)$: The clean, ground-truth image at coordinate $\mathbf{x}$ and time $t$.
  • $\mathcal{T}$: The tilt operator, which causes geometric warping.
  • $\mathcal{B}$: The blur operator, which causes high-order aberrations.
  • $I(\mathbf{x}, t)$: The observed corrupted image. The goal of TMT is to invert this process.

• Stage 1: Tilt-Removal Module

  • Principle: This module aims to align the input frames to remove the dominant geometric warping. It uses a lightweight U-Net architecture with depth-wise 3D convolutions. This choice allows it to process multiple frames simultaneously and efficiently capture temporal jitter.
  • Architecture: It is a four-level U-Net. The encoder uses max-pooling for downsampling, and the decoder uses transposed convolutions for upsampling. It outputs warping grids at three different scales to progressively rectify the input image sequence.

• Multi-Scale Loss for Tilt Removal

  • Principle: To train the tilt-removal module effectively, the authors introduce a multi-scale loss. The intuition is that tilt is easier to correct at lower resolutions. By enforcing correctness at multiple scales, the network learns a more robust and physically consistent alignment.
  • Procedure:
    1. The simulator generates both the fully distorted (tilt+blur) sequence $I$ and a corresponding "blur-only" sequence $\widetilde{J}_{\ell}$ at multiple scales $\ell=1, \dots, L$.
    2. The tilt-removal module takes $I$ and produces an estimated tilt-free (but still blurry) image $\widehat{J}_{\ell}$ at each scale.
    3. The loss function minimizes the difference between the network's estimate and the blur-only ground truth at each scale.
  • Formula: The loss is a weighted sum of Charbonnier losses across the scales: $$\mathcal{L}_{\text{tilt}} = \sum_{\ell=1}^{L} \gamma_{\ell} \cdot \mathcal{L}_{\text{char}}(\widehat{J}_{\ell}(\mathbf{x}, t), \widetilde{J}_{\ell}(\mathbf{x}, t))$$
    • $\widehat{J}_{\ell}$: The network's estimated tilt-free output at scale $\ell$.
    • $\widetilde{J}_{\ell}$: The ground-truth blur-only image at scale $\ell$.
    • $\mathcal{L}_{\text{char}}$: The Charbonnier loss, a smooth variant of the L1 loss that is less sensitive to outliers.
    • $\gamma_{\ell}$: Weights that prioritize different scales (e.g., $\gamma_1=0.6, \gamma_2=0.3, \gamma_3=0.1$).

• Stage 2: Blur-Removal Module

  • Principle: This module takes the aligned (but still blurry) frames from the first stage and performs deblurring. The core of this module is a new, efficient transformer block featuring Temporal-Channel Joint Attention (TCJA).

  • Temporal-Channel Joint Attention (TCJA): The key innovation for handling multiple frames efficiently. Instead of computing attention across all spatial and temporal positions (which is computationally infeasible), TCJA operates on the temporal and channel dimensions for each spatial location independently. This allows for full temporal connection across all frames without quadratic complexity in the number of pixels.

  • TCJA Variants: The paper proposes two variants of the TCJA block, as shown below.

    该图像是论文“Imaging through the Atmosphere using Turbulence Mitigation Transformer”中的结构示意图，展示了Transformer块内的通道-时序联合注意力机制及其两种连接方式(a) Vanilla channel-temporal connection和(b) Channel-temporal shuffle connection的具体流程。

    • (a) Vanilla Connection: A straightforward approach where linear projections are applied separately to the temporal dimension before attention.
    • (b) Channel-Temporal Shuffle Connection: This more advanced variant, inspired by ShuffleNet, splits channels into groups and shuffles them. This allows features in different channels to interact across frames over several attention blocks, promoting better information flow and leading to slightly better performance.

The Improved Turbulence Simulator

The quality of the deep learning model is highly dependent on the training data. The authors made several key improvements to a state-of-the-art turbulence simulator (DF-P2S [54]).

1. Temporal Correlation: Real turbulence is temporally correlated. To simulate this efficiently, the authors use an auto-regressive model on the random noise vector $\mathbf{w}_t$ that generates the turbulence phase screens: $$\mathbf{w}_t = \alpha \mathbf{w}_{t-1} + \sqrt{1 - \alpha^2} \mathbf{z}$$
   • $\mathbf{w}_t$: The noise vector at time $t$.
   • $\alpha$: A correlation coefficient ($0 \le \alpha \le 1$). A value close to 1 means strong temporal correlation.
   • $\mathbf{z}$: A fresh random Gaussian noise vector. This simple but effective method creates a temporally smooth evolution of turbulence without the massive computational cost of a full spatio-temporal simulation.
2. Flexible Kernel Size: Previous simulators were limited to a fixed blur kernel size (e.g., $33 \times 33$). The authors regenerated the underlying PSF basis functions at a higher resolution, allowing them to be resized to various kernel sizes ($9 \times 9$ to $33 \times 33$) without introducing artifacts. This creates a more diverse dataset that better covers real-world conditions.
3. The TMT Dataset: Using the improved simulator, the authors generated a large-scale dataset for both static and dynamic scenes.
   • Sources: Places dataset for static scenes, SVW and TSRWGAN ground truths for dynamic scenes.
   • Parameters: Turbulence parameters (aperture, distance, etc.) were sampled from weak, medium, and strong levels to ensure diversity, as detailed in Table III of the paper.
   • Size: The dataset contains over 9,000 static image sequences and over 4,600 dynamic videos, making it a significant contribution to the field.

5. Experimental Setup

• Datasets:

  • Training Data: The newly created TMT dataset, split into static and dynamic scenes.
  • Quantitative Testing Data: The held-out test split of the TMT dataset.
  • Qualitative (Generalization) Testing Data: Several real-world datasets were used to test how well models trained on synthetic data perform in practice. These include OTIS [56] (static patterns), CLEAR [51] (dynamic scenes), and the UG2+ Challenge dataset [57] (text patterns).

• Evaluation Metrics:

  1. PSNR (Peak Signal-to-Noise Ratio): Measures the pixel-wise difference between the ground truth and the restored image. A higher PSNR indicates better reconstruction quality.
     • Conceptual Definition: It's the ratio of the maximum possible power of a signal to the power of corrupting noise. It's measured on a logarithmic scale (dB).
     • Mathematical Formula: $$\text{PSNR} = 10 \cdot \log_{10} \left( \frac{\text{MAX}_I^2}{\text{MSE}} \right)$$
     • Symbol Explanation: $\text{MAX}_I$ is the maximum possible pixel value (e.g., 255 for 8-bit images). $\text{MSE}$ is the Mean Squared Error between the ground truth and the reconstructed image.
  2. SSIM (Structural Similarity Index Measure): Measures the perceptual similarity between two images, considering luminance, contrast, and structure.
     • Conceptual Definition: It is designed to be more consistent with human visual perception than PSNR. Values range from -1 to 1, where 1 indicates a perfect match.
     • Mathematical Formula: For two image windows $x$ and $y$: $$\text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$$
     • Symbol Explanation: $\mu_x, \mu_y$ are the average pixel values; $\sigma_x^2, \sigma_y^2$ are the variances; $\sigma_{xy}$ is the covariance; $c_1, c_2$ are small constants for stability.
  3. CW-SSIM (Complex Wavelet SSIM): An extension of SSIM that is more robust to small geometric distortions like shifts and rotations.
  4. LPIPS (Learned Perceptual Image Patch Similarity): Uses a pre-trained deep neural network to extract features from two image patches and computes the distance between them.
     • Conceptual Definition: It better captures perceptual similarity than traditional metrics. A lower LPIPS score means the images are more similar perceptually.

• Baselines:

  • Turbulence-Specific: TurbNet [30] (single-frame), TSRWGAN [12] (multi-frame).
  • General Video Restoration: VRT [38], $BasicVSR++$ [60].
  • Classical Methods: CLEAR [4], Mao et al. [5].

6. Results & Analysis

• Core Results:

  Table V: Comparison on the Static Scene Dataset (transcribed)

  Methods / # frames	PSNR	SSIM	CW-SSIM	LPIPS(↓)
  TurbNet [30] / 1	22.7628	0.6923	0.8230	0.4012
  BasicVSR++ [60] / 12	26.5055	0.8121	0.9189	0.2587
  TSRWGAN [12] / 15	25.2888	0.7784	0.8982	0.2243
  VRT [38] / 12	27.4556	0.8287	0.9338	0.1877
  TMT [ours] / 12	27.7309	0.8341	0.9376	0.1815
  TMT [ours] / 20	28.4421	0.8580	0.9452	0.1693

  Table VI: Comparison on the Dynamic Scene Dataset (transcribed)

  Methods / # frames	PSNR	SSIM	CW-SSIM	LPIPS(↓)
  TurbNet [30] / 1	24.2229	0.7149	0.8072	0.4445
  BasicVSR++ [60] / 12	27.0231	0.8073	0.8653	0.2492
  TSRWGAN [12] / 15	26.3262	0.7957	0.8596	0.2606
  VRT [38] / 12	27.6114	0.8300	0.8691	0.2485
  TMT [ours] / 12	27.8816	0.8318	0.8705	0.2475
  TMT [ours] / 20	28.0124	0.8352	0.8741	0.2412

  Analysis: TMT consistently outperforms all competing methods, including specialized turbulence models and general video restoration models, across all metrics on the synthetic test set. Importantly, the $TMT / 20$ results show that the model's performance scales effectively with more input frames, which is a direct benefit of the efficient TCJA module.

• Computational Cost:

  Table VII: Comparison of Computational Consumption (transcribed)

  Methods	# parameters (M)	FLOPs/frame (G)	speed (s)
  Mao et al. [5]	-	-	~5500
  CLEAR [4]	-	-	~20
  BasicVSR++ [60]	9.76	338.4	0.08
  TSRWGAN [12]	46.28	2471	1.15
  VRT [38]	18.32	7756	5.88
  TMT [ours]	26.04	1826	1.52

  Analysis: TMT achieves its superior performance with significantly less computation than its strongest competitor, VRT (1826 vs. 7756 GFLOPs). This highlights the efficiency of the TCJA design. While slower than $BasicVSR++$, it offers a much higher restoration quality.

• Ablation Studies:

  Table VIII: Ablation Study on Design Choices (showing key results, transcribed)

  Dataset	Methods	PSNR	# Params (M)	FLOPs (G)
  Static	MS-TMTb w.o. warp (No tilt-removal stage)	27.5003	23.92	1304
  Static	MS-TMTb [Proposed] (With tilt-removal)	27.5422	26.04	1490
  Static	MS-TMTa (Vanilla TCJA)	27.5215	25.82	1392
  Static	MS-TMTb [Proposed] (Shuffle TCJA)	27.5422	26.04	1490
  Static	SS-TMTb (Single-scale input)	27.3836	25.07	1700
  Static	MS-TMTb [Proposed] (Multi-scale input)	27.5422	26.04	1490

  Analysis: The ablations confirm the value of each design choice:

  1. Two-stage design: Adding the explicit tilt-removal module (w.o. warp vs. proposed) provides a performance boost.

  2. Channel shuffle: The shuffle variant of TCJA (TMTb) slightly outperforms the vanilla version (TMTa).

  3. Multi-scale input: Using multi-scale inputs (MS-TMT) provides a significant gain of ~0.15 dB PSNR over single-scale input (SS-TMT) with lower computational cost.

  4. Number of frames (Fig. 5): The performance consistently improves as more frames are used, particularly for static scenes where averaging over time is more effective.

     该图像是图表，展示了输入帧数对静态场景和动态场景图像恢复性能（以PSNR为指标）的影响。随着输入帧数的增加，静态场景的PSNR明显提升，动态场景的PSNR提升趋缓且较稳定。

• Generalization To Real-World Data: This is a key strength of the paper.

  • Visual Quality: Across various real-world datasets (text, patterns, dynamic scenes), TMT produces visibly sharper and more stable results than all baselines, as shown in Figures 6, 7, 8, and 10.

    该图像是论文中图4的视觉示例，展示了在合成静态场景数据集上不同方法的去模糊及复原效果。(a) 输入图片 (b) 真实图像 (c-f) 分别为BasicVSR++、TSRWGAN、VRT和本文提出的TMT方法的输出结果。TMT在细节和PSNR指标上表现最佳。

    该图像是图像复原对比示意图，展示了受大气湍流影响的车辆图像通过多种方法（包括WGAN、BasicVSR++、VRT及本文提出的TMT）复原的效果。结果显示TMT在细节和清晰度上优于其他方法。

  • Superiority of the Simulator: Figure 9 is a crucial piece of evidence. When the competitor model TSRWGAN is fine-tuned on the new TMT dataset, its performance on real-world data improves dramatically. This strongly suggests that the high-quality synthetic data from the new simulator is the primary reason for TMT's excellent generalization.

    该图像是论文中对比实验的示意图，展示了输入图像(a)以及两种去除大气湍流畸变效果的放大细节图(b和c)。红色箭头标注出湍流影响明显的区域，c图显示了作者方法在去除倾斜畸变上的效果更优。

    This comparison shows that TMT's lightweight tilt-removal module, trained on the superior TMT dataset, outperforms a more complex model from a competing work [51] on real-world videos.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully presents a comprehensive solution for multi-frame atmospheric turbulence mitigation. The three core contributions—the physics-informed TMT model, the efficient TCJA attention module, and the advanced simulator and dataset—work in synergy to deliver state-of-the-art performance. The method is not only more accurate than previous work but also more computationally efficient and, most importantly, demonstrates superior generalization from synthetic training data to challenging real-world scenarios.

• Limitations & Future Work:

  • Approximated Temporal Correlation: The auto-regressive model for temporal correlation is an efficient approximation but may not capture all the complex dynamics described by Taylor's frozen flow hypothesis.
  • Supervised Learning Dependency: The method still relies entirely on a simulator. While the simulator is state-of-the-art, any remaining gap between simulation and reality (the "domain gap") will limit performance. Unsupervised or self-supervised methods could be a future direction.
  • Other Degradations: The paper focuses on turbulence. In real-world long-range imaging, other factors like camera shake, sensor noise, and haze are also present. While the authors add some noise during training, a model that explicitly handles a wider range of combined degradations could be more robust.

• Personal Insights & Critique:

  • Holistic Problem-Solving: The paper's greatest strength is its holistic approach. The authors correctly identified that progress was stalled not just by model architecture but also by the quality of training data. By advancing both the model and the simulator, they achieved a significant breakthrough.
  • Physics-Informed AI: This work is an excellent example of how incorporating domain knowledge (in this case, the physics of optics) into a deep learning model can lead to better performance and generalization than a generic, black-box approach. The two-stage design is simple but powerful.
  • Impact on the Field: The public release of the TMT model, code, and especially the large-scale dataset is a major contribution that will likely accelerate research in this area. It provides a strong baseline and a valuable resource for the community. The TCJA module is a clever architectural innovation that could be adopted in other video processing tasks where long-term temporal modeling is needed without prohibitive computational costs.