1. Bibliographic Information

1.1. Title

UltraFusion: Ultra High Dynamic Imaging using Exposure Fusion

1.2. Authors

Zixuan Chen (Zhejiang University, Shanghai AI Laboratory), Yujin Wang (Shanghai AI Laboratory), Xin Cai (The Chinese University of Hong Kong), Zhiyuan You (The Chinese University of Hong Kong), Zheming Lu (Zhejiang University), Fan Zhang (Shanghai AI Laboratory), Shi Guo (Shanghai AI Laboratory), and Tianfan Xue (The Chinese University of Hong Kong, Shanghai AI Laboratory).

1.3. Journal/Conference

This paper was published on arXiv on January 20, 2025. Given the affiliations and the quality of the work, it is part of the research output from prominent institutions like the Shanghai AI Laboratory and CUHK, which are leading centers for computer vision research.

1.4. Publication Year

2025

1.5. Abstract

Capturing high dynamic range (HDR) scenes is a critical challenge in camera design. Most cameras use exposure fusion to merge images of different exposure levels. However, standard techniques typically only handle differences of 3-4 stops. In very high dynamic range scenes, these methods fail due to misalignment, lighting inconsistencies, or tone mapping artifacts. This paper introduces UltraFusion, the first technique capable of merging images with a 9-stop exposure difference. By modeling fusion as a guided inpainting problem, the model uses an under-exposed image as soft guidance to fill in highlights in an over-exposed image. This approach is robust to alignment issues and generates natural tone mapping using generative image priors. The authors also contribute the UltraFusion dataset, a real-world benchmark with up to 9-stop differences.

1.6. Original Source Link

• Original Source Link: https://arxiv.org/abs/2501.11515
• PDF Link: https://arxiv.org/pdf/2501.11515v4.pdf

2. Executive Summary

2.1. Background & Motivation

The dynamic range of a scene—the ratio between the brightest highlights and the darkest shadows—often exceeds the capabilities of modern camera sensors. To capture such scenes, cameras take multiple photos at different exposure levels.

• The Problem: Traditional exposure fusion works well for small exposure differences (3-4 stops). However, for "Ultra" high dynamic range scenes (e.g., a dark room with a view of the sun through a window), a difference of 8-9 stops is needed.
• Challenges:
  1. Alignment: Moving objects cause "ghosting" artifacts because they are in different positions in each frame.
  2. Lighting Variation: Objects look different at different exposure levels, making simple merging look unnatural.
  3. Tone Mapping: High-dynamic images must be compressed to be viewed on standard screens, often leading to loss of contrast or "halo" effects.

2.2. Main Contributions / Findings

• UltraFusion Model: A novel framework that treats exposure fusion as guided inpainting. It uses a diffusion model to "fill in" over-exposed areas using the information from a darkly exposed photo.
• Robustness to Motion: By using the under-exposed image as "soft guidance" rather than a strict requirement, the model can ignore misaligned pixels and generate realistic textures.
• Decompose-and-Fuse Branch: A specialized neural network structure that separates color and structural information from the under-exposed image to guide the generation process more effectively.
• Fidelity Control Branch: An additional module designed to ensure the final output remains faithful to the original scene's colors and textures, preventing the AI from "hallucinating" fake details.
• UltraFusion Dataset: A new benchmark of 100 real-world image pairs with extreme exposure differences (up to 9 stops) to test the limits of HDR algorithms.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, several core concepts in imaging and AI are required:

• Stops and Exposure Value (EV): In photography, a "stop" represents a doubling or halving of the amount of light. An 8-stop difference means one image is $2^8 = 256$ times brighter than the other.
• High Dynamic Range (HDR) Imaging: The process of capturing and displaying scenes with a high range of luminosity.
• Exposure Fusion (MEF): A technique that directly blends multiple Low Dynamic Range (LDR) images into a single high-quality LDR image, skipping the complex step of creating an intermediate mathematical HDR file.
• Diffusion Models: A type of generative AI (like Stable Diffusion) that learns to create images by reversing a process of adding noise. They are excellent at creating realistic textures.
• Image Inpainting: The process of reconstructing lost or deteriorated parts of images. In this paper, over-exposed (white) pixels are treated as "deteriorated" areas to be filled.
• Optical Flow: A technique to track how pixels move between two frames. The paper uses RAFT (Recurrent All-Pairs Field Transforms) for this.

3.2. Previous Works

• Traditional HDR (e.g., HDR+): Used by Google and others, these typically handle 3 stops. They rely on rigid alignment, which fails when objects move quickly or exposures differ too much.
• ControlNet: A popular framework for adding conditions (like edges or poses) to Stable Diffusion. The authors find that standard ControlNet fails here because it doesn't know which image to prioritize as the "reference," leading to artifacts.
• Tone Mapping Operators (TMOs): Algorithms that squash HDR data into LDR. Most TMOs are mathematical filters that can sometimes make images look "flat" or "cartoonish."

3.3. Technological Evolution

HDR research has moved from simple pixel-averaging to flow-based alignment, then to CNN (Convolutional Neural Network) based fusion, and finally to Transformers (like HDR-Transformer). UltraFusion represents the next step: using Generative AI (Diffusion) to solve the "impossible" cases where data is missing or pixels are completely misaligned.

4. Methodology

4.1. Principles

The core intuition is that an over-exposed image $I_{oe}$ is perfect in the shadows but "broken" (pure white) in the highlights. Conversely, the under-exposed image $I_{ue}$ is perfect in the highlights but "broken" (pure black) in the shadows. UltraFusion treats the white areas of $I_{oe}$ as a hole to be filled, using $I_{ue}$ as a "hint" or guidance for what should be inside that hole.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Stage 1: Pre-alignment

Before the AI can merge the images, it needs a rough idea of how they overlap.

1. Intensity Mapping: Since one image is dark and the other is bright, standard motion tracking fails. The system first adjusts the brightness of $I_{ue}$ to match $I_{oe}$ using an intensity mapping function.
2. Flow Estimation: It uses the RAFT network to estimate the bidirectional flow $f_{oe \to ue}$ (how pixels move from bright to dark) and $f_{ue \to oe}$ (dark to bright).
3. Backward Warping: The under-exposed image is "warped" to match the position of the over-exposed image.
4. Consistency Check: To avoid "ghosting" in areas where objects moved or were hidden (occluded), a mask $\mathcal{M}$ is created. The final output of this stage is the pre-aligned image $I_{ue \to oe}$: $I_{ue \to oe} = (1 - \mathcal{M}) \cdot \mathcal{W}(I_{ue}, f_{oe \to ue})$

• $\mathcal{W}$: The warping function.

• $\mathcal{M}$: The occlusion mask (1 where pixels are missing/hidden, 0 otherwise).

• $f_{oe \to ue}$: The motion path from the bright image to the dark image.

  The following figure (Figure 3 from the paper) illustrates the two-stage pipeline:

  该图像是示意图，展示了 UltraFusion 中的预对齐阶段和引导修复阶段的流程。左侧部分包含了过曝和欠曝图像的流估计方法，而右侧部分则描述了生成模型的编码解码过程，并展示了如何结合控制分支以重建高质量图像。

4.2.2. Stage 2: Guided Inpainting

This stage uses a modified Stable Diffusion model. It takes three inputs: the over-exposed image $I_{oe}$, the aligned dark guidance $I_{ue \to oe}$, and the current noise latent $z_t$.

A. Decompose-and-Fuse Control Branch (DFCB)

The authors found that dark images are too dim for the AI to "see" clearly. To fix this, they decompose the dark image into Structure and Color.

1. Structure Extraction ($S_{ue}$): They normalize the luminance channel to remove brightness bias: $S_{ue} = (Y_{ue} - \mu(Y_{ue})) / \sigma(Y_{ue})$
   • $Y_{ue}$: The brightness (Luminance) channel.
   • $\mu(\cdot)$: The average intensity.
   • $\sigma(\cdot)$: The standard deviation (contrast).
2. Color Extraction ($C_{ue}$): They use the Chroma (UV) channels.
3. Multi-scale Cross Attention: These features are fused into the main network. Instead of just adding them, the model uses Attention to decide which parts of the dark image are actually useful for the current "hole" it is filling. The cross-attention output $X_{out}$ is calculated as: $X_{out} = X_{oe} + \mathbf{Conv_{1 \times 1}}(V \mathbf{Softmax}(\frac{Q^T K}{\tau}))$

• Q, K, V: The Query (from $X_{oe}$), Key, and Value (from $X_{ue}$) matrices common in Transformer architectures.

• $\tau$: A learnable scaling factor to control the strength of the guidance.

• $\mathbf{Conv_{1 \times 1}}$: A small neural layer to adjust dimensions.

  The DFCB architecture is shown below:

  该图像是示意图，展示了我们提出的分解与融合控制分支的详细架构。图中显示了对不同曝光图像的引导提取与融合过程，包括主提取器、跨注意力机制等模块。

B. Fidelity Control Branch (FCB)

Diffusion models are known to "dream up" textures that weren't there. To keep the image realistic, the FCB acts as a shortcut. It takes the original image and sends high-frequency details directly to the final VAE Decoder (the part of the AI that turns math back into a visible image), ensuring the output isn't too "smooth" or "fake."

4.3. Training Data Synthesis

Since there aren't many "Ultra HDR" videos with ground truth, the authors invented a way to synthesize data:

1. Take a standard video (e.g., Vimeo-90K).
2. Select two frames far apart to simulate "large motion."
3. Take a high-quality static HDR pair from the SICE dataset.
4. Apply the motion from the video to the HDR pair to create a "fake" dynamic HDR scene.

5. Experimental Setup

5.1. Datasets

• SICE: A static dataset used for training.
• MEFB: A benchmark for static exposure fusion.
• RealHDRV: A dynamic dataset containing 50 scenes with motion.
• UltraFusion Benchmark (New): 100 pairs captured by the authors using DSLR (Canon R8) and various phones (iPhone 13, Redmi, etc.). It features differences up to 9 stops.

5.2. Evaluation Metrics

The paper uses several metrics to measure quality without needing a "perfect" reference image:

1. MUSIQ / DeQA-Score / PAQ2PIQ / HyperIQA: These are "no-reference" image quality assessors. They use deep learning to give a score based on how "good" or "natural" an image looks to a human.
2. MEF-SSIM (Multi-Exposure Fusion Structural Similarity): $MEF\text{-}SSIM(X, Y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}$
   • Quantifies how much of the structural information from the input images is preserved in the fused result.
3. TMQI (Tone Mapped Quality Index): Specifically designed for HDR. It combines structural fidelity and statistical naturalness.

5.3. Baselines

The method is compared against:

• HDR Reconstruction: HDR-Transformer, SCTNet, SAFNet.
• Exposure Fusion: Deepfuse, MEF-GAN, U2Fusion, HSDS-MEF.

6. Results & Analysis

6.1. Core Results Analysis

UltraFusion consistently outperforms all competitors, especially in dynamic scenes and extreme exposure differences.

6.1.1. Performance on Static Data (MEFB)

The following are the results from Table 1 of the original paper:

6.1.2. Performance on Dynamic and Ultra HDR Data

The following are the results from Table 2 of the original paper:

6.2. Ablation Studies

The authors proved that every part of the model is necessary:

• Without Alignment: The model produces blurry results in motion areas.
• Without DFCB (Structure/Color Branch): The model ignores the dark image hint and "invents" its own highlights, which may not match reality.
• Without FCB (Fidelity Branch): The image loses sharp textures and looks "AI-generated."

7. Conclusion & Reflections

7.1. Conclusion Summary

UltraFusion successfully breaks the 4-stop barrier of traditional exposure fusion, reaching up to 9 stops. By using a diffusion-based guided inpainting approach, it creates images that are not only high-contrast but also look natural and are free of motion artifacts. The introduction of the UltraFusion dataset provides a much-needed benchmark for extreme dynamic range imaging.

7.2. Limitations & Future Work

• Speed: It takes about 3.3 seconds to process a $512 \times 512$ image on an expensive NVIDIA RTX 4090 GPU. This is too slow for real-time smartphone use.
• Single Image Fallback: If the occlusion (hidden area) is too large, the model relies entirely on AI "imagination" (diffusion prior), which can be unreliable for scientific or forensic purposes.
• Future Work: The authors suggest looking for faster implementations and more robust motion estimation algorithms.

7.3. Personal Insights & Critique

This paper is a brilliant application of Generative AI to a "hardware" problem. Instead of trying to build a better sensor, the authors are building a "smarter" brain for the camera.

• Innovation: The move from "merging" to "inpainting" is a paradigm shift. Inpainting acknowledges that some data is truly lost and needs to be reconstructed using intelligent priors.
• Potential Issue: The 3.3-second processing time is a significant hurdle. While the results are beautiful, the practical application in consumer electronics is currently limited to "post-processing" apps rather than the native camera shutter.
• Aesthetic vs. Accurate: Because it uses a diffusion model, there is always a risk that the sun or a light bulb in the fused image is a "perfect AI version" of a sun, not the actual sun that was there. For photography, this is great; for scientific data, it might be problematic.