In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck due to the long sequence length and the quadratic complexity. We find that attention weights can be separated into two parts: a small fraction of large weights with high rank and the remaining weights with very low rank. This naturally suggests applying sparse acceleration to the first part and low-rank acceleration to the second. Based on this finding, we propose SLA (Sparse-Linear Attention), a trainable attention method that fuses sparse and linear attention to accelerate diffusion models. SLA classifies attention weights into critical, marginal, and negligible categories, applying O(N^2) attention to critical weights, O(N) attention to marginal weights, and skipping negligible ones. SLA combines these computations into a single GPU kernel and supports both forward and backward passes. With only a few fine-tuning steps using SLA, DiT models achieve a 20x reduction in attention computation, resulting in significant acceleration without loss of generation quality. Experiments show that SLA reduces attention computation by 95% without degrading end-to-end generation quality, outperforming baseline methods. In addition, we implement an efficient GPU kernel for SLA, which yields a 13.7x speedup in attention computation and a 2.2x end-to-end speedup in video generation on Wan2.1-1.3B.

1. Bibliographic Information

• Title: SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
• Authors: Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, Joseph E. Gonzalez, Jun Zhu, Jianfei Chen.
• Affiliations: The authors are from Tsinghua University and UC Berkeley.
• Journal/Conference: The paper is available on arXiv, which is a preprint server. This means it has not yet undergone formal peer review for a conference or journal at the time of this analysis. The future-dated citations (e.g., "2025") suggest it is a very recent work submitted for a 2025 conference.
• Publication Year: The citations suggest a submission for a conference in 2025. The version analyzed is $v1$.
• Abstract: The paper addresses the attention latency bottleneck in Diffusion Transformer (DiT) models, particularly for video generation. The authors observe that attention weights can be split into two components: a small set of high-rank, large-magnitude weights and a large set of low-rank, small-magnitude weights. Based on this, they propose SLA (Sparse-Linear Attention), a trainable method that fuses sparse and linear attention. SLA classifies attention weights into three categories: critical weights are computed with standard quadratic attention, marginal weights with linear attention, and negligible weights are skipped. The method is implemented in a single GPU kernel with forward and backward passes. After a short fine-tuning process, DiT models with SLA can achieve a 20x reduction in attention computation (95% sparsity) without sacrificing generation quality. On the Wan2.1-1.3B video generation model, SLA achieves a 13.7x speedup in the attention kernel and a 2.2x end-to-end speedup.
• Original Source Link: https://arxiv.org/pdf/2509.24006

2. Executive Summary

Background & Motivation (Why)

• Core Problem: The self-attention mechanism, a cornerstone of Transformer models, has a computational complexity of $O(N^2)$, where $N$ is the sequence length. In generative models like Diffusion Transformers (DiTs), especially for high-resolution video, the sequence length can be very large (10K-100K tokens), making attention the primary performance bottleneck.
• Gaps in Prior Work: Existing solutions fall into two camps, each with significant limitations:
  1. Linear Attention: These methods reduce complexity to $O(N)$ but often cause a severe degradation in generation quality, particularly for complex tasks like video generation. They struggle to approximate the full attention matrix, which is often high-rank.
  2. Sparse Attention: These methods compute only a subset of the attention scores. However, to maintain quality, they typically cannot achieve very high sparsity (e.g., beyond 80-85%), limiting their acceleration potential. Pushing sparsity further leads to a sharp drop in performance.
• Fresh Angle / Innovation: The paper's key insight is that the attention matrix is not uniformly complex. It can be decomposed into a sparse, high-rank component (a few important scores) and a dense, low-rank component (many less important scores). This insight explains why neither pure sparse nor pure linear attention works perfectly. SLA proposes a hybrid approach: use the right tool for each component. It applies computationally expensive sparse attention only to the critical scores and uses cheap linear attention to approximate the remaining marginal scores, while ignoring the negligible ones.

Main Contributions / Findings (What)

• A Novel Hybrid Attention Mechanism (SLA): The paper introduces Sparse-Linear Attention (SLA), which dynamically classifies attention blocks into three types (critical, marginal, negligible) and applies a different computational strategy to each, effectively fusing sparse and linear attention.
• High Efficiency without Quality Loss: SLA achieves a 95% reduction in attention computation (a 20x reduction in FLOPs) on a video DiT model while maintaining generation quality comparable to the original full-attention model. This significantly surpasses the efficiency-quality trade-off of previous methods.
• An Efficient, Trainable GPU Kernel: The authors implemented SLA as a single, efficient GPU kernel supporting both forward and backward passes. This is crucial for practical speedups and enables the model to be fine-tuned to adapt to the new attention mechanism.
• Demonstrated End-to-End Acceleration: The custom kernel provides a 13.7x speedup for the attention computation itself and translates into a 2.2x end-to-end speedup for video generation with the Wan2.1-1.3B model, making the attention part of the computation almost negligible.

3. Prerequisite Knowledge & Related Work

Foundational Concepts

• Transformer: A neural network architecture that relies heavily on the self-attention mechanism to process sequential data. It excels at capturing long-range dependencies.
• Diffusion Models: A class of generative models that learn to create data (like images or videos) by reversing a gradual noising process. They start with random noise and iteratively "denoise" it to produce a clean sample.
• Diffusion Transformer (DiT): A specific type of diffusion model that uses a Transformer architecture as its backbone to perform the denoising steps. This is particularly effective for high-resolution generation.
• Self-Attention: The core operation in a Transformer. For a sequence of input tokens, it computes three vectors for each token: a Query (Q), a Key (K), and a Value (V). The attention score between two tokens is calculated from their Q and K vectors. These scores determine how much "attention" each token should pay to every other token when producing its output. The standard computation is: $$\mathrm{Attention}(Q, K, V) = \mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ The $QK^\top$ matrix multiplication is the source of the $O(N^2)$ complexity.
• Sparse Attention: An approximation of self-attention where only a subset of the scores in the $QK^\top$ matrix are computed. The rest are masked out (treated as zero). This reduces computation but can harm performance if important scores are ignored. For GPU efficiency, this is often done at a block level, as in FlashAttention.
• Linear Attention: A family of methods that reformulate the attention calculation to avoid computing the $N \times N$ matrix. They typically introduce a feature map $\phi(\cdot)$ and reorder the operations to achieve $O(N)$ complexity. For example, instead of $(QK^\top)V$, they compute $Q(K^\top V)$. However, this reformulation is an approximation and often struggles to capture the full expressiveness of standard attention.

Differentiation from Related Work

• Versus Sparse Attention (e.g., VSA, VMoBa, SpargeAttn): These methods only distinguish between important and unimportant scores. To maintain quality, they must keep a relatively large fraction of scores, limiting sparsity. SLA goes a step further by introducing a third "marginal" category, which is approximated by linear attention instead of being computed sparsely or dropped entirely. This allows SLA to achieve much higher sparsity (e.g., 95%) while retaining the information from these marginal scores.
• Versus Linear Attention (e.g., SANA, Dig): These methods apply a low-rank approximation to the entire attention matrix. This fails when the matrix has a significant high-rank component, which the paper shows is the case for the most important scores. SLA avoids this pitfall by applying the low-rank (linear) approximation only to the part of the matrix that is already low-rank, while handling the high-rank part exactly.
• Novelty of Fusion: SLA is novel in its principled fusion of sparse and linear attention based on the observed rank properties of different parts of the attention matrix. It's not a simple sum of two independent mechanisms but an integrated system where linear attention serves as a "learnable compensation" for the information lost at very high sparsity levels.

4. Methodology (Core Technology & Implementation)

Principles

The core idea of SLA is based on the empirical observation that an attention matrix $P$ can be decomposed into two distinct parts:

1. A small number of large-valued weights that form a sparse, high-rank matrix.

2. The vast majority of small-valued weights that form a dense, extremely low-rank matrix.

   该图像是图3，展示了Wan2.1模型中注意力权重的分解。完整的注意力权重（秩=6226）被分解为前8%（秩=6230），用于稀疏注意力，以及后92%（秩=9），用于低秩（线性）注意力，揭示了两者在秩上的显著差异，为加速模型提供了依据。

As shown in Figure 3, the top 8% of weights have a rank (6230) comparable to the full matrix (6226), while the remaining 92% of weights have a tiny rank (9). This motivates a hybrid strategy:

• Use sparse attention for the high-rank component.

• Use a low-rank approximation (linear attention) for the low-rank component.

  The formal decomposition is: $$P = \underbrace { P \odot M } _ { \mathrm { sparse\:component } } + \underbrace { P \odot ( 1 - M ) } _ { \mathrm { low-rank\:component } }$$ where $M$ is a binary mask identifying the critical weights. SLA approximates this decomposition.

Steps & Procedures

The SLA pipeline, illustrated in Figure 4, consists of the following steps:

该图像是图4，SLA的示意图。展示注意力权重预测后，分类为关键($O(N^2)$)、边缘($O(N)$)及可忽略(跳过)。右侧详述SLA前向算法，结合稀疏FlashAttention处理关键权重，线性注意力处理边缘权重，最终输出 $O_1 = O^s + Proj(O^l)$。

1. Block-wise Attention Prediction: Instead of computing the full $N \times N$ attention matrix to decide which parts are important, SLA predicts importance at a block level. It first pools the query and key matrices, $Q$ and $K$, along the token dimension to create smaller representations. It then computes a compressed, low-resolution attention map $P_c$: $$P _ { c } = { \mathrm { Softmax } } ( { \mathrm { pool } } ( Q ) { \mathrm { pool } } ( K ) ^ { \top } / { \sqrt { d } } )$$ This $P_c$ serves as a cheap proxy for the full attention matrix.

2. Block Classification: Each block in $P_c$ is classified into one of three categories based on its value, controlled by hyperparameters $k_h$ and $k_l$:

   • Critical (Label 1): The top $k_h\%$ of blocks in each row of $P_c$. These are deemed most important.
   • Negligible (Label -1): The bottom $k_l\%$ of blocks in each row of $P_c$. These are ignored.
   • Marginal (Label 0): All other blocks. These are moderately important. This classification is stored in a compressed mask matrix $M_c$.

3. Computation based on Classification:

   • For Critical blocks ($M_c[i,j] = 1$): Standard sparse attention is computed using a highly optimized kernel like FlashAttention. The output is denoted $O^s$.
   • For Marginal blocks ($M_c[i,j] = 0$): Linear attention is used. The computation is reformulated to avoid the $N \times N$ matrix. The output is denoted $O^l$.
   • For Negligible blocks ($M_c[i,j] = -1$): Computation is skipped entirely.

4. Final Output Fusion: The outputs from the sparse and linear components are combined. A learnable linear projection (Proj) is applied to the linear attention output $O^l$ to help align its distribution with the sparse attention output $O^s$. $$O = O ^ { s } + \mathrm { Proj } ( O ^ { l } )$$ This Proj layer is fine-tuned along with the rest of the model, allowing the model to learn how to best integrate the information from the marginal weights.

Algorithms and Passes

The paper provides detailed pseudocode for the forward and backward passes, which are fused into a single GPU kernel for maximum efficiency.

• Forward Pass (Algorithm 1):

  • It takes Q, K, V and their feature-mapped versions ($Q^\phi, K^\phi$) as input.
  • It pre-computes intermediate values for linear attention ($h_j, z_j$) for all key/value blocks.
  • It computes the compressed mask $M_c$.
  • It iterates through query blocks ($i$) and key blocks ($j$).
    • If $M_c[i,j] = 1$, it performs the online softmax computation for the sparse attention component.
    • If $M_c[i,j] = 0$, it accumulates the pre-computed linear attention intermediates ($h_j, z_j$).
  • Finally, it computes the outputs $O_i^s$ and $O_i^l$ for each query block.

• Backward Pass (Algorithm 2):

  • It takes the gradients from the output ($dO^s, dO^l$) and the saved values from the forward pass.
  • It computes the gradients for both the sparse and linear components separately but within a fused loop structure.
  • For the sparse component, the gradient calculation follows that of FlashAttention.
  • For the linear component, it uses the chain rule to backpropagate gradients through the linear attention formulation.
  • The final gradients for Q, K, and V are the sum of gradients from both components where applicable.

5. Experimental Setup

Datasets

• Video Generation: A private dataset of 20,000 5-second videos at 480p resolution, collected from sources like Pexels and Common Crawl. The model used is Wan2.1-1.3B.
• Image Generation: The standard ImageNet dataset at $512 \times 512$ resolution. The model used is LightningDiT-1.0B.

Evaluation Metrics

• Video Quality:
  • VBench metrics:
    • Imaging Quality (IQ): Measures visual quality aspects like clarity and artifacts.
    • Overall Consistency (OC): Measures temporal consistency across video frames.
    • Aesthetic Quality (AQ): Measures the artistic or aesthetic appeal of the video.
    • Subject Consistency (SC): Measures whether the main subject remains consistent throughout the video.
  • Vision Reward (VR): A metric trained to reflect human preferences for video quality.
  • Aesthetic Video Quality (VA) and Technical Video Quality (VT): Other established metrics for assessing video quality from aesthetic and technical standpoints.
• Image Quality:
  • Fréchet Inception Distance (FID): A standard metric for evaluating the quality of generated images. It measures the distance between the feature distributions of real and generated images. A lower FID score indicates higher quality and better realism. $$\mathrm{FID}(x, g) = \left\| \mu_x - \mu_g \right\|_2^2 + \mathrm{Tr}\left( \Sigma_x + \Sigma_g - 2(\Sigma_x \Sigma_g)^{1/2} \right)$$
    • $\mu_x, \mu_g$: Mean of feature vectors for real ($x$) and generated ($g$) images.
    • $\Sigma_x, \Sigma_g$: Covariance matrices of feature vectors for real and generated images.
    • $\mathrm{Tr}$: The trace of a matrix.
• Efficiency:
  • FLOPs (Floating Point Operations): A measure of the total amount of computation required. Reported in Tera-FLOPs (T). A lower value is better.
  • FLOPS (Floating-Point Operations Per Second): A measure of computational speed or throughput of the GPU kernel. A higher value is better.
  • End-to-end Latency: The total wall-clock time required to generate a sample, measured in seconds. A lower value is better.

Baselines

• Full Attention: The original, unmodified DiT model.
• VSA and VMoBa: State-of-the-art sparse attention methods that require training/fine-tuning.
• Sparge-F (Training-Free) and Sparge-T (Trainable): Implementations of the SpargeAttn method.
• Linear Only: A baseline using only linear attention.
• Sparse Only: An ablation using only the sparse component of SLA (i.e., marginal weights are dropped).
• $L+S$: A naive baseline that directly sums the outputs of independent sparse and linear attention mechanisms.

6. Results & Analysis

Core Results

The main results on video generation with Wan2.1-1.3B are summarized in Table 1.

This is a manual transcription of Table 1 from the paper.

• Analysis: SLA at 95% sparsity achieves video quality scores that are on par with, or even slightly better than, the Full Attention baseline across all metrics. In contrast, all other sparse methods (Sparge-T, VMoBa, VSA) show a significant drop in quality even at lower sparsity levels (84%-89%). In terms of efficiency, SLA reduces computation from 52.75T FLOPs to just 2.74T FLOPs, a 19.3x reduction. This is over twice as efficient as the next best method, VSA.

Efficiency Analysis

The kernel and end-to-end speedups are shown in the bar charts.

该图像是对比不同注意力机制前向与后向核函数速度的柱状图。左侧图显示了前向核函数的FLOPS，其中SLA (95%) 达到2996 FLOPS，相较于FlashAttn (219 FLOPS) 实现了13.7倍的加速。右侧图展示了后向核函数的FLOPS，SLA (95%) 同样以1479 FLOPS表现出最佳性能，优于FlashAttn (218 FLOPS) 和其他基线方法。这表明SLA在扩散模型中的高效性。

该图像是一个水平条形图，展示了不同注意力机制（如VMoBa、VSA和SLA）与原始模型在计算时间上的对比。图中将总时间分为“其他”部分（62秒）和“注意力”部分。SLA（稀疏度=95%）将注意力计算时间从原始模型的97秒大幅减少至11秒，从而实现了2.2倍的端到端加速。

• Kernel Speed (Figure 6): The SLA kernel is exceptionally fast. In the forward pass, it achieves a 13.7x speedup over the highly optimized FlashAttention-2. It is also significantly faster than other sparse attention kernels like VSA and VMoBa at similar sparsity levels.
• End-to-End Latency (Figure 7): The kernel speedup translates to a substantial real-world performance gain. SLA reduces the attention portion of generation time from 97s to just 11s. This leads to a 2.2x total speedup in video generation time (from 159s to 73s).

Ablation Studies

Table 2 presents the ablation results, which validate SLA's design choices.

This is a manual transcription of Table 2 from the paper.

• Fusion Strategy: The Linear Only and Sparse Only baselines perform poorly, confirming that neither method is sufficient on its own. The naive $L+S$ sum is also much worse than SLA. This demonstrates that SLA's integrated, learnable fusion is critical to its success.
• Impact of $k_h$: Varying the percentage of critical blocks ($k_h$) shows a trade-off. Using just the top 5% of blocks ($k_h=5\%$) is enough to match full attention quality while being twice as efficient as using the top 10%.

Visual Examples

该图像是图2，展示了在Wan2.1模型上，使用全注意力、线性注意力、稀疏注意力和SLA四种方法进行视频生成的结果示例。全注意力（稀疏度0%）作为基准，生成了高质量的图像。线性注意力（稀疏度100%）和稀疏注意力（稀疏度90%）的生成效果不佳，图像模糊或充满噪点，并带有红色叉号。相比之下，SLA方法（稀疏度95%）在实现高稀疏度的同时，成功生成了与全注意力质量相当的清晰图像，并带有绿色对勾，表明其在保持生成质量方面表现出色。

该图像是图5，展示了使用SLA和基线方法（如S+L、Sparge-T、Linear Only等）对Wan2.1模型微调后的视频生成效果对比。SLA和Full Attention展示了连贯的视频序列，而其他基线方法因视频质量不足仅显示单帧或质量较差的帧，突出了SLA的优越性。

该图像是图7，展示了Wan2.1模型使用SLA及多种基线方法生成的完整视频示例。视频内容涵盖‘北极熊弹吉他’、‘太平洋海岸海浪’和‘小鸟筑巢’。该图对比了SLA（稀疏度95%）与Full Attention、S+L等方法的生成质量。SLA在实现高稀疏度的同时，能保持与Full Attention相近的视频生成质量。相比之下，Linear Only方法生成效果明显较差。

The visual examples corroborate the quantitative results. Videos generated with SLA are visually indistinguishable from those using full attention. In contrast, other methods produce severe artifacts, noise, or incoherent frames, highlighting the effectiveness of the SLA approach in preserving visual quality.

7. Conclusion & Reflections

Conclusion Summary

The paper introduces SLA, a novel and highly effective attention mechanism for accelerating Diffusion Transformers. By identifying that attention weights can be decomposed into a high-rank sparse part and a low-rank dense part, SLA intelligently applies sparse and linear attention where each is most suitable. This hybrid approach, combined with a custom GPU kernel and a short fine-tuning stage, allows for a massive reduction in computational cost (20x) and significant end-to-end speedups (2.2x) for video generation, all without any discernible loss in quality. SLA represents a significant step forward in making large-scale generative models, especially for video, more efficient and accessible.

Limitations & Future Work

The paper does not explicitly state limitations, but some can be inferred:

• Fine-tuning Requirement: Although the fine-tuning process is short (2000 steps), it is still an extra step compared to training-free inference methods. This adds a small computational overhead and requires access to the training dataset.
• Hyperparameter Sensitivity: The performance of SLA depends on the choice of $k_h$ and $k_l$. While the paper shows good results with a specific setting (5% and 10%), these may need to be tuned for different models or tasks.
• Implementation Complexity: Developing and maintaining a custom fused GPU kernel is complex and may pose challenges for adoption and integration into standard deep learning frameworks compared to methods that build on existing primitives.
• Reliance on Private Data: The primary video experiments are conducted on a private dataset, which limits the direct reproducibility of the main results by the wider research community.

Personal Insights & Critique

• Elegance of the Core Idea: The central insight—decomposing the attention matrix by rank and applying tailored approximations—is both intuitive and powerful. It provides a clear theoretical justification for why a hybrid approach should outperform pure sparse or pure linear methods. This is a strong contribution that could influence future work on efficient attention.
• Practicality and Engineering: The work is not just a theoretical proposal; the authors have clearly invested significant effort in creating a high-performance implementation. The fused kernel and the end-to-end speedup results demonstrate a deep understanding of practical system performance, which is often a missing piece in algorithmic papers.
• Generalizability: The positive results on both video (Wan2.1) and image (LightningDiT) generation suggest that the underlying principle is robust and likely applicable to other domains where Transformers with long sequences are used, such as large language models (LLMs).
• A New Baseline for Efficiency: SLA sets a new, high bar for the trade-off between efficiency and quality in generative Transformers. Future work on attention approximation will likely need to compare against this hybrid sparse-linear approach. The paper convincingly argues that simply pursuing sparsity is not enough; the nature of the approximated values matters.