1. Bibliographic Information

• Title: Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Caching
• Authors: Zhixin Zheng, Xinyu Wang, Chang Zou, Shaobo Wang, and Linfeng Zhang. The authors are affiliated with Shanghai Jiao Tong University, University of Electronic Science and Technology of China, and Shandong University. Linfeng Zhang, the corresponding author, appears to lead a research group focused on model efficiency and acceleration.
• Journal/Conference: The paper is submitted to the 33rd ACM International Conference on Multimedia (MM '25). ACM Multimedia is a premier international conference in the field of multimedia, known for its high standards and significant impact. Acceptance at this venue indicates a strong contribution to the field.
• Publication Year: 2025 (as per the submission reference).
• Abstract: The paper addresses the high computational cost of diffusion transformers used for image and video generation. While prior "feature caching" methods accelerate these models by reusing computations from previous timesteps (leveraging temporal similarity), they ignore similarities between different spatial regions (tokens) within the same timestep. The authors introduce Cluster-Driven Feature Caching (ClusCa), a method that performs spatial clustering on tokens. In each step, it computes only one representative token per cluster and propagates its features to the other tokens in the same cluster, reducing the number of computed tokens by over 90%. ClusCa is training-free and can be applied to any diffusion transformer. On the FLUX model, it achieves a 4.96x speedup while improving the ImageReward score, demonstrating its effectiveness.
• Original Source Link:
  • Preprint: https://arxiv.org/abs/2509.10312
  • PDF Link: http://arxiv.org/pdf/2509.10312v1
  • Status: The paper is available as a preprint on arXiv. The provided reference format suggests it is under review or accepted for publication at ACM MM '25.

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: Diffusion Transformers (DiTs), such as those used in state-of-the-art models like Sora and FLUX, produce stunningly high-quality images and videos. However, their iterative denoising process, which requires hundreds of passes through a large neural network, makes them computationally expensive and slow.
  • Existing Gaps: Current acceleration techniques, particularly feature caching, have successfully exploited temporal redundancy—the observation that features don't change much between consecutive denoising steps. These methods cache and reuse features across time. However, they completely overlook spatial redundancy—the fact that many tokens (patches of the image) within the same timestep are highly similar (e.g., different parts of a blue sky). This untapped source of redundancy represents a significant opportunity for further acceleration.
  • Fresh Angle: This paper introduces a novel perspective: what if we could accelerate inference by exploiting both temporal and spatial similarities simultaneously? The core innovation is to cluster similar tokens in the spatial dimension and compute updates for only a small, representative subset of them.

• Main Contributions / Findings (What):

  • Spatial Similarity Analysis: The paper is the first to systematically investigate and demonstrate that tokens in diffusion transformers exhibit strong spatial clustering. Tokens within the same cluster not only have similar features but also follow similar evolutionary paths during the denoising process.
  • ClusCa Method: The paper proposes Cluster-Driven Feature Caching (ClusCa), a training-free algorithm that:
    1. Clusters all image tokens into a small number of groups (e.g., 16).
    2. In most timesteps, it only performs the expensive computation for a single token per cluster.
    3. It then intelligently updates the features of all other tokens by blending their old (cached) features with the new features from their cluster's representative token.
  • State-of-the-Art Performance: ClusCa significantly improves the trade-off between speed and quality. For example, it achieves a 4.96x speedup on the FLUX text-to-image model while actually improving the output quality (ImageReward score of 99.49%). This is particularly effective at high acceleration ratios where previous methods suffer from severe quality degradation. The method's effectiveness is demonstrated on diverse models including DiT, FLUX, and HunyuanVideo.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Diffusion Models: These are generative models that learn to create data, like images, by reversing a noise-adding process.
    • Forward Process: Starts with a clean image and gradually adds Gaussian noise over many timesteps ($T$) until only pure noise remains.
    • Backward Process: A neural network is trained to reverse this process. Starting from random noise, it iteratively removes the predicted noise at each timestep ($t$) to gradually recover a clean image. This iterative denoising is what makes diffusion models slow.
  • Diffusion Transformer (DiT): A powerful architecture for the denoising network in diffusion models. Instead of the commonly used U-Net, a DiT uses a Transformer, which has proven to be highly scalable and effective. It treats an image as a sequence of patches or "tokens," similar to how Transformers process words in a sentence.
  • Feature Caching: An acceleration technique based on the idea of temporal redundancy. Since the input to the denoising network at step $t$ is very similar to the input at step t-1, the intermediate features (activations) inside the network are also very similar. Feature caching methods compute these features once at a specific step, cache them, and then reuse them for the next few steps, skipping the expensive computation.

• Previous Works:

  • Step Reduction: Methods like DDIM and DPM-Solver aim to reduce the total number of denoising steps required, but often at the cost of quality.
  • Temporal Feature Caching:
    • DeepCache first introduced this idea for U-Net based models.
    • FORA, Δ-DiT, ToCa, and DuCa adapted this concept for DiTs. These methods operate on a "cache-then-reuse" cycle. For example, in a cycle of $N$ steps, they perform a full computation at the first step and reuse the cached features for the next N-1 steps.
    • TaylorSeer improved upon this by "forecasting" what the features would be in future steps using a Taylor series expansion, rather than just reusing old ones. This helps reduce the error accumulated from caching.

• Differentiation: ClusCa is fundamentally different from all previous feature caching methods because it introduces a spatial dimension to the caching strategy. While others treat each token independently and focus only on its evolution over time, ClusCa recognizes the relationships between tokens at a single point in time. It is orthogonal and complementary to previous methods. Instead of reusing a token's own past feature (temporal reuse), it also uses features from other spatially similar tokens computed in the current timestep (spatial reuse). This dual-reuse mechanism is the key innovation that allows ClusCa to maintain high quality even at extreme speedups.

4. Methodology (Core Technology & Implementation)

The core idea of ClusCa is to exploit spatial redundancy by grouping similar tokens and only computing a few representatives. The method operates in cycles.

• Principles & Observations: The method is built on two key empirical observations about tokens in DiTs:

  1. Intra-Cluster Similarity (Observation 1): Tokens can be grouped into clusters where members are highly similar to each other. As shown in Figure 2(a), the feature distance between tokens within the same cluster is about 100 times smaller than the average distance between all tokens. Furthermore, Figure 2(b) shows that tokens in the same cluster not only have similar features but also follow similar trajectories during the denoising process.

     Figure 1: Visualization of similarity in two dimensions Figure 2: (a) The distributions of distance between tokens in the same cluster (1/2/3) and all the tokens in the current timestep, showing that tokens in the same cluster have significantly lower distance. (b) PCA visualization on the evolution of tokens in different clusters. Points in the same line denote the same token in different timesteps. Tokens with the same color denote tokens in the same cluster.

  2. Cluster Stability (Observation 2): The way tokens are grouped into clusters remains relatively stable across adjacent timesteps. As shown in Figure 5 from the paper, the clustering structure doesn't change drastically from one step to the next. This means the expensive clustering step doesn't need to be performed at every timestep, making the method efficient.

• Steps & Procedures: ClusCa operates in "cache cycles" of length $N$.

  1. Full Calculation & Clustering (First Timestep of a Cycle):

     • At the first timestep of a cycle (e.g., step $t$), the model performs a full computation, processing all tokens as usual.
     • The output features from the final layer are then fed into a K-Means clustering algorithm. This algorithm groups all tokens into $K$ clusters based on feature similarity. For example, if there are 1024 tokens and $K=16$, each token is assigned to one of 16 clusters.
     • The computed features for all tokens are stored in a cache.

  2. Partial Calculation (Subsequent Timesteps in a Cycle):

     • For the next N-1 timesteps (from t-1 to t-N+1), the model performs a partial computation.
     • Instead of computing all tokens, it first randomly selects one token from each of the $K$ clusters.
     • Only these $K$ representative tokens are passed through the DiT network for computation. This drastically reduces the computational load (e.g., from 1024 tokens to just 16).

  3. Spatiotemporal Feature Reuse & Propagation:

     • This is the key step where all tokens get updated, even those that were not computed.
     • For the $K$ computed tokens, their entries in the cache are updated with their new, freshly computed features.
     • For the vast majority of uncomputed tokens, their features are updated using a weighted average that combines temporal and spatial information.

• Mathematical Formulas & Key Details: The cache update rule for a token $x_i$ is the centerpiece of the method: $$C ( x _ { i } ) = \left\{ \begin{array} { l l } { { \mathcal F } ( x _ { i } ) , } & { i \in { \mathcal { I } } _ { \mathrm { Compute } } } \\ { \gamma \cdot \mu _ { ( i ) } + ( 1 - \gamma ) C ( x _ { i } ) , } & { i \notin { \mathcal { I } } _ { \mathrm { Compute } } } \end{array} \right.$$ Let's break this down:

  • $C(x_i)$: The feature cache for token $x_i$.
  • $\mathcal{F}(x_i)$: The result of passing token $x_i$ through the computationally expensive network layers.
  • $\mathcal{I}_{\mathrm{Compute}}$: The set of indices for the $K$ tokens that were selected for computation.
  • Top Case (Computed Tokens): If token $i$ is in the computed set, its cache is directly updated with the new feature $\mathcal{F}(x_i)$.
  • Bottom Case (Uncomputed Tokens): If token $i$ was not computed, its new cache value is a mix of two parts:
    • $(1-\gamma)C(x_i)$: This is the temporal reuse component. It's the token's own feature from the previous timestep, scaled by $(1-\gamma)$. It provides historical context.
    • $\gamma \cdot \mu_{(i)}$: This is the spatial reuse component. $\mu_{(i)}$ is the average of the newly computed features of the representative token(s) belonging to the same cluster as token $i$. This term propagates the new information from the computed representative to all its cluster-mates.
  • $\gamma$: The propagation ratio, a small hyperparameter that balances the influence of spatial reuse versus temporal reuse. The authors find that a small, non-zero $\gamma$ is optimal. If $\gamma=0$, the method becomes similar to previous temporal-only caching methods. If $\gamma$ is too large, it discards historical information, which can harm quality.
  • $\mu_{(i)}$: The mean feature of computed tokens in the same cluster as token $i$. Formally: $$\mu _ { ( i ) } = \frac { \sum _ { j \in \mathcal { I _ { \mathrm { C o m p u te } } } } \mathcal { F } ( x _ { j } ) } { \sum _ { j \in \mathcal { I _ { \mathrm { C o m p u te } } } } [ \mathbf { I } _ { j } = \mathbf { I } _ { i } ] }$$ Where $\mathbf{I}_i$ is the cluster index for token $i$, and $[\cdot]$ is the Iverson bracket (1 if true, 0 if false). Since only one token is computed per cluster, this effectively becomes the feature of that one representative.

5. Experimental Setup

• Datasets:

  • ImageNet: A large-scale image classification dataset. Used for class-conditional image generation with DiT-XL/2 to generate 50,000 images at $256 \times 256$ resolution.
  • DrawBench: A set of 200 challenging text prompts designed to test the capabilities of text-to-image models. Used with FLUX.
  • VBench: A comprehensive benchmark with 946 prompts for evaluating text-to-video generation models. Used with Hunyuan-Video.

• Evaluation Metrics:

  1. FID (Fréchet Inception Distance):

     • Conceptual Definition: Measures the quality and diversity of generated images. It calculates the "distance" between the feature distribution of generated images and real images. A lower FID score is better, indicating the generated images are more realistic.
     • Mathematical Formula: $$\mathrm{FID}(x, g) = \left\| \mu_x - \mu_g \right\|_2^2 + \mathrm{Tr}\left(\Sigma_x + \Sigma_g - 2(\Sigma_x \Sigma_g)^{1/2}\right)$$
     • Symbol Explanation:
       • $\mu_x, \mu_g$: Mean of the feature vectors for real and generated images, respectively.
       • $\Sigma_x, \Sigma_g$: Covariance matrices of the feature vectors for real and generated images.
       • $\mathrm{Tr}(\cdot)$: The trace of a matrix (sum of diagonal elements).

  2. sFID (spatial FID): A variant of FID that is more sensitive to spatial artifacts and object placement errors. Lower is better.

  3. ImageReward:

     • Conceptual Definition: A metric that uses a trained model to predict human preference for generated images. It scores how "good" an image is based on aesthetics and prompt alignment, mimicking human judgment. Higher is better.

  4. CLIP Score:

     • Conceptual Definition: Measures the semantic similarity between a given text prompt and a generated image. It uses the CLIP model to embed both the text and image into a shared space and calculates their cosine similarity. A higher CLIP score indicates better alignment with the prompt.

  5. VBench Score: A comprehensive score for video generation that aggregates multiple metrics evaluating aspects like video quality, temporal consistency, motion realism, and text-video alignment. Higher is better.

• Baselines: The paper compares ClusCa against several strong baselines:

  • DDIM Step Reduction: Simply using fewer denoising steps. This is the most basic form of acceleration.
  • FORA, ToCa, DuCa, Δ-DiT: State-of-the-art temporal feature caching methods for DiTs.
  • TaylorSeer: An advanced feature forecasting method that represents the previous state-of-the-art in caching-based acceleration.
  • TeaCache: Another relevant caching-based acceleration method.

6. Results & Analysis

• Core Results:

  • Text-to-Image (FLUX): As shown in the transcribed Table 1, ClusCa demonstrates a superior trade-off between speed and quality.

    • At a ~4.14x FLOPs speedup, $ClusCa (N=5, O=2, K=16)$ achieves an ImageReward of 0.9961, significantly outperforming ToCa (0.9802), Δ-DiT (0.8721), and TaylorSeer (0.9857) at similar speedups.

    • Even more impressively, at a 4.96x speedup, ClusCa maintains an extremely high ImageReward of 0.9949, which is better than the original, unaccelerated model (0.9898). Other methods see a significant quality drop at this level of acceleration.

    • The qualitative results in Figure 6 show this clearly. In the "oil painting of a couple" example, only ClusCa correctly generates an umbrella with proper geometric structure at 4.96x acceleration, while other methods produce distorted or incomplete objects.

      Figure 6: Qualitative comparison on FLUX. ClusCa excels in generating complex scenes and produces images with significantly richer content.

  • Text-to-Video (Hunyuan-Video): The transcribed Table 2 shows that ClusCa also excels in video generation.

    • At a 5.54x speedup, ClusCa achieves a VBench score of 79.96%, outperforming TaylorSeer (79.78%) and other caching methods.
    • At an even higher 6.21x speedup, it still maintains a strong score of 79.60%. This is crucial for video, where computational costs are even more prohibitive.

  • Class-to-Image (DiT): Figure 3 provides a clear visual summary of the results from the transcribed Table 3.

    • The plot shows that across all speedup ratios, ClusCa's curve (in dark blue) is consistently lower (better FID/sFID) than all other methods.

    • While other methods like ToCa and DuCa see their FID scores skyrocket beyond a 3.3x speedup, ClusCa's performance degrades much more gracefully, demonstrating its robustness to high acceleration. For instance, at a ~4.6x speedup, ClusCa has an FID of 3.20, whereas FORA is at 6.58.

      Figure 3: Comparison between existing feature caching methods and ClusCa on DiT with FID and sFID (lower is better).

• Ablations / Parameter Sensitivity:

  • Propagation Ratio ($\gamma$): Figure 10 and the transcribed Table 4 show the impact of $\gamma$.

    • The FID score first improves as $\gamma$ increases from 0, then starts to degrade. This confirms the hypothesis that a balance is needed: $\gamma=0$ ignores the beneficial spatial information, while a large $\gamma$ discards too much temporal history.

    • An optimal, small, non-zero $\gamma$ provides the best results, validating the effectiveness of the spatial propagation mechanism.

      Figure 10: ClusCa with different propagation ratios.

  • Number of Clusters ($K$): The transcribed Table 5 shows that increasing $K$ (from 4 to 32) generally improves FID. This is expected, as more clusters mean more representative tokens are computed, and the intra-cluster similarity is higher. However, this comes at the cost of reduced speedup. The authors chose $K=16$ as a good balance for their experiments.

  • Cache Interval ($N$): The transcribed Table 8 shows that as the cache interval $N$ increases (from 3 to 7), the speedup increases, but the FID also increases (quality degrades). However, the paper notes that even with a large interval like $N=7$, ClusCa still outperforms TaylorSeer using a smaller interval of $N=6$, showcasing its robustness.

• Visualization on Feature Trajectories: Figure 9 uses PCA to visualize the evolution of features over time.

  • The "DiT" trajectory represents the ground truth (no acceleration).

  • The trajectories of other methods diverge significantly from the ground truth, indicating error accumulation.

  • The ClusCa trajectory stays remarkably close to the ground truth, providing a clear visual confirmation that its update mechanism successfully minimizes error accumulation while accelerating computation.

    Figure 9: PCA visualization on the trajectories of features in different timesteps. "DiT" here denotes the original model without acceleration, which can be considered as the ground-truth trajectory.

• Time Cost Analysis: Figure 11 shows that the overhead from the new components is minimal.

  • The Clustering step takes up only 2.9%-4.1% of the total time.

  • The Propagation step takes 5.6%-8.0%.

  • The vast majority of the time (around 90%) is still spent on the core network computation, which is exactly what the method aims to reduce. This demonstrates that the management overhead of ClusCa is small.

    Figure 11: Time consumption visualization of each part

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces ClusCa, a novel and highly effective framework for accelerating diffusion transformers. By being the first to systematically exploit spatial similarity in addition to the well-studied temporal similarity, ClusCa sets a new state-of-the-art in the trade-off between inference speed and generation quality. Its core mechanism—clustering tokens, computing a small set of representatives, and propagating their information—is shown to be robust, efficient, and broadly applicable across different models and tasks (image and video generation) without any need for retraining.

• Limitations & Future Work:

  • The authors do not explicitly state limitations, but some can be inferred. The optimal number of clusters ($K$) and the propagation ratio ($\gamma$) are hyperparameters that may need tuning for different models, resolutions, or datasets.
  • The clustering is performed on the features of the last layer. It's possible that clustering at an intermediate layer could provide different or even better results.
  • The selection of a single random token per cluster is simple and effective, but a more intelligent selection strategy (e.g., choosing the token closest to the cluster centroid) could potentially yield further improvements.

• Personal Insights & Critique:

  • Simplicity and Elegance: The core idea of ClusCa is intuitive and powerful. The observation that tokens representing similar visual content (like patches of the sky) can be treated as a group is a fundamental insight that was overlooked by prior work. The implementation via K-Means and a simple weighted average is elegant.

  • High Impact Potential: As diffusion transformers become larger and more ubiquitous (e.g., for video generation), efficient inference becomes paramount. ClusCa provides a practical, training-free solution that can significantly reduce computational barriers, making these powerful models more accessible for real-world applications.

  • Transferability: The concept of exploiting spatial redundancy via clustering is not limited to diffusion models. It could potentially be applied to accelerate other vision transformer models in tasks like video understanding or dense prediction, where there is significant spatial correlation in the input.

  • Open Question: How does the optimal clustering strategy change with the content of the image? For an image with simple, large regions (like a landscape), a small $K$ might suffice. For a highly detailed, complex scene, a larger $K$ might be necessary. An adaptive method that determines $K$ on-the-fly could be a promising direction for future research.

    ---

Transcribed Tables

This section contains manual transcriptions of the tables from the paper, as no image files were provided for them.

Table 1: Quantitative comparison in text-to-image generation for FLUX on Image Reward.

Table 2: Quantitative comparison in text-to-video generation on VBench

Table 3: Quantitative comparison in class-to-image generation on ImageNet with DiT-XL/2.

Table 4: Ablation Study with Different Configurations on ImageNet with DiT-XL/2.

Table 5: Ablation study on number of clusters K.

Table 6: Comparison of additional fidelity scores in video generation (HunyuanVideo).

Table 7: Comparison of VBench motion sub-metrics in video generation (HunyuanVideo).

Table 8: Ablation study on cache cycle length N.