We introduce DeepSeek-V3.2-Exp, an experimental sparse-attention model, which equips DeepSeek-V3.1-Terminus with DeepSeek Sparse Attention (DSA) through continued training. With DSA, a fine-grained sparse attention mechanism powered by a lightning indexer, DeepSeek-V3.2-Exp achieves significant efficiency improvements in both training and inference, especially in long-context scenarios. The model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp.

1. Bibliographic Information

• Title: DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention
• Authors: DeepSeek-AI
• Journal/Conference: This paper is presented as a technical report and is not formally published in a peer-reviewed journal or conference. It is available as a preprint.
• Publication Year: The paper references works from 2024 and 2025 (preprints), suggesting it was released in 2024.
• Abstract: The authors introduce DeepSeek-V3.2-Exp, an experimental large language model that enhances its predecessor, DeepSeek-V3.1-Terminus, with a new mechanism called DeepSeek Sparse Attention (DSA). DSA is a fine-grained sparse attention method featuring a "lightning indexer" that selects the most relevant tokens for attention calculation. This modification, applied through continued training, leads to substantial efficiency gains in both training and inference, particularly for tasks involving long contexts, without a significant drop in performance.
• Original Source Link: The paper is available at /files/papers/68e088fc18b383404984cb25/paper.pdf, and the model checkpoints are hosted on Hugging Face at https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp. Its status is a technical report or preprint.

2. Executive Summary

• Background & Motivation (Why):
  • Core Problem: The standard attention mechanism in Transformer models, which underpins most modern Large Language Models (LLMs), has a computational complexity of $O(L^2)$, where L is the sequence length. This quadratic scaling makes processing very long contexts (e.g., hundreds of thousands of tokens) prohibitively expensive and slow in terms of memory and computation.
  • Importance: As LLMs are increasingly used for tasks requiring long-context understanding—such as analysing lengthy documents, writing code for large repositories, or maintaining long conversations—the efficiency bottleneck of standard attention has become a critical challenge.
  • Innovation: This paper introduces DeepSeek Sparse Attention (DSA), a method to approximate the full attention mechanism by intelligently selecting a small, fixed-size subset of relevant key-value pairs for each query token to attend to. This reduces the complexity from quadratic to near-linear ($O(L \cdot k)$, where k is the small subset size), dramatically improving efficiency for long sequences.
• Main Contributions / Findings (What):
  • A Novel Sparse Attention Mechanism (DSA): The paper proposes DeepSeek Sparse Attention, which comprises two key components:
    1. A lightning indexer that efficiently calculates relevance scores between tokens.
    2. A fine-grained token selection mechanism that uses these scores to pick the top-k most relevant key-value entries for each query.
  • An Efficient Long-Context Model (DeepSeek-V3.2-Exp): The authors successfully integrated DSA into a powerful existing model (DeepSeek-V3.1-Terminus) through a carefully designed continued training process.
  • Demonstrated Efficiency Gains: The resulting model shows significant reductions in inference costs (both for processing prompts and generating responses) in long-context scenarios, without substantial degradation in performance on a wide range of benchmarks.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:
  • Transformer Architecture: The dominant architecture for LLMs. Its core component is the self-attention mechanism, which allows a model to weigh the importance of different tokens in a sequence when processing a specific token.
  • Self-Attention: For each token (query), the standard mechanism computes a score with every other token (key) in the sequence. These scores are then used to create a weighted sum of all tokens' values. The need to compute scores for all pairs of tokens leads to the $O(L^2)$ complexity.
  • Sparse Attention: A class of techniques designed to overcome the quadratic complexity of self-attention. Instead of attending to all tokens, each query token attends to a smaller, pre-defined or dynamically selected subset of key tokens. This makes the computation much more efficient.
  • Multi-Head Latent Attention (MLA): An attention mechanism introduced in previous DeepSeek models. Instead of using tokens directly as keys and values, it uses a set of "latent vectors" to represent the key-value information, which can be more compressed and efficient. The paper builds DSA on top of MLA.
  • Multi-Query Attention (MQA): An attention variant where all attention heads share a single key and value projection, while each head retains its own query projection. This reduces the memory footprint and speeds up decoding. The paper implements DSA based on the MQA mode of MLA.
  • KL-Divergence (Kullback-Leibler Divergence): A statistical measure of how one probability distribution differs from a reference distribution. In this paper, it is used as a loss function to train the lightning indexer to mimic the attention patterns of the original, dense attention mechanism.
• Differentiation: The proposed DeepSeek Sparse Attention (DSA) is distinct from other sparse attention methods. While many methods rely on fixed patterns (e.g., local windows, strided attention), DSA is fully dynamic and data-dependent. For every query token, it uses a lightweight but powerful "lightning indexer" to compute relevance scores across the entire context and selects the top k most relevant tokens. This allows for a more flexible and potentially more accurate approximation of full attention.

4. Methodology (Core Technology & Implementation)

The core innovation is DeepSeek Sparse Attention (DSA), which is integrated into the DeepSeek-V3.1-Terminus model.

Principles of DSA

DSA operates on a simple but effective principle: instead of calculating costly attention scores between a query and all previous keys, first use a highly efficient proxy—the lightning indexer—to identify the k most promising keys, and then perform the standard attention calculation only on this small subset.

Steps & Procedures

The overall architecture, as shown in Figure 1, can be broken down into these steps for each query token:

1. Lightning Indexer: A lightweight module computes an "index score" between the current query token and all preceding key tokens.

2. Top-k Selector: This component selects the key-value pairs corresponding to the top-k highest index scores.

3. Core Attention: The standard attention mechanism is applied, but only between the query token and the k selected key-value pairs.

   Figure 1: Attention architecture of DeepSeek-V3.2-Exp. This diagram illustrates how the Lightning Indexer computes scores to guide the Top-k Selector, which then feeds a sparse set of key-value entries into the core attention mechanism (instantiated under MLA).

Mathematical Formulas & Key Details

1. The Lightning Indexer

The index score $I_{t,s}$ between a query token at position t and a preceding token at position s is calculated as: $$I _ { t , s } = \sum _ { j = 1 } ^ { H ^ { I } } w _ { t , j } ^ { I } \cdot \mathrm { R e L U } \left( \mathbf { q } _ { t , j } ^ { I } \cdot \mathbf { k } _ { s } ^ { I } \right) ,$$

• $I_{t,s}$: The index score, a scalar value indicating the relevance of token s to token t.

• $\mathbf{h}_t, \mathbf{h}_s$: The hidden states (vectors) for tokens at positions t and s, respectively.

• $H^I$: The number of heads in the indexer, which is kept small for efficiency.

• $\mathbf{q}_{t,j}^I \in \mathbb{R}^{d^I}$: The j-th indexer query vector, derived from the query token's hidden state $\mathbf{h}_t$.

• $\mathbf{k}_s^I \in \mathbb{R}^{d^I}$: The indexer key vector, derived from the preceding token's hidden state $\mathbf{h}_s$.

• $w_{t,j}^I \in \mathbb{R}$: A weight for the j-th indexer head, also derived from $\mathbf{h}_t$.

• $\mathrm{ReLU}$: The Rectified Linear Unit activation function, chosen for its computational efficiency.

  The authors note that the indexer is very fast because it has few heads ($H^I$) and can be implemented using low-precision FP8 arithmetic.

2. Fine-grained Token Selection & Attention

After computing all index scores $I_{t,:}$ for a given query token t, the attention output $\mathbf{u}_t$ is computed as: $$\mathbf { u } _ { t } = \mathrm { A t t n } \big ( \mathbf { h } _ { t } , \big \{ \mathbf { c } _ { s } \big | I _ { t , s } \in \mathrm { T o p } { - } \mathrm { k } \big ( I _ { t , : } \big ) \big \} \big ) .$$

• $\mathrm{Top-k}(I_{t,:})$: This function returns the set of k largest index scores for the query t.
• $\{\mathbf{c}_s\}$: The set of key-value entries. The formula selects only those entries whose corresponding index scores are in the top k.
• $\mathrm{Attn}(\cdot)$: The standard attention function, which now operates on a much smaller set of inputs.

Training Procedure

The model is created by continuing the training of DeepSeek-V3.1-Terminus in two stages:

Stage 1: Dense Warm-up

• Goal: To initialize the lightning indexer so its scoring aligns with the original model's attention patterns.
• Method:
  • The parameters of the main model are frozen. Only the indexer is trained.
  • The model continues to use dense (full) attention.
  • The training objective for the indexer is to minimize the KL-divergence between its output distribution and the main attention head distribution.
  • The loss function is: $$\mathcal { L } ^ { I } = \sum _ { t } \mathbb { D } _ { \mathrm { K L } } \big ( p _ { t , : } \big \| \mathrm { S o f t m a x } \big ( I _ { t , : } \big ) \big ) .$$
  • $p_{t,:}$ is the target distribution, created by summing the attention scores of the main model's heads and normalizing.
• Duration: Very short, only 1000 steps (2.1B tokens).

Stage 2: Sparse Training

• Goal: To adapt the entire model to the sparse attention pattern.
• Method:
  • All model parameters (main model and indexer) are unfrozen and trained.
  • The top-k selection mechanism is activated, making attention sparse. For this model, k is set to 2048.
  • The indexer continues to be trained with a KL-divergence loss, but now the comparison is only done over the set of selected tokens $S_t$.
  • The main model is trained with the standard language modeling loss. The indexer's computation is detached from the main model's computational graph to keep their optimizations separate.
• Duration: 15,000 steps (943.7B tokens).

Post-Training

After pre-training, the model undergoes alignment using the same pipeline as its predecessor, ensuring a fair comparison. This includes Specialist Distillation (training domain-specific models and using them to generate data for the final model) and Mixed RL Training using Group Relative Policy Optimization (GRPO).

5. Experimental Setup

• Datasets: The model's capabilities are evaluated on a comprehensive suite of benchmarks covering various domains:
  • General Knowledge: MMLU-Pro, GPQA-Diamond, Humanity's Last Exam (HLE).
  • Search Agent: BrowseComp, BrowseComp_zh, SimpleQA.
  • Coding: LiveCodeBench, Codeforces, Aider-Polyglot.
  • Code Agent: SWE Verified, SWE-bench Multilingual, Terminal-bench.
  • Math: AIME 2025, HMMT 2025.
• Evaluation Metrics: Standard metrics for each benchmark are used, including Exact Match (EM), Pass@1, Accuracy (Acc.), and Rating.
• Baselines: The primary baseline for comparison is the model's direct predecessor, DeepSeek-V3.1-Terminus, which uses a dense attention mechanism.

6. Results & Analysis

Core Results: Model Capabilities

The performance comparison between DeepSeek-V3.2-Exp (with DSA) and DeepSeek-V3.1-Terminus (dense) is presented in Table 1 of the paper.

Summary of Table 1:

• Overall, DeepSeek-V3.2-Exp demonstrates no substantial performance degradation compared to its dense counterpart across most benchmarks.
• For example, on MMLU-Pro, both models score 85.0. On BrowseComp, the sparse model actually performs slightly better (40.1 vs. 38.5).
• On a few tasks like GPQA and HMMT 2025, the sparse model shows a slight performance dip. The authors attribute this to the model generating fewer reasoning tokens (i.e., being more concise), and note that this gap closes with checkpoints that generate a similar number of tokens. This suggests the performance difference is not necessarily a fundamental limitation of the sparse architecture itself.

Core Results: Training Stability

![](/files/papers/68e088fc18b383404984cb25/images/2.jpg)

![](/files/papers/68e088fc18b383404984cb25/images/3.jpg)
*Figure 2: RL training curves on (a) BrowseComp and (b) SWE Verified. The solid lines (accuracy) for both DeepSeek-V3.1-Terminus (blue) and DeepSeek-V3.2-Exp (orange) show very similar upward trends, indicating that the introduction of DSA did not negatively impact training stability.*

As seen in the figures above, the Reinforcement Learning training curves for both models are closely aligned. Both models show steady improvement in accuracy throughout training, confirming that DSA is a stable architecture that can be effectively trained with advanced techniques like RL.

Core Results: Inference Costs

This is the most significant result of the paper.

Figure 3: Inference costs of DeepSeek-V3.1-Terminus and DeepSeek-V3.2-Exp. The costs are benchmarked on H800 GPUs. Both for prefilling (a) and decoding (b), the cost of the sparse DeepSeek-V3.2-Exp (orange line) grows much more slowly with sequence length than the dense DeepSeek-V3.1-Terminus (blue line).

The charts clearly demonstrate the efficiency benefits of DSA:

• Reduced Complexity: DSA reduces the core attention complexity from $O(L^2)$ to $O(L \cdot k)$, where $k=2048$ is much smaller than the context length $L=128K$. Although the lightning indexer is still $O(L^2)$, its computation is much lighter.
• Practical Speedup: The end-to-end inference cost for DeepSeek-V3.2-Exp is dramatically lower than for the dense model, especially as the token position (context length) increases. At a context of 128K tokens, the cost per million tokens is roughly 3-4 times lower for the sparse model.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces DeepSeek-V3.2-Exp, an experimental model that leverages DeepSeek Sparse Attention (DSA) to significantly improve long-context efficiency. By continuing the training of a strong baseline model with a two-stage process, the authors demonstrate that it is possible to achieve near-linear time complexity for attention without a meaningful sacrifice in performance. The results show massive cost savings for inference on long sequences, marking a promising direction for building more scalable and accessible LLMs.
• Limitations & Future Work: The authors candidly state that while internal evaluations are promising, the model requires further large-scale testing in real-world scenarios to uncover potential limitations of the sparse architecture. The "Exp" (experimental) in the model's name underscores this ongoing validation process.
• Personal Insights & Critique:
  • Pragmatic Innovation: This paper is a strong example of pragmatic engineering. Instead of designing a new model from scratch, the authors cleverly and effectively modified a state-of-the-art existing model. The two-stage training process (warm-up and sparse fine-tuning) is a well-reasoned approach to integrating a new architectural component.
  • Lack of Ablations: As a concise technical report, the paper lacks detailed ablation studies. It would be insightful to see how performance changes with different values of k (the number of selected tokens) or different architectures for the lightning indexer.
  • Comparison Scope: The primary comparison is against its own predecessor. While this is a fair and direct evaluation of DSA's impact, a comparison against other sparse attention models from the literature would have provided broader context on its relative novelty and effectiveness.
  • Future Impact: The success of DSA in a powerful model like DeepSeek provides strong evidence that dynamic, fine-grained sparse attention is a viable path forward for scaling LLMs to even longer contexts. The open-sourcing of the model allows the community to build upon and further validate this approach.