1. Bibliographic Information

1.1. Title

DeepSeek-V3 Technical Report

1.2. Authors

DeepSeek-AI (research@deepseek.com). The appendix lists numerous contributors under "Research & Engineering", "Data Annotation", and "Business & Compliance". Within each role, authors are listed alphabetically by first name.

• Research & Engineering: Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Djian Yang, Deli Chen, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honhuil Ding, Huajian Xin, Huazuo Gao, Hui Qu, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, KKai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Panpan Huuang, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shirong Ma, Shhiy Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Tao Yun, Tian Pei, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, X in Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, XinyuYang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y.K. Li, Y.Q. Wang, Y.. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Ya aohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, YYu Wu, Yuan Ou, Yuduan Wang, Yue Gong, Yuhng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z.F. Wu, Z.Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhan, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhiyu Wu, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, Zizheng Pan.
• Data Annotation: Bei Feng, Hui Li, J.L. Cai, Jiaqi Ni, Lei Xu, Meng Li, Ning Tian, R.J. Chen, R.L. Jin, Ruyi Chen, S.S. Li, Shuang Zhou, Tianyu Sun, X.Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Y.X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Zhen Huang, Zhipeng Xu, Zhongyu Zhang.
• Business & Compliance: Dongjie Ji, Jian Liang, Jin Chen, Leyi Xia, Miaojun Wang, Mingming Li, Peng Zhang, Shaoqing Wu, Shengfeng Ye, T. Wang, W.L. Xiao, Wei An, Xianzu Wang, Xinxia Shan, Ying Tang, Yukun Zha, Yuting Yan, Zhen Zhang.

1.3. Journal/Conference

arXiv preprint. Published on arXiv.org, which is an open-access repository for scholarly articles. It is a prominent platform for rapid dissemination of research, particularly in fields like artificial intelligence, before or in parallel with formal peer review.

1.4. Publication Year

2024

1.5. Abstract

The paper introduces DeepSeek-V3, a Mixture-of-Experts (MoE) language model with 671B total parameters, of which 37B are activated per token. It leverages Multi-head Latent Attention (MLA) and DeepSeekMoE architectures from DeepSeek-V2 for efficient inference and cost-effective training. DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and uses a multi-token prediction objective for enhanced performance. The model is pre-trained on 14.8 trillion diverse tokens, followed by Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). Evaluations show DeepSeek-V3 surpasses other open-source models and rivals leading closed-source models. Its full training, remarkably stable and without loss spikes or rollbacks, cost only 2.788M H800 GPU hours.

1.6. Original Source Link

Official Source: https://arxiv.org/abs/2412.19437 PDF Link: https://arxiv.org/pdf/2412.19437v2.pdf Publication Status: Preprint.

2. Executive Summary

2.1. Background & Motivation

The field of Large Language Models (LLMs) is rapidly advancing towards Artificial General Intelligence (AGI). While closed-source models often lead, open-source models are making significant strides but still face a performance gap. A major challenge in scaling LLMs is the prohibitive cost of training and inference, especially for Mixture-of-Experts (MoE) architectures, which, while parameter-efficient, can introduce complexities like load imbalance and communication overhead.

The core problem DeepSeek-V3 aims to solve is to further push the boundaries of open-source model capabilities by scaling up model size and training data, while simultaneously achieving cost-effective training and efficient inference. The paper seeks to bridge the performance gap between open-source and closed-source models, emphasizing economical costs, stable training, and superior performance, particularly in specialized domains like code and math.

2.2. Main Contributions / Findings

DeepSeek-V3 presents several key contributions:

• Innovative Load Balancing Strategy: It pioneers an auxiliary-loss-free strategy for MoE load balancing, aiming to improve model performance by minimizing the adverse impact typically caused by auxiliary losses designed to ensure expert load distribution. This leads to better expert specialization.
• Multi-Token Prediction (MTP) Objective: Introduces MTP as a training objective to extend prediction scope to multiple future tokens, enhancing training signals and potentially enabling better representation planning, leading to stronger overall performance. It can also be repurposed for speculative decoding.
• Extreme Training Efficiency through Infrastructure Co-Design:
  • FP8 Mixed Precision Training: Validates the feasibility and effectiveness of FP8 training on an extremely large-scale model (671B parameters), achieving accelerated training and reduced GPU memory usage.
  • Advanced Training Framework: Develops DualPipe for efficient pipeline parallelism with fewer bubbles and overlaps computation/communication. Implements efficient cross-node all-to-all communication kernels to overcome MoE communication bottlenecks, achieving near-full computation-communication overlap and near-zero all-to-all communication overhead.
  • Memory Optimization: Achieves significant memory savings, allowing training without costly Tensor Parallelism (TP).
• Knowledge Distillation from DeepSeek-R1: Introduces a novel methodology to distill reasoning capabilities from long Chain-of-Thought (CoT) models (specifically, DeepSeek-R1 series) into standard LLMs, significantly improving reasoning performance while maintaining output style and length control.
• Superior Performance at Economical Cost:
  • Pre-trained on 14.8T tokens with only 2.664M H800 GPU hours. The total training (pre-training, context extension, post-training) required only 2.788M H800 GPU hours, costing approximately $5.576M.
  • DeepSeek-V3-Base is presented as the strongest open-source base model, particularly in code and math.
  • Its chat version DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models like GPT-4o and Claude-3.5-Sonnet across standard and open-ended benchmarks, including breaking the Arena-Hard 85% barrier for open-source models.
  • The training process was remarkably stable, with no irrecoverable loss spikes or rollbacks.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Large Language Models (LLMs): LLMs are neural networks, typically Transformer-based, trained on vast amounts of text data to understand, generate, and process human language. They learn complex patterns and relationships in language, enabling tasks like translation, summarization, question answering, and creative writing.
• Transformer Architecture: Introduced by Vaswani et al. (2017), the Transformer is a neural network architecture that relies heavily on the self-attention mechanism to process sequential data, eschewing traditional recurrent or convolutional layers. It consists of an encoder and decoder stack (though LLMs often use only the decoder part) where each layer typically contains a Multi-Head Attention sub-layer and a Feed-Forward Network (FFN) sub-layer.
• Multi-Head Attention (MHA): A key component of the Transformer, MHA allows the model to jointly attend to information from different representation subspaces at different positions. It computes attention weights multiple times in parallel, each with different learned linear projections to queries (Q), keys (K), and values (V). These multiple "heads" then concatenate their outputs, which are linearly projected once more. The core calculation for a single attention head is: $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ where $Q$ is the query matrix, $K$ is the key matrix, $V$ is the value matrix, and $d_k$ is the dimension of the keys.
• Rotary Positional Embedding (RoPE): A type of positional embedding that encodes absolute position information with a rotation matrix and naturally incorporates relative position dependencies. Unlike absolute positional embeddings that add vectors directly to token embeddings, RoPE rotates the query and key vectors in a way that the dot product naturally incorporates relative position. This allows for better generalization to longer sequence lengths during inference than seen in training.
• Mixture-of-Experts (MoE): An MoE layer replaces the dense Feed-Forward Network (FFN) layer in a Transformer with multiple specialized FFNs (called "experts") and a gating network. For each input token, the gating network decides which experts to route the token to. Typically, only a small number of experts (e.g., 2 or 4) are activated per token, leading to a model with a massive total parameter count but a much smaller number of active parameters per token, enabling computational efficiency during training and inference.
• Pipeline Parallelism (PP): A distributed training strategy where different layers or blocks of a model are placed on different GPUs or devices. A mini-batch of data is split into smaller micro-batches, which are then processed sequentially through the pipeline. This helps reduce memory requirements per device and allows for larger models. The "pipeline bubble" refers to periods when some devices are idle, waiting for data from previous or subsequent stages.
• Expert Parallelism (EP): In MoE models, EP involves distributing the different experts across multiple devices. This allows MoE models to scale to billions or trillions of parameters, as each device only needs to store a subset of the total experts.
• Data Parallelism (DP) & ZeRO-1: Data Parallelism involves replicating the entire model on multiple devices and feeding each device a different subset of the data. Gradients are then aggregated (e.g., averaged) across devices. ZeRO-1 (Zero Redundancy Optimizer Stage 1) is an optimization technique that partitions the optimizer states (e.g., Adam states) across DP ranks, reducing memory footprint compared to traditional DP where each device stores a full copy of the optimizer states.
• Mixed Precision Training (FP8, BF16, FP32): Training neural networks using a combination of different numerical precisions (e.g., FP8 for computations, BF16 for weights, FP32 for master weights/optimizer states). FP8 (8-bit floating point) offers significant memory and speed benefits but requires careful handling of numerical stability. BF16 (bfloat16) is a 16-bit floating point format with a wider dynamic range than FP16 (half-precision float), making it more numerically stable for LLM training. FP32 (single-precision float) is the standard precision.
• Supervised Fine-Tuning (SFT): After pre-training on a large corpus, LLMs are typically fine-tuned on a smaller, high-quality dataset of instruction-response pairs. This aligns the model's behavior with specific instructions and desired output formats.
• Reinforcement Learning (RL) from Human Feedback (RLHF) / Alignment: A technique used to further align LLMs with human preferences and instructions. It involves training a reward model on human preferences (e.g., comparisons of model outputs), which then provides a scalar reward signal to an RL algorithm (like PPO) that optimizes the LLM's policy to maximize these rewards.

3.2. Previous Works

• DeepSeek-V2: The direct predecessor, DeepSeek-V2, validated the core architectural components of Multi-head Latent Attention (MLA) for efficient inference and DeepSeekMoE for economical training. DeepSeek-V3 builds upon these validated architectures, retaining their benefits while introducing further innovations.
• GShard (Lepikhin et al., 2021): A foundational work in MoE architectures, GShard introduced the concept of conditional computation and automatic sharding for scaling LLMs. It often relied on an auxiliary loss to encourage balanced expert usage, a common approach that DeepSeek-V3 seeks to improve upon with its auxiliary-loss-free strategy.
• YaRN (Peng et al., 2023a): YaRN (Yet another RoPE extension) is a technique for extending the context window of LLMs that use Rotary Positional Embeddings (RoPE). DeepSeek-V3 adopts YaRN for its long context extension.
• Speculative Decoding (Leviathan et al., 2023; Xia et al., 2023): A method to accelerate LLM inference by using a smaller, faster "draft" model to predict a sequence of tokens, which are then quickly verified by the larger, more powerful "target" model. DeepSeek-V3's Multi-Token Prediction (MTP) objective can be repurposed for this purpose, indicating its relevance to inference acceleration techniques.
• Open-source LLMs (LLaMA, Qwen, Mistral series): These models represent the state-of-the-art in open-source LLMs. DeepSeek-V3 aims to surpass their performance, as demonstrated in its comparative evaluations.
• Closed-source LLMs (GPT-4o, Claude-3.5-Sonnet): These models are considered the frontier in LLM capabilities. DeepSeek-V3 aims to achieve performance comparable to these models, closing the gap between open and closed-source AI.

3.3. Technological Evolution

The evolution of LLMs has seen a continuous push towards larger models, better performance, and greater efficiency. Initial Transformer models were dense, meaning all parameters were active for every token. MoE architectures, pioneered by works like GShard, emerged as a way to scale total parameters to trillions while keeping active parameters manageable, thus improving computational efficiency. However, MoE introduces challenges like load balancing and communication overhead. Techniques like Multi-Head Latent Attention (MLA) from DeepSeek-V2 addressed KV cache memory issues for efficient inference. DeepSeek-V3 continues this trend by refining MoE (auxiliary-loss-free balancing), improving training signals (MTP), and integrating system-level optimizations (FP8 training, DualPipe for pipeline parallelism, efficient all-to-all communication) to make the training of such massive MoE models both performant and cost-effective. The integration of YaRN addresses long-context capabilities, while distillation techniques (DeepSeek-R1) enhance reasoning. DeepSeek-V3 represents a significant step in making frontier LLM capabilities accessible in the open-source domain by aggressively optimizing across architecture, training methodology, and infrastructure.

3.4. Differentiation Analysis

DeepSeek-V3 differentiates itself from prior LLMs and MoE models through several key innovations:

• Auxiliary-Loss-Free Load Balancing: Unlike traditional MoE models (e.g., GShard) that rely heavily on auxiliary loss to balance expert loads, DeepSeek-V3 pioneers an auxiliary-loss-free strategy. This is a crucial distinction as auxiliary losses can sometimes negatively impact model performance. By using a dynamic bias term for routing, DeepSeek-V3 aims for better load balance without performance degradation and encourages greater expert specialization.
• Multi-Token Prediction (MTP) Training Objective: While MTP has been explored for speculative decoding, DeepSeek-V3 explicitly uses it as a training objective to densify training signals and enable the model to pre-plan representations. Its sequential prediction approach, maintaining a complete causal chain, differs from other MTP implementations that might use independent heads. This is novel for enhancing training effectiveness rather than just inference speed.
• Large-Scale FP8 Training Validation: DeepSeek-V3 is one of the first to successfully validate FP8 mixed precision training on an extremely large-scale model (671B parameters) across 14.8T tokens. This involved custom fine-grained quantization, improved accumulation precision (promoting to CUDA Cores), and specialized low-precision storage/communication, addressing numerical stability challenges that often limit FP8 adoption at scale.
• Comprehensive Infrastructure Co-design: The paper highlights a meticulous co-design of algorithms, frameworks, and hardware. The DualPipe pipeline parallelism algorithm, with its efficient computation-communication overlap and reduced pipeline bubbles, and the highly optimized cross-node all-to-all communication kernels, specifically tailored for IB and NVLink bandwidths, provide a level of engineering depth rarely detailed in LLM papers. These optimizations enable near-zero all-to-all communication overhead even with fine-grained experts across nodes.
• Knowledge Distillation from DeepSeek-R1 for Reasoning: The innovative methodology to distill reasoning capabilities from specialized long Chain-of-Thought (CoT) models (DeepSeek-R1) into a general LLM (DeepSeek-V3) is a unique post-training contribution. This allows DeepSeek-V3 to gain advanced reasoning skills without becoming a specialized CoT model itself, leading to significant performance boosts in math and code.
• Cost-Effectiveness at Scale: Despite its massive parameter count and strong performance, DeepSeek-V3 emphasizes its economical training costs (2.788M H800 GPU hours total). This positions it as a highly efficient option compared to other open-source models of similar or even smaller activated parameter sizes.

4. Methodology

4.1. Principles

The core principles underpinning DeepSeek-V3's methodology are:

1. Efficiency through Sparsity and Low-Rank Approximation: Leveraging Mixture-of-Experts (MoE) for economical training by activating only a subset of parameters per token, and Multi-head Latent Attention (MLA) for efficient inference by reducing Key-Value (KV) cache memory.
2. Performance Enhancement via Training Objectives: Introducing a Multi-Token Prediction (MTP) objective to provide denser training signals and encourage better internal representations, leading to stronger downstream performance.
3. Load Balancing without Performance Degradation: Pioneering an auxiliary-loss-free strategy for MoE to maintain balanced expert utilization without impairing model performance, thereby fostering expert specialization.
4. System-Algorithm Co-Design for Scalability and Stability: Meticulously co-designing FP8 mixed precision training with DualPipe pipeline parallelism and optimized cross-node communication kernels to overcome bottlenecks, reduce memory footprint, accelerate training, and ensure training stability even at extreme scales.
5. Targeted Capability Enhancement through Post-Training: Employing Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) with innovative knowledge distillation from specialized reasoning models (DeepSeek-R1) to unlock and align the model's full potential, particularly in reasoning-heavy tasks.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Architecture

The basic architecture of DeepSeek-V3 maintains the Transformer framework and builds upon the efficiency innovations from DeepSeek-V2: Multi-head Latent Attention (MLA) for inference efficiency and DeepSeekMoE for training economy.

4.2.1.1. Multi-Head Latent Attention

For the attention mechanism, DeepSeek-V3 adopts MLA. The core idea of MLA is to use low-rank joint compression for attention keys (K) and values (V) to significantly reduce the Key-Value (KV) cache during inference, which is a major memory bottleneck for long contexts.

Given attention input $\mathbf{h}_t \in \mathbb{R}^d$ for the $t$-th token at a given attention layer, where $d$ is the embedding dimension, $n_h$ is the number of attention heads, and $d_h$ is the dimension per head:

First, keys and values are compressed into a latent vector $\mathbf{c}_t^{KV}$: $ \left[ \mathbf{c}_t^{KV} \right] = W^{DKV} \mathbf{h}_t $ Here, $\mathbf{c}_t^{KV} \in \mathbb{R}^{d_c}$ is the compressed latent vector for keys and values. $d_c$ is the KV compression dimension, where $d_c \ll d_h n_h$. $W^{DKV} \in \mathbb{R}^{d_c \times d}$ is the down-projection matrix.

Next, this compressed latent vector is up-projected to form the compressed keys $\mathbf{k}_t^C$ and compressed values $\mathbf{v}_t^C$: $ [ \mathbf{k}{t,1}^C ; \mathbf{k}{t,2}^C ; ... ; \mathbf{k}{t,n_h}^C ] = \mathbf{k}t^C = W^{UK} \mathbf{c}t^{KV} $ $ [ \mathbf{v}{t,1}^C ; \mathbf{v}{t,2}^C ; ... ; \mathbf{v}{t,n_h}^C ] = \mathbf{v}_t^C = W^{UV} \mathbf{c}_t^{KV} $ Here, $W^{UK}$ and $W^{UV} \in \mathbb{R}^{d_h n_h \times d_c}$ are the up-projection matrices for keys and values, respectively. The notation $[ \cdot ; \cdot ]$ denotes concatenation.

A decoupled key $\mathbf{k}_t^R$ is generated, which carries Rotary Positional Embedding (RoPE): $ \left[ \mathbf{k}_t^R \right] = \mathrm{RoPE}(W^{KR} \mathbf{h}_t) $ $W^{KR} \in \mathbb{R}^{d_h^R \times d}$ is the matrix used to produce this decoupled key, and $\mathrm{RoPE}(\cdot)$ applies the Rotary Positional Embedding. $d_h^R$ is the dimension of the decoupled key.

The final key for each head $i$ is formed by concatenating the compressed key and the decoupled key: $ \mathbf{k}{t,i} = [ \mathbf{k}{t,i}^C ; \mathbf{k}_t^R ] $ During generation (inference), only the compressed latent vector $\mathbf{c}_t^{KV}$ and the decoupled key $\mathbf{k}_t^R$ need to be cached, significantly reducing KV cache size.

For attention queries, a similar low-rank compression is performed to reduce activation memory during training: $ \mathbf{c}t^Q = W^{DQ} \mathbf{h}t $ $ [ \mathbf{q}{t,1}^C ; \mathbf{q}{t,2}^C ; ... ; \mathbf{q}{t,n_h}^C ] = \mathbf{q}t^C = W^{UQ} \mathbf{c}t^Q $ $ [ \mathbf{q}{t,1}^R ; \mathbf{q}{t,2}^R ; ... ; \mathbf{q}{t,n_h}^R ] = \mathbf{q}t^R = \mathrm{RoPE}(W^{QR} \mathbf{c}t^Q) $ $ \mathbf{q}{t,i} = [ \mathbf{q}{t,i}^C ; \mathbf{q}_{t,i}^R ] $ Here, $\mathbf{c}_t^Q \in \mathbb{R}^{d_c'}$ is the compressed latent vector for queries, $d_c' \ll d_h n_h$ is the query compression dimension. $W^{DQ} \in \mathbb{R}^{d_c' \times d}$ and $W^{UQ} \in \mathbb{R}^{d_h n_h \times d_c'}$ are the down-projection and up-projection matrices for queries. $W^{QR} \in \mathbb{R}^{d_h^R n_h \times d_c'}$ is for producing decoupled queries that carry RoPE.

Finally, the attention queries $\mathbf{q}_{t,i}$, keys $\mathbf{k}_{j,i}$, and values $\mathbf{v}_{j,i}^C$ are combined to produce the attention output $\mathbf{u}_t$: $ \mathbf{o}{t,i} = \displaystyle \sum{j=1}^t \mathrm{Softmax}j \left( \frac{\mathbf{q}{t,i}^T \mathbf{k}{j,i}}{\sqrt{d_h + d_h^R}} \right) \mathbf{v}{j,i}^C $ $ \mathbf{u}t = W^O [ \mathbf{o}{t,1} ; \mathbf{o}{t,2} ; ... ; \mathbf{o}{t,n_h} ] $ Here, $\mathbf{o}_{t,i}$ is the output of head $i$, normalized by $\sqrt{d_h + d_h^R}$. $W^O \in \mathbb{R}^{d \times d_h n_h}$ is the output projection matrix.

4.2.1.2. DeepSeekMoE with Auxiliary-Loss-Free Load Balancing

For Feed-Forward Networks (FFNs), DeepSeek-V3 employs the DeepSeekMoE architecture, which uses finer-grained experts and isolates some as shared.

Given the FFN input $\mathbf{u}_t$ of the $t$-th token, the FFN output $\mathbf{h}_t'$ is computed as: $ \mathbf{h}_t' = \mathbf{u}t + \sum{i=1}^{N_s} \mathrm{FFN}i^{(s)}(\mathbf{u}t) + \sum{i=1}^{N_r} g{i,t} \mathrm{FFN}_i^{(r)}(\mathbf{u}_t) $ Here, $N_s$ and $N_r$ are the numbers of shared experts and routed experts, respectively. $\mathrm{FFN}_i^{(s)}(\cdot)$ and $\mathrm{FFN}_i^{(r)}(\cdot)$ denote the $i$-th shared and routed expert FFN, respectively. $g_{i,t}$ is the gating value for the $i$-th expert.

The gating values $g_{i,t}$ are derived from token-to-expert affinity scores $s_{i,t}$: $ g_{i,t} = \frac{g_{i,t}'}{\sum_{j=1}^{N_r} g_{j,t}'} $ where $ g_{i,t}' = \left{ \begin{array}{ll} s_{i,t} , & s_{i,t} \in \mathrm{Topk}({s_{j,t}}{1 \le j \le N_r}, K_r) \ 0 , & \mathrm{otherwise} \end{array} \right. $ and $ s{i,t} = \mathrm{Sigmoid}(\mathbf{u}_t' \mathbf{e}_i) $ $K_r$ denotes the number of activated routed experts. $\mathbf{e}_i$ is the centroid vector of the $i$-th routed expert. $\mathrm{Topk}(\cdot, K)$ returns the set of $K$ highest scores. DeepSeek-V3 uses the sigmoid function to compute affinity scores and normalizes selected scores to produce gating values.

Auxiliary-Loss-Free Load Balancing: To prevent routing collapse and computational inefficiency from unbalanced expert loads without negatively impacting performance, DeepSeek-V3 introduces an auxiliary-loss-free load balancing strategy. A bias term $b_i$ is introduced for each expert and added to its affinity score $s_{i,t}$ to determine top-K routing: $ g_{i,t}' = \left{ \begin{array}{ll} s_{i,t} , & s_{i,t} + b_i \in \mathrm{Topk}({s_{j,t} + b_j | 1 \le j \le N_r}, K_r) \ 0 , & \mathrm{otherwise} \end{array} \right. $ Crucially, $b_i$ is only used for routing decisions, not for the final gating value calculation, which still uses the original $s_{i,t}$. During training, $b_i$ is dynamically adjusted: decreased by $\gamma$ if the expert is overloaded, increased by $\gamma$ if underloaded. $\gamma$ is the bias update speed.

Complementary Sequence-Wise Auxiliary Loss: While primarily auxiliary-loss-free, a small complementary sequence-wise balance loss $\mathcal{L}_{\mathrm{Bal}}$ is used to prevent extreme imbalance within any single sequence: $ \mathcal{L}{\mathrm{Bal}} = \alpha \sum{i=1}^{N_r} f_i P_i $ where $ f_i = \frac{N_r}{K_r T} \sum_{t=1}^T \mathbf{1}(s_{i,t} \in \mathrm{Topk}({s_{j,t} | 1 \le j \le N_r}, K_r)) $ $ s_{i,t}' = \frac{s_{i,t}}{\sum_{j=1}^{N_r} s_{j,t}} $ $ P_i = \frac{1}{T} \sum_{t=1}^T s_{i,t}' $ Here, $\alpha$ is a small hyper-parameter, $\mathbf{1}(\cdot)$ is the indicator function, and $T$ is the number of tokens in a sequence. This loss encourages balanced expert load within each sequence.

Node-Limited Routing: Similar to DeepSeek-V2's device-limited routing, DeepSeek-V3 restricts each token to be sent to at most $M$ nodes. These nodes are selected based on the sum of the highest $K_r/M$ affinity scores of experts distributed on each node. This limits communication costs and enables near-full computation-communication overlap.

No Token-Dropping: Due to effective load balancing strategies in both training and inference, DeepSeek-V3 does not drop tokens.

The following figure (Figure 2 from the original paper) illustrates the basic architecture of DeepSeek-V3:

该图像是DeepSeek-V3的基本架构示意图。图中展示了DeepSeekMoE和多头潜在注意力（MLA）的结构，强调了在高效推理和经济训练中的应用。

4.2.1.3. Multi-Token Prediction (MTP)

Inspired by Gloeckle et al. (2024), DeepSeek-V3 uses a Multi-Token Prediction (MTP) objective during training. Instead of predicting only the immediate next token, it predicts multiple future tokens at each position. This densifies training signals and allows the model to "pre-plan" representations.

The MTP implementation uses $D$ sequential modules to predict $D$ additional tokens. The $k$-th MTP module consists of:

• A shared embedding layer $\mathrm{Emb}(\cdot)$.

• A shared output head $\mathrm{OutHead}(\cdot)$.

• A Transformer block $\mathrm{TRM}_k(\cdot)$.

• A projection matrix $M_k \in \mathbb{R}^{d \times 2d}$.

  For the $i$-th input token $t_i$, at the $k$-th prediction depth: First, a combined representation $\mathbf{h}_i^{\prime k}$ is formed by linearly projecting the representation of the $i$-th token at depth (k-1) and the embedding of the $(i+k)$-th token: $ \mathbf{h}_i^{\prime k} = M_k [ \mathrm{RMSNorm}(\mathbf{h}i^{k-1}) ; \mathrm{RMSNorm}(\mathrm{Emb}(t{i+k})) ] $ Here, $[ \cdot ; \cdot ]$ denotes concatenation. RMSNorm is the Root Mean Square Normalization. When $k=1$, $\mathbf{h}_i^{k-1}$ refers to the representation from the main model. The embedding layer Emb is shared with the main model.

This combined representation then serves as input to the Transformer block $\mathrm{TRM}_k(\cdot)$ to produce the output representation $\mathbf{h}_i^k$ at the current depth: $ \mathbf{h}_{1:T-k}^k = \mathrm{TRM}k ( \mathbf{h}{1:T-k}^{\prime k} ) $ Here, $T$ is the input sequence length, and i:j denotes slicing.

Finally, the shared output head $\mathrm{OutHead}(\cdot)$ uses $\mathbf{h}_i^k$ to predict the $(i+k+1)$-th token, producing prediction probabilities $P_{i+k+1}^k \in \mathbb{R}^V$: $ P_{i+k+1}^k = \mathrm{OutHead}(\mathbf{h}_i^k) $ $V$ is the vocabulary size. The output head is shared with the main model. The OutHead maps representations to logits and then applies Softmax to get probabilities.

The MTP loss for the $k$-th module, $\mathcal{L}_{\mathrm{MTP}}^k$, is a CrossEntropy loss: $ \mathcal{L}{\mathrm{MTP}}^k = \mathrm{CrossEntropy}(P{2+k:T+1}^k, t_{2+k:T+1}) = - \frac{1}{T} \sum_{i=2+k}^{T+1} \log P_i^k [ t_i ] $ Here, $t_i$ is the ground-truth token at position $i$, and $P_i^k[t_i]$ is its predicted probability by the $k$-th MTP module.

The overall MTP loss $\mathcal{L}_{\mathrm{MTP}}$ is the average of individual MTP losses across depths, weighted by $\lambda$: $ \mathcal{L}{\mathrm{MTP}} = \frac{\lambda}{D} \sum{k=1}^D \mathcal{L}_{\mathrm{MTP}}^k $

MTP in Inference: During inference, the MTP modules can be discarded, allowing the main model to function independently. Alternatively, these modules can be repurposed for speculative decoding to accelerate generation.

The following figure (Figure 3 from the original paper) illustrates the MTP implementation:

该图像是示意图，展示了多token预测（MTP）实现的结构。图中描述了主模型与两个MTP模块的关系，以及输入tokens和目标tokens的交互关系，包括交叉熵损失的计算流程。

4.2.2. Infrastructures

4.2.2.1. Compute Clusters

DeepSeek-V3 was trained on a cluster with 2048 NVIDIA H800 GPUs. Each H800 node has 8 GPUs connected by NVLink and NVSwitch. Inter-node communication uses InfiniBand (IB).

4.2.2.2. Training Framework

The HAI-LLM framework supports DeepSeek-V3's training, utilizing 16-way Pipeline Parallelism (PP), 64-way Expert Parallelism (EP) across 8 nodes, and ZeRO-1 Data Parallelism (DP).

DualPipe and Computation-Communication Overlap: DualPipe is an innovative pipeline parallelism algorithm designed to:

1. Reduce pipeline bubbles (idle time).

2. Overlap computation and communication phases, especially crucial for heavy cross-node communication in MoE.

   DualPipe achieves this by overlapping the forward and backward chunks. Each chunk is divided into attention, all-to-all dispatch, MLP, and all-to-all combine. For backward, attention and MLP are further split into backward for input and backward for weights. A PP communication component is also present. By rearranging these components and adjusting GPU Streaming Multiprocessors (SMs) ratio for communication vs. computation, both all-to-all and PP communication can be fully hidden. This overlap ensures that all-to-all communication overhead remains near-zero even as the model scales up.

The following figure (Figure 4 from the original paper) illustrates the overlapping strategy for a pair of individual forward and backward chunks:

该图像是图示，展示了前向和后向计算块的重叠策略。图中橙色表示前向计算，绿色表示输入的后向计算，蓝色表示权重的后向计算，紫色表示PP通信，红色表示障碍。这些计算和通信操作在时间上被有效重叠。

The full DualPipe scheduling employs a bidirectional pipeline scheduling, feeding micro-batches from both ends simultaneously, further overlapping communications.

The following figure (Figure 5 from the original paper) shows an example DualPipe scheduling:

该图像是图表，展示了8个PP排名和20个微批次在两个方向上的DualPipe调度。每个设备的前向和反向计算过程以不同颜色表示，图中还突出显示了重叠的前向与反向计算。此图清晰展示了时间进度中的并行操作。

The following are the results from Table 2 of the original paper, comparing pipeline bubbles and memory usage across different pipeline parallel methods:

$F$ denotes the execution time of a forward chunk, $B$ denotes the execution time of a full backward chunk, $W$ denotes the execution time of a "backward for weights" chunk, and F&B denotes the execution time of two mutually overlapped forward and backward chunks. DualPipe significantly reduces pipeline bubbles compared to ZB1P and 1F1B, with a minor increase in peak activation memory (by $\frac{1}{\text{PP}}$ times) and requiring two copies of model parameters.

Efficient Implementation of Cross-Node All-to-All Communication: Custom cross-node all-to-all communication kernels (dispatching and combining) were developed. The implementation is co-designed with the MoE gating algorithm and cluster network topology (IB for cross-node, NVLink for intra-node).

• Each token is dispatched to at most 4 nodes to reduce IB traffic (IB bandwidth is ~1/3.2 of NVLink).
• Tokens are first transmitted via IB to GPUs with the same in-node index on target nodes.
• Upon reaching target nodes, they are instantly forwarded via NVLink to specific GPUs hosting target experts, overlapping IB and NVLink communications.
• This allows efficient selection of up to 13 experts (4 nodes * 3.2 experts/node) without additional communication overhead.
• Only 20 SMs are sufficient to fully utilize IB and NVLink bandwidths.
• Warp specialization technique partitions SMs into communication channels, with dynamic warp allocation for IB sending, IB-to-NVLink forwarding, NVLink receiving (dispatching), and NVLink sending, NVLink-to-IB forwarding/accumulation, IB receiving/accumulation (combining).
• Custom PTX (Parallel Thread Execution) instructions and auto-tuned communication chunk size reduce L2 cache usage and interference with other SMs.

Extremely Memory Saving with Minimal Overhead:

• Recomputation of RMSNorm and MLA Up-Projection: RMSNorm operations and MLA up-projections are recomputed during back-propagation instead of persistently storing their output activations, significantly reducing memory.
• Exponential Moving Average (EMA) in CPU: EMA of model parameters is stored in CPU memory and updated asynchronously, avoiding GPU memory or time overhead.
• Shared Embedding and Output Head for Multi-Token Prediction: By deploying the shallowest (embedding) and deepest (output head) layers on the same PP rank, physical sharing of parameters and gradients between MTP modules and the main model is enabled, enhancing memory efficiency.

4.2.2.3. FP8 Training

DeepSeek-V3 uses a fine-grained mixed precision framework with FP8 data format for training.

The following figure (Figure 6 from the original paper) illustrates the overall mixed precision framework with FP8 data format:

Mixed Precision Framework:

• FP8 for Core Computations: Most compute-intensive operations, particularly GEMM (General Matrix Multiplication) operations, are conducted in FP8. This includes Fprop (forward pass), Dgrad (activation backward pass), and Wgrad (weight backward pass). FP8 inputs yield BF16 or FP32 outputs, theoretically doubling computational speed and reducing memory (as activations can be stored in FP8 for Wgrad).

• Higher Precision for Sensitive Operations: BF16 or FP32 is retained for:

  • Embedding module
  • Output head
  • MoE gating modules
  • Normalization operators
  • Attention operators This balances efficiency with numerical stability.

• High-Precision Storage: Master weights, weight gradients, and optimizer states are stored in higher precision to ensure numerical stability, with memory overhead minimized by efficient sharding across DP ranks.

  Improved Precision from Quantization and Multiplication: Strategies to enhance FP8 training accuracy.

The following figure (Figure 7 from the original paper) illustrates fine-grained quantization and improved FP8 GEMM precision:

该图像是图表，展示了细粒度量化方法以缓解特征异常导致的量化误差，以及提升 FP8 GEMM 精度的方法。图中包括了输入、权重和输出的表示，以及通过 $N_{C}=128$ 元素的 MMA 来提高积累精度的策略。

• Fine-Grained Quantization: Addresses outliers that can degrade FP8 quantization accuracy.

  • For activations: Grouping and scaling elements on a 1x128 tile basis (per token per 128 channels).
  • For weights: Grouping and scaling elements on a 128x128 block basis (per 128 input channels per 128 output channels). This granular scaling adapts better to outliers. Per-group scaling factors are introduced along the inner dimension of GEMM operations, supported by FP32 accumulation.

• Increasing Accumulation Precision: Addresses underflow issues and limited accumulation precision of FP8 GEMM on NVIDIA H800 GPUs (around 14 bits, lower than FP32).

  • Promotion to CUDA Cores: During MMA (Matrix Multiply-Accumulate) execution on Tensor Cores, intermediate results are accumulated with limited bit width. Once an interval of $N_C$ elements (e.g., $N_C = 128$) is reached, partial results are copied to FP32 registers on CUDA Cores for full FP32 accumulation.
  • Scaling factors from fine-grained quantization are efficiently multiplied on CUDA Cores during dequantization. This process overlaps MMA and promotion operations on H800, maintaining Tensor Core utilization.

• Mantissa over Exponents: DeepSeek-V3 adopts the E4M3 format (4-bit exponent, 3-bit mantissa) for all tensors in FP8, unlike hybrid formats (E4M3 for Fprop, E5M2 for Dgrad/Wgrad) used in prior work. This is feasible due to fine-grained quantization, which effectively shares exponent bits among grouped elements, mitigating dynamic range limitations.

• Online Quantization: Instead of delayed quantization (inferring scale from past iterations), DeepSeek-V3 calculates the maximum absolute value online for each activation tile ($\pm 128$) or weight block ($128 \times 128$) to derive scaling factors and quantize on the fly.

  Low-Precision Storage and Communication: Further reduces memory and communication overhead.

• Low-Precision Optimizer States: BF16 is used for AdamW optimizer's first and second moments instead of FP32, without observable performance degradation. Master weights and gradients (for batch size accumulation) remain in FP32 for numerical stability.

• Low-Precision Activation: Activations are cached in FP8 for the backward pass of Linear operators.

  • Special considerations: Inputs of Linear after attention use a customized E5M6 format and round-scaled (integral power of 2) scaling factors for 1x128 to 128x1 quantization during backward.
  • Inputs of SwiGLU operator in MoE are cached in FP8 with fine-grained quantization.

• Low-Precision Communication: Activations before MoE up-projections are quantized to FP8 (with round-scaled factors) for dispatch. Similarly for activation gradients before MoE down-projections. Forward and backward combine components retain BF16 for critical precision.

4.2.3. Inference and Deployment

Deployment for DeepSeek-V3 is on the H800 cluster, with NVLink intra-node and IB inter-node. Prefilling and decoding stages are separated for Service-Level Objective (SLO) and high throughput.

Prefilling:

• Minimum unit: 4 nodes with 32 GPUs.
• Attention: 4-way Tensor Parallelism (TP4) with Sequence Parallelism (SP), combined with 8-way Data Parallelism (DP8). Small TP size limits communication overhead.
• MoE: 32-way Expert Parallelism (EP32), ensuring large batch size per expert for efficiency.
• MoE all-to-all communication: Same as training; IB cross-node, NVLink intra-node.
• Dense MLPs in shallow layers: 1-way Tensor Parallelism to save TP communication.
• Load Balancing for Experts: Redundant experts are deployed (duplicating high-load experts) based on online statistics, rearranged within nodes to balance load without increasing cross-node communication. 32 redundant experts for prefilling.
• Throughput Optimization: Two micro-batches with similar computational workloads are processed simultaneously, overlapping attention and MoE of one with dispatch and combine of another.
• Future Exploration: Dynamic redundancy strategy for experts.

Decoding:

• Minimum unit: 40 nodes with 320 GPUs.
• Shared expert is treated as a routed expert, so 9 experts are selected per token.
• Attention: TP4 with SP, combined with DP80.
• MoE: EP320. Each GPU hosts one expert, with 64 GPUs for redundant and shared experts.
• MoE all-to-all communication: Direct point-to-point transfers over IB for low latency, leveraging IBGDA (InfiniBand GPUDirect Async) technology.
• Load Balancing for Experts: Redundant experts are periodically determined based on statistical load.
• Future Exploration: Dynamic redundancy for decoding, processing two micro-batches simultaneously (overlapping attention of one with $dispatch+MoE+combine$ of another, as attention is a larger bottleneck in decoding).

4.2.4. Suggestions on Hardware Design

Based on their experience, the authors offer suggestions for future AI hardware:

• Communication Hardware:
  • Offload all-to-all communication tasks (data forwarding between IB/NVLink, data transport between RDMA buffers and input/output buffers, reduce operations, fine-grained memory management) from SMs to a dedicated GPU co-processor or network co-processor.
  • Unify IB (scale-out) and NVLink (scale-up) networks from the perspective of computation units, providing simple primitives for communication requests.
• Compute Hardware:
  • Higher FP8 GEMM Accumulation Precision in Tensor Cores: Address the H800's limitation of 14-bit accumulation precision in FP8 GEMM. Future chips need to adopt higher precision natively.
  • Support for Tile- and Block-Wise Quantization: Integrate native support for fine-grained quantization within Tensor Cores, allowing them to receive scaling factors and implement MMA with group scaling to avoid frequent data movements between Tensor Cores and CUDA Cores.
  • Support for Online Quantization: Fuse FP8 cast and TMA (Tensor Memory Accelerator) access into a single operation, completing quantization during data transfer from HBM to shared memory. Also, support warp-level cast instructions or near-memory computing.
  • Support for Transposed GEMM Operations: Enable direct transposed reads of matrices from shared memory before MMA operations, fusing FP8 format conversion and TMA access to streamline quantization workflow.

5. Experimental Setup

5.1. Datasets

DeepSeek-V3's pre-training and post-training stages involve diverse datasets.

5.1.1. Pre-Training Data

The pre-training corpus for DeepSeek-V3 consists of 14.8 trillion high-quality and diverse tokens in their custom tokenizer.

• Composition: Enhanced ratio of mathematical and programming samples, expanded multilingual coverage beyond English and Chinese.
• Processing: Refined pipeline to minimize redundancy while maintaining corpus diversity.
• Document Packing: Implemented document packing for data integrity but without cross-sample attention masking.
• Fill-in-Middle (FIM) Strategy: Incorporated at a rate of 0.1 using the Prefix-Suffix-Middle (PSM) framework, structuring data as: <|fim_begin|>fpre<|fim_hole|>fsuf<|fim_end|>fmiddle<|eos_token|> This strategy helps the model predict middle text based on context.
• Tokenizer: Uses Byte-level BPE with an extended vocabulary of 128K tokens. Pretokenizer and training data were modified for multilingual compression efficiency. Addresses token boundary bias by randomly splitting combined punctuation/line break tokens during training.

5.1.2. Post-Training Data (Supervised Fine-Tuning)

The instruction-tuning dataset for SFT comprises 1.5 million instances across multiple domains, created with tailored methods.

• Reasoning Data:
  • Generated using an internal DeepSeek-R1 model.
  • Methodology: Expert models (for code, math, general reasoning) are trained via SFT and RL and serve as data generators.
  • Two types of SFT samples generated: $<problem, original response>$ and $<system prompt, problem, R1 response>$. The system prompt guides R1 to produce responses with reflection and verification mechanisms.
  • During RL, the model samples high-temperature outputs integrating R1 patterns.
  • Rejection sampling is used to curate high-quality SFT data for the final model, retaining DeepSeek-R1's strengths while ensuring conciseness.
• Non-Reasoning Data: (e.g., creative writing, role-play, simple QA)
  • Generated using DeepSeek-V2.5.
  • Human annotators verify accuracy and correctness.

5.2. Evaluation Metrics

For DeepSeek-V3, a comprehensive set of metrics is used, tailored to different tasks.

• Accuracy / Exact Match (EM):
  • Conceptual Definition: Measures the percentage of predictions that exactly match the ground truth. It's a strict metric often used in multiple-choice questions, factoid question answering, and math problems where a precise answer is expected.
  • Mathematical Formula: $ \mathrm{EM} = \frac{\text{Number of exact matches}}{\text{Total number of samples}} \times 100% $
  • Symbol Explanation:
    • Number of exact matches: The count of instances where the model's output is identical to the reference answer.
    • Total number of samples: The total count of questions or tasks evaluated.
• F1 Score (F1):
  • Conceptual Definition: A harmonic mean of precision and recall, commonly used in information retrieval and question answering to assess the quality of generated text compared to a reference. It balances false positives and false negatives.
  • Mathematical Formula: $ \mathrm{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}} $ $ \mathrm{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}} $ $ \mathrm{F1} = 2 \times \frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}} $
  • Symbol Explanation:
    • True Positives: Correctly identified positive instances (e.g., words correctly extracted from the answer).
    • False Positives: Incorrectly identified positive instances.
    • False Negatives: Positive instances that were missed.
• Pass@1:
  • Conceptual Definition: A metric specifically used for code generation tasks. It measures the percentage of generated code solutions that pass all provided unit tests on the first attempt (without retry or multiple generations).
  • Mathematical Formula: $ \mathrm{Pass@1} = \frac{\text{Number of solutions passing all tests}}{\text{Total number of problems}} \times 100% $
  • Symbol Explanation:
    • Number of solutions passing all tests: The count of problems for which the model's single generated code solution successfully executed and passed all associated test cases.
    • Total number of problems: The total count of coding problems evaluated.
• Bits-Per-Byte (BPB):
  • Conceptual Definition: A common metric for evaluating language models, particularly for their ability to compress data or predict the next byte accurately. Lower BPB values indicate better compression or prediction efficiency. It measures the average number of bits needed to encode one byte of the input text given the model's probabilities.
  • Mathematical Formula: $ \mathrm{BPB} = - \frac{1}{N \cdot \log_2(e)} \sum_{i=1}^N \log P(x_i | x_{$P(x_i | x_{<i})$ is the probability assigned by the model to the $i$-th byte given all preceding bytes, and the summation is over all $N$ bytes in the text.
  • Symbol Explanation:
    • $N$: Total number of bytes in the evaluated text.
    • $P(x_i | x_{<i})$: The conditional probability assigned by the language model to the $i$-th byte $x_i$, given the sequence of preceding bytes $x_{<i}$.
    • $\log_2(e)$: Conversion factor from natural logarithm to base-2 logarithm.
• Correct:
  • Conceptual Definition: Similar to Exact Match, this metric indicates whether the model's answer is factually correct, often used in question-answering tasks focusing on factual recall.
• Accuracy (Acc.):
  • Conceptual Definition: A general term for the proportion of correct predictions among the total number of cases examined. Used across various benchmarks to denote correct classification or prediction.
• Percentile:
  • Conceptual Definition: Used in the context of coding competition benchmarks (like Codeforces) to indicate the model's performance relative to a distribution of human competitors. A higher percentile suggests better performance compared to others.
• Resolved:
  • Conceptual Definition: Used in engineering-focused tasks (like SWE-Bench Verified) where the model needs to identify and fix issues in a codebase. This metric indicates whether the model successfully resolved the given problem, often verified by passing tests or applying correct patches.

5.3. Baselines

The paper compares DeepSeek-V3 against a comprehensive set of strong baselines, including both open-source and leading closed-source models.

5.3.1. Open-Source Baselines

• DeepSeek-V2-Base (DeepSeek-AI, 2024c): The previous iteration of the DeepSeek series, an MoE model.
• DeepSeek-V2-0506 / DeepSeek-V2.5-0905: Specific chat versions of DeepSeek-V2 used in post-training evaluations.
• Qwen2.5 72B Base (Qwen, 2024b) / Qwen2.5 72B Instruct: A strong dense LLM from Alibaba, notable for its performance in Chinese.
• LLaMA-3.1 405B Base (AI@Meta, 2024b) / LLaMA-3.1 405B Instruct: The largest open-source model, from Meta, serving as a key benchmark for general capabilities.

5.3.2. Closed-Source Baselines

• GPT-4o-0513 (OpenAI, 2024a): OpenAI's flagship multimodal model, representing top-tier performance.

• Claude-3.5-Sonnet-1022 (Anthropic, 2024): Anthropic's competitive model, known for strong reasoning and safety.

  These baselines are representative because they cover the spectrum of current LLM capabilities, from smaller efficient models to the largest open-source models, and the leading closed-source models, providing a robust comparative evaluation of DeepSeek-V3's advancements.

6. Results & Analysis

6.1. Core Results Analysis

6.1.1. Training Costs and Efficiency

The paper emphasizes the economical training costs of DeepSeek-V3 due to its optimized co-design of algorithms, frameworks, and hardware.

The following are the results from Table 1 of the original paper, summarizing the training costs of DeepSeek-V3:

Assuming an H800 GPU rental price of 2 per GPU hour, the total training cost for DeepSeek-V3 is5.576M. The pre-training stage alone, on 14.8T tokens, cost 2664K H800 GPU hours, which translates to only 180K H800 GPU hours per trillion tokens. This allowed the pre-training to be completed in less than two months on a cluster with 2048 H800 GPUs. This demonstrates exceptionally high training efficiency and cost-effectiveness for a model of its scale (671B total parameters, 37B activated). The training process was also notably stable, with no irrecoverable loss spikes or rollbacks.

6.1.2. Base Model Performance

The DeepSeek-V3-Base model is evaluated against state-of-the-art open-source base models.

The following are the results from Table 3 of the original paper, comparing DeepSeek-V3-Base and other representative open-source base models:

• Overall Dominance: DeepSeek-V3-Base (37B activated parameters, 671B total) emerges as the strongest open-source model. It comprehensively outperforms DeepSeek-V2-Base (21B activated, 236B total) and Qwen2.5 72B Base (72B activated, 72B total). Crucially, it surpasses LLaMA-3.1 405B Base (405B activated, 405B total) in the majority of benchmarks, despite LLaMA-3.1 having 11 times more activated parameters.
• Strengths in Math and Code: DeepSeek-V3-Base shows exceptional performance in math and code tasks, outperforming all compared baselines in HumanEval, MBPP, LiveCodeBench-Base, CRUXEval-I, CRUXEval-O, GSM8K, MATH, MGSM, and CMath. This indicates strong reasoning capabilities in these technical domains.
• English & Chinese Benchmarks: It shows competitive or better performance across English benchmarks like BBH, MMLU-series, and DROP. In Chinese benchmarks, DeepSeek-V3-Base also performs very well, outperforming Qwen2.5 72B in most, except CMMLU where it's slightly lower. For C-Eval, it achieves 90.1 EM, and for CCPM, 92.0 EM.
• Multilingual Performance: It significantly outperforms other models in MMMLU-non-English.
• Efficiency Advantage: The ability of DeepSeek-V3 to achieve superior performance with fewer activated parameters than dense models like LLaMA-3.1 405B highlights the efficiency of its MoE architecture and engineering optimizations.

6.1.3. Chat Model Performance (Post-Training)

The post-trained DeepSeek-V3 chat model is evaluated against top open-source and closed-source models.

The following are the results from Table 6 of the original paper, comparing DeepSeek-V3 and other representative chat models:

• Overall Strong Performance: DeepSeek-V3 stands as the best-performing open-source model and is highly competitive with frontier closed-source models like GPT-4o and Claude-3.5-Sonnet.
• English Benchmarks:
  • Knowledge: Achieves 88.5 EM on MMLU, 89.1 EM on MMLU-Redux, and 75.9 EM on MMLU-Pro, outperforming all open-source models and closely trailing or surpassing leading closed-source models. On GPQA-Diamond, it ranks just behind Claude-3.5-Sonnet.
  • Long Context: Demonstrates strong capabilities in long-context understanding with an impressive 91.6 F1 on DROP and top performance on LongBench v2 and FRAMES, closely trailing GPT-4o on FRAMES.
  • Factuality: While behind GPT-4o and Claude-3.5-Sonnet on SimpleQA, its strength in Chinese knowledge is noted.
• Code and Math Benchmarks:
  • Code: Emerges as the top performer in algorithmic coding tasks (e.g., HumanEval-Mul, LiveCodeBench), outperforming all baselines. In engineering tasks (SWE-Bench Verified, Aider), it trails Claude-3.5-Sonnet but significantly outperforms other open-source models.
  • Math: Achieves exceptional performance, setting a new state-of-the-art for non-$o1$-like models, significantly outperforming the second-best model (Qwen2.5 72B) by approximately 10% in AIME, MATH-500, and CNMO 2024.
• Chinese Benchmarks: On C-SimpleQA, DeepSeek-V3 surpasses Qwen2.5-72B by 16.4 points, showcasing superior Chinese factual knowledge. It exhibits similar strong performance levels to Qwen2.5-72B on C-Eval and CLUEWSC.

6.1.4. Open-Ended Evaluation

DeepSeek-V3 is also evaluated on open-ended generation tasks using LLMs as judges.

The following are the results from Table 7 of the original paper, showing English open-ended conversation evaluations:

• Arena-Hard: DeepSeek-V3 achieves an impressive win rate of 85.5% against GPT-4-0314, performing on par with Claude-3.5-Sonnet-1022 and becoming the first open-source model to surpass 85% on this benchmark. This highlights its robust capabilities in complex prompts, coding, and debugging.
• AlpacaEval 2.0: DeepSeek-V3 showcases exceptional performance with 70.0% win rate, outperforming both closed-source and open-source models, demonstrating outstanding proficiency in writing and straightforward question-answering.

6.1.5. DeepSeek-V3 as a Generative Reward Model

DeepSeek-V3's judgment ability is compared with state-of-the-art models on RewardBench.

The following are the results from Table 8 of the original paper, showing performances of GPT-4o, Claude-3.5-sonnet and DeepSeek-V3 on RewardBench:

• DeepSeek-V3 (87.0 average) achieves performance on par with GPT-4o-0806 (86.7 average) and Claude-3.5-Sonnet-1022 (88.7 average), especially when voting (maj@6) is applied, reaching an average of 89.6. This indicates its strong capability as a generative reward model for providing self-feedback and enhancing alignment.

6.2. Ablation Studies / Parameter Analysis

6.2.1. Ablation Studies for Multi-Token Prediction

The MTP strategy is validated through ablation studies on two different model scales.

The following are the results from Table 4 of the original paper, showing ablation results for the MTP strategy:

The MTP strategy consistently enhances model performance across most evaluation benchmarks for both small (15.7B total params) and large (228.7B total params) MoE models. Notable improvements are seen in BBH, MMLU, DROP, HumanEval, GSM8K, and MATH. This confirms the benefit of MTP as a training objective, even when the MTP module is discarded during inference (meaning no extra inference cost).

6.2.2. Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy

Ablations for the auxiliary-loss-free balancing strategy are conducted against purely auxiliary-loss-based methods.

The following are the results from Table 5 of the original paper, showing ablation results for the auxiliary-loss-free balancing strategy:

The auxiliary-loss-free strategy consistently achieves better model performance on most evaluation benchmarks compared to the purely auxiliary-loss-based method. This supports the hypothesis that removing direct auxiliary losses for load balancing can improve model quality by minimizing performance degradation. Improvements are seen in BBH, MMLU, HumanEval, GSM8K, and MATH.

6.2.3. Batch-Wise Load Balance vs. Sequence-Wise Load Balance

The paper discusses the distinction between auxiliary-loss-free balancing (batch-wise) and sequence-wise auxiliary loss. The batch-wise approach, being more flexible, allows experts to specialize better in different domains.

The following figure (Figure 9 from the original paper) shows expert load of auxiliary-loss-free and auxiliary-loss-based models on three domains in the Pile test set:

该图像是图表，展示了在 Pile 测试集中辅助损失自由模型与基于辅助损失模型在三个领域的专家负载。图中显示了不同层次的相对专家负载，突显出辅助损失自由模型在专家专业化模式上表现更佳。

Figure 9 illustrates that the auxiliary-loss-free model demonstrates greater expert specialization patterns on different domains (Wikipedia (en), Github, DM Mathematics) in the Pile test set compared to the auxiliary-loss-based model. This visual evidence supports the claim that the flexibility of batch-wise balancing (no explicit in-domain balance enforcement per sequence) promotes expert specialization. Further experiments with 1B and 3B MoE models showed that batch-wise auxiliary loss can achieve similar validation losses to the auxiliary-loss-free method, confirming its performance advantage over sequence-wise auxiliary loss.

6.2.4. FP8 vs. BF16 Training

The FP8 mixed precision framework is validated against BF16 training on two model scales.

The following figure (Figure 10 from the original paper) shows loss curves comparison between BF16 and FP8 training:

该图像是图表，展示了在16B和230B的DeepSeek-V2模型中BF16与FP8训练的损失曲线比较。结果通过指数移动平均（EMA）平滑处理，显示不同训练配置下损失的变化情况。

Figure 10 (a) and (b) show that for both small (16B) and large (230B) scale MoE models, FP8 training achieves comparable loss curves to BF16 training. The relative loss error remains consistently below 0.25%, which is within acceptable training randomness. This validates the feasibility and effectiveness of the proposed FP8 mixed precision framework for large-scale LLM training, confirming that the fine-grained quantization and high-precision accumulation strategies successfully mitigate the numerical stability challenges of FP8.

6.2.5. Distillation from DeepSeek-R1

The contribution of knowledge distillation from DeepSeek-R1 is ablated based on DeepSeek-V2.5.

The following are the results from Table 9 of the original paper, showing the contribution of distillation from DeepSeek-R1:

Distillation from DeepSeek-R1 significantly improves performance on LiveCodeBench-CoT (from 31.1 to 37.4 Pass@1) and MATH-500 (from 74.6 to 83.2 Pass@1). This demonstrates the effectiveness of the knowledge distillation technique for enhancing reasoning capabilities in code and math. However, it also leads to a substantial increase in average response length, indicating a trade-off between accuracy and conciseness that requires careful tuning.

6.2.6. Multi-Token Prediction Evaluation (Inference Aspect)

Although MTP is primarily a training objective, its potential for speculative decoding was evaluated. The acceptance rate of the second token prediction ranges between 85% and 90% across various generation topics. This high acceptance rate enables DeepSeek-V3 to achieve a significantly improved decoding speed, delivering 1.8 times TPS (Tokens Per Second).

6.3. Long Context Extension

The Needle In A Haystack (NIAH) test assesses the model's ability to retrieve specific information from long contexts.

The following figure (Figure 8 from the original paper) shows evaluation results on the "Needle In A Haystack" (NIAH) tests:

该图像是图表，展示了DeepSeek-V3在"Needle In A Haystack"测试中的压力测试结果。图中显示了不同上下文长度下的文档深度百分比，范围从2K到128K，DeepSeek-V3在各个上下文长度上均表现良好。

Figure 8 illustrates that DeepSeek-V3 performs well across all context window lengths up to 128K, demonstrating consistent robustness. This confirms the effectiveness of the two-phase YaRN-based context extension from 4K to 32K and then to 128K.

7. Conclusion & Reflections

7.1. Conclusion Summary

The DeepSeek-V3 technical report introduces a powerful Mixture-of-Experts (MoE) language model, DeepSeek-V3, with 671B total parameters and 37B activated per token. It builds upon the proven Multi-head Latent Attention (MLA) and DeepSeekMoE architectures from DeepSeek-V2 for efficient inference and training. Key innovations include an auxiliary-loss-free strategy for MoE load balancing, which enhances expert specialization without performance compromise, and a Multi-Token Prediction (MTP) training objective for stronger performance and speculative decoding potential.

A significant achievement is the successful implementation and validation of FP8 mixed precision training on an extremely large scale, coupled with a highly optimized DualPipe pipeline parallelism framework and efficient cross-node all-to-all communication. These infrastructure co-designs resulted in remarkably stable and cost-effective training, requiring only 2.788M H800 GPU hours (approximately $5.576M) for its full training on 14.8T tokens, without any irrecoverable loss spikes or rollbacks.

Post-training involved Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), incorporating a novel knowledge distillation method from DeepSeek-R1 models to boost reasoning capabilities. Comprehensive evaluations demonstrate that DeepSeek-V3 surpasses all other open-source models, establishing itself as the strongest in this category, particularly excelling in code and math. Furthermore, it achieves performance comparable to leading closed-source models such as GPT-4o and Claude-3.5-Sonnet, significantly narrowing the gap, and notably became the first open-source model to exceed 85% on the Arena-Hard benchmark.

7.2. Limitations & Future Work

The authors acknowledge several limitations and propose future research directions:

• Deployment Unit Size: The recommended deployment unit for DeepSeek-V3 is relatively large, which might be a burden for small teams or resource-constrained environments.

• Inference Speed: While significant progress has been made (2x speedup over DeepSeek-V2), there is still potential for further enhancement in end-to-end generation speed.

• Hardware Dependency: These limitations are expected to be naturally addressed with the development of more advanced hardware.

  Future research directions include:

• Model Architectures: Continuously refining architectures to improve training/inference efficiency, aiming for efficient support of "infinite context length," and breaking Transformer architectural limitations.

• Training Data: Iterating on the quantity and quality of training data, and exploring additional training signal sources to drive data scaling across more dimensions.

• Deep Thinking Capabilities: Exploring ways to enhance models' intelligence and problem-solving abilities by expanding reasoning length and depth.

• Evaluation Methods: Developing more comprehensive and multi-dimensional model evaluation methods to prevent optimizing for a fixed set of benchmarks, which can create misleading impressions of capabilities.

7.3. Personal Insights & Critique

DeepSeek-V3 stands out as a testament to the power of holistic engineering and algorithmic co-design in pushing the boundaries of LLMs. The paper meticulously details not just architectural innovations but also the intricate system-level optimizations required to train such a massive model efficiently and stably.

The auxiliary-loss-free load balancing is particularly insightful. It's a clever way to address a known performance trade-off in MoE models, moving from explicit regularization that can hinder specialization to a more adaptive, dynamic adjustment. This suggests a growing maturity in MoE research, moving beyond basic load balancing to optimizing for expert specialization itself. Similarly, utilizing MTP as a training objective, rather than just for speculative decoding, indicates a deeper understanding of how to leverage predictive tasks to enrich model representations.

The detailed breakdown of FP8 training and DualPipe pipeline parallelism is a valuable contribution for the broader AI community, offering concrete strategies for overcoming memory and communication bottlenecks in large-scale distributed training. Their specific hardware suggestions to AI vendors, stemming directly from their challenges, highlight the critical need for tighter hardware-software co-optimization in AI accelerators.

The knowledge distillation from DeepSeek-R1 is a pragmatic approach to inject advanced reasoning capabilities without making the main model overly complex or slow. This "borrowing" of intelligence from specialized models for specific tasks (like math and code) could become a standard practice for creating more versatile general-purpose LLMs.

One potential area for further exploration or critique is the exact impact of the "dynamic redundancy strategy" for MoE inference. While mentioned as future work for decoding, and being explored for prefilling, the real-world latency and throughput benefits, especially under variable load, would be crucial. The dependence on "periodically adjusted" redundant experts based on statistics might introduce a slight lag or suboptimal routing until adjustments are made. Furthermore, while the cost-effectiveness is impressive, the absolute figure of ~$5.5M is still substantial, implying that such cutting-edge LLM development remains largely confined to well-funded organizations.

Overall, DeepSeek-V3 represents a significant leap for open-source LLMs, demonstrating that with deep technical expertise across algorithms, architecture, and systems, open models can indeed rival closed-source leaders. Its emphasis on efficiency, stability, and specialized performance makes it a highly valuable contribution to the field.