1. Bibliographic Information

1.1. Title

Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication

1.2. Authors

Lewei Jin, Kui Zhang, Yongqi Chen, Yifan Zhuo, Renjie Li, Yi Gao, Bowei Yang, Zhengong Cai, Wei Dong from Zhejiang University. Their research backgrounds appear to be in computer science, likely focusing on distributed systems, machine learning, and potentially network optimization, given the paper's subject matter.

1.3. Journal/Conference

The paper does not explicitly state the journal or conference it was published in, but the publication date and abstract suggest it is a recent academic publication. Given the nature of the research, it would likely target top-tier conferences or journals in distributed systems, machine learning systems, or computer architecture.

1.4. Publication Year

2025

1.5. Abstract

The abstract introduces the challenge of costly Large Language Model (LLM) serving due to the requirement for multiple high-end GPUs. It highlights the opportunity presented by cheaper consumer-grade GPUs. While prior distributed serving efforts focused on model deployment, communication efficiency remains a challenge, particularly the imbalance in data transfer between prefill and decode phases, where prefill can involve significantly larger data volumes. This imbalance leads to decode delays and server underutilization. The paper proposes MoLink, an efficient distributed LLM serving system. MoLink addresses this by splitting large prefill transmissions into smaller chunks and carefully scheduling them. It comprises two main components: (i) a transmission scheduling algorithm that fairly determines whether to transmit prefill or decode requests, ensuring prefill is not starved; and (ii) a chunking determination algorithm that adaptively decides the prefill chunk volume just-in-time to avoid blocking decode requests. Evaluations show that MoLink reduces Time to First Token (TTFT), Time per Output Token (TPOT), and overall latency by up to 46% compared to state-of-the-art distributed LLM serving systems.

1.6. Original Source Link

/files/papers/6914a1e059f6bf3b040db300/paper.pdf (This link indicates an internal file path, suggesting it might be a preprint or an internally hosted version of the paper.)

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the high cost and inefficiency of serving Large Language Models (LLMs) in distributed environments, particularly when leveraging consumer-grade Graphics Processing Units (GPUs).

• Why is this problem important?

  • Cost-Efficiency: Modern LLMs, such as OPT-175B, require vast amounts of memory (e.g., 350 GB for OPT-175B), necessitating multiple expensive high-end GPUs (e.g., A100s). Consumer-grade GPUs (e.g., RTX 4090) offer comparable computational power at a significantly lower cost (e.g., 4x lower hourly pricing than A100s in cloud markets). There's a massive untapped resource in widely deployed but underutilized consumer-grade GPUs (e.g., 101 million PC GPUs shipped in Q4 2021). Efficiently utilizing these cheaper resources can democratize LLM serving and reduce operational expenses.
  • Communication Inefficiency: Prior distributed LLM serving efforts primarily focused on model deployment strategies (e.g., partitioning models across GPUs). However, a critical challenge emerges from communication efficiency, specifically the severe imbalance in data transfer volumes between the two primary phases of LLM inference: prefill and decode. Prefill requests, which process the input prompt, can involve up to 1000 times more data transfer than decode requests, which generate one token at a time.
  • Performance Bottleneck: This imbalance leads to transmission competition. When a large prefill request occupies the network bandwidth, smaller, more frequent decode requests are delayed, even if they require only milliseconds to transmit. This causes server underutilization, as GPUs wait idly for decode data, leading to increased Time to First Token (TTFT), Time per Output Token (TPOT), and overall latency.

• What is the paper's entry point or innovative idea? The paper's innovative idea is to address the transmission competition and communication inefficiency by intelligently managing the transmission of prefill requests. Instead of sending an entire large prefill request in one go, MoLink (the proposed system) splits the prolonged transmission volume of prefill requests into smaller chunks. These chunks are then carefully scheduled for transmission to interleave with decode requests, minimizing their blocking effect. This strategy aims to reconcile the computation and communication demands in a bandwidth-constrained, distributed environment using consumer-grade GPUs.

2.2. Main Contributions / Findings

The paper presents MoLink, an efficient distributed LLM serving system, with the following primary contributions and findings:

• Identification of Performance Bottleneck: The paper identifies transmission competition among requests from different inference phases (prefill and decode) as a significant performance bottleneck in distributed LLM serving, especially in bandwidth-constrained environments.
• Novel Transmission Scheduling Strategy: It proposes a chunk transmission strategy to mitigate this competition. This involves splitting large prefill transmission volumes into smaller chunks and carefully scheduling their transmission alongside decode requests.
• MoLink System Design:
  • Weighted-Priority Transmission Scheduling Algorithm: This algorithm intelligently determines when to transmit prefill or decode requests. It prioritizes decode requests but includes a waiting weight (W) mechanism to prevent prefill requests from being starved.
  • Just-in-Time Chunk Determination Algorithm: This adaptive algorithm dynamically determines the optimal size of prefill chunks to transmit. It predicts the available time intervals based on the current and upcoming decode execution times, ensuring that decode requests are not unduly blocked.
  • Micro-Batch Extension: MoLink extends the number of micro-batches beyond the pipeline degree to improve GPU utilization by overlapping transmission time with execution.
• Empirical Validation and Performance Improvements:
  • Evaluation against state-of-the-art distributed LLM serving systems (vLLM, Helix) demonstrates significant performance gains.
  • MoLink reduces TTFT (Time to First Token), TPOT (Time per Output Token), and overall end-to-end latency.
  • The system achieves a maximum reduction of up to 46% in these metrics, particularly benefiting medium workloads where transmission competition is most pronounced.
  • The benefits are observed across a range of request rates, bandwidths, and network delays.
• Platform Support: MoLink is designed to support heterogeneous environments, including both Linux servers and Windows PCs, facilitating the use of widely available consumer-grade GPUs.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully grasp the innovations presented in this paper, it's crucial to understand several foundational concepts related to Large Language Models (LLMs), distributed computing, and performance metrics.

• Large Language Models (LLMs): These are advanced artificial intelligence models, typically based on the Transformer architecture, designed to understand, generate, and process human language. They are "large" due to their massive number of parameters (ranging from billions to hundreds of billions), requiring significant computational resources. Examples include GPT-3, LLaMA, and Qwen.
• LLM Inference Phases: When an LLM generates text, the process is typically divided into two distinct phases:
  • Prefill Phase (Prompt Processing): This is the initial phase where the model processes the input prompt (the user's query or initial text). During prefill, all tokens in the input sequence are processed in parallel to generate their corresponding hidden representations and key-value (KV) caches. This phase often involves a large number of input tokens, leading to a substantial computational workload and, importantly for this paper, a large volume of data (activation tensors) that might need to be transferred between distributed GPUs.
  • Decode Phase (Token Generation): After the prefill phase, the model enters the decode phase. In this phase, the model generates one new token at a time based on the input prompt and previously generated tokens. Each new token requires a new inference step, processing only one token. Compared to prefill, the decode phase involves much smaller data transfers per step but is highly iterative (auto-regressive).
• GPU TFLOPS (Tera Floating-Point Operations Per Second): TFLOPS is a measure of a computer's processing speed, specifically how many trillion floating-point operations it can perform per second. A higher TFLOPS value indicates greater computational power.
• FP16 (Half-Precision Floating Point): This refers to a data format for floating-point numbers that uses 16 bits (2 bytes) for representation, compared to standard 32-bit (FP32) or 64-bit (FP64) formats. Using FP16 reduces memory consumption and can speed up computations on GPUs that have specialized hardware (like Tensor Cores) for half-precision operations, often with minimal loss in model accuracy.
• Distributed LLM Serving: Because LLMs are so large, a single GPU often cannot hold the entire model's parameters or the intermediate activations needed for inference. Distributed serving involves distributing parts of the model or computations across multiple GPUs or servers to collectively perform inference. This introduces challenges related to communication between these distributed components.
  • Tensor Parallelism (TP): A model parallelism strategy where individual layers (e.g., weight matrices) of an LLM are partitioned across multiple GPUs. For example, a large weight matrix might be split into columns, with each GPU holding a part. This requires frequent communication (e.g., all-reduce operations) between GPUs within the same layer, making it highly sensitive to network conditions.
  • Pipeline Parallelism (PP): A model parallelism strategy where different layers or stages of an LLM are assigned to different GPUs or servers, forming a "pipeline." Data (specifically, activation tensors, which are intermediate results) flows sequentially through these stages. To keep the pipeline busy and improve throughput, inputs are often divided into micro-batches. While one GPU processes micro-batch A of layer $N$, another GPU can process micro-batch A of layer N-1 or micro-batch B of layer $N$ (if the latter has completed layer N-1).
    • Activation Tensor: In the context of neural networks, an activation tensor is the output of a layer after applying its operations and activation function. When using pipeline parallelism, these tensors (intermediate results) need to be transferred between different servers/GPUs that host consecutive layers of the model.
• Pipeline Bubbles: In pipeline parallelism, a pipeline bubble refers to periods of idle time in some GPUs within the pipeline. This often occurs at the beginning and end of a sequence of micro-batches or when communication overheads (e.g., transferring activation tensors) are high, causing a receiving GPU to wait for data from the previous stage. These bubbles reduce the overall efficiency and throughput of the system.
• Performance Metrics for LLM Serving:
  • Time to First Token (TTFT): This metric measures the time elapsed from when a request is submitted until the first output token is generated and delivered. It primarily reflects the efficiency of the prefill phase, as a faster prefill leads to a quicker TTFT.
  • Time per Output Token (TPOT): This metric measures the average time taken to generate each subsequent output token after the first one. It reflects the efficiency of the decode phase. A lower TPOT means faster token generation and a smoother user experience.
  • End-to-end Latency: This is the total time elapsed from when a request is submitted until the very last output token for that request is generated and delivered. It encompasses both TTFT and the time taken for all subsequent decode steps.

3.2. Previous Works

The paper mentions several prior studies related to distributed computing, LLM serving, and model deployment. These works lay the groundwork for understanding the current challenges and MoLink's contributions.

• Distributed Computing at Scale:
  • Folding@Home [22]: A distributed computing project that uses volunteer computing to simulate protein folding. It demonstrates the potential of sourcing computational power from a vast number of heterogeneous, consumer-grade devices (e.g., 40,000 Nvidia and AMD GPUs), highlighting the feasibility of distributed GPU utilization.
• Fault-Tolerant and Decentralized LLM Serving:
  • Petals [4]: Explores fault-tolerance for serving LLMs on unsteady, decentralized servers. It focuses on model allocation and request scheduling in dynamic device groups. While similar in using distributed GPUs, Petals primarily addresses fault tolerance and dynamic environments, whereas MoLink focuses on communication efficiency in a more stable, fixed device group.
  • HexGen [10]: Optimizes the deployment of LLMs in decentralized environments. This work likely deals with how to place model layers or parameters across a network of unreliable or diverse machines.
• Optimal LLM Deployment and Scheduling:
  • Helix [14]: Discovers optimal LLM deployment and request scheduling under heterogeneous clusters. Helix is a high-throughput serving system that sequentially sends prefill or decode volumes into a ZeroMQ message queue, which then asynchronously sends messages. MoLink is compared against Helix as a baseline, showing Helix suffers from transmission competition due to its lack of explicit awareness of prefill/decode imbalances.
• LLM Serving Systems (General Optimizations):
  • Orca [28]: Proposed iteration-level scheduling to release resources once a request is finished. This is an internal scheduling optimization within a single server or a cluster.
  • vLLM [12]: A widely used LLM serving system that introduced PageAttention to reduce memory consumption by allocating the exact number of pages a request requires. vLLM uses a concurrent transmission schedule, asynchronously sending prefill or decode volumes via sockets. MoLink also uses vLLM as a baseline, highlighting that vLLM's concurrent transmission still leads to transmission competition without prefill/decode specific scheduling.
  • Speculative Inference [13, 15]: Applies a smaller, faster model (a "draft model") to predict multiple output tokens simultaneously. These tokens are then verified by the larger, more accurate model. This can significantly speed up decode phase, but it's orthogonal to MoLink's communication focus.
  • Splitwise [18] and DistServe [29]: These works disaggregate the prompt (prefill) and decode phases of requests. This means they might process these phases on different hardware or using different strategies. While related to phase separation, MoLink specifically tackles the communication implications of this separation in a distributed setup.
  • Sarathi [1]: Introduced chunked prefill, which allocates a budget to the prompt phase. This is conceptually close to MoLink's chunking idea for prefill. However, Sarathi does not optimize the transmission of these prefill chunks by dynamically scheduling their volume to interleave with decode requests, which is MoLink's key innovation.
• Distributed ML Task Optimization:
  • [9, 17]: Co-design model partition and placement on heterogeneous clusters. This addresses how to optimally divide a model and place its parts given varying hardware capabilities.
  • Learninghome [21] and DeDLOC [5]: Studied network-aware routing on decentralized clusters. This focuses on optimizing data paths in dynamic, unreliable networks.
  • SWARM [20]: Optimizes pipeline communication in a heterogeneous network. This is highly relevant as MoLink also focuses on pipeline communication, but with a specific emphasis on the prefill/decode imbalance.
  • [26] and [7]: Efforts on using approximations to reduce network communication or synchronization. These are general strategies to reduce communication overhead.
  • SkyPilot [27] and Mélange [6]: Select the best type of GPUs for a request. These focus on resource allocation and cost optimization by picking the right hardware.

3.3. Technological Evolution

The evolution of LLM serving technologies can be broadly categorized:

1. Single-GPU Era: Early LLMs could often fit on a single high-end GPU. The primary focus was on optimizing inference on a single device.
2. Distributed Serving - Basic Model Parallelism: As LLMs grew larger, they exceeded the memory capacity of single GPUs. This led to the development of model parallelism techniques like Tensor Parallelism and Pipeline Parallelism to distribute models across multiple GPUs/servers. Initial efforts focused on how to split the model and how to schedule requests to keep GPUs busy, often assuming high-bandwidth, low-latency interconnections (e.g., within a data center rack).
3. Advanced Distributed Serving - Resource Optimization: Works like vLLM and Orca introduced more sophisticated scheduling and memory management techniques to improve throughput and reduce latency within a distributed setup, often still assuming relatively robust network conditions. Petals, HexGen, and Helix further explored deployment and scheduling in more diverse or decentralized environments.
4. Communication-Aware Distributed Serving (Current Focus): This paper highlights a critical gap in the previous stages: communication efficiency, especially in bandwidth-constrained environments and with consumer-grade GPUs. The growing recognition of the prefill/decode imbalance and its impact on performance, particularly in real-world networks, has pushed the focus towards optimizing data transfer itself, rather than just computational scheduling. MoLink fits squarely into this category.

3.4. Differentiation Analysis

Compared to the main methods in related work, especially vLLM and Helix, MoLink introduces several core differences and innovations:

• Explicit Awareness of Prefill/Decode Imbalance: Unlike vLLM (which uses concurrent transmission) and Helix (which uses ZeroMQ for asynchronous sending), MoLink is explicitly designed around the imbalance in data transfer volumes between the prefill and decode phases. Prior systems treat all network transmissions somewhat generically; MoLink recognizes that prefill is communication-bound and decode is computation-bound and designs its strategy accordingly.
• Chunked Transmission for Prefill: MoLink's most significant innovation is the chunking of prefill requests. Instead of sending the entire potentially massive prefill activation tensor at once, it breaks it into smaller, manageable chunks. This allows for fine-grained control over network bandwidth usage. Sarathi also proposed chunked prefill, but MoLink optimizes the transmission of these chunks.
• Intelligent, Fair Transmission Scheduling: MoLink employs a weighted-priority transmission scheduling algorithm that actively prioritizes decode requests while ensuring prefill requests are not starved. This is a proactive scheduling mechanism, in contrast to the more passive asynchronous sending used by baselines, which can still lead to transmission competition.
• Just-in-Time Adaptive Chunk Determination: MoLink dynamically calculates the optimal size of prefill chunks based on the predicted available network time, considering the timing of upcoming decode executions. This adaptive approach ensures that prefill chunks are sized precisely to fit into network "gaps" without significantly delaying decode requests. This level of dynamic, just-in-time adjustment is not present in generic asynchronous transmission systems.
• Micro-Batch Extension for Utilization: While not unique to MoLink, its application of extending micro-batches beyond the pipeline degree is crucial for maximizing GPU utilization, especially when network delays create pipeline bubbles that would otherwise leave GPUs idle. This synergizes with its communication optimizations.
• Target Environment: MoLink explicitly targets consumer-grade GPUs and bandwidth-constrained environments, making its optimizations particularly relevant for cost-efficient LLM serving outside of high-end data centers with specialized interconnects.

4. Methodology

4.1. Principles

The core idea behind MoLink is to mitigate transmission competition between prefill and decode requests in distributed LLM serving, especially in bandwidth-constrained environments. It achieves this by recognizing the fundamentally different communication characteristics of these two inference phases: prefill involves large, bursty data transfers, while decode involves small, frequent transfers. The underlying principles are:

1. Chunking Large Transmissions: Break down the large activation tensor generated during the prefill phase into smaller, manageable chunks. This prevents a single large prefill transmission from monopolizing network bandwidth for an extended period.
2. Prioritized and Fair Scheduling: Develop a scheduling mechanism that intelligently prioritizes the transmission of small, latency-sensitive decode requests. Simultaneously, ensure that the chunked prefill requests are not indefinitely delayed or starved, maintaining overall system progress.
3. Adaptive Resource Utilization: Dynamically determine the size of prefill chunks based on real-time network conditions and the predicted timing of decode requests. This "just-in-time" adaptation aims to fill network idle times with prefill data without impacting decode performance.
4. Overlapping Communication and Computation: Maximize system throughput by actively creating opportunities to overlap communication (especially prefill chunk transmissions) with computation (especially decode executions or other prefill computations), thereby reducing pipeline bubbles and server idle times.

4.2. Core Methodology In-depth (Layer by Layer)

MoLink is structured with two main components: a transmission scheduling algorithm and a chunking determination algorithm. It also incorporates a strategy for extending the number of micro-batches.

4.2.1. Architecture

The overall architecture of MoLink is designed for distributed LLM serving using pipeline parallelism.

The following figure (Figure 4 from the original paper) shows the architecture of MoLink:

该图像是一个示意图，展示了在MoLink系统中，多个k8s pod和WSL VM之间的微批处理过程。图中表示了模型层之间的数据传输、网络连接及微批次调度的过程，突出了分布式LLM服务的结构与调度算法。

The figure illustrates that different parts of the LLM layers are deployed on multiple workers (servers) using a pipeline parallelism strategy. This means that a request progresses through the layers sequentially, with each worker responsible for a subset of the model's layers. Intermediate activation tensors are transferred between these workers. MoLink supports deployment on both Linux servers (managed via Kubernetes and Docker containers) and Windows PCs (using lightweight Kubernetes-like functionality for resource management in containerized environments like AutoDL). This broad platform support highlights its focus on utilizing diverse, potentially consumer-grade, hardware.

4.2.2. Transmission Scheduling

To mitigate the transmission competition between prefill and decode requests, MoLink implements a weighted-priority transmission scheduling algorithm. The goal is to maximize throughput while guaranteeing that prefill requests are not indefinitely starved.

Algorithm 1, as presented in the paper, outlines this scheduling policy:

1 Initialize volume queue vq_1, vq_2 ← ∅
2 Initialize waiting weight W = 0
3: Initialize max waiting weight N = 30
4: While True do
5: While v_new = get_next_volume() do
6: if v_new in phase.decode:
7: add v_new to vq_1
8: else
9: add v_new to vq_2
10: if vq_1 ≠ ∅ and vq_2 ≠ ∅
11: W = W + 1
12: if vq_1 ≠ ∅ and W < N
13: send(vq_1[0])
14: pop(vq_1)
15: elif vq_2 ≠ ∅
16: if W >= N
17: v_t = vq_2[0].left
18: else
19: v_t = chunk(vq_2[0].left)
20: vq_2[0].left = vq_2[0].left - v_t
21: if vq2[0].lef t == 0
22: pop(vq2)
23: send(vt)
24: W = 0

Explanation of Algorithm 1 (Weighted-priority transmission scheduling):

• Line 1-3 (Initialization):
  • Two queues are initialized: $vq_1$ for decode request volumes and $vq_2$ for prefill request volumes. These queues hold the data (activation tensors) that are ready to be transmitted to the next server in the pipeline.
  • A waiting weight W is initialized to 0. This counter tracks how many times decode requests have been prioritized over prefill requests when both were available.
  • A max waiting weight N is set (e.g., 30). This threshold determines how many times decode can be prioritized before prefill must be served to prevent starvation.
• Line 4 (Main Loop): The system continuously runs this loop to manage transmissions.
• Line 5-9 (Volume Enqueueing):
  • get_next_volume(): This function continuously checks for newly computed activation tensors that are ready for transmission.
  • If a new volume v_new belongs to the decode phase, it is added to $vq_1$ (the decode queue).
  • Otherwise (if it belongs to the prefill phase), it is added to $vq_2$ (the prefill queue).
• Line 10-11 (Increment Waiting Weight): If both decode and prefill queues ($vq_1$ and $vq_2$) are non-empty, meaning there's a potential transmission competition, the waiting weight W is incremented. This signifies that decode might be prioritized, potentially delaying prefill.
• Line 12-14 (Prioritize Decode):
  • If $vq_1$ (decode queue) is not empty AND $W$ is less than the max waiting weight N:
    • The first decode request in $vq_1$ (vq_1[0]) is sent.
    • This decode request is then removed from $vq_1$.
  • This logic ensures that decode requests are prioritized as long as prefill hasn't been waiting for too long (i.e., $W < N$).
• Line 15-23 (Handle Prefill):
  • This elif block is executed if either $vq_1$ is empty (no decode requests to send) OR $W$ has reached or exceeded $N$ (meaning prefill must be served to avoid starvation).
  • If $vq_2$ (prefill queue) is not empty:
    • Line 16-17 (Starvation Prevention): If $W >= N$, it means prefill has been waiting for a significant amount of time. In this case, the entire remaining left volume of the first prefill request ($vq_2[0].left$) is designated for transmission ($v_t$). This ensures the prefill request is fully processed to prevent starvation.
    • Line 18-19 (Chunking): Otherwise (if $W < N$, meaning decode was prioritized but now either $vq_1$ is empty or it's prefill's turn without starvation condition), a chunk() function is called to determine a smaller volume ($v_t$) from the prefill request ($vq_2[0].left$). This is where the just-in-time chunk determination algorithm comes into play.
    • Line 20-22 (Update and Pop Prefill): The remaining left volume of the prefill request is updated by subtracting $v_t$. If the left volume becomes 0, the entire prefill request has been transmitted, and it's removed from $vq_2$.
    • Line 23 (Send Chunk): The determined volume ($v_t$, either full remaining prefill or a chunk) is sent.
    • Line 24 (Reset Waiting Weight): After any prefill transmission (either a full remaining or a chunk), the waiting weight W is reset to 0. This gives decode requests an opportunity to regain priority.

4.2.3. Just-in-Time Chunk Determination

This algorithm adaptively determines the volume of a prefill chunk ($v_t$ in Algorithm 1, line 19) based on the predicted available time interval for transmission. The goal is to transmit prefill chunks without delaying decode requests.

The following table (Table 1 from the original paper) defines the variables used:

The core idea is to predict $T_a$, the available time for prefill chunk transmission, which depends on when the next decode transmission/execution will occur. $T_a$ is calculated as: $ T_a = t_s + T_d - t_c $ where $T_d$ represents the duration of the current (or next) decode execution.

The following figure (Figure 5 from the original paper) illustrates the three cases for determining chunk volume:

该图像是一个示意图，显示了在不同服务器上处理预填和解码请求的时间安排。图中展示了三个服务器在处理请求时的情况，标记了时间点 $t_c$ 和 $t_s$，并列出了三种不同的情况（案例1、案例2、案例3），说明了请求的到达和处理时间的关系。

Case 1: The start time of transmission ($t_c$) equals the start time of the current execution ($t_s$).

• Scenario: This typically happens when the first chunk of a prefill request is transmitted, and the execution of a subsequent decode batch begins simultaneously.
• Calculation: The finish time of the decode execution, $t_f$, is $t_s + T_d$. Since $t_c = t_s$, the available time interval $T_a$ simplifies to: $ T_a = t_f - t_c = (t_s + T_d) - t_s = T_d $ Here, $T_d(x)$ is the decode execution duration, expressed as a function of the number of tokens $x$, derived from system profiling.

Case 2: The start time of transmission ($t_c$) is later than the start time of the current execution ($t_s$).

• Scenario: This occurs when a prefill chunk transmission begins after a decode batch has already started executing (e.g., if one of multiple serial decode batches has completed, and a prefill chunk fits in before the next decode stage).
• Calculation: The finish time of the decode execution, $t_f$, is still $t_s + T_d$. However, since $t_c > t_s$, the available time interval $T_a$ is reduced: $ T_a = t_f - t_c = (t_s + T_d) - t_c $ This value is smaller than in Case 1 due to the delayed start of the prefill chunk transmission relative to the decode execution.

Case 3: The start time of transmission ($t_c$) is earlier than the start time of the current execution ($t_s$).

• Scenario: This situation arises when a prefill chunk transmission begins after the final decode batch in a sequence has completed, meaning there's currently no decode batch executing. In this case, the system needs to consider the next decode execution in the upcoming iteration.
• Prediction: Because LLMs are auto-regressive, the execution of the next iteration can be predicted from the previous one. If $\{B_1, B_2, \ldots, B_N\}$ are the decode batches in the latest iteration, $B_1$ is expected to be the next to execute ($\mathrm{dec}^*$ in Figure 5).
• Modeling Overheads: The system models both transmission overhead ($T_m$) and computation overhead ($T_o$) for $B_1$ to complete its iteration across servers.
  • Computation Overhead ($T_o$): $ T_o = \sum_{i=1}^{M} T_{d_i}(n) $ where:
    • $M$: The pipeline degree (number of servers in the pipeline).
    • $T_{d_i}(n)$: The execution time on server $i$ for $n$ tokens. This is a function derived from profiling data for each server.
  • Transmission Overhead ($T_m$): $ T_m = \sum_{i=1}^{M} \mathrm{Lat}i + \sum{i=1}^{M-1} \frac{\mathrm{act_sz} \times n}{\mathrm{Band}_i} + \frac{\mathrm{tok_sz} \times n}{\mathrm{Band}_M} $ where:
    • $M$: The pipeline degree.
    • $\mathrm{Lat}_i$: The network latency between server $i$ and server $i+1$ (for $i \le M-1$), or between server $M$ and server 1 (for $i = M$, likely closing a loop if the pipeline is circular or for the final output).
    • $\mathrm{Band}_i$: The corresponding network bandwidth between these servers.
    • $n$: The number of tokens processed in the current decode batch.
    • $\mathrm{act\_sz}$: The size of the activation tensor (e.g., 13312B for LLaMa-30B).
    • $\mathrm{tok\_sz}$: The size of a single token (e.g., 2B for LLaMa-30B). The last term in the sum likely accounts for the final token output transmission.
• Calculation: The estimated finish time of the next decode execution ($t_f$) is: $ t_f = t_p + T_o + T_m $ where $t_p$ is the completion time of the first decode batch in the latest iteration. Therefore, the available time interval $T_a$ is: $ T_a = t_f - t_c = t_p + T_o + T_m - t_c $ This value is typically larger than in Case 1 due to the longer gap before the next decode execution.

Once $T_a$ is determined, MoLink can calculate the chunk volume ($v_t$) that can be transmitted within this duration, considering the available network bandwidth.

4.2.4. Extending the Number of Micro-Batches

This optimization addresses the issue of pipeline bubbles and underutilization in pipeline parallelism under bandwidth-constrained environments.

The following figure (Figure 6 from the original paper) illustrates the impact of micro-batch number in the pipeline:

该图像是一个示意图，展示了在分布式 LLM 服务中，两个服务器（server1 和 server2）在不同时间点上处理请求 B1 和 B2 的调度情况。图中同时展示了网络传输的时间，反映了预填充请求与解码请求的调度策略。

• Problem (Figure 6a): Existing systems often set the number of micro-batches equal to the pipeline degree ($N$). This works well when transmission overhead is negligible. However, with limited bandwidth, transmission time for activation tensors becomes significant. As shown in Figure 6a, transmission process (e.g., for B1 and B2 to Server 2) can delay the execution of micro-batches on the target server. This leads to server idling (e.g., Server 2 idles after finishing B2 because B1 is still arriving), as there are no more micro-batches to fill the idle time.
• Solution (Figure 6b): MoLink proposes to extend the number of micro-batches to be larger than the pipeline degree. Figure 6b shows an example where a new micro-batch (B3) is added. When B2 finishes on Server 2, B3 can start executing, and its execution time can overlap with the transmission time of B1 to Server 2. This effectively reduces the idle time of servers and improves utilization.
• Optimization: The optimal number of micro-batches depends on hardware and network conditions. MoLink emulates different numbers of micro-batches within a limited search space (from $N$ to 2N, where $N$ is the pipeline degree) to find an optimal value.

4.3. Platform Support

MoLink is designed to be versatile, supporting both Windows PCs and Linux servers. For Linux environments, it leverages Kubernetes to manage Docker containers, providing robust orchestration. For Windows PCs and containerized environments like AutoDL [3], it implements lightweight Kubernetes-like functionality to manage resources and deployments, making it adaptable to a wide range of consumer-grade setups.

5. Experimental Setup

5.1. Datasets

The experiments in the paper utilize a specific trace and model to evaluate MoLink's performance.

• Model: Qwen 7B [19], a representative and popular open-source Transformer model, is used for evaluating system performance. Inference is performed using half-precision (FP16).
• Trace: An Azure Conversation trace [2] is employed to simulate the arrival of requests. This trace is described as representative of LLM inference invocations, providing realistic patterns of input and output tokens.

  • Data Characteristics: The following figure (Figure 7 from the original paper) shows the length distribution of the datasets.

    该图像是一个直方图，展示了输入长度（蓝色）和输出长度（红色）的比例分布。横轴表示长度，纵轴表示比例，数据包括长度为0到2000的范围。该图反映了输入与输出长度的差异情况。

    The histogram illustrates the distribution of input length (blue) and output length (red). The input length distribution is broad, with a significant proportion of requests having input lengths up to 500 tokens, and some extending beyond 1500. The output length distribution is more concentrated at shorter lengths, mostly below 200 tokens. This indicates that prefill requests often involve longer sequences, while decode requests generate shorter outputs in each step.

  • Arrival Rate: The following figure (Figure 8 from the original paper) shows the arrival rate.

    该图像是一个示意图，展示了在时间（秒）上到达率（请求/秒）的变化情况。图中显示到达率在0到20之间波动，具有明显的高峰和低谷，反映了请求流量的动态特性。

    The diagram shows the arrival rate (requests per second) fluctuating over time. The rate varies dynamically, peaking at around 15-20 requests/s at certain points, but also experiencing periods of lower activity. This dynamic workload simulates real-world usage patterns where request arrival is not constant.

• Preprocessing: Requests with input lengths larger than 2048 or output lengths larger than 1024 are removed. The frequency of requests arrival is scaled to match the capacity of the GPUs used in the cluster, which is referred to as the arrival rate in the experiments.
• Experimental Duration: The cluster is warmed up for 1 minute before testing, and experiments run for 30 minutes.

5.2. Evaluation Metrics

The performance of the LLM service is measured using three key metrics:

1. Time to First Token (TTFT):

   • Conceptual Definition: TTFT quantifies the responsiveness of the LLM service. It measures the duration from the moment a user's request (prompt) is submitted to the system until the very first output token generated by the LLM is returned. This metric is crucial for user experience, as a quick TTFT makes the service feel snappy and responsive. It is predominantly influenced by the efficiency of the prefill phase, as the model must process the entire input prompt before it can generate the first output token.
   • Mathematical Formula: The paper does not provide an explicit formula for TTFT, but it is generally defined as: $ \mathrm{TTFT} = \text{Time}{\text{first_output_token_generated}} - \text{Time}{\text{request_submitted}} $
   • Symbol Explanation:
     • $\text{Time}_{\text{first\_output\_token\_generated}}$: The timestamp when the first output token is successfully generated and ready to be sent back.
     • $\text{Time}_{\text{request\_submitted}}$: The timestamp when the user's request (prompt) was initially received by the serving system.

2. Time per Output Token (TPOT):

   • Conceptual Definition: TPOT measures the average time taken to generate each subsequent output token after the first one. This metric reflects the throughput and steady-state generation speed of the LLM in its decode phase. A lower TPOT indicates that the model can generate tokens quickly, leading to a faster completion of the overall response once it has started.
   • Mathematical Formula: The paper does not provide an explicit formula for TPOT, but it is typically calculated as: $ \mathrm{TPOT} = \frac{\text{Time}{\text{last_output_token_generated}} - \text{Time}{\text{first_output_token_generated}}}{\text{Number of Output Tokens} - 1} $ (Note: If only one token is generated, TPOT is not well-defined or can be considered 0).
   • Symbol Explanation:
     • $\text{Time}_{\text{last\_output\_token\_generated}}$: The timestamp when the final output token for a given request is generated.
     • $\text{Time}_{\text{first\_output\_token\_generated}}$: The timestamp when the first output token was generated.
     • $\text{Number of Output Tokens}$: The total count of tokens generated in response to the request.

3. End-to-end Latency:

   • Conceptual Definition: End-to-end latency represents the total time a user has to wait for a complete response from the LLM. It is the duration from the submission of a request until all output tokens have been generated and the request is fully completed. This metric encompasses both the prefill and decode phases.
   • Mathematical Formula: The paper does not provide an explicit formula for End-to-end Latency, but it is generally defined as: $ \text{End-to-end Latency} = \text{Time}{\text{last_output_token_generated}} - \text{Time}{\text{request_submitted}} $
   • Symbol Explanation:

     • $\text{Time}_{\text{last\_output\_token\_generated}}$: The timestamp when the final output token for a given request is generated.

     • $\text{Time}_{\text{request\_submitted}}$: The timestamp when the user's request (prompt) was initially received by the serving system.

       These metrics are averaged over the entire serving duration (30 minutes after warmup) to provide a comprehensive view of the service's performance.

5.3. Baselines

MoLink is compared against two state-of-the-art distributed LLM serving systems to demonstrate its effectiveness:

1. vLLM [12]:

   • Description: vLLM is a widely recognized and used LLM serving system in both academia and industry. It is known for its PageAttention mechanism, which efficiently manages key-value (KV) caches to reduce memory consumption.
   • Transmission Strategy: vLLM employs a concurrent transmission schedule. This means it asynchronously sends the activation volumes of both prefill and decode requests using sockets. However, critically, it does so without explicit awareness of the differential characteristics or priorities between prefill and decode requests.
   • Version Used: The paper specifies v0.7.2 [25], which is stated to be the same basic implementation version upon which MoLink builds.
   • Representativeness: It serves as a strong baseline for systems that optimize memory and execution but lack explicit communication scheduling for the prefill/decode imbalance.

2. Helix [14]:

   • Description: Helix is another high-throughput serving system designed for distributed clusters. It focuses on optimal LLM deployment and request scheduling in heterogeneous environments.

   • Transmission Strategy: Helix sequentially sends prefill or decode volumes into a ZeroMQ message queue. ZeroMQ then handles the asynchronous sending of these messages. Similar to vLLM, Helix does not inherently differentiate or prioritize prefill and decode transmissions based on their distinct data volumes and latency sensitivities.

   • Representativeness: It represents systems that leverage robust asynchronous messaging libraries for distributed communication but may still suffer from transmission competition due to a lack of specialized prefill/decode scheduling.

     By comparing against these two baselines, the paper aims to highlight that while vLLM and Helix are effective at general distributed LLM serving, their lack of explicit communication optimization for the prefill/decode imbalance makes them susceptible to the transmission competition that MoLink specifically addresses.

6. Results & Analysis

6.1. Core Results Analysis

The evaluation demonstrates MoLink's effectiveness across various operational conditions. The results highlight its ability to significantly reduce TTFT, TPOT, and end-to-end latency by intelligently managing communication between prefill and decode requests.

6.1.1. Impact of Request Rate

The following are the results from Table 2 of the original paper:

• Overall Performance: MoLink consistently achieves the lowest TTFT, TPOT, and end-to-end latency across all tested request rates compared to vLLM and Helix. The maximum reduction is up to 46% (e.g., TTFT at 0.7 req/s for MoLink is 54% of vLLM's value, implying a 46% reduction).
• Impact of Workload:
  • Medium Workloads (e.g., 0.3 req/s): MoLink demonstrates its most significant improvements at medium request rates. For instance, at 0.3 req/s, MoLink reduces TTFT by 16% (84% of vLLM), TPOT by 23% (77% of vLLM), and end-to-end latency by 17% (83% of vLLM). This is because, at these workloads, prefill and decode requests are processed concurrently in a balanced manner, leading to pronounced transmission competition. MoLink's intelligent scheduling effectively resolves this competition.
  • High Workloads (e.g., 0.7 req/s): While still outperforming baselines, the benefits of MoLink decrease. At 0.7 req/s, TPOT for all systems (including MoLink) exceeds 3s, indicating an unacceptable serving scenario where the system is saturated. In such a congested state, the underlying transmission competition becomes less distinguishable as all requests are bottlenecked by overall resource availability, reducing the relative impact of MoLink's fine-grained scheduling.
  • Low Workloads (e.g., 0.1 req/s): The benefits also decrease at low workloads. When one type of request (e.g., prefill or decode) dominates the execution, or when requests are sparse, there is less opportunity for transmission competition to occur. Therefore, the specialized scheduling of MoLink has less impact.

6.1.2. Impact of Bandwidth

The following are the results from Table 3 of the original paper:

• Consistent Benefits: MoLink consistently shows improvements across a wide range of bandwidths (from 60 Mbps to 400 Mbps), maintaining smaller TTFT, TPOT, and end-to-end latency values than vLLM and Helix.
• Communication Overhead Persistence: Even as bandwidth increases (e.g., from 100 Mbps to 400 Mbps), which naturally mitigates transmission competition, MoLink's optimizations remain beneficial. This indicates that while higher bandwidth reduces the severity of communication overhead, the volume of data transferred (especially for long prefill prompts in scenarios like summarization tasks) is still substantial enough to warrant MoLink's chunking and scheduling approach.
• Relevance to Consumer-Grade GPUs: This is particularly relevant for consumer-grade GPUs often connected via standard networks, where bandwidth might be a more significant bottleneck than in specialized data centers.

6.1.3. Impact of Network Delay

The following are the results from Table 4 of the original paper:

• Decreasing Benefits with Increasing Delay: As network delay increases (from 10 ms to 50 ms), the performance gap between MoLink and the baselines tends to narrow, meaning MoLink's relative benefits decrease.
• Impact of Pipeline Bubbles: Higher network delay leads to more frequent and longer pipeline bubbles. When micro-batches are significantly delayed in arriving at the next server, the pipeline becomes less efficient, making it harder for MoLink's fine-grained scheduling to fully compensate. The system used a fixed number of 5 micro-batches in this experiment. The paper suggests that increasing the number of micro-batches could further mitigate the impact of delay.

6.2. Ablation Studies / Parameter Analysis

An ablation study was conducted to isolate the contribution of each proposed technique within MoLink.

The following are the results from Table 5 of the original paper:

• Baseline (No Optimizations): The scenario "No optimizations" likely represents a basic vLLM-like system without MoLink's specific enhancements, achieving TTFT of 3.56s, TPOT of 0.48s, and Latency of 107.45s.
• Individual Contributions:
  • "w/ chunk transmission": This row likely refers to MoLink without the chunk transmission algorithm, or perhaps a configuration where it's less optimally applied (the wording is slightly ambiguous but implies removal of this key feature). If "w/ chunk transmission" means only chunk transmission is enabled (or without it), then the relative increase in TTFT (from 3.30 to 3.37) and TPOT (0.41 to 0.44) and Latency (93.07 to 99.25) suggests that chunk transmission (when present in "All optimizations") is beneficial. The table label "w/ chunk transmission" likely means a configuration without the chunk transmission as designed in MoLink, reverting to a more basic approach, which leads to worse performance than "All optimizations". If interpreted this way, removing chunk transmission degrades performance (TTFT: 3.37s, TPOT: 0.44s, Latency: 99.25s).
  • "w/ micro-batch extending": Similarly, this row likely means MoLink without the micro-batch extending optimization. Removing this feature also degrades performance (TTFT: 3.60s, TPOT: 0.43s, Latency: 97.23s).
• Combined Effect:

  • The "All optimizations" row represents the full MoLink system, achieving the best performance (TTFT: 3.30s, TPOT: 0.41s, Latency: 93.07s).

  • The ablation study clearly shows that both individual techniques contribute to performance improvement.

  • For example, chunk transmission (by its absence, causing degradation) contributes to reducing TPOT from 0.44s to 0.41s (a reduction of approx. 6.8%). Micro-batch extending (by its absence, causing degradation) contributes to reducing TPOT from 0.43s to 0.41s (a reduction of approx. 4.6%).

  • Crucially, the combined effect is more significant than the sum of individual contributions. Compared to "No optimizations" (TPOT: 0.48s), chunk transmission reduces TPOT by 8.4% (0.48 to 0.44, assuming "w/ chunk transmission" means no chunking). Micro-batch extending reduces TPOT by 10.5% (0.48 to 0.43, assuming "w/ micro-batch extending" means no micro-batch extending). When both techniques are combined (All optimizations), they reduce TPOT from 0.48s to 0.41s, which is a 14.6% reduction. This demonstrates the synergistic efficiency of these optimizations when used together.

    The results confirm that both chunk transmission and micro-batch extending are vital components of MoLink and that their combined application yields the most substantial performance benefits.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces MoLink, an efficient distributed Large Language Model (LLM) serving system specifically designed for cost-effective deployment on consumer-grade GPUs. The core contribution of MoLink lies in its innovative approach to reconciling computation and communication by addressing the critical challenge of transmission competition arising from the imbalanced data transfer volumes between prefill and decode inference phases.

MoLink achieves this through:

1. A novel weighted-priority transmission scheduling algorithm that intelligently prioritizes small, frequent decode requests while preventing starvation of large, chunked prefill requests.

2. A just-in-time chunk determination algorithm that adaptively calculates the optimal size of prefill chunks to transmit, utilizing available network bandwidth without delaying decode operations.

3. An optimization to extend the number of micro-batches beyond the pipeline degree to maximize GPU utilization and minimize pipeline bubbles in bandwidth-constrained environments.

   Experimental evaluations against state-of-the-art baselines like vLLM and Helix demonstrate that MoLink significantly improves LLM serving performance. It achieves a maximum reduction of up to 46% in key metrics such as Time to First Token (TTFT), Time per Output Token (TPOT), and overall end-to-end latency, particularly benefiting medium workloads where transmission competition is most prevalent. These improvements are sustained across varying bandwidths and network delays, validating MoLink's efficacy in real-world distributed settings with consumer-grade hardware.

7.2. Limitations & Future Work

The authors acknowledge specific limitations of the current MoLink design and propose future research directions:

• Network Fluctuation Adaptation:
  • Limitation: The current MoLink design assumes network conditions are static. In reality, networks are highly dynamic, with bandwidth and latency constantly fluctuating.
  • Future Work: Future research could explore adaptive mechanisms for MoLink to dynamically adjust its scheduling and chunking strategies in response to real-time network fluctuations. This would ensure more consistent performance under varying network conditions.
• Fault Tolerance:
  • Limitation: The current design assumes that devices in the distributed cluster are reliable. However, in real-world deployments, especially when using potentially less stable consumer-grade GPUs or decentralized environments, device failures or faults are possible.
  • Future Work: Future work should investigate incorporating fault-tolerance mechanisms into MoLink. This would enhance system robustness and reliability, allowing it to continue operating effectively even if some devices fail.

7.3. Personal Insights & Critique

This paper presents a very practical and timely solution to a critical problem in the burgeoning field of LLM serving. The core insight—that prefill and decode have fundamentally different communication profiles and should be handled distinctly—is elegant and well-justified by the data.

Inspirations and Applications:

• Democratization of LLM Serving: The focus on consumer-grade GPUs is a significant step towards democratizing access to powerful LLM inference. By making serving more cost-efficient and accessible, MoLink could enable a wider range of individuals and smaller organizations to deploy custom LLMs without prohibitive infrastructure costs. This could foster more innovation and diverse applications.
• Edge AI and Hybrid Clouds: The principles of MoLink could be highly applicable to edge computing scenarios where LLMs might be deployed on local, less powerful hardware with varying network connectivity. Similarly, in hybrid cloud architectures, where some LLM components reside on-premises and others in the cloud, MoLink's communication-aware scheduling could optimize performance across these disparate environments.
• Beyond LLMs: The concept of chunking large data transfers and prioritized scheduling based on the nature of the data (e.g., bursty vs. streamable, latency-sensitive vs. batch-tolerant) could be generalized to other distributed machine learning tasks or even general-purpose distributed computing where heterogeneous communication patterns exist.

Potential Issues/Unverified Assumptions/Areas for Improvement:

• Profiling Accuracy and Overhead: The just-in-time chunk determination relies on profiling data (Td(x)) for execution times and accurate estimation of Tm and To. The accuracy of these estimations is crucial. If network conditions change rapidly, or if the profiling data becomes stale, the chunk determination might become sub-optimal. The overhead of continuously gathering and using this profiling data also needs to be considered.

• Scalability of $W$ (Waiting Weight): The fixed $max waiting weight N=30$ (in Algorithm 1) is a heuristic. While effective for the tested scenarios, its optimality might vary significantly with different LLM sizes, network topologies, request arrival patterns, or even the type of workload (e.g., conversational vs. long document summarization). An adaptive $N$ could be a valuable future extension.

• Complexity of Chunking Logic: While beneficial, the chunking determination algorithm involves several cases and predictions. Implementing and maintaining this logic robustly in a dynamic distributed system can be complex. There might be scenarios (e.g., unexpected network congestion) where the predictions are inaccurate, leading to temporary performance degradation.

• Overhead of Micro-Batch Extension Search: The paper mentions emulating the number of micro-batches to find an optimal value. While the search space is limited ($N$ to 2N), the overhead of this emulation (e.g., how often it runs, how it affects live service) is not explicitly discussed. Continuous online optimization might be beneficial but also introduce overhead.

• Heterogeneity of Consumer-Grade GPUs: While the paper mentions supporting Windows PCs and Linux servers, the full extent of heterogeneity (e.g., mixing RTX 3090 with RTX 4090, different memory sizes, different PCIe generations) and how MoLink specifically adapts to these differences is not deeply explored beyond generic pipeline parallelism. The Td(n) function being per server accounts for some of this, but network-specific heterogeneity might need more fine-grained adaptation.

  Overall, MoLink represents a strong step forward in making distributed LLM serving more efficient and accessible. Its focused approach on communication bottlenecks, often overlooked in favor of computational optimizations, addresses a critical practical challenge.