1. Bibliographic Information

1.1. Title

The central topic of the paper is "LoRAFusion: Efficient LoRA Fine-Tuning for LLMs," which focuses on improving the efficiency of Low-Rank Adaptation (LoRA) for fine-tuning Large Language Models (LLMs).

1.2. Authors

The authors are Zhanda Zhu, Qidong Su, Yaoyao Ding, Kevin Song, Shang Wang, and Gennady Pekhimenko. Their affiliations include the University of Toronto, Vector Institute, and NVIDIA, suggesting a strong background in both academic research and industry application in the fields of machine learning systems, distributed computing, and LLMs.

1.3. Journal/Conference

The paper is published at the "21st European Conference on Computer Systems (EUROSYS '26), April 27-30, 2026, Edinburgh, Scotland UK." EuroSys is a highly reputable and influential conference in the field of computer systems, known for publishing cutting-edge research in operating systems, distributed systems, and networked systems. Publication at such a venue indicates significant contributions and rigorous peer review.

1.4. Publication Year

The paper was published at (UTC) 2025-09-30T19:26:22.000Z, with the ACM reference format indicating the conference year as 2026.

1.5. Abstract

This paper introduces LoRAFusion, an efficient system designed to accelerate LoRA fine-tuning for Large Language Models (LLMs). The authors identify two key inefficiencies in existing LoRA fine-tuning systems: substantial runtime overhead from redundant memory accesses on large activation tensors, and the missed opportunity to concurrently fine-tune multiple independent LoRA adapters that share the same base model on the same set of GPUs.

LoRAFusion addresses these issues through a two-pronged approach. At the kernel level, it proposes a graph-splitting method to fuse memory-bound operations. This design eliminates unnecessary memory accesses and maintains the performance of compute-bound General Matrix Multiplications (GEMMs) without incurring recomputation or synchronization costs. At the scheduling level, LoRAFusion introduces an adaptive batching algorithm for multi-job fine-tuning. This algorithm groups LoRA adapters to stagger batch execution across jobs and then uses a bin-packing problem formulation to generate balanced, dependency-aware microbatches.

The system achieves significant performance improvements, with up to $1.96\times$ ($1.47\times$ on average) end-to-end speedup compared to Megatron-LM and up to $1.46\times$ ($1.29\times$ on average) improvement over mLoRA, which is the state-of-the-art multi-LoRA fine-tuning system. The proposed fused kernel alone provides up to $1.39\times$ ($1.27\times$ on average) kernel performance improvement and can be used as a drop-in replacement in existing LoRA systems. The authors open-source LoRAFusion at https://github.com/CentML/lorafusion.

1.6. Original Source Link

The official source link is https://arxiv.org/abs/2510.00206, and the PDF link is https://arxiv.org/pdf/2510.00206v1.pdf. This indicates it is a preprint, likely submitted to arXiv before its formal publication at EuroSys '26.

2. Executive Summary

2.1. Background & Motivation

The rapid advancement of pre-trained Large Language Models (LLMs) like GPT and LLaMa has made fine-tuning these models essential for adapting them to specific, personalized, or domain-specific tasks (e.g., biomedical analysis, specialized chatbots). However, traditional full-model fine-tuning, which updates all model parameters, demands exorbitant hardware resources. For instance, fine-tuning LLaMa-3.1-70B requires roughly 1120GB of GPU memory just for model states (parameters, gradients, optimizer states), making it prohibitively expensive.

To counter this, Parameter-Efficient Fine-Tuning (PEFT) methods have emerged. These methods freeze the majority of pre-trained LLM parameters and only train a small subset of injected trainable parameters, called adapters. Among PEFT methods, Low-Rank Adaptation (LoRA) has become a leading technique due to its simplicity and effectiveness. LoRA introduces a low-rank decomposition of weight updates, drastically reducing the number of trainable parameters. For example, LLaMa-3.1-70B with LoRA rank 16 requires only 142GB of GPU memory, a substantial reduction while maintaining model quality.

Despite LoRA's algorithmic benefits in reducing memory footprint, current LoRA fine-tuning systems mainly reuse optimizations from traditional full-model fine-tuning. The authors identify two critical inefficiencies that these systems fail to address:

1. Substantial Runtime Overhead from Redundant Memory Access: Although LoRA adapters add less than 1% of parameters, they introduce significant runtime overhead. Profiling shows a ~40% reduction in training throughput compared to a frozen linear layer. This is largely due to memory-bandwidth-bound operations in the small LoRA projection layers and repeated loading/storing of large activation tensors, leading to a ~2.64x increase in GPU global memory traffic.

2. Missed Opportunities for Multi-Job Fine-Tuning: Existing systems typically fine-tune each LoRA adapter independently, even when sharing the same base model. This overlooks the potential to concurrently fine-tune multiple adapters on the same GPUs, which could yield performance gains by reducing distributed training overhead (e.g., pipeline bubbles, communication, load imbalance). While mLoRA attempts multi-LoRA fine-tuning, it has limitations, such as not addressing kernel-level memory bottlenecks, failing to balance variable sequence lengths across GPUs, and relying on inefficient communication for pipeline parallelism.

   The paper's innovative idea is to address these inefficiencies at both the low-level kernel execution and high-level job scheduling. By doing so, LoRAFusion aims to unlock the full potential of LoRA for efficient, multi-job fine-tuning on modern GPU clusters.

2.2. Main Contributions / Findings

LoRAFusion makes the following primary contributions:

1. Identification of Key Bottlenecks: The paper rigorously identifies and quantifies the two core limitations in existing LoRA fine-tuning systems: high runtime overhead due to redundant memory accesses (bottlenecked by memory bandwidth) and missed opportunities for optimizing multi-job training scenarios, leading to distributed parallelism overhead and GPU load imbalance.

2. Novel Kernel-Level Optimization (FusedLoRA & FusedMultiLoRA): It proposes a graph-splitting method that fuses memory-bound operations within the LoRA computation graph. This design strategically splits the graph at small, intermediate tensors to eliminate unnecessary memory accesses without recomputing or synchronizing large tensors, while preserving the optimal performance of compute-bound General Matrix Multiplications (GEMMs).

3. Adaptive Job-Level Scheduling (Multi-LoRA Scheduler): It introduces a hierarchical adaptive batching algorithm for multi-job fine-tuning. This scheduler first groups LoRA adapters to intentionally stagger batch execution across jobs, and then employs a two-stage Mixed Integer Linear Programming (MILP)-based bin-packing algorithm to generate balanced, dependency-aware microbatches. This strategy significantly improves GPU load balance and reduces pipeline bubbles in distributed training.

4. Comprehensive System Implementation and Evaluation: The authors implement LoRAFusion on top of Megatron-LM and evaluate it across diverse LLMs (LLaMa-3.1-8B, Qwen-2.5-32B, LLaMa-3.1-70B), realistic datasets, and NVIDIA H100/L40S GPUs.

   The key conclusions and findings are:

• LoRAFusion achieves substantial end-to-end speedups: up to $1.96\times$ ($1.47\times$ on average) compared to Megatron-LM and up to $1.46\times$ ($1.29\times$ on average) over mLoRA.
• The FusedLoRA kernel alone delivers an average $1.27\times$ (up to $1.39\times$) kernel performance improvement and can serve as a plug-and-play replacement in existing LoRA systems.
• The kernel fusion effectively reduces GPU global memory read/write traffic by $34\%-37\%$.
• The job-level scheduling significantly reduces pipeline bubbles from 44.17% (single adapter) to 11.09% (four adapters), and improves GPU utilization from 65% to 89%.
• The system demonstrates strong scalability and robustness, even in challenging heterogeneous multi-job settings. These findings collectively solve the problem of inefficient LoRA fine-tuning by optimizing both low-level memory access and high-level distributed training coordination, making LLM adaptation more accessible and cost-effective.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

3.1.1. Large Language Models (LLMs)

Large Language Models (LLMs) are deep learning models, typically based on the transformer architecture, trained on vast amounts of text data to understand and generate human-like text. They excel at tasks like text generation, question answering, and code generation. Their pre-training involves learning general language patterns and knowledge, while fine-tuning adapts them to specific downstream tasks or domains. Examples include GPT and LLaMa.

3.1.2. Low-Rank Adaptation (LoRA)

LoRA (Low-Rank Adaptation) is a leading Parameter-Efficient Fine-Tuning (PEFT) method. Instead of fine-tuning all parameters of a pre-trained LLM, LoRA freezes the original, large pre-trained weights. For specific layers (typically linear layers), it injects a small, trainable adapter composed of two low-rank linear layers.

Formally, for a pre-trained weight matrix $W \in \mathbb{R}^{k \times n}$ (where $k$ is the input dimension and $n$ is the output dimension), LoRA introduces two new trainable matrices: $A \in \mathbb{R}^{k \times r}$ and $B \in \mathbb{R}^{r \times n}$. Here, $r$ is the LoRA rank, and $r \ll \min(n, k)$. The update to the original layer's output is then represented as the product of these two low-rank matrices, scaled by a constant $\alpha$. The output $Y$ of a LoRA-equipped linear layer is calculated as: $$Y = XW + \alpha SB = XW + \alpha (\widehat{X}A)B$$ Where:

• $X \in \mathbb{R}^{m \times k}$ is the input tensor, where $m$ is the number of aggregated tokens (batch size multiplied by sequence length).

• $W \in \mathbb{R}^{k \times n}$ represents the frozen (non-trainable) base model weights.

• $\widehat{X} \in \mathbb{R}^{m \times k}$ is the input tensor after dropout (a regularization technique where random elements of the input are set to zero during training to prevent overfitting).

• $A \in \mathbb{R}^{k \times r}$ is the first trainable LoRA weight matrix (down-projection).

• $B \in \mathbb{R}^{r \times n}$ is the second trainable LoRA weight matrix (up-projection).

• $S = \widehat{X}A$ is an intermediate result.

• $\alpha$ is a constant scalar that scales the output of the LoRA branch.

  The key benefit is that only $A$ and $B$ (and their corresponding gradients and optimizer states) need to be stored and updated, dramatically reducing memory usage and computational cost compared to fine-tuning $W$.

3.1.3. Parameter-Efficient Fine-Tuning (PEFT)

PEFT methods are a family of techniques designed to adapt large pre-trained models to new tasks with minimal computational and memory resources. Instead of fine-tuning all parameters, PEFT methods typically introduce a small number of new, trainable parameters (adapters) or modify existing ones in a parameter-efficient way, while keeping the bulk of the pre-trained model frozen. LoRA is one of the most popular PEFT methods. Others include Prefix-tuning, Prompt tuning, and AdapterDrop.

3.1.4. Distributed Training

Distributed training involves training a model across multiple computational devices (e.g., GPUs) or multiple machines (nodes) to handle large models or datasets that cannot fit into a single device's memory or to accelerate training. Common strategies include:

• Data Parallelism (DP): The model is replicated on each GPU, and the training data is partitioned across GPUs. Each GPU computes gradients on its data subset, and then gradients are aggregated (e.g., averaged) across all GPUs to update the model copies.
• Fully Sharded Data Parallelism (FSDP) / ZeRO-3: An advanced form of data parallelism where not just data, but also model states (parameters, gradients, and optimizer states), are sharded (partitioned) across GPUs. This drastically reduces the memory footprint per GPU, allowing larger models to be trained. Model states are only gathered to a GPU when needed for computation.
• Tensor Parallelism (TP): Individual layers (e.g., linear layers or attention layers) within the model are split across multiple GPUs. For instance, a weight matrix $W$ might be split into $W_1$ and $W_2$ across two GPUs, and the input $X$ would be distributed accordingly ($X_1, X_2$). Computations are performed in parallel, and intermediate results are communicated between GPUs to form the complete output.
• Pipeline Parallelism (PP): The model's layers are divided into sequential stages, and each stage is assigned to a different GPU or device. Data batches are then processed in a pipeline fashion, where different GPUs work on different stages of the model for different microbatches simultaneously. This reduces communication overhead but can introduce pipeline bubbles (idle time) if stages are imbalanced or microbatches are not perfectly aligned.

3.1.5. Kernel Fusion

Kernel fusion is an optimization technique that merges multiple elementary computational operations (called kernels) into a single, larger kernel. This reduces overhead associated with:

• Memory Transfers: By fusing operations, intermediate results can remain in faster on-chip memory (e.g., shared memory or registers) rather than being written to and read from slower global memory (DRAM).
• Kernel Launch Overhead: Each kernel launch incurs a small overhead on the GPU. Fusing multiple operations into one reduces the number of launches. Examples include FlashAttention, which fuses attention operations.

3.1.6. On-the-fly Data Packing

In LLM fine-tuning, training data often consists of sequences of variable lengths.

• Traditional Batch Padding: Shorter sequences are padded with special padding tokens to match the length of the longest sequence in a batch (Figure 2(a)). This leads to wasted computations on padding tokens.
• Dataset Pre-packing: Sequences are pre-processed and concatenated into fixed-length blocks offline (Figure 2(b)). While efficient, it can introduce variable sample counts per batch and might affect training stability or randomness if not handled carefully.
• On-the-fly Packing: Sequences are dynamically concatenated within each batch to fill a fixed token capacity, avoiding wasted computations from padding while maintaining a consistent number of actual training samples per batch (Figure 2(c)). This method is commonly adopted for its effectiveness.

3.2. Previous Works

3.2.1. PEFT Library

The PEFT Library [55] from Hugging Face is a popular open-source library that provides implementations of various Parameter-Efficient Fine-Tuning methods, including LoRA. It simplifies the application of PEFT techniques to pre-trained transformer models, making it a common tool for researchers and practitioners. LoRAFusion integrates with this library for model architecture and LoRA adaptation.

3.2.2. Megatron-LM

Megatron-LM [81] is a state-of-the-art distributed training framework developed by NVIDIA, designed for training very large transformer models. It implements various parallelization strategies, including Tensor Parallelism (TP) and Pipeline Parallelism (PP), to enable efficient training across many GPUs and nodes. LoRAFusion builds its multi-adapter pipeline parallelism on top of Megatron-LM.

3.2.3. mLoRA

mLoRA [98] is a prior system designed for multi-LoRA fine-tuning. It focuses on grouping multiple LoRA adapters that share the same base model from separate tasks into a single batched operation. Its primary motivation was to reduce the memory footprint of replicated pre-trained models and improve training efficiency, particularly in pipeline parallelism by filling pipeline bubbles with independent groups of samples from different adapters. However, LoRAFusion critically analyzes mLoRA's limitations, which include:

• Reliance on generic LoRA kernels that are bottlenecked by redundant memory accesses.
• Lack of handling for load imbalance from variable sequence lengths in real workloads.
• Narrow focus on Pipeline Parallelism and reliance on inefficient CPU-based communication, limiting scalability.

3.2.4. LoRA Serving Systems (Punica, S-LoRA, dLoRA)

Works like Punica [9], S-LoRA [79], and dLoRA [94] are system-level optimizations for LoRA, but specifically for inference and serving scenarios. They focus on efficiently serving thousands of concurrent LoRA adapters by batching requests to increase arithmetic intensity during autoregressive single-token decoding. While they also use multi-LoRA grouping, their motivations and challenges differ significantly from fine-tuning, which already processes full sequences with sufficient arithmetic intensity.

3.2.5. Kernel Fusion Works (FlashAttention, Triton, Mirage)

FlashAttention [13] is a prominent example of kernel fusion that significantly improves the efficiency of the attention mechanism in transformers by fusing operations to reduce memory transfers. Triton [84] is an open-source domain-specific language and compiler for writing highly optimized GPU kernels with fine-grained control, which LoRAFusion uses to implement its FusedLoRA and FusedMultiLoRA kernels. Mirage [95] explores multi-level superoptimizers for tensor programs, showing benefits for LoRA serving, but it doesn't fully address the complexities of LoRA fine-tuning (e.g., dropout, backward computation, multi-LoRA kernels).

3.3. Technological Evolution

The evolution of fine-tuning LLMs has progressed through several stages:

1. Full-Model Fine-Tuning: Initial approaches involved updating all parameters of a pre-trained LLM. While effective for task adaptation, this method is extremely memory-intensive and computationally expensive, requiring vast GPU memory and powerful distributed training setups.
2. Parameter-Efficient Fine-Tuning (PEFT): To address resource constraints, PEFT methods emerged. These techniques, such as Adapter-tuning, Prefix-tuning, and LoRA, reduce the number of trainable parameters by keeping most of the base model frozen and introducing small, task-specific adapters. LoRA, in particular, gained popularity due to its simplicity and effectiveness, significantly reducing memory footprint for gradients and optimizer states.
3. System-Level Optimizations for PEFT: As PEFT methods became prevalent, the focus shifted to optimizing the systems that execute them. Initially, these systems reused distributed training techniques (like FSDP, TP, PP) developed for full-model pre-training. However, these generic optimizations often failed to fully leverage the unique characteristics of lightweight LoRA adapters.
4. Specialized Optimizations for LoRA (Inference): The inference side of LoRA saw specialized system optimizations (e.g., Punica, S-LoRA) to efficiently serve multiple adapters concurrently, primarily by batching requests during autoregressive decoding to improve GPU utilization.
5. Specialized Optimizations for LoRA (Fine-Tuning): The fine-tuning side, however, lagged. While mLoRA started exploring multi-adapter fine-tuning, it still relied on generic kernel performance and lacked robust scheduling for variable sequence lengths. This is where LoRAFusion enters the timeline, providing specialized kernel fusion and adaptive scheduling specifically for LoRA fine-tuning, addressing its unique memory-bandwidth-bound bottlenecks and distributed parallelism challenges.

3.4. Differentiation Analysis

LoRAFusion differentiates itself from prior works by addressing two fundamental inefficiencies in LoRA fine-tuning that existing systems largely overlook:

1. Kernel-Level Efficiency for LoRA Operations:

   • Differentiation from Megatron-LM and standard LoRA implementations: Megatron-LM and other frameworks (like those using Hugging Face PEFT Library) rely on generic kernel implementations for LoRA. As shown in the paper, these are memory-bandwidth-bound due to redundant memory accesses on large activation tensors, leading to significant overhead despite LoRA's small parameter count. torch.compile also offers limited benefits.
   • LoRAFusion's innovation: LoRAFusion introduces FusedLoRA and FusedMultiLoRA kernels that employ a novel graph-splitting fusion strategy. This strategy fuses memory-bound operations (e.g., dropout, element-wise additions) around full-sized activation tensors without recomputing them or introducing expensive synchronization. It strategically splits the graph at small, intermediate LoRA rank tensors, preserving the optimal performance of compute-bound GEMMs (the base model's large matrix multiplications). This directly tackles the memory traffic bottleneck, reducing global memory read/write by $34\%-37\%$.

2. Adaptive Scheduling for Multi-Job LoRA Fine-Tuning:

   • Differentiation from Megatron-LM (single-job focus): Megatron-LM (with FSDP or PP) is designed for single-job, large-scale training and does not natively support efficient concurrent fine-tuning of multiple LoRA adapters. Running multiple jobs would typically mean sequential execution or inefficient resource partitioning.
   • Differentiation from mLoRA (incomplete multi-job optimization): While mLoRA attempts multi-LoRA fine-tuning by batching adapters and filling pipeline bubbles, it has key limitations:
     • It still uses generic LoRA kernels, ignoring the memory-bandwidth bottleneck.
     • It fails to adequately address load imbalance caused by variable sequence lengths in real datasets, which can lead to pipeline stalls and idle GPUs. mLoRA assumes uniform adapter grouping and schedules based on memory, not sequence length variability.
     • Its BatchLoRA kernel reduces kernel launch overhead but doesn't solve the core memory access bottleneck.
     • It's narrowly focused on Pipeline Parallelism and relies on inefficient CPU-based communication.
   • LoRAFusion's innovation: LoRAFusion introduces a sophisticated Multi-LoRA Scheduler that goes beyond simple adapter grouping. It uses a two-stage hierarchical strategy:

     1. Adapter Grouping with Bubble Lemma consideration: Groups adapters based on sample length distributions to ensure global batches from the same adapter are sufficiently spaced, satisfying pipeline parallelism dependencies.

     2. Adaptive Batching with Two-Stage MILP: Solves a bin-packing problem using Mixed Integer Linear Programming (MILP) (with greedy fallback) to create balanced, dependency-aware microbatches that minimize the total number of microbatches and maximize space in underfilled ones. This significantly reduces pipeline bubbles (from 34.11% in mLoRA to 11.09% in LoRAFusion) and improves GPU load balance by intelligently packing variable-length samples across jobs.

        In essence, LoRAFusion provides a more holistic and deeply optimized solution for LoRA fine-tuning by tackling both the micro-level (kernel execution) and macro-level (job scheduling) inefficiencies, leading to superior end-to-end throughput and resource utilization.

4. Methodology

4.1. Principles

The core idea of LoRAFusion is to address the identified inefficiencies in LoRA fine-tuning by applying multi-level fusion:

1. Kernel-level Optimization (FusedLoRA & FusedMultiLoRA): The primary principle here is to reduce redundant memory accesses to large activation tensors by fusing memory-bound operations within the LoRA computation graph. This is achieved through a graph-splitting strategy that isolates compute-bound General Matrix Multiplications (GEMMs) (from the base model) to preserve their optimal performance, while combining related memory-bound operations (from the LoRA branch and element-wise operations) that share large inputs/outputs. This strategic fusion minimizes memory traffic without incurring the costs of recomputation or synchronization for large intermediate tensors.

2. Job-level Scheduling (Multi-LoRA Scheduler): The principle is to maximize GPU utilization and mitigate distributed parallelism overhead by intelligently scheduling multiple independent LoRA fine-tuning jobs. This involves adaptively batching samples from different jobs into balanced microbatches, considering sequence length variability and respecting data dependencies (especially in pipeline parallelism). The goal is to reduce pipeline bubbles and load imbalance across GPUs, which are major bottlenecks in multi-GPU and multi-node training.

   These two levels of optimization work synergistically. The fused kernels make individual LoRA operations more efficient, while the scheduler ensures that these efficient operations are fed with optimally packed data across multiple concurrent jobs, maximizing system throughput.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. FusedLoRA and FusedMultiLoRA

4.2.1.1. Design Considerations and Challenges

Traditional kernel fusion aims to merge multiple operations into a single GPU kernel to reduce memory transfers and kernel launch overhead. However, applying full fusion to LoRA modules presents specific challenges:

• Sensitivity of Compute-Bound GEMMs: The base model's GEMM operation ($Y_1 = XW$) is typically compute-bound and highly optimized. Fusing other operations with it could disrupt its optimal tiling strategies and GPU resource usage (e.g., registers, shared memory), leading to performance degradation.

• Recomputation or Synchronization Overhead: Fusing operations with producer-consumer dependencies, such as the down-projection $\widehat{X}A$ followed by the up-projection $(\widehat{X}A)B$, might require recomputing intermediate results (costly for large inputs) or introducing thread block synchronization (adds coordination overhead).

  A good fusion strategy must therefore reduce memory access without compromising the performance of compute-bound operations and avoid expensive synchronization or recomputation.

4.2.1.2. Full Graph Fusion vs. Split Graph Fusion

The paper explores different strategies for handling the intermediate tensor $S = \widehat{X}A$ (the output of the LoRA down-projection) in the forward pass, as illustrated in Figure 9.

• Full Graph Fusion (Recompute S): One option is to recompute $S$ within each tile of a fully fused kernel. However, this repeatedly loads the entire $A$ matrix, which becomes expensive for large batch sizes $M$.

• Full Graph Fusion (Synchronize to Share S): Another option is to fuse the computation and use synchronization across thread blocks to share $S$. Here, a single Mtile (a block along the token dimension) computes the intermediate $S$ tiles and writes them to global memory, while other tiles wait. This adds coordination overhead.

• Split Graph Fusion (LoRAFusion's approach): LoRAFusion adopts a third approach: explicitly storing and reloading $S$ from GPU global memory. This is feasible because $S$ (of size $m \times r$) is much smaller than other tensors (which are $m \times k$ or $m \times n$) due to the small LoRA rank $r$. The cost of reading and writing this small tensor is low. By splitting the graph at $S$, LoRAFusion avoids both recomputation for large inputs and synchronization overhead, while still reducing memory traffic associated with full-sized activation tensors. This also preserves GPU resources for optimal tiling of the compute-bound XW operation.

  The following figure (Figure 9 from the original paper) illustrates the full graph fusion approach versus the split graph fusion approach:

  该图像是展示 LoRA 模块在前向传递中的融合策略的示意图，比较了完全图融合方法与分裂图融合方法。图中展示了不同选项，包括融合与重计算、融合与同步等操作，以优化内存访问和计算效率。

4.2.1.3. FusedLoRA Design

The FusedLoRA kernels are designed to fuse memory-heavy operations while maintaining the efficiency of compute-bound operations. Figure 10 illustrates this design for both the forward and backward passes.

The following figure (Figure 10 from the original paper) illustrates the LoRA kernel design:

Forward Pass (Figure 10(a)):

1. Operation $\pmb{\mathbb{0}}$ (Dropout + Down-projection): This kernel fuses the dropout operation on the input tensor $X$ to produce $\widehat{X}$ with the down-projection $\widehat{X}A$. By doing so, it eliminates the need to load the full-sized activation tensor $X$ twice and saves an intermediate write/read of $\widehat{X}$ to global memory. The output of this fused kernel is the intermediate tensor $S = \widehat{X}A$.
2. Operation $\pmb{\theta}$ (Base GEMM + LoRA Up-projection + Addition): This kernel combines the compute-bound base model GEMM ($Y_1 = XW$) with the memory-bound LoRA up-projection ($Y_2 = \alpha SB$) and the final element-wise addition of their results ($Y = Y_1 + Y_2$). This fusion is crucial. It eliminates redundant memory operations by directly accumulating the partial results of XW and $\alpha SB$. Specifically, it saves one read and one write of the full-sized output tensor $Y_1$ (or $Y_2$) by performing the addition within the same kernel. This is achieved without affecting the base GEMM performance, as the fusion happens after the XW computation. The small $S$ tensor (the output of $\pmb{\mathbb{0}}$) is read into this kernel.

Backward Pass (Figure 10(b)): The backward pass similarly applies fusion principles to efficiently compute gradients.

1. Operation $\otimes$ (Gradient of S + Gradient of B): This kernel fuses the computation of the gradient of $S$ ($\nabla S$) and the gradient of $B$ ($\nabla B$) from the output gradient dY. By operating directly on dY and $S$, it eliminates the need to reload dY for separate gradient computations, reducing memory traffic.

2. Operation $\bullet$ (Gradient of A): This operation remains separate. It computes the gradient of $A$ ($\nabla A$) using the masked input $\widehat{X}$ and $\nabla S$. Since it operates on the relatively small masked input and intermediate gradient, fusion provides minimal additional benefit here.

3. Operation $\pmb{\mathbb{6}}$ (Gradient of Base Model + Gradient of LoRA Path + Addition): This kernel horizontally fuses the compute-intensive gradient computation for the base model weights $W$ ($\nabla W$) with the memory-bound gradient computation from the LoRA path and the addition of their partial gradients. Similar to the forward pass, this prevents redundant reads and writes of partial output gradients ($dY_1$ and $dY_2$) by directly accumulating them, while preserving the performance of the base model's gradient GEMM.

   The key insight is to identify operations that share large activation tensors ($\pmb{\mathbb{0}}$, $\pmb{\theta}$, $\otimes$, and $\pmb{\mathbb{6}}$) and fuse them. This significantly reduces memory bottlenecks while allowing compute-bound operations to use optimal tiling strategies.

4.2.1.4. Extending to FusedMultiLoRA

To support concurrent fine-tuning of multiple LoRA adapters, FusedLoRA is extended to FusedMultiLoRA. This allows the fused kernels to operate on mixed-adapter batches from different jobs efficiently.

As shown in Figure 11, tile-level routing is employed:

• Each input Mtile (a block of tokens) is tagged with an adapter ID and its configuration (e.g., LoRA rank, scaling factor $\alpha$, dropout ratio).

• This information is stored in a lightweight lookup table.

• During execution, the frozen model computation (XW) is shared across all tokens, regardless of their adapter.

• However, adapter-specific logic (e.g., applying $A$ and $B$ matrices, scaling, dropout) is applied dynamically per Mtile. For each (Mtile, Ntile) of the output, the kernel loads the appropriate $A$ and $B$ matrices based on the adapter ID associated with that Mtile.

• In the backward pass, the same mapping mechanism is used to route gradients to their respective adapters without interference.

  This tile-level routing enables efficient execution of heterogeneous adapters within a single fused kernel, avoiding redundant kernel launches per adapter and maintaining high GPU utilization across multiple jobs. The system dynamically chooses between FusedLoRA (if only one adapter is in the batch) and FusedMultiLoRA (for multiple adapters).

The following figure (Figure 11 from the original paper) illustrates FusedMultiLoRA in the forward pass:

该图像是多种 LoRA 适配器的前向传播示意图，展示了 MultiLoRA A 矩阵和 B 矩阵的路由关系。映射查找表在其中起到关键作用，确保适配器间的信息传递与高效调度。

4.2.2. Multi-LoRA Scheduler

The Multi-LoRA Scheduler in LoRAFusion orchestrates the grouping of adapters and adaptive batching of their samples across multiple fine-tuning jobs to optimize GPU load balance and minimize distributed parallelism overhead. The overall workflow is depicted in Figure 12.

The following figure (Figure 12 from the original paper) illustrates the Multi-LoRA adapter scheduling workflow:

该图像是示意图，展示了多LoRA适配器的调度工作流程。顶部展示了根据序列长度统计的适配器分组，中间部分显示了微批次创建的双阶段MILP优化，而底部则展示了对不足填充微批次的跨批次合并。

4.2.2.1. Granularity

The scheduling operates at the global batch level. Each adapter's dataset is conceptually divided into global batches based on a user-specified global batch size. The scheduler then aggregates all samples belonging to the same global batch index across all active adapters and packs them into multiple microbatches for actual processing.

4.2.2.2. Bubble Lemma & Adapter Grouping

A critical challenge in pipeline parallelism is respecting data dependencies between consecutive global batches. Specifically, for an adapter $i$, if a sample $s$ from global batch $j$ completes its forward pass in microbatch $k$, its backward pass will only begin after S-1 other microbatches (where $S$ is the number of pipeline stages) complete their forward passes. To ensure correctness, no sample from global batch $j+1$ of the same adapter can start its forward pass before microbatch k+S-1 (i.e., before sample $s$'s backward pass completes). This is termed the bubble lemma.

To manage this, LoRAFusion first groups LoRA adapters before batching samples:

• Strict Ordering between Groups: Adapters are grouped such that there is strict ordering in their execution. This creates natural gaps or staggering between global batches from different adapter groups.
• Flexible Merging within Groups: Within each group, samples can be flexibly merged into microbatches.
• Head-Tail Pairing for Load Balance: To enhance load balance within groups, adapters are sorted by their mean token length. Short-sequence adapters are then paired with long-sequence adapters. This head-tail pairing strategy aims to create more balanced microbatches within a group. This grouping approach balances the need to respect pipeline dependencies with the desire for flexible batching to improve load balance.

4.2.2.3. Data Batching with Two-Stage MILP

After adapter grouping, the scheduler solves a bin-packing problem to pack samples into microbatches, each constrained by a fixed token capacity. The goal is twofold: (i) minimize the total number of microbatches needed to pack all samples, and (ii) make the smallest microbatch as empty as possible to facilitate merging in later stages.

The paper uses a two-stage Mixed Integer Linear Programming (MILP) formulation, as outlined in Algorithm 1. For notation:

• $P$: Padding multiple, a user-specified parameter to pad the sequence length of samples from the same adapter to a multiple of $P$ (e.g., 64 or 128).
• $x_{s,b} \in \{0, 1\}$: Binary variable indicating if sample $s$ is assigned to bin (microbatch) $b$.
• $k_{a,b} \in \mathbb{N}$: The number of padded multiples contributed by adapter $a$ in bin $b$.
• $z_b \in \{0, 1\}$: Binary variable indicating if bin $b$ is used.

Algorithm 1: Data Batching & Merging (Per Group)

1 foreach global batch b in parallel do
2   (Bg, {mi}) ← GreedyPacking(b, C) // Greedy fallback as baseline
3   B* ← MILP_MinBins(b, C, timeout = t) // Stage 1: minimize number of microbatches
4   if B* ≥ Bg then
5     B* ← Bg
6   end
7   {B1 ← MILP_MinSmallestBin(b, B*, C, timeout = t)} // Stage 2: minimize smallest bin tokens
8   if B* = Bg and {m_i*} ≥ {m_i} then
9     return GreedyPacking(b,C)
10  end
11 end
12 foreach consecutive batch pairs (b, b+1) do
13   Shift tokens from b+1 into b if bubble lemma is preserved
14 end
15 VerifyAndFix(schedule) // Insert no-ops where needed
16 return Scheduled microbatches

Stage 1: Minimize the Number of Microbatches (MILP_MinBins) This stage aims to find the minimum number of microbatches required to pack all samples within a global batch. The optimization problem is formulated as: $$\begin{array}{cl} \displaystyle \operatorname*{argmin}_{\boldsymbol{x}_{s,b}, \boldsymbol{k}_{a,b}, \boldsymbol{z}_b} & \displaystyle \sum_{b=1}^{B} z_b \\ \mathrm{s.t.} & \displaystyle z_{b+1} \leq z_b \quad \forall b < B \\ & \displaystyle \sum_{b=1}^{B} x_{s,b} = 1 \quad \forall s \in \mathrm{samples} \\ & \displaystyle \sum_{s \in \mathrm{adapter}(a)} \mathrm{len}(s) \cdot x_{s,b} \leq k_{a,b} \cdot P \quad \forall a, b \\ & \displaystyle z_b \leq \sum_{a} k_{a,b} \cdot P \leq \mathrm{capacity} \cdot z_b \quad \forall b \end{array}$$ Where:

• $\sum_{b=1}^{B} z_b$: The objective function, which minimizes the total number of used bins $B$.
• $z_{b+1} \leq z_b \quad \forall b < B$: Ensures that used bins are contiguous from the start (i.e., if bin $b$ is used, bin b-1 must also be used).
• $\sum_{b=1}^{B} x_{s,b} = 1 \quad \forall s \in \mathrm{samples}$: Ensures each sample $s$ is assigned to exactly one bin $b$.
• $\sum_{s \in \mathrm{adapter}(a)} \mathrm{len}(s) \cdot x_{s,b} \leq k_{a,b} \cdot P \quad \forall a, b$: This constraint ensures that for each adapter $a$ and bin $b$, the total length of samples from $a$ assigned to $b$ does not exceed its padded contribution $k_{a,b} \cdot P$.
• $z_b \leq \sum_{a} k_{a,b} \cdot P \leq \mathrm{capacity} \cdot z_b \quad \forall b$: This links the usage of bin $b$ ($z_b$) to its total token count. If $z_b=1$, the bin must contain tokens, and its total padded token count ($\sum_{a} k_{a,b} \cdot P$) must not exceed the token capacity of a microbatch. If $z_b=0$, the bin is empty.

Stage 2: Minimize Smallest Bin Tokens (MILP_MinSmallestBin) After the first stage determines the optimal number of bins $B^*$, the second stage fixes $B = B^*$ and aims to minimize the smallest total token count among all bins. This leaves more slack or empty space in the least-full microbatch, making it more amenable to merging with tokens from subsequent global batches. The optimization problem is: $$\begin{array}{l} \displaystyle \operatorname*{argmin}_{\boldsymbol{x}_{s,b}, \boldsymbol{k}_{a,b}} \quad \underset{b \in [1, B^*]}{\operatorname{min}} \sum_a k_{a,b} \cdot P \\ \mathrm{s.t.} \quad \quad \displaystyle \sum_{b=1}^{B^*} x_{s,b} = 1 \quad \forall s \in \mathrm{samples} \\ \displaystyle \sum_{s \in \mathrm{adapter}(a)} \mathrm{len}(s) \cdot x_{s,b} \leq k_{a,b} \cdot P \quad \forall a, b \\ \displaystyle \sum_a k_{a,b} \cdot P \leq \mathrm{capacity} \quad \forall b \end{array}$$ Where:

• $\underset{b \in [1, B^*]}{\operatorname{min}} \sum_a k_{a,b} \cdot P$: The objective function, which minimizes the minimum total padded token count across all $B^*$ bins.
• The constraints are similar to Stage 1, ensuring each sample is assigned to one bin, adapter-specific token counts respect padding multiples, and bin capacity is not exceeded for the fixed $B^*$.

Runtime Efficiency Techniques: To improve the runtime of the MILP solver, LoRAFusion employs two techniques:

• Greedy Fallback: A timeout $t$ is set for the MILP solver. If the solver exceeds this time, it falls back to a simpler greedy bin-packing algorithm (Algorithm 1, lines 2, 5, and 9), balancing optimality with computational cost.
• Multiprocessing: Since global batches are independent, the bin-packing optimization for different global batches can be parallelized using multiprocessing (Algorithm 1, line 1). This allows efficient scheduling of all training data.

4.2.2.4. Merging & Verification

After microbatch packing, the final microbatch in a global batch might be underfilled, leading to reduced GPU efficiency and increased pipeline bubbles.

• Greedy Merge Pass: A greedy merge pass (Figure 12 bottom, Algorithm 1, lines 12-14) attempts to shift tokens from the next global batch into the current batch's final microbatch. This is done only if the token capacity is not exceeded and the bubble lemma (data dependency constraint) is preserved.
• Verification and Fix: A final verification step ensures that no constraint is violated. If any bubble condition is not met, no-op microbatches (empty microbatches) are inserted into the schedule (Algorithm 1, line 15) to restore correctness and maintain pipeline consistency.

4.2.2.5. Parallelism Profiler

The Multi-LoRA Scheduler requires a specific token capacity for its microbatches. This capacity depends on the underlying parallelism strategy (e.g., FSDP + PP) and hardware. LoRAFusion includes a lightweight parallelism profiler. This profiler benchmarks the runtime under different model parallelism configurations using fixed-length inputs and collects throughput data. The configuration yielding the best throughput determines the token capacity passed to the data batching stage, ensuring the scheduler's packing aligns with the system's optimal performance characteristics. This decouples the low-level parallelism tuning from the high-level scheduling logic.

4.2.3. System Workflow

Figure 8 provides an overview of the LoRAFusion system workflow:

1. Input: A set of fine-tuning jobs is provided, each specifying its LoRA adapter and dataset.

2. Dataset Statistics Extraction: LoRAFusion first extracts dataset statistics, particularly sample length distributions, which are crucial for adaptive batching.

3. Parallelism Simulation & Token Budget Proposal: A parallelism simulator is used to propose an optimal microbatch token budget based on the available hardware and parallelism strategy.

4. Adapter Grouping: The Multi-LoRA Scheduler groups LoRA adapters based on dataset statistics (e.g., sequence length distributions) and bubble lemma considerations.

5. Microbatch Construction: Within each group, the scheduler constructs microbatches using the two-stage MILP-based adaptive batching algorithm, aiming for balanced token counts and respecting dependencies.

6. Simulation & Iteration: The proposed grouping and batching configuration is re-evaluated through simulation. If a higher-throughput configuration can be found, the process iterates.

7. Execution with Fused Kernels: Once an optimal schedule is determined, the fine-tuning jobs are executed by the Executor using the FusedLoRA and FusedMultiLoRA kernels.

8. Multi-Adapter Runtime Coordinator: A multi-adapter runtime coordinator ensures token-to-adapter consistency (i.e., each token is processed by its correct adapter), manages resource sharing, and tracks gradients across job boundaries.

   Through this combined approach, LoRAFusion systematically addresses both memory bandwidth bottlenecks at the kernel level and distributed training overhead at the job scheduling level.

5. Experimental Setup

5.1. Datasets

The authors evaluate LoRAFusion on three public summarization datasets, chosen for their diverse length distributions (as illustrated in Figure 13), which stress the batching and scheduling capabilities under realistic variable-length scenarios:

The following figure (Figure 13 from the original paper) shows the distribution of sample lengths across the XSum, CNN/DailyMail, and WikiSum datasets:

该图像是图表，展示了在XSum、CNN/DailyMail和WikiSum数据集上样本长度的分布情况。横轴为样本长度（以# Tokens表示），纵轴为密度。不同颜色的曲线代表各数据集的分布，虚线表示对应的均值。

• XSum [61]: A dataset for extreme summarization, where summaries are very short (single sentence) and abstractive. It typically features shorter input and output sequences.
• CNN/DailyMail (CNNDM) [78]: A widely used dataset for abstractive summarization, consisting of news articles and accompanying multi-sentence summaries. This dataset generally has medium-length sequences.
• WikiSum [12]: A dataset for coherent summarization, which likely contains longer documents and summaries, posing challenges for token capacity and memory management.

Dataset Characteristics: As seen in Figure 13:

• XSum has a distribution centered around shorter sample lengths (e.g., < 512 tokens).

• CNN/DailyMail has a broader distribution, with many samples in the mid-range (e.g., 512-1024 tokens) and some extending to longer lengths.

• WikiSum exhibits a distribution with a significant number of longer sequences (e.g., > 1024 tokens), demonstrating higher sequence length variability and larger token counts per sample.

  These diverse distributions are effective for validating the method's performance because:

• They represent realistic fine-tuning workloads where variable sequence lengths are common.

• They allow LoRAFusion to demonstrate its ability to mitigate load imbalance and efficiently pack microbatches under different token distributions.

• The presence of longer sequences (e.g., in WikiSum) specifically challenges batching and memory management, highlighting the benefits of LoRAFusion's optimized kernels and scheduler.

Workload Settings for Multi-LoRA Experiments: For multi-LoRA experiments, four LoRA adapters are trained in parallel, with varying dataset configurations:

• XSum, CNN/DailyMail, WikiSum Configurations: All four adapters are trained independently on the same respective dataset (e.g., 4 adapters all on XSum).
• Mixed Setting: Each of the four adapters is trained on a dataset that combines samples from all three XSum, CNN/DailyMail, and WikiSum.
• Heterogeneous (Het) Setting: The four adapters are trained on different datasets: one adapter on XSum, one on CNN/DailyMail, one on WikiSum, and one on the Mixed dataset. This is the most challenging scenario, testing the scheduler's ability to handle highly diverse workloads concurrently.

5.2. Evaluation Metrics

The primary evaluation metric used in the paper is throughput, specifically measured in trained tokens per second.

5.2.1. Conceptual Definition

Throughput in the context of LLM training measures the efficiency of the training system. When dealing with inputs of variable sequence lengths (common in LLM fine-tuning), simply counting "samples per second" can be misleading because samples can have vastly different computational costs. "Tokens per second" provides a more accurate and normalized measure of computational work completed per unit of time, reflecting the system's ability to process the actual information content. A higher tokens per second indicates better system efficiency and faster training.

5.2.2. Mathematical Formula

While the paper does not explicitly provide a formula for "tokens per second," it is generally calculated as: $$\text{Throughput (tokens/sec)} = \frac{\sum_{i=1}^{N_{\text{samples}}} \text{length}(\text{sample}_i)}{\text{Total training time (seconds)}}$$

5.2.3. Symbol Explanation

• $\text{Throughput (tokens/sec)}$: The metric quantifying the number of tokens processed by the training system per second.

• $N_{\text{samples}}$: The total number of training samples processed during the measurement period.

• $\text{length}(\text{sample}_i)$: The number of tokens in the $i$-th training sample. This sum represents the total number of actual tokens processed.

• $\text{Total training time (seconds)}$: The total wall-clock time taken to process all $N_{\text{samples}}$ samples, including computation, communication, and any idle time.

  Additionally, the paper uses other derived metrics to analyze components:

• Speedup: Calculated as (Throughput of LoRAFusion) / (Throughput of Baseline).

• Memory Traffic Reduction: Measured as a ratio of DRAM read/write traffic using NVIDIA Nsight Compute (NCU).

• Pipeline Bubble Ratio: Measures the percentage of idle time in pipeline parallelism due to pipeline bubbles.

• Tuning Time: Time taken by the scheduler to determine the optimal batching strategy.

5.3. Baselines

LoRAFusion is compared against three representative baselines to demonstrate its performance advantages:

1. Megatron-LM [81] with Fully Sharded Data Parallelism (FSDP):

   • Description: Megatron-LM is a state-of-the-art distributed training framework. FSDP (or ZeRO-3) is a memory optimization technique that shards all model states (parameters, gradients, optimizer states) across data-parallel ranks, drastically reducing GPU memory consumption.
   • Representativeness: This is a strong baseline for memory-efficient distributed training of large models. It represents a common and effective way to scale LLM fine-tuning in terms of memory.
   • Limitation (for LoRA): Megatron-LM does not natively support multi-LoRA fine-tuning, meaning multiple jobs would typically be trained sequentially, failing to exploit multi-job concurrency or specific LoRA characteristics.

2. Megatron-LM [81] with Pipeline Parallelism (PP):

   • Description: Pipeline Parallelism divides the model layers into sequential stages, with each stage assigned to a different GPU. Data microbatches flow through these stages in a pipeline, reducing communication overhead but potentially introducing pipeline bubbles (idle time).
   • Representativeness: PP is another critical distributed training strategy, especially for models too large to fit on a single GPU even with FSDP, and is often combined with FSDP for hybrid parallelism.
   • Limitation (for LoRA): Similar to FSDP, it lacks native multi-LoRA support, and its efficiency is susceptible to pipeline bubbles and load imbalance, especially with variable sequence lengths.

3. mLoRA [98]:

   • Description: mLoRA is the state-of-the-art multi-LoRA fine-tuning system prior to LoRAFusion. It specifically aims to group multiple LoRA adapters sharing the same base model to improve training efficiency, primarily by filling pipeline bubbles in pipeline parallelism.

   • Representativeness: This is the most direct and relevant baseline for multi-LoRA fine-tuning, as it attempts to solve a similar problem.

   • Authors' Reimplementation: The paper notes that the original mLoRA used inefficient Python RPC for inter-GPU communication. To ensure a fair comparison, the authors reimplemented mLoRA within their system using high-performance communication primitives. They also optimistically assume mLoRA's BatchLoRA kernel has the same performance as a naive single LoRA kernel, as it doesn't provide a unique multi-LoRA CUDA kernel. This ensures that the comparison focuses on the core scheduling and architectural differences rather than implementation details.

     Common Software Stack: All experiments use PyTorch 2.6, CUDA Toolkit 12.4, Triton 3.2.0, and Megatron-Core 0.11.0, ensuring a consistent environment.

5.4. Hardware Settings

The experiments are primarily conducted on modern GPU clusters:

• NVIDIA H100 (80GB) GPUs:
  • Configuration: Each node is equipped with 8 NVIDIA H100 GPUs, connected via NVLink (high-speed interconnect for intra-node communication). Nodes are interconnected via InfiniBand for multi-node communication. Each node also has 208 vCPUs.
  • Importance: H100 GPUs represent the cutting-edge in AI accelerators, offering high compute FLOPS and memory bandwidth. Testing on H100s demonstrates LoRAFusion's performance on powerful, modern hardware, where memory bandwidth can still be a bottleneck relative to compute capability.
• NVIDIA L40S (48GB) GPUs:

  • Configuration: Each server contains 4 L40S GPUs, connected over PCIe (a slower interconnect compared to NVLink), and 128 vCPUs.

  • Importance: L40S GPUs offer a different hardware profile, typically with lower compute-to-memory bandwidth ratios compared to H100s. Evaluating on L40S demonstrates the generalizability of LoRAFusion's benefits across different GPU generations and interconnect types, showing that memory bandwidth optimization is valuable even on less premium hardware.

    GPU Allocation: Most experiments use 1, 2, or 4 GPUs per job, representing typical deployment scenarios. The paper notes that assigning fewer GPUs per job and running more independent jobs often leads to better efficiency by reducing inter-GPU communication and synchronization overhead, a finding explored in their Scalability Studies.

6. Results & Analysis

6.1. Core Results Analysis

6.1.1. End-to-End Results

The end-to-end throughput (measured in tokens per second) demonstrates LoRAFusion's significant performance improvements across various models and hardware.

6.1.1.1. Speedup on H100 GPUs

The following figure (Figure 14 from the original paper) reports the end-to-end throughput of training 4 LoRA adapters on 1, 2, and 4 H100 GPUs:

该图像是图表，展示了在不同数据集上，使用 1、2 和 4 个 H100 GPU 训练 4 个 LoRA 适配器的端到端训练吞吐量（tokens/sec）。图中比较了不同方法的吞吐量，包括 mLoRA 和 LoRAFusion。

• Overall Performance: LoRAFusion consistently outperforms all baselines by $1.19 - 1.96\times$.
• LLaMa-3.1-8B (Single H100 GPU): For LLaMa-3.1-8B, which fits on a single H100 GPU, LoRAFusion achieves an average $1.26\times$ speedup (up to $1.43\times$). This improvement is primarily attributed to the FusedLoRA kernel's ability to reduce memory traffic. Since single-GPU setups inherently do not suffer from load imbalance or distributed parallelism overhead, this gain directly reflects the kernel-level optimizations.
• Qwen-2.5-32B and LLaMa-3.1-70B (Distributed Training): For larger models like Qwen-2.5-32B and LLaMa-3.1-70B, which necessitate distributed training (2 or 4 H100 GPUs), LoRAFusion achieves even higher average speedups of $1.42\times$ and $1.64\times$ (up to $1.64\times$ and $1.81\times$), respectively. This indicates that larger models benefit more from LoRAFusion's combined kernel and scheduling optimizations, as pipeline stalls and load imbalance become more pronounced with higher parallelism.
• Dataset Impact: The WikiSum dataset, known for its large variance in sample lengths, shows high speedups. While baseline methods struggle with out-of-memory errors or severe slowdowns, LoRAFusion achieves stable and efficient packing.
• Heterogeneous Setting (Het): In the most challenging heterogeneous setting, where each of the four adapters is trained on a different dataset, LoRAFusion still maintains strong performance. This highlights the robustness and adaptability of its Multi-LoRA Scheduler.

6.1.1.2. Speedup on L40S GPUs

The following figure (Figure 15 from the original paper) presents results on NVIDIA L40S GPUs:

该图像是一个柱状图，展示了在 Llama-3.1-8B 和 Qwen-2.5-32B 模型下，使用不同方法（如 LoRAFusion、mLoRA 等）进行混合和异构训练的吞吐量（tokens/sec）。图中显示了 LoRAFusion 在两种模型下的显著性能提升。

• Overall Performance: LoRAFusion achieves $1.19 - 1.91\times$ average speedup for LLaMa-3.1-8B and Qwen-2.5-32B, respectively.
• LLaMa-3.1-8B on L40S: The benefit for LLaMa-3.1-8B is slightly smaller on L40S compared to H100. This is attributed to the limited memory capacity (48GB vs. 80GB) on a single L40S GPU, which constrains the batch size and thus limits the full effectiveness of kernel fusion. Despite these constraints, LoRAFusion still delivers consistent improvements, demonstrating its generalizability across different model sizes and hardware platforms. The performance gains from memory bandwidth optimization are particularly relevant on hardware where memory bandwidth is a more limiting factor.

6.1.2. Scalability Studies

The scalability of LoRAFusion is evaluated across 4, 8, and 16 H100 GPUs under two scaling strategies: DP scaling (more GPUs per job) and job-level scaling (more concurrent jobs). Global batch sizes are scaled proportionally with the GPU count for fair comparison.

The following figure (Figure 16 from the original paper) shows the scalability of LoRAFusion across 4, 8, and 16 H100 GPUs:

该图像是一个图表，展示了在同时训练4个LoRA适配器时，LoRAFusion与Megatron-LM的性能对比。图中列出了不同GPU数量（4、8、16）下的吞吐量（K tokens/s）以及相应的缩放因子，显示了LoRAFusion在DP和Job缩放中的优越性能。

• Job-Level Scaling vs. DP Scaling: The results clearly show that job-level scaling consistently outperforms DP scaling. This is attributed to better load balance when running multiple independent jobs. Job-level scaling achieves $1.18\times$ and $1.25\times$ higher throughput on 8 and 16 GPUs, respectively, compared to DP scaling. This implies that, for LoRA fine-tuning, efficiently running more independent jobs across available GPUs is often more beneficial than simply increasing the data parallelism degree for a single job, as it better utilizes resources by reducing inter-GPU communication and synchronization overhead.
• Compatibility and Performance with DP Scaling: Even under DP scaling, LoRAFusion demonstrates strong performance. It achieves an average $1.78\times$ speedup over Megatron-LM and $1.50\times$ over mLoRA. This confirms that LoRAFusion is fully compatible with traditional data parallelism and multi-node fine-tuning setups, while still providing significant gains through its kernel and scheduling optimizations.

6.1.3. Effectiveness of FusedLoRA Kernel

6.1.3.1. Kernel Performance

The following figure (Figure 17 from the original paper) shows the throughput of FusedLoRA and FusedMultiLoRA kernels compared to the standard Torch LoRA implementation:

该图像是图表，展示了不同模型（Torch LoRA、FusedLoRA、FusedMultiLoRA）在不同 Token 数量下的规范化吞吐量。图中显示了在 N=K=4096、N=K=5120 和 N=K=8192 的条件下，各模型的性能变化趋势，FusedLoRA 在大多数情况下表现优越。

• FusedLoRA achieves an average speedup of $1.27\times$ (up to $1.39\times$).
• FusedMultiLoRA achieves an average speedup of $1.17\times$ (up to $1.24\times$).
• In the forward pass, FusedMultiLoRA performs similarly to FusedLoRA because the majority of the computation (the base model GEMM) is shared across tokens regardless of the adapter.
• In the backward pass, FusedMultiLoRA incurs a slight overhead. This is due to the additional complexity of accumulating gradients across different adapters and performing extra element-wise operations (e.g., routing gradients to the correct adapter weights).
• Despite this slight overhead in the backward pass, both fused kernels consistently outperform the baseline Torch LoRA implementation across different token sizes and model configurations. This validates the effectiveness of the graph-splitting fusion strategy in reducing memory access bottlenecks.

6.1.3.2. Layer-wise Performance

The following figure (Figure 18 from the original paper) compares the speedup across different linear layers in various models:

该图像是一个图表，展示了不同模型在不同批量大小下的标准化吞吐量。图表中包括了三种方法：Torch LoRA、FusedLoRA 和 FusedMultiLoRA，分别针对 Llama-3.1-8B、Qwen2.5-32B 和 Llama-3.1-70B 模型进行比较。

• FusedLoRA achieves an average speedup of $1.21\times$ (up to $1.30\times$).
• FusedMultiLoRA achieves an average speedup of $1.13\times$ (up to $1.17\times$).
• These results are based on microbatches containing four adapters. The paper notes that in practical fine-tuning workloads, each microbatch often contains only one or two adapters. In such scenarios, FusedMultiLoRA's performance would be even closer to FusedLoRA due to less overhead from multi-adapter gradient accumulation. This further confirms the robustness of the fused kernels across different layers and model sizes.

6.1.3.3. Memory Traffic Reduction

The following figure (Figure 19 from the original paper) illustrates the DRAM read and write traffic from NVIDIA Nsight Compute (NCU) across representative GEMM shapes:

该图像是一个示意图，展示了不同 LoRA 方法在 DRAM 读取/写入性能上的比较。数据显示，在不同的维度配置下，Fused LoRA 和 Fused MultiLoRA 相较于传统的 Torch LoRA 具有更低的内存使用量。具体表现为在矩阵尺寸 $8192 \times 4096 \times 4096$ 时，Torch LoRA 为 1.00 倍，而 Fused LoRA 为 0.63 倍。

• Both FusedLoRA and FusedMultiLoRA consistently reduce memory usage compared to Torch LoRA.
• For a large GEMM shape of $8192 \times 4096 \times 4096$, the total DRAM traffic is reduced to $0.63\times$ (i.e., a $37\%$ reduction).
• Across all tested settings, the DRAM traffic is reduced by $34\% - 37\%$. This directly confirms that the graph-splitting fusion design effectively addresses the redundant memory access bottleneck identified in the motivations, leading to substantial memory bandwidth savings.

6.1.3.4. Performance Insights Across Diverse Hardware

The paper emphasizes that the FusedLoRA and FusedMultiLoRA kernels' benefits (reducing redundant memory access for large activation tensors) are particularly pronounced on hardware where memory bandwidth is a limiting factor compared to compute FLOPS. As modern accelerators continue to increase compute FLOPS faster than memory bandwidth [27], the relative performance gains from LoRAFusion's fused kernels are expected to grow in future systems.

6.1.4. Effectiveness of Job-Level Scheduling

6.1.4.1. Pipeline Bubble Reduction

The following figure (Figure 20 from the original paper) illustrates how LoRAFusion helps reduce pipeline bubbles by scheduling multiple adapters together:

该图像是图表，展示了不同方法下的管道气泡比率。Megatron-LM表现出48.79%，而mLoRA则为34.11%。在使用一个适配器的LoRAFusion中，管道气泡比率为44.17%；使用两个适配器的LoRAFusion为15.00%；三个适配器为12.23%；而四个适配器的比率则降至11.09%。

The analysis reveals three key observations:

1. Single Adapter Performance: With only one adapter, the pipeline bubble ratio remains high at 44.17%, which is close to Megatron-LM's 48.79%. This demonstrates that adapter grouping is ineffective when only a single dataset/job is available, highlighting the importance of multi-LoRA fine-tuning to enable improved scheduling flexibility.
2. Impact of Multiple Adapters: As more adapters are trained concurrently, the bubble ratio steadily decreases: 15.00% for 2 adapters, 12.23% for 3 adapters, and 11.09% for 4 adapters. In contrast, mLoRA (a baseline for multi-LoRA) only achieves 34.11%. This confirms that LoRAFusion's grouping and adaptive batching strategies significantly reduce pipeline idle time by effectively filling pipeline bubbles with useful work from other jobs.
3. Residual Bubbles: Even with four adapters, a bubble ratio of 11.09% remains. The authors attribute this to uneven execution times across pipeline stages, specifically the last stage taking longer due to an extra linear layer and cross-entropy loss computation. This limitation is inherent to the model's structure and pipeline partitioning, and thus is not addressable by the scheduler alone.

6.1.4.2. Tuning Time

The following figure (Figure 21 from the original paper) shows how tuning (scheduling) and computation time grow with the number of training samples for a 4-stage pipeline with 4 adapters:

该图像是图表，展示了在具有4个适配器的4阶段流水线中，调优时间与样本数量之间的关系。随着样本数量的增加，计算时间呈现出明显递增的趋势，而调优时间则相对平稳。

• Linear Scalability: The scheduling time (for the MILP-based optimization) increases nearly linearly with the number of samples, from 15.74 seconds at 640 samples to 102.12 seconds at 25600 samples. This demonstrates the linear scalability of the scheduler.
• Negligible Overhead: The scheduling overhead is negligible compared to the overall computation time. This is due to three factors:
  1. Overlap: The CPU-based scheduling runs in parallel with GPU training of the preceding global batch. With a linear scaling of ~4ms per sample on the CPU and a much larger magnitude difference in execution time between CPU and GPU, the scheduler's latency is fully hidden by this overlap.
  2. Saturation of Gains: Performance gains from adding more adapters saturate at around 4 adapters, meaning practical deployments can operate with a small, constant number of adapters.
  3. Timeout and Fallback: The MILP solver has a timeout mechanism and falls back to a greedy bin-packing algorithm if it takes too long. This ensures the scheduling overhead remains within a controllable range.

6.1.4.3. Effectiveness of the Merging & Greedy Fallback

An evaluation on 4 adapters of LLaMa-3.1-70B fine-tuned on four H100 GPUs quantifies the contribution of the merging pass and two-stage MILP optimization:

• The merging pass (which shifts tokens from the next global batch into an underfilled microbatch) improves throughput by 4.34%.
• The two-stage MILP optimization (compared to pure greedy bin-packing) provides an additional 3.82% improvement.
• The MILP solver path is selected for 77.4% of global batches (with a timeout of 10 seconds), indicating its effectiveness in reducing token counts for underfilled microbatches. These modest improvements suggest that most microbatches are already well-packed, and these algorithms primarily optimize the final microbatch in each global batch. Given that scheduling overhead is hidden by parallel GPU execution, these optimizations push performance closer to the hardware limit without introducing additional latency.

6.1.5. Speedup Breakdown

The following figure (Figure 22 from the original paper) shows the contribution of each component in LoRAFusion on LLaMa-3.1-70B with 4 GPUs:

该图像是图表，展示了在使用4个GPU时，LoRAFusion在LLaMa3.1-70B上的相对加速效果。各项加速效果与1F1B PP的基准1.00x进行对比，其中平衡的Multi-LoRA ZeRO Bubble PP + FusedMultiLoRA达到最高的2.05x加速效果。

• Baseline (1F1B PP in Megatron-LM): This serves as the reference point ($1.00\times$ speedup).
• 1F1B PP + FusedLoRA: Adding FusedLoRA alone yields a 1.13x speedup. This gain is modest because load imbalance and suboptimal token shapes (which limit kernel efficiency) are still present.
• Multi-LoRA Zero-Bubble PP: Replacing the 1F1B PP with a Multi-LoRA zero-bubble pipeline parallelism (without FusedLoRA) improves throughput to 1.50x. This substantial gain comes from eliminating pipeline stalls by filling pipeline bubbles with microbatches from independent adapters.
• Multi-LoRA Zero-Bubble PP + FusedMultiLoRA: Further adding the FusedMultiLoRA kernel (which enables multi-adapter microbatches and reduces redundant memory access) raises the speedup to 1.72x.
• Balanced Multi-LoRA ZeRO Bubble PP (with Scheduler): Applying the Multi-LoRA Scheduler to rebalance token distribution across microbatches (even without fusion) achieves 1.57x speedup. This shows the significant impact of the scheduler in reducing load imbalance.
• Balanced Multi-LoRA ZeRO Bubble PP + FusedMultiLoRA (LoRAFusion's Full System): Combining the adaptive scheduling with the fused kernels achieves the highest speedup of 2.05x. This clearly demonstrates the importance of jointly optimizing kernel efficiency, parallelism, and workload balance.

Speedup over mLoRA: The speedup over mLoRA is driven by two main optimizations:

1. Kernel Fusion: Our kernel fusion (comparing Multi-LoRA Zero-Bubble PP to Multi-LoRA Zero-Bubble PP + FusedMultiLoRA) yields a 1.15x speedup ($1.72 / 1.50 \approx 1.15$). This is consistent with Figure 17 ($1.17\times$ average speedup) and Figure 18 (1.13x average speedup).
2. Adaptive Batching: Our adaptive batching (comparing Multi-LoRA Zero-Bubble PP + FusedMultiLoRA to Balanced Multi-LoRA ZeRO Bubble PP + FusedMultiLoRA) mitigates load imbalance, providing an additional 1.19x speedup ($2.05 / 1.72 \approx 1.19$). This is supported by Figure 20, which shows a 23.02% reduction in pipeline bubbles with LoRAFusion's scheduling compared to mLoRA.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper effectively identifies and addresses two crucial performance bottlenecks in LLM LoRA fine-tuning: the substantial runtime overhead caused by redundant memory accesses in LoRA modules and the missed optimization opportunities inherent in grouping multiple concurrent LoRA jobs. The proposed LoRAFusion system offers a dual-level solution:

1. Kernel-level Optimization: A novel horizontal fusion technique (FusedLoRA and FusedMultiLoRA) is introduced. This method judiciously splits the computation graph to fuse memory-bound operations while preserving the efficiency of compute-bound GEMMs, resulting in a significant reduction of memory traffic by up to 37%.

2. Job-level Scheduling: An adaptive multi-LoRA scheduling strategy is presented. This scheduler intelligently groups adapters and uses a two-stage MILP-based bin-packing algorithm to create balanced, dependency-aware microbatches. This significantly improves GPU utilization from 65% to 89% and reduces pipeline bubbles from 44.17% to 11.09%.

   Combined, these optimizations achieve an impressive end-to-end speedup of up to $1.96\times$ (average $1.47\times$) compared to Megatron-LM and up to $1.46\times$ (average $1.29\times$) over mLoRA, the previous state-of-the-art multi-LoRA fine-tuning system. LoRAFusion not only enhances performance but also improves the accessibility and efficiency of LLM LoRA fine-tuning for both researchers and practitioners.

7.2. Limitations & Future Work

The authors acknowledge several areas for future work and discuss the generalizability of their approach:

7.2.1. Generalizability to LoRA Variants

The proposed kernel fusion design is inherently extensible to other popular LoRA variants such as DoRA [52] and VeRA [37]. These variants typically introduce pre- or post-processing functions around the core LoRA computation. The authors state that LoRAFusion's optimizations are orthogonal to these modifications, meaning users could define prologue/epilogue functions to extend the existing kernels manually. For a more general approach, they plan to integrate these fusion patterns into a compiler framework, specifically by adding compiler annotations as hints for torch.compile. This would automate the optimization process for both existing and future LoRA variants, removing the need for manual kernel development and specialized system expertise from users.

7.2.2. Generalizability to Quantization

The kernels developed in LoRAFusion can be directly applied to 4-bit QLoRA [14]. Current QLoRA implementations typically dequantize 4-bit weights to half-precision (e.g., FP16) before performing the LoRA computation. This means LoRAFusion's kernels can operate without modification on these dequantized weights. While it might be possible to fuse dequantization with the LoRA path, recent research suggests that two-step approaches (dequantize then compute) are often more performant for large token counts.

7.3. Personal Insights & Critique

7.3.1. Strengths and Innovations

• Holistic Approach: The paper's greatest strength lies in its holistic approach, tackling inefficiencies at both the low-level kernel execution and high-level job scheduling. This multi-level optimization is crucial for maximizing performance in complex distributed systems.
• Rigorous Bottleneck Analysis: The detailed profiling in Section 3, clearly identifying memory bandwidth as the primary bottleneck for LoRA, is highly insightful. This rigorous analysis guides the elegant graph-splitting fusion strategy, which is a key innovation.
• Practicality of Kernel Fusion: The FusedLoRA and FusedMultiLoRA kernels offer a practical and immediate benefit. Their plug-and-play nature means existing LoRA systems can integrate them with minimal effort, offering immediate throughput gains. The tile-level routing for FusedMultiLoRA is particularly clever for handling heterogeneous jobs.
• Sophisticated Scheduling: The two-stage MILP-based adaptive batching algorithm is a significant advancement over prior multi-LoRA scheduling. It directly addresses the critical issues of load imbalance and pipeline bubbles caused by variable sequence lengths and data dependencies, making distributed fine-tuning much more efficient. The use of greedy fallbacks and multiprocessing for MILP demonstrates a pragmatic approach to optimize for both optimality and runtime.
• Strong Evaluation: The extensive evaluation across diverse LLMs, datasets, and GPU platforms (H100 and L40S) under various workload configurations (homogeneous, mixed, heterogeneous) provides robust evidence for the system's effectiveness and generalizability.

7.3.2. Potential Issues, Unverified Assumptions, or Areas for Improvement

• Complexity of MILP: While effective, the MILP-based scheduler can be computationally intensive for extremely large numbers of samples or adapters, even with timeouts and greedy fallbacks. The paper shows linear scaling, but the constant factor might be significant for very dynamic or extremely large-scale scheduling scenarios with frequent re-scheduling. Further work could explore more lightweight heuristics or reinforcement learning-based schedulers for extreme scale.
• Dependency on Parallelism Profiler: The scheduler relies on an external parallelism profiler to determine the optimal token capacity. While this decouples concerns, it introduces an extra profiling step and assumes that this token capacity remains optimal throughout training, which might not always be true if workload characteristics shift significantly. Integrating this profiling more dynamically into the scheduling loop could be an enhancement.
• Fixed Pipeline Stage Imbalance: The paper acknowledges that residual pipeline bubbles ($11.09\%$) are due to uneven execution times across pipeline stages, a limitation not solvable by their scheduler. This points to an area where dynamic pipeline re-balancing or stage-aware kernel fusion could offer further improvements, potentially beyond the current scope of LoRAFusion.
• Triton Kernel Maintenance: While Triton offers fine-grained control for kernel optimization, it also requires manual tuning and maintenance (as seen in the Artifact Appendix's tune_kernels.py script). As hardware evolves, these kernels might need re-tuning. The proposed future work on integrating with compiler frameworks like torch.compile is crucial to mitigate this.
• Specific to LoRA: The optimizations are highly tailored to LoRA. While this yields significant gains, adapting them to other PEFT methods (beyond just DoRA or VeRA variants) might require substantial re-engineering.

7.3.3. Transferability and Broader Impact

• Memory-Bound Optimization: The graph-splitting fusion strategy for memory-bound operations could be transferable to other domains where small computational additions cause disproportionate memory traffic overhead. This principle could apply to other adapter-based architectures or even specific layers in full models that are memory-bandwidth-bound.
• Adaptive Batching for Variable Workloads: The adaptive batching and scheduling algorithm for variable sequence lengths and multi-job concurrency is highly transferable. This approach can benefit any distributed training or inference system dealing with heterogeneous, variable-length workloads, not just LLMs or LoRA. For example, in multi-modal models or graph neural networks with variable graph sizes, similar scheduling challenges arise.
• Cost Reduction in AI: By significantly improving the efficiency of LLM fine-tuning, LoRAFusion directly contributes to reducing the operational costs and hardware requirements for adapting LLMs. This democratizes access to advanced AI capabilities, making it more feasible for smaller organizations and individual researchers to fine-tune high-quality models.
• Foundation for Future Systems: LoRAFusion establishes a strong foundation for future LLM fine-tuning systems, demonstrating the power of combining low-level kernel engineering with high-level system scheduling for specialized workloads. This multi-level optimization paradigm will likely inspire further research in system-level AI acceleration.