1. Bibliographic Information

1.1. Title

K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs

1.2. Authors

Ziheng Ouyang, Zhen Li†, Qibin Hou (VCIP, School of Computer Science, Nankai University) Contact: {zihengouyang666, zhenli1031}@gmail.com

1.3. Journal/Conference

The paper is published at (UTC) 2025-02-25T18:59:12.000Z, indicating it is likely a preprint (e.g., on arXiv) and may be submitted to or under review for a conference or journal in computer vision, machine learning, or graphics. Given the affiliations and the domain of the research (computer vision, diffusion models), it targets high-impact venues in these fields.

1.4. Publication Year

2025

1.5. Abstract

The paper addresses the challenge of combining different LoRAs to jointly generate learned style and content in diffusion models. Existing methods often struggle to preserve both original subject and style simultaneously or necessitate additional training. The authors argue that the intrinsic properties of LoRA can effectively guide diffusion models in merging these elements. Based on this insight, they propose K-LoRA, a simple yet effective training-free LoRA fusion approach. K-LoRA operates in each attention layer by comparing the Top-K elements of the LoRAs to be fused, thereby determining which LoRA to select for optimal fusion. This selection mechanism is designed to ensure that the most representative features of both subject and style are retained and balanced throughout the fusion process. Experimental results demonstrate that K-LoRA effectively integrates subject and style information from original LoRAs, outperforming state-of-the-art training-based approaches in both qualitative and quantitative assessments.

1.6. Original Source Link

https://arxiv.org/abs/2502.18461 (Preprint)

1.7. PDF Link

https://arxiv.org/pdf/2502.18461v2.pdf (Preprint)

2. Executive Summary

2.1. Background & Motivation

The field of image generation, particularly with diffusion models, has seen significant advancements in personalization (generating specific subjects) and stylization (applying specific artistic styles). Low-Rank Adaptation (LoRA) has emerged as a highly efficient fine-tuning technique for these large models, allowing for the independent training of subject-specific and style-specific adaptors.

The core problem the paper aims to solve is the effective and efficient fusion of these independently trained subject and style LoRAs. While various methods have explored combining LoRAs, existing approaches face two main limitations:

1. Preservation Challenge: Many methods fail to consistently preserve both the distinct characteristics of the original subject (content) and the intricate details of the desired style simultaneously. This often leads to concept dilution, blurring of fine details, or inconsistent object characteristics.

2. Training Requirement: A significant number of current fusion techniques either require additional training (e.g., learning fusion ratios, fine-tuning specific modules) or involve extensive manual hyperparameter tuning and seed selection, making them less user-friendly and computationally intensive.

   The paper's entry point is the insight that the intrinsic properties of LoRA itself, specifically its low-rank structure and the differing roles of diffusion timesteps, can be leveraged to guide the merging process without requiring any further training. This addresses the need for a training-free, robust, and high-quality LoRA fusion method.

2.2. Main Contributions / Findings

The paper introduces K-LoRA, a novel approach for fusing subject and style LoRAs. Its primary contributions and findings are:

• Training-Free Fusion: K-LoRA is a simple yet highly effective optimization technique that seamlessly merges content and style LoRAs without any additional training or complex hyperparameter tuning. This significantly improves user-friendliness and reduces computational overhead compared to existing methods.
• Top-K Selection Mechanism: The core of K-LoRA lies in its novel Top-K selection process. In each attention layer, it compares the importance of elements within the content LoRA and the style LoRA (quantified by the sum of their Top-K absolute values). This selective approach ensures that only the most representative features from either the subject or the style are chosen, effectively balancing their contributions and preventing concept dilution.
• Dynamic Scaling Factor: K-LoRA incorporates a scaling factor that adapts based on the diffusion timestep. This mechanism leverages the insight that early diffusion steps are crucial for reconstructing the object and larger texture details, while later steps focus on refining finer details and style. The scaling factor dynamically emphasizes content LoRA in early stages and style LoRA in later stages, leading to more coherent and high-fidelity fusions.
• Superior Performance: Experimental results demonstrate that K-LoRA effectively integrates both subject and style information, producing stable generative outputs and preserving intricate details. It quantitatively and qualitatively outperforms state-of-the-art training-based approaches in terms of subject and style similarity, as well as user preference.
• Key Findings on LoRA Behavior: The research provides empirical evidence supporting two critical insights: (i) applying LoRA to only a subset of layers per step can achieve comparable effects to applying it to all layers, and (ii) subject LoRAs are more effective in earlier diffusion steps, while style LoRAs are more impactful in later steps for achieving stylistic details without distorting content.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand K-LoRA, a grasp of the following fundamental concepts is essential for a beginner:

• Diffusion Models: These are generative models that learn to reverse a diffusion process. They start with random noise and progressively denoise it over several steps to generate a coherent image. The process typically involves a U-Net architecture that predicts the noise to be removed at each step. Diffusion models have become central to high-quality image synthesis.
• LoRA (Low-Rank Adaptation): LoRA is a parameter-efficient fine-tuning technique designed to adapt large pre-trained models (like diffusion models or large language models) to specific tasks or data with minimal computational cost. Instead of fine-tuning all parameters of the large model, LoRA introduces small, low-rank matrices (adapters) alongside the original weights. During fine-tuning, the original weights are frozen, and only these much smaller adapter matrices are trained.
  • How LoRA works: For a pre-trained weight matrix $W_0 \in \mathbb{R}^{m \times n}$, LoRA assumes that the update $\Delta W$ (the change needed during fine-tuning) has a low intrinsic rank $r$. This means $\Delta W$ can be approximated by factoring it into two much smaller matrices, $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$, where $r \ll \min(m, n)$. The update is then represented as $\Delta W = BA$. During inference, the original weight matrix $W_0$ is combined with the trained LoRA matrices as $W_0 + BA$. This significantly reduces the number of trainable parameters.
• Subject LoRA vs. Style LoRA:
  • Subject LoRA (Content LoRA): A LoRA fine-tuned to embed a specific object or subject into the model's generation capabilities. For example, training a LoRA on images of a particular dog breed to generate that specific dog.
  • Style LoRA: A LoRA fine-tuned to capture a specific artistic style (e.g., impressionistic painting, watercolor, pop art). It learns the visual attributes like color palettes, textures, brushstrokes, and patterns associated with that style.
• Attention Mechanism: A core component in many modern neural networks, including diffusion models. It allows the model to focus on specific parts of its input (e.g., an image or text prompt) when making predictions. In diffusion models, cross-attention layers often link the text prompt (conditioning) to the visual features, and self-attention layers help the model understand spatial relationships within the image features themselves. LoRAs are frequently applied to the weight matrices within these attention layers.
• Diffusion Timesteps: In the iterative denoising process of a diffusion model, each step is referred to as a timestep. The process typically goes from a high timestep (more noise, early stages of generation) to a low timestep (less noise, later stages of refinement). The paper highlights that different timesteps play distinct roles in reconstructing content and applying style.

3.2. Previous Works

The paper discusses several categories of prior research related to diffusion models and LoRA combinations:

• Diffusion Models for Customization: These methods enable users to personalize image generation.

  • Textual Inversion [1, 29, 40]: Fine-tunes embeddings (text tokens) to represent new concepts from a few images. The model learns a new "word" (embedding) for a specific object or style.
  • DreamBooth [24]: Fine-tunes the entire model (or significant parts) using a few images of a subject, often with a unique identifier word. It excels at subject-driven generation.
  • Custom Diffusion [16]: Focuses on fine-tuning cross-attention layers within the diffusion model to learn new concepts efficiently.
  • Training-free Inference Methods [2, 27, 31, 32]: Approaches that utilize pre-trained modules without additional training during inference, but may be suboptimal for specialized tasks.
  • LoRA and its Variants [11, 15, 22, 39, 43, 44]: Emphasized as efficient fine-tuning methods for large models, making them a popular choice.

• LoRA Combination in Image Generation: This is directly relevant to K-LoRA.

  • Object Integration (Multiple Objects): Studies focusing on combining multiple subject LoRAs to generate images with diverse objects. Examples include methods that fine-tune subject LoRAs and use masking techniques for object layout [7, 10, 14, 18, 36].
  • Content-Style Fusion: Methods specifically designed to merge subject (content) and style LoRAs.
    • MergingLoRA [25]: A foundational approach that often involves direct arithmetic merging of LoRA weights, sometimes with adjustable coefficients. The paper notes that this can lead to concept dilution or require extensive manual tuning.
    • Mixture-of-Subspaces [30]: Proposes methods for learning fusion matrices or hyperparameters to combine LoRA weight layers.
    • ZipLoRA [26]: Attempts to train a fusion ratio vector to balance different LoRAs, aiming to combine subjects and styles effectively. K-LoRA directly compares against ZipLoRA as a state-of-the-art method.
    • B-LoRA [8]: Identifies distinct roles for attention modules in the generative process and fine-tunes only two attention modules to achieve object-style decoupling. K-LoRA also compares against B-LoRA.
    • LoRA Composition [41]: Uses a cyclic update of the model's LoRA modules, allowing multiple LoRAs to collaboratively guide the model. K-LoRA refers to this work as inspiration for its experiments on applying LoRA at different timesteps.

3.3. Technological Evolution

The field has evolved from:

1. Full Model Retraining: Early methods required retraining large generative models for each new concept or style, which was computationally prohibitive.

2. Parameter-Efficient Fine-Tuning (PEFT): Techniques like LoRA revolutionized this by allowing efficient adaptation with minimal trainable parameters, making personalization and stylization more accessible.

3. Single Concept LoRAs: Initially, LoRAs were primarily used to embed a single concept (either a subject or a style).

4. Multi-LoRA Integration: The next challenge became combining multiple LoRAs (e.g., multiple subjects, or a subject and a style) to create complex scenes or stylized content. This led to research into fusion strategies.

5. Training-Based Fusion: Many early fusion methods involved some form of additional training to learn how to combine LoRAs effectively (e.g., learning weights, fusion matrices).

6. Training-Free, Intelligent Fusion (K-LoRA): K-LoRA represents a step towards more intelligent and efficient fusion by leveraging intrinsic LoRA properties and diffusion dynamics, eliminating the need for further training.

   K-LoRA's work fits into the current state by providing a highly efficient, training-free solution to a known problem in multi-LoRA integration, directly building on insights from how LoRAs function within the diffusion process.

3.4. Differentiation Analysis

Compared to the main methods in related work, K-LoRA's core differences and innovations are:

• Training-Free Nature: Unlike ZipLoRA [26] which trains a fusion ratio vector, or B-LoRA [8] which fine-tunes specific modules, K-LoRA requires no additional training. It directly operates on pre-trained content and style LoRA weights, making it highly efficient and user-friendly.

• Adaptive Layer-wise Selection: Instead of element-wise merging (which can lead to smoothing or loss of details as observed in the paper), K-LoRA employs a Top-K selection mechanism at each attention layer. This allows it to dynamically choose the most important elements from either the subject or style LoRA based on their magnitude and representation power, rather than simply averaging or blending them.

• Timestep-Dependent Scaling: K-LoRA explicitly leverages the distinct roles of diffusion timesteps for content and style generation. It introduces a scaling factor that emphasizes content in early stages and style in later stages. This is a more nuanced control than static fusion ratios or periodic integration (like LoRA Composition [41]).

• Preservation of Fidelity: The paper claims K-LoRA consistently produces higher-quality outputs that preserve both the original shape, color, and stylistic features without overfitting to color or focusing on background elements like some baselines.

• Robustness: K-LoRA is designed to be robust to seed variations and discrepancies between community LoRAs, requiring minimal parameter adjustments.

  In essence, K-LoRA differentiates itself by offering a smart, selective, and dynamic fusion strategy that avoids the pitfalls of previous methods by leveraging intrinsic LoRA characteristics and the generative dynamics of diffusion models, all without incurring additional training costs.

4. Methodology

4.1. Principles

The core idea behind K-LoRA is that the Low-Rank Adaptation (LoRA) modules inherently possess sparse and concentrated information, meaning only a few key elements within their weight matrices are most influential for concept representation. Building on this, the method further leverages the understanding that different stages of the diffusion process (early vs. late timesteps) are responsible for generating distinct aspects of an image (object structure vs. fine-grained style details).

The theoretical basis and intuition are:

1. Sparsity of LoRA Weights: LoRA matrices, particularly their update $\Delta W$, often contain many small or near-zero elements, indicating that only a small subset of dominant elements truly carries the concept information. K-LoRA aims to identify and utilize these dominant elements for fusion.
2. Sequential Generation in Diffusion: The diffusion denoising process is not uniform. Early steps primarily establish the overall structure and main object features, while later steps refine details and apply textures/styles. K-LoRA aligns its fusion strategy with this inherent generative flow.
3. Adaptive Selection: Instead of fixed or learned blending ratios, K-LoRA proposes to adaptively select either the subject LoRA or the style LoRA for each attention layer and timestep based on the significance of their dominant elements. This selective approach aims to prevent the "smoothing" or "dilution" that can occur with element-wise merging.

4.2. Core Methodology In-depth (Layer by Layer)

K-LoRA's method is built upon two key findings derived from experiments:

• (i) Subset Application: Applying LoRA to only a subset of layers per diffusion step can achieve comparable effects to applying it to all layers. This suggests that not all LoRA elements are equally important, and selective application is viable. Figure 3(a) illustrates this, showing that when more than 50% of LoRA attention layers are randomly applied, the results are almost indistinguishable from using all layers, but performance degrades significantly below 25%.
• (ii) Timestep-Dependent Roles:
  • Applying style LoRA in earlier timesteps significantly impacts object reconstruction, potentially distorting the content. However, applying it in later timesteps effectively preserves style information without affecting the original object structure.
  • For content LoRA, applying it in earlier timesteps yields significantly better results, indicating its importance in establishing the subject's form. Figure 2 visually supports this by showing how the placement of LoRA layers across initial and latter timesteps influences generation quality. Early steps are for object reconstruction and larger texture details, while later steps refine finer details and style.

The overall architecture of K-LoRA is illustrated in Figure 4. It operates by adaptively selecting which LoRA (content or style) to apply at each attention layer during the diffusion process.

The following figure (Figure 4 from the original paper) shows the overall architecture of K-LoRA:

该图像是一个示意图，展示了K-LoRA方法如何通过比较两个LoRA的Top-K元素在注意力层中进行无训练融合，选择具有更大Top-K元素的LoRA以保留内容和风格的关键特征，实现最终图像的合成。

The methodology proceeds as follows:

4.2.1. LoRA Preliminaries and Objective

Let $D$ be a base diffusion model, and $W_0$ denote the pre-trained weights to be updated by a LoRA layer. A specific concept can be adapted by adding a trained LoRA weight set $\Delta W_x$ to the model weights, resulting in $D' = W_0 + \Delta W_x$. Given two independently trained LoRA weight sets, $\Delta W_c$ (for content/subject) and $\Delta W_s$ (for style), the objective of K-LoRA is to effectively fuse these two sets. This fusion is expressed as: $$\Delta W_x = K(\Delta W_c, \Delta W_s)$$ Here, $K$ represents the K-LoRA method, which integrates the contributions of the content LoRA and style LoRA.

4.2.2. Importance Assessment via Absolute Values

To determine the significance of elements within each LoRA layer, K-LoRA first takes the absolute value of each element. This is because small values might be less important, and their signs could complicate direct comparison of "strength." $$\begin{array}{r} \Delta W_c' = | \Delta W_c | , \\ \Delta W_s' = | \Delta W_s | , \end{array}$$ where $\Delta W_c$ and $\Delta W_s$ are the raw content and style LoRA weights, respectively. The resulting $\Delta W_c'$ and $\Delta W_s'$ matrices contain the absolute magnitudes of these weights.

4.2.3. Top-K Element Selection for Layer Importance

Since a small subset of dominant elements can achieve the original generation effect, and the overall data distribution of LoRA weights shows a large proportion of smaller elements (as illustrated in Figure 3(b)), K-LoRA focuses on the largest elements to represent the importance of each layer. Specifically, for a given attention layer, K-LoRA selects the Top-K elements with the highest values from $\Delta W_c'$ and $\Delta W_s'$, respectively. The importance of the two matrices at that layer is then assessed by summing these Top-K elements: $$\begin{array}{l} S_c = \sum_{i \in \mathrm{Top-K}(\Delta W_c')} \Delta W_{c,i}' , \\ S_s = \sum_{j \in \mathrm{Top-K}(\Delta W_s')} \Delta W_{s,j}' , \end{array}$$ where Top-K() is a function that returns the indices of the largest $K$ values in the input matrix, and $\Delta W_{c,i}'$ and $\Delta W_{s,j}'$ are the absolute values of the elements at those indices.

4.2.4. Determining the Value of K

The value of $K$ is crucial. The paper aligns $K$ with the rank of each LoRA, recognizing that the rank reflects the amount of information contained within the matrix: $K = r_c \cdot r_s ,$ where $r_c$ and $r_s$ represent the ranks of the content and style LoRA layers, respectively. This formulation aims to dynamically set $K$ based on the complexity of the LoRAs being fused.

The following figure (Figure 8 from the original paper) illustrates the impact of different values of K:

该图像是论文中的图表，展示了K-LoRA方法中不同K值选择对图像融合效果的影响。图中通过融合内容与风格的例子，比较了从K=4到K=16384及全部元素的差异，体现了不同K值对融合细节保留程度的影响。

As shown in Figure 8, when $K$ is too small, neither style nor object characteristics are prominent. As $K$ increases, performance improves. However, if $K$ becomes excessively large (e.g., using all elements), style might not be preserved, and object shape can distort. This suggests an optimal range for $K$ which the proposed formulation aims to find.

4.2.5. Initial LoRA Selection

Based on the calculated sums of Top-K elements, K-LoRA makes an initial decision on which LoRA to select for the current attention layer: $$C(S_c, S_s) = \left\{ \begin{array}{ll} \Delta W_c, & \mathrm{if} \ S_c \geq S_s \\ \Delta W_s . & \mathrm{otherwise} \end{array} \right.$$ This means if the sum of the Top-K absolute values of the content LoRA is greater than or equal to that of the style LoRA, the content LoRA is chosen; otherwise, the style LoRA is chosen.

4.2.6. Timestep-Dependent Scaling Factor

To leverage finding (ii) about the distinct roles of object and style across diffusion timesteps, a scaling factor $S$ is introduced. This factor enhances object content in early stages and gradually emphasizes style in later stages. $$S = \alpha \cdot \frac{t_{now}}{t_{all}} + \beta ,$$ where:

• $t_{now}$ is the current step in the backward denoising process (ranging from total steps down to 0).

• $t_{all}$ is the total number of diffusion steps.

• $\alpha$ and $\beta$ are hyperparameters. The paper sets $\alpha = 1.5$ and $\beta = 0.5$.

  This linear scaling ensures that at early timesteps (small $t_{now}/t_{all}$), $S$ is closer to $\beta$, and at later timesteps (large $t_{now}/t_{all}$), $S$ is larger, emphasizing style.

4.2.7. Balancing Factor for Community LoRAs

To account for potential weight disparities between LoRA models from different sources (e.g., locally trained vs. community models from Hugging Face), a balancing factor $\gamma$ is introduced. Such disparities, as shown in Figure 3(b), can make the Top-K selection ineffective. First, $\gamma$ is computed by summing the absolute values of elements across all layers for both content and style LoRAs: $$\gamma = \frac{\sum_l \sum_i \Delta W_{c_{l,i}}'}{\sum_l \sum_j \Delta W_{s_{l,j}}'} .$$ Here, $l$ iterates over all layers, and i, j iterate over elements within a layer. $\Delta W_{c_{l,i}}'$ and $\Delta W_{s_{l,j}}'$ are the absolute values of elements in the content and style LoRAs at layer $l$. This $\gamma$ essentially normalizes the overall "strength" of the content LoRA relative to the style LoRA across the entire model. The normalized scaling factor $S'$ is then calculated: $S' = \gamma \cdot S .$

The following figure (Figure 5 from the original paper) shows the ratio visualization of dominant components after considering $\gamma$:

该图像是图表，展示了不同层中Top 64元素和的比值变化，反映了style和content的比例关系。图中多个位置的比值显著偏离1，说明两者贡献存在明显差异。

Figure 5 illustrates that even after introducing $\gamma$, there are significant differences in the proportions of dominant components' sums across various forward layers, reinforcing the need for per-layer selection.

4.2.8. Final LoRA Selection

The adjusted scaling factor $S'$ is applied to the style LoRA's importance sum: $S_s' = S_s \cdot S' .$ Finally, the decision to select the content or style LoRA is made based on the updated $S_s'$: $$\Delta W_x = C(S_c, S_s') = \left\{ \begin{array}{ll} \Delta W_c, & \mathrm{if} \ S_c \geq S_s' \\ \Delta W_s . & \mathrm{otherwise} \end{array} \right.$$ This decision is made for each attention layer at each diffusion timestep.

The following figure (Figure 6 from the original paper) shows the selection proportions for content (blue) and style (green) across different layers and timesteps:

该图像是一个示意图，展示了LoRA权重选择在不同注意力层（layer index）和优化步骤（step）中的分布情况，体现了K-LoRA方法中Top-K权重的选择过程。

Figure 6 visually confirms that early layers/timesteps are predominantly chosen for objects (blue bars), while later layers/timesteps favor styles (green bars), with a smooth transition, supporting the core findings of the paper.

4.2.9. Algorithm Pseudocode

The process is summarized in Algorithm 1, presented in a PyTorch-like style:

# Algorithm 1 Pseudocode in a PyTorch-like style.

# timestep: current timestep (t_now in paper's math, but here used for scaling)
# content_lora_weight, style_lora_weight: input LoRA weight matrices for the current layer
# alpha, beta, gamma: scaling factors (hyperparameters)
# all_timesteps: total number of diffusion steps (t_all in paper's math)

# Set k based on rank (K = r_c * r_s)
# Note: The paper implies r_c and r_s are ranks of the *layer*, not overall LoRA.
# For simplicity in pseudocode, let's assume 'rank' refers to a combined effective rank.
# In implementation, this 'k' needs to be determined for each specific layer based on its ranks.
k = rank * rank # This 'rank' needs to be derived from r_c and r_s for the specific layer.
                # Assuming 'rank' is some effective rank for the current layer.
                # More accurately, it should be K = current_layer_rc * current_layer_rs

# Sum of TopK content values
abs_content_matrix = abs(content_lora_weight)
# .fl() is a placeholder for flatten or similar operation to get a 1D array for topk
topk_content_values = topk(abs_content_matrix.fl(), k)
sum_topk_content = sum(topk_content_values)

# Sum of TopK style values
abs_style_matrix = abs(style_lora_weight)
topk_style_values = topk(abs_style_matrix.fl(), k)
sum_topk_style = sum(topk_style_values)

# Compute and apply scaling factor S
# Note: 'timestep' here is t_now, and 'all_timesteps' is t_all.
# The calculation in the paper is S = alpha * (t_now / t_all) + beta
scale = alpha * (timestep / all_timesteps) + beta

# Apply balancing factor gamma
scale = scale * gamma

# Apply scaled factor to style's Top-K sum
sum_topk_style *= scale

# Compare and return the result
if sum_topk_content >= sum_topk_style:
    return content_lora_weight
else:
    return style_lora_weight

Symbol Explanation for Algorithm 1:

• timestep: Represents $t_{now}$, the current diffusion timestep (from the total all_timesteps).

• content_lora_weight: The $\Delta W_c$ matrix for the current attention layer.

• style_lora_weight: The $\Delta W_s$ matrix for the current attention layer.

• alpha, beta, gamma: Hyperparameters and balancing factor as defined in the paper.

• all_timesteps: Represents $t_{all}$, the total number of diffusion steps.

• $k$: The value $K = r_c \cdot r_s$ determined for the specific LoRA layer being processed.

• abs(): Element-wise absolute value function, producing $\Delta W_c'$ and $\Delta W_s'$.

• .fl(): A conceptual function representing flattening the matrix into a 1D array to apply topk.

• topk(array, k): A function that returns the $k$ largest values from the array.

• sum(): Summation function.

• sum_topk_content: Corresponds to $S_c$.

• sum_topk_style: Corresponds to $S_s$.

• scale: Corresponds to the unadjusted scaling factor $S$.

• scale * gamma: Corresponds to the adjusted scaling factor $S'$.

• sum_topk_style *= scale: Updates $S_s$ to $S_s'$.

• return content_lora_weight or return style_lora_weight: The selected LoRA weight matrix for the current attention layer and timestep, which will be added to the base model weights ($W_0 + \Delta W_x$).

  The following figure (Figure 19 from the original paper) shows an ablation study on Top-K selection and scaling factor, highlighting K-LoRA's benefits.

  该图像是多组对比示意图，展示了K-LoRA方法与B-LoRA在融合不同主题和风格下的生成效果。每组展示从内容、风格到不同场景（如骑车、睡觉、坐船、开车）的融合结果，直观体现了K-LoRA在保持主体与风格的平衡和细节表现上的优势。

5. Experimental Setup

5.1. Datasets

The experiments primarily use two types of datasets:

• Content (Subject) LoRAs: Obtained by training on images from the DreamBooth [24] dataset. Each subject typically consists of 4-5 images. DreamBooth is widely used for subject-driven generation.
  • Example Data Sample: A set of 4-5 images of a specific dog, a particular toy, or a unique person, intended to capture the identity of that subject.
• Style LoRAs: Selected from the dataset provided by the authors of StyleDrop [28]. This includes classic masterpieces and modern innovative styles. For each style, only a single image is used for training. StyleDrop focuses on generating images in arbitrary styles.

  • Example Data Sample: A single painting by Van Gogh (e.g., "Starry Night") for a "Van Gogh style" LoRA, or a single image exhibiting a distinct graphic design style.

    The choice of these datasets is standard in the field for evaluating personalization and stylization, allowing for a diverse set of subject-style combinations.

5.2. Evaluation Metrics

The paper employs both quantitative and qualitative metrics to assess the performance of K-LoRA:

5.2.1. Quantitative Metrics

• Style Similarity (Style Sim ↑): Measured using CLIP (Contrastive Language-Image Pre-training) [21].

  • Conceptual Definition: CLIP is a neural network trained on a vast dataset of image-text pairs. It learns to embed images and text into a shared latent space such that semantically similar image-text pairs are closer together. Style similarity here likely refers to the cosine similarity between the CLIP embeddings of the generated image and the target style image (or a textual description of the style). A higher value indicates better style preservation.
  • Mathematical Formula (for Cosine Similarity, commonly used for CLIP): $ \text{Similarity}(\mathbf{A}, \mathbf{B}) = \frac{\mathbf{A} \cdot \mathbf{B}}{|\mathbf{A}| |\mathbf{B}|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2}\sqrt{\sum_{i=1}^{n} B_i^2}} $
  • Symbol Explanation:
    • $\mathbf{A}$: CLIP embedding vector of the generated image.
    • $\mathbf{B}$: CLIP embedding vector of the target style image (or its textual description).
    • $A_i, B_i$: The $i$-th component of vectors $\mathbf{A}$ and $\mathbf{B}$, respectively.
    • $\|\mathbf{A}\|, \|\mathbf{B}\|$: The Euclidean (L2) norms of vectors $\mathbf{A}$ and $\mathbf{B}$.
    • $n$: Dimensionality of the CLIP embedding vectors.

• Subject Similarity (CLIP Score ↑): Also measured using CLIP [21].

  • Conceptual Definition: Similar to style similarity, but here it assesses how well the generated image preserves the identity and characteristics of the original subject. It typically measures the cosine similarity between the CLIP embedding of the generated image and the CLIP embedding of the original subject image or a textual description of the subject. A higher value indicates better subject preservation.
  • Mathematical Formula: Same cosine similarity formula as above.
  • Symbol Explanation:
    • $\mathbf{A}$: CLIP embedding vector of the generated image.
    • $\mathbf{B}$: CLIP embedding vector of the original subject image (or its textual description).

• Subject Similarity (DINO Score ↑): Measured using DINO (Self-supervised learning with Vision Transformers) [38].

  • Conceptual Definition: DINO is a self-supervised learning method for Vision Transformers that learns robust visual features. It is particularly effective at capturing fine-grained structural and semantic details of objects. The DINO score measures the similarity between the DINO features of the generated image and the original subject image, providing an alternative and often more perceptually aligned measure of subject preservation than CLIP. A higher value indicates better subject preservation.
  • Mathematical Formula (for Cosine Similarity, commonly used for DINO features): $ \text{Similarity}(\mathbf{F}{\text{gen}}, \mathbf{F}{\text{orig}}) = \frac{\mathbf{F}{\text{gen}} \cdot \mathbf{F}{\text{orig}}}{|\mathbf{F}{\text{gen}}| |\mathbf{F}{\text{orig}}|} $
  • Symbol Explanation:
    • $\mathbf{F}_{\text{gen}}$: DINO feature vector of the generated image.
    • $\mathbf{F}_{\text{orig}}$: DINO feature vector of the original subject image.

5.2.2. Qualitative Metrics

• User Preference: A user study where participants are asked to identify which method best preserves both style and subject from a set of generated images by different methods. This provides a subjective yet crucial measure of perceived quality.
• GPT-4o Feedback: An assessment performed by the advanced AI model, GPT-4o, for similar comparative evaluation. This offers an objective (though AI-driven) qualitative assessment.

5.3. Baselines

The paper compares K-LoRA against several popular and state-of-the-art methods for LoRA fusion:

• Direct Arithmetic Merging (Direct): This typically involves simply adding or averaging the LoRA weights, possibly with fixed coefficients (e.g., $W_0 + w_c \Delta W_c + w_s \Delta W_s$). The paper refers to it as MergingLoRA [25] in qualitative comparisons.

• Joint Training (Joint): This implies training a single model or a fusion mechanism with both subject and style objectives simultaneously, or fine-tuning existing LoRAs together.

• B-LoRA [8]: A technique that focuses on training only two attention modules to facilitate style transfer, achieving object-style decoupling.

• ZipLoRA [26]: A method that attempts to learn a fusion ratio vector to balance different LoRAs, aiming for effective subject-style combinations.

• Fixed Selection: An ablation baseline proposed in the paper, where content LoRA is selected if the scaling factor is greater than one, otherwise style LoRA is chosen. This is also referred to as an extension of Multi-LoRA Composition [41].

• Random Selection: An ablation baseline where content or style LoRA is chosen randomly with certain probabilities (e.g., 1/3 for content, 2/3 for style).

• StyleID [5]: An additional comparison in the supplementary material, which achieves style transfer while preserving texture but may lead to blurred objects or less distinct styles.

  The experiments are conducted using the SDXL v1.0 base model and the FLUX model, testing with both locally trained LoRAs and community-available LoRAs from Hugging Face. For the scaling factor parameters, $\alpha = 1.5$ and $\beta = 0.5$ are used, found to be effective across most cases.

6. Results & Analysis

6.1. Core Results Analysis

6.1.1. Quantitative Comparisons

The paper presents quantitative results comparing K-LoRA with several baselines across style similarity and subject similarity metrics (CLIP Score and DINO Score).

The following are the results from Table 1 of the original paper:

Analysis:

• Subject Similarity (CLIP Score and DINO Score): K-LoRA achieves the highest CLIP Score (69.4%) and DINO Score (46.9%) among all methods. This indicates its superior ability to preserve the identity and characteristics of the original subject during the fusion process. Notably, the Joint training method performs poorly on DINO Score (17.4%), suggesting that while it might capture some style, it struggles to maintain subject details. Direct merging also shows good CLIP Score but is outpaced by K-LoRA.
• Style Similarity (Style Sim): K-LoRA achieves a Style Sim of 58.7%, which is competitive. While Joint training has the highest style similarity (68.2%), it comes at a significant cost to subject preservation. K-LoRA demonstrates a strong balance, achieving high subject fidelity while maintaining satisfactory style transfer, which is a key challenge in this domain. The results suggest K-LoRA successfully mitigates the trade-off between style and content preservation.

6.1.2. Qualitative Comparisons

Qualitative results are primarily shown in Figure 7, with additional comparisons in the supplementary material (Figures 11-19).

The following figure (Figure 7 from the original paper) shows qualitative comparisons:

Analysis from Figure 7 and text:

• Direct Arithmetic Merging [25]: Often struggles to preserve the original shape, color, and stylistic features, especially without extensive parameter adjustments. This validates the problem of concept dilution when simply combining weights.
• B-LoRA [8]: Tends to capture the color and appearance of objects, but can lead to overfitting of color, making it difficult to distinguish original objects or resulting in a less distinct style.
• ZipLoRA [26] and Joint Training: While incorporating some stylistic textures, these methods often focus on background elements of the style rather than capturing the essence of the style itself, leading to a lower success rate in truly stylizing the subject.
• K-LoRA (Ours): Addresses these limitations by producing higher-quality output images with stable performance across various seed variations. It effectively balances subject and stylistic features, leading to seamless integration without distortion. This highlights K-LoRA's ability to maintain fidelity for both components.

6.1.3. User Study and GPT-4o Feedback

User studies and AI-driven assessments provide further qualitative validation.

The following are the results from Table 2 of the original paper:

Analysis:

• K-LoRA is the most preferred method by human users (52.7%) and overwhelmingly favored by GPT-4o (83.3%). This strong preference, particularly from GPT-4o which can perform detailed visual assessments, further substantiates the superiority of K-LoRA in combining subject and style effectively and aesthetically.

6.2. Ablation Studies / Parameter Analysis

The paper conducts ablation studies to validate the effectiveness of its key components: Top-K selection and the scaling factor.

The following figure (Figure 9 from the original paper) shows ablation results for Top-K selection and scaling factor:

该图像是图表，展示了论文中Top-K选择机制与缩放因子消融实验的结果。图中分为SD和FLUX两部分，比较了不同选择方法对融合效果的影响，通过六列对比展示了内容、风格、Top-K选择、随机选择、固定选择和作者方法下的融合图像差异。

6.2.1. Top-K Selection Ablation

Two alternatives to the Top-K selection were tested:

• Fixed Selection: In this approach, if the scaling factor $S'$ (which incorporates $S$ and $\gamma$) is greater than one, the content LoRA is selected; otherwise, the style LoRA is chosen.
  • Results (Figure 9): While it can yield satisfactory results in some cases, it leads to object blurring or alterations in content appearance under specific style LoRA conditions. This shows that a simple threshold-based decision is insufficient compared to the more nuanced Top-K comparison.
• Random Selection: In this setup, the model randomly selects content (1/3 probability) or style (2/3 probability) attention.
  • Results (Figure 9): Generated images often retain only a single object or style feature, or fail to maintain either altogether. This underscores the importance of the adaptive, non-random selection process and validates the paper's finding (ii) about timestep-dependent roles.

Impact of K (Figure 8):

• Small K: When $K$ is too small, neither the style nor the object characteristics are sufficiently prominent, indicating a loss of important information.
• Large K: If $K$ becomes excessively large (e.g., using all elements), the style may not be preserved, and the shape of the object can undergo significant distortions.
• Optimal K: The choice of $K = r_c \cdot r_s$ aims to find a balance, ensuring enough information is considered without diluting the distinct contributions.

6.2.2. Scaling Factor Ablation

• Removal of Scaling Factor: When the scaling factor module (which includes $S$ and $\gamma$) is entirely removed, and only the original Top-K approach is used.
  • Results (Figure 9): Although Top-K alone can yield good results occasionally, broader experiments reveal instances of object distortion and style loss. This demonstrates the critical role of the dynamic, timestep-dependent emphasis provided by the scaling factor in guiding the fusion process for better overall fidelity.
• Significance of Gamma ($\gamma$): To assess $\gamma$'s importance, LoRA models from distinct sources with significant numerical differences in their element sums were tested.

  • Results (Figure 9, bottom row): Without $\gamma$, Top-K selection fails to capture the style accurately, and the fusion quality in Fixed Selection is noticeably weaker. This confirms that $\gamma$ is essential for balancing numerical discrepancies between disparate LoRAs, ensuring fair comparison of Top-K sums.

    The following figure (Figure 10 from the original paper) compares results with different scaling factors:

    该图像是论文中的插图，展示了不同内容和风格图像经K-LoRA方法融合后的生成结果。每行显示不同内容图像与风格图像结合，S与S结果分别体现了两种不同融合策略下的输出效果。*

6.2.3. Alternative Scale $S^*$

The paper also explores an alternative scale factor, $S^*$, compared to its primary scale $S$. $$S = \alpha \cdot \frac{t_{now}}{t_{all}} + \beta$$ $$S^* = \left( \alpha' \cdot \frac{t_{now}}{t_{all}} + \beta' \right) \% \alpha$$ With $\alpha' = 1.5$ and $\beta' = 1.3$, $S^*$ enhances style information at the beginning of the generation process.

• Results (Figure 10): $S^*$ results show that while the generated images capture background and color block information from the style LoRA due to early style emphasis, this comes at the cost of weakened learning of texture and brushstroke information. This demonstrates a trade-off, allowing users to choose scales based on preference, but reinforcing the chosen $S$ for overall fidelity.

6.2.4. Hyperparameter ($\alpha$, $\beta$) Selection

The paper conducts an additional ablation to find optimal $\alpha$ and $\beta$ for the scaling factor. They compute the sum of CLIP similarity scores (for both object and style) for 18 random combinations across different $\alpha$ and $\beta$ values.

The following are the results from the ablation table in section F of the original paper:

Analysis: The highest summed CLIP similarity score (128.1%) is achieved when $\alpha = 1.5$ and $\beta = 0.5$. This empirically validates the chosen hyperparameters as optimal for balancing content and style preservation, making the method robust and reducing the need for user adjustments.

In conclusion, the ablation studies consistently demonstrate that both the Top-K selection mechanism and the dynamic scaling factor (including $\gamma$) are critical components, each contributing significantly to K-LoRA's superior generative performance. Their removal or naive alternatives lead to a decrease in the quality and fidelity of the fused images.

6.3. Additional Visual Results

The supplementary material provides extensive visual results on SDXL and FLUX models, using both locally trained and community LoRAs, further demonstrating K-LoRA's robustness and generalizability (Figures 11-19). These figures show consistent high-quality fusion across diverse subjects and styles, the ability to modify object actions via prompts, and stability across random seeds.

The following figure (Figure 11 from the original paper) shows various generated images using K-LoRA:

该图像是示意图，展示了通过K-LoRA方法实现不同内容（content）与风格（style）LoRA的训练自由融合效果。图中以猫、青蛙、动物等内容为基底，分别融入多种艺术风格，体现了方法在保持原始内容同时成功转换风格的能力。

The following figure (Figure 12 from the original paper) shows more generated images using K-LoRA:

该图像是多组动画风格插图示意图，展示了不同风格和内容LoRA模型融合的效果。图中按内容和风格排列，显示对不同主体（如猫、青蛙、狗等）应用多种风格后的融合表现，验证了K-LoRA在无训练条件下实现风格与内容融合的能力。

The following figure (Figure 13 from the original paper) shows results generated by applying K-LoRA with different LoRAs:

该图像是一张示意图，展示了不同内容与风格的融合效果。图中横向第一排是多种内容图像，纵向第一列是多种风格图像，其余部分为内容与风格结合后的生成结果，体现了K-LoRA方法在训练免费条件下融合多样内容和风格的能力。

The following figure (Figure 14 from the original paper) shows additional results generated by applying K-LoRA with different LoRAs:

该图像是一个示意图，展示了不同内容（Content）和风格（Style）LoRA融合后生成的图像效果。每一行代表不同的风格LoRA，每一列代表不同的内容LoRA，图像通过训练自由的K-LoRA方法融合，清晰保留了对应的内容和风格特征。

The following figure (Figure 15 from the original paper) shows a comparison with StyleID:

该图像是论文中多个内容与风格融合对比示意图，展示了K-LoRA方法在不同内容与风格组合下的生成效果，包含原始内容与风格图像、StyleID、放大局部细节和K-LoRA生成结果，突出该方法在保持内容和风格特征上的优势。

The following figure (Figure 16 from the original paper) shows results comparing K-LoRA with other methods for generalizability and robustness:

该图像是论文中的示意图，展示了K-LoRA方法与其他融合策略（Fixed Selection、Direct Merge、LoRA Switch）在不同风格和内容融合上的对比效果，突出K-LoRA在保留内容与风格上的优势。

The following figure (Figure 17 from the original paper) shows robustness validation with random seeds:

该图像是一组图像对比示意图，展示了不同主题和风格LoRA融合后的效果。每组第一张为原始图像，后续依次为不同艺术风格渲染，突出融合技术对保持原始内容和风格的表现力。

The following figure (Figure 18 from the original paper) demonstrates prompt control capabilities of K-LoRA:

该图像是对比不同方法融合主体和风格效果的示意图，展示了猫、狗等不同主体融合多种风格和道具（球、飞盘、帽子、冠冕）后的生成结果。图中“ Ours”和“B-LoRA”两种方法效果对比，突出本文所提方法在保持主体特征和风格方面的优势。

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper successfully introduces K-LoRA, a novel and highly effective training-free approach for fusing independently trained subject and style LoRA models. K-LoRA addresses key limitations of existing methods, particularly the need for additional training and the challenge of simultaneously preserving both content and style fidelity. Its core innovations include a Top-K selection mechanism that adaptively chooses between content and style LoRAs at each attention layer based on element importance, and a dynamic scaling factor that guides the fusion process by emphasizing content in early diffusion steps and style in later steps. Extensive experiments demonstrate that K-LoRA not only effectively integrates subject and style information but also quantitatively and qualitatively outperforms state-of-the-art training-based approaches, providing stable, high-quality generative outputs without the need for retraining or manual hyperparameter tuning.

7.2. Limitations & Future Work

The paper does not explicitly state a dedicated "Limitations" section, but potential areas for future exploration can be inferred:

• Sensitivity of K: While the formula $K = r_c \cdot r_s$ is proposed, the optimal value of $K$ might still be sensitive to the specific characteristics of different LoRAs or extremely diverse subject-style combinations. The ablation study shows the balance is delicate.

• Generalizability of Hyperparameters: The empirically determined $\alpha = 1.5$ and $\beta = 0.5$ for the scaling factor are shown to work effectively across many cases, but there might be specific niche cases or extreme styles/subjects where these fixed values are suboptimal.

• Complexity of Fusion Scenarios: K-LoRA focuses on one subject and one style. More complex scenarios involving multiple subjects, multiple styles, or hierarchical style applications might require extensions beyond the current pairwise selection.

• Implicit Assumptions: The method relies on the assumption that "importance" can be accurately captured by the magnitude of Top-K elements and that the diffusion timesteps consistently behave in the hypothesized manner (early for content, late for style). While supported by experiments, these are empirical observations that could be further theoretically grounded.

  Future work could involve:

• Adaptive K Selection: Developing a more dynamic and data-driven approach to determine $K$ for each layer or even element.

• Learned Scaling Factors: Instead of fixed $\alpha, \beta$, exploring meta-learning or a lightweight network to predict optimal scaling parameters based on input LoRAs and prompts.

• Multi-Concept Fusion: Extending K-LoRA to handle the simultaneous fusion of more than two LoRAs (e.g., two subjects with one style, or one subject with two styles).

• Theoretical Analysis: Providing deeper theoretical insights into why the magnitude of LoRA elements and the timestep-dependent application work so effectively.

7.3. Personal Insights & Critique

This paper presents an elegant and practical solution to a common problem in generative AI. The training-free nature of K-LoRA is its most compelling feature, significantly lowering the barrier for users to combine diverse LoRAs without specialized machine learning knowledge or computational resources. The core idea of leveraging Top-K element importance and timestep-dependent scaling is intuitive and well-supported by the ablation studies. It’s a smart way to perform a "surgical" fusion rather than a "blending" one, which often leads to better preservation of distinct features.

The idea that different diffusion timesteps are responsible for different aspects of image generation (content vs. style) is a powerful insight, and K-LoRA effectively operationalizes this. The use of a simple linear scaling factor for this temporal emphasis is effective, and the introduction of $\gamma$ to normalize LoRAs from different sources demonstrates a practical understanding of real-world LoRA usage.

A potential area for critique or deeper exploration could be the definition of $K = r_c \cdot r_s$. While it intuitively links to the complexity of the LoRAs, it's a fixed product. Could a more adaptive or context-aware $K$ further improve results, perhaps by considering the semantic content or visual complexity? Also, while "training-free" is a huge advantage, the fixed hyperparameters ($\alpha, \beta$) still represent a form of tuning, albeit a simple one. Future work might explore how to make even these parameters more dynamic.

Overall, K-LoRA provides a valuable contribution by offering a robust, efficient, and user-friendly method for LoRA fusion, directly addressing practical challenges faced by generative AI practitioners and researchers. Its principles of selective fusion and temporal adaptation could inspire further research in fine-grained control of diffusion models.