1. Bibliographic Information

1.1. Title

FreeFuse: Multi-Subject LoRA Fusion via Auto Masking at Test Time

1.2. Authors

• Yaoli Liu (State Key Laboratory of CAD&CG, Zhejiang University)
• Yao-Xiang Ding (State Key Laboratory of CAD&CG, Zhejiang University)
• Kun Zhou (State Key Laboratory of CAD&CG, Zhejiang University)

1.3. Journal/Conference

The paper is published as a preprint on arXiv. The listed publication date (2025-10-27T16:54:08.000Z) suggests it is a very recent or upcoming work. While arXiv is a reputable platform for disseminating research quickly, it is not a peer-reviewed journal or conference proceeding itself. Papers published on arXiv are typically submitted to conferences or journals later. The affiliations indicate a strong academic background from Zhejiang University, which is a well-regarded institution in computer science and related fields.

1.4. Publication Year

2025 (as indicated by the arXiv timestamp, though this is a future date, implying it's a recent submission for upcoming review cycles).

1.5. Abstract

This paper introduces FreeFuse, a novel approach for generating images with multiple subjects from text prompts. It works by automatically fusing multiple subject LoRAs (Low-Rank Adaptations) without requiring any additional training. Unlike existing methods that merge LoRA weights before inference or use complex techniques like segmentation models and noise blending, FreeFuse leverages the insight that dynamic, context-aware subject masks can be automatically extracted from cross-attention layer weights during inference. Mathematical analysis supports that applying these masks directly to LoRA outputs during inference closely approximates the ideal scenario where each subject LoRA is used individually for its masked region within the diffusion model. FreeFuse is highly practical and efficient, as it needs no extra training, no modifications to existing LoRAs, no auxiliary models, and no specific user-defined prompt templates or region specifications. Users only need to provide LoRA activation words to integrate it into standard workflows. Extensive experiments show that FreeFuse surpasses current methods in both generation quality and ease of use for multi-subject generation tasks.

1.6. Original Source Link

https://arxiv.org/abs/2510.23515 (Preprint) PDF Link: https://arxiv.org/pdf/2510.23515v1.pdf

2. Executive Summary

2.1. Background & Motivation

The core problem FreeFuse aims to solve is the effective and high-quality generation of multi-subject text-to-image content using Low-Rank Adaptations (LoRAs). Large-scale Text-to-Image (T2I) models like FLUX.1-dev have shown impressive performance in general image generation. To personalize these models, LoRAs have become a popular choice due to their efficiency and precision in fine-tuning.

However, a significant challenge arises when attempting to combine multiple subject LoRAs (e.g., for different characters) within a single image. A straightforward combination often leads to performance degradation, characterized by feature conflicts and quality deterioration. This happens because LoRAs tend to compete strongly in key regions (like faces), causing features from one subject to bleed into another or leading to a confused, averaged appearance.

Prior research has attempted to address this with various techniques:

• Pre-inference LoRA weight merging (e.g., ZipLoRA).

• Methods requiring retraining or additional trainable parameters (e.g., Mix-of-Show).

• Approaches relying on external segmentation models or complex noise blending (e.g., OMG, Concept Weaver).

• Methods demanding user-defined template prompts or region specifications (e.g., CLoRA, Mix-of-Show).

  These existing solutions often struggle with complex multi-subject interactions, lack efficiency, or impose significant burdens on the user, making them less practical for everyday use. The paper identifies a clear gap: a need for a training-free, user-friendly, and efficient method that can effectively mitigate feature conflicts for multi-subject generation in advanced DiT models.

2.2. Main Contributions / Findings

The paper's primary contributions are threefold:

1. Analysis of Feature Conflicts in Multi-Subject LoRA Inference: The authors provide an in-depth analysis revealing that the main issue stems from subject LoRAs interfering with regions designated for other subjects during joint inference. They mathematically demonstrate that constraining each LoRA's output to its target region via masks can effectively alleviate these conflicts. This finding underpins the entire FreeFuse approach.

2. General Solution for Mitigating LoRA Interference: FreeFuse offers a novel solution that effectively isolates conflicts between LoRAs in DiT models. It preserves the original LoRA weights, even in cases of overfitting, by using masks automatically derived from attention maps. Crucially, this solution requires no new trainable parameters, no modifications to LoRA modules, no auxiliary models, and no additional prompts from users, making it highly compatible and practical.

3. Portable and Highly Efficient Framework (FreeFuse): The paper introduces FreeFuse as an end-to-end framework for multi-subject scene generation. It operates in two stages:

   • Mask Generation: Automatically computes high-quality subject masks from cross-attention maps by addressing the attention sink phenomenon, exploiting the locality of self-attention, and applying patch-level voting. This is done very efficiently, using only one step and one attention block during inference.
   • Mask Application: Directly constrains LoRA outputs to their respective masked regions during inference, avoiding complex strategies like feature replacement or noise blending. Experimental results demonstrate that FreeFuse significantly outperforms previous methods in terms of alleviating feature conflicts, enhancing image quality, maintaining subject fidelity, and adhering to complex prompts, all while being highly practical and seamlessly integratable into standard T2I workflows.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand FreeFuse, a foundational understanding of Text-to-Image (T2I) diffusion models and Low-Rank Adaptations (LoRAs) is essential.

• Diffusion Models: These are a class of generative models that learn to reverse a gradual diffusion (noise-adding) process. During training, they learn to denoise data (e.g., images) corrupted with Gaussian noise. During inference, they start with random noise and iteratively remove noise, guided by a text prompt, to generate a coherent image. The core idea is to transform noise into meaningful data over several steps.

  • Denoising Diffusion Probabilistic Models (DDPMs): A prominent type of diffusion model that performs denoising over many discrete steps.
  • Latent Diffusion Models (LDMs): A variant that applies the diffusion process in a compressed latent space rather than directly on pixel space, significantly reducing computational cost and allowing for higher-resolution image generation. Models like Stable Diffusion are LDMs.

• Text-to-Image (T2I) Generation: The task of generating an image from a given text description. Diffusion models are currently state-of-the-art for T2I generation.

• Diffusion Transformers (DiT): An advanced architecture for diffusion models that replaces the traditional U-Net backbone with a Transformer network. Transformers are powerful neural network architectures that have shown great success in natural language processing and, more recently, in computer vision. DiT models process image latents as sequences of patches (tokens), allowing them to scale efficiently to higher resolutions and leverage the Transformer's capacity for global reasoning. FLUX.1-dev is an example of a DiT-based model.

• Low-Rank Adaptation (LoRA): A parameter-efficient fine-tuning technique. Instead of fine-tuning all the weights of a large pre-trained model (like a T2I model), LoRA injects small, trainable rank-decomposition matrices into the Transformer layers. This means that for a pre-trained weight matrix $W_0$, the update is represented as $W_0 + BA$, where $B$ and $A$ are low-rank matrices ($B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with $r \ll \min(d, k)$). Only $A$ and $B$ are trained, significantly reducing the number of trainable parameters and computational cost while retaining high performance. LoRAs are widely used for personalized generation, allowing users to train specific concepts (e.g., a particular character, object, or style) efficiently.

• Cross-Attention Mechanism: A key component in Transformer architectures, especially in T2I models. It allows information to flow between different modalities, typically between text embeddings (from the prompt) and image latent tokens. In cross-attention, Query (Q) vectors are derived from one modality (e.g., image latents), and Key (K) and Value (V) vectors are derived from another modality (e.g., text embeddings). The attention weights indicate how much each image latent token "attends" to each word in the text prompt, effectively localizing semantic information in the image.

  $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ where:

  • $Q$ is the Query matrix.
  • $K$ is the Key matrix.
  • $V$ is the Value matrix.
  • $d_k$ is the dimension of the Key vectors, used for scaling to prevent very small gradients.
  • $\mathrm{softmax}$ normalizes the attention scores.
  • The Attention mechanism computes a weighted sum of Value vectors, where the weights are determined by the similarity between Query and Key vectors.

• Self-Attention Mechanism: Similar to cross-attention, but here the Query, Key, and Value vectors all come from the same modality (e.g., image latent tokens). It allows each latent token to attend to all other latent tokens within the same image, capturing long-range dependencies and global contextual information within the image itself.

• Attention Sink: A phenomenon observed in Transformer-based models, particularly in attention maps. It refers to the tendency for a few tokens (often boundary or trivial tokens) to accumulate a disproportionately high amount of attention weights, effectively "sinking" attention that should be distributed more meaningfully across important semantic regions. This can lead to misleading or noisy attention maps if not handled.

• Superpixels (SLIC - Simple Linear Iterative Clustering): A method for over-segmenting an image into small, perceptually uniform regions (superpixels). Instead of processing individual pixels, SLIC groups pixels into compact, nearly uniform regions, which can then be treated as individual units. This reduces the number of entities to process and often aligns better with object boundaries than rigid grid-based segmentation. It is often used for simplifying image processing tasks and improving computational efficiency by leveraging local image coherence.

3.2. Previous Works

The paper contextualizes FreeFuse by discussing existing approaches to multi-concept generation using LoRAs in T2I models, drawing parallels with similar challenges in Large Language Models (LLMs).

• In LLMs: Multi-LoRA inference in LLMs also faced quality drops. Solutions included:

  • Clustering LoRAs in advance [Zhao et al., 2024].
  • Introducing gating networks [Wu et al., 2024].
  • Retraining with conflict-mitigation objectives [Feng et al., 2025].

• In T2I Models (LoRA-based):

  • Pre-inference LoRA Weight Merging: Methods like ZipLoRA [Shah et al., 2024] and K-LoRA [Ouyang et al., 2025] fuse multiple LoRAs before inference. While successful for tasks like style transfer, they struggle with multi-concept generation (especially characters) due to inherent conflicts that arise from simply summing weights.
  • Inference-time Conflict Mitigation:
    • Multi-LoRA [Zhong et al., 2024]: Uses switch and composite strategies during inference. Showed promise for character-object compositions but struggled with multi-character scenarios.
    • OMG (Occlusion-friendly Multi-concept Generation) [Kong et al., 2024]: Introduces an auxiliary model for character localization and applies noise blending. Its reliance on an external model and LoRA's redraw tendency can lead to failures with complex interactions.
    • Mix-of-Show [Gu et al., 2023]: Requires retraining LoRAs and manually specifying spatial constraints (e.g., bounding boxes). This is labor-intensive and lacks cross-LoRA awareness, failing in close interaction scenes.
    • Concept Weaver [Kwon et al., 2024]: Mitigates issues with Fusion Sampling but still heavily relies on the quality of segmentation.
    • CLoRA [Meral et al., 2024]: Leverages attention maps to derive concept masks. However, it requires template prompts for mask extraction, limiting its performance in complex multi-concept scenes.

3.3. Technological Evolution

The field of image generation has seen rapid advancements:

1. Early Generative Models (GANs): Starting with Generative Adversarial Networks (GANs) [Goodfellow et al., 2014], models learned to generate realistic images by having two networks (generator and discriminator) compete. Subsequent improvements like WGANs [Arjovsky et al., 2017] and StyleGANs [Karras et al., 2019, 2020] pushed the boundaries of image quality and control.

2. U-Net Based Diffusion Models: The emergence of Denoising Diffusion Probabilistic Models (DDPMs) [Ho et al., 2020] marked a paradigm shift. These models, often using U-Net [Ronneberger et al., 2015] as their backbone, showed superior synthesis quality and diversity. Latent Diffusion Models (LDMs) [Rombach et al., 2022] further refined this by performing diffusion in a latent space, making them computationally more efficient for high-resolution images.

3. DiT-Based Diffusion Models: The latest evolution involves replacing the U-Net backbone with Transformer architectures, leading to Diffusion Transformers (DiT) [Podell et al., 2023; Peebles and Xie, 2023]. Models like FLUX.1-dev [Labs, 2024; Labs et al., 2025] leverage the scalability and global reasoning capabilities of Transformers for even better performance in T2I tasks.

4. Personalized Generation and LoRAs: Alongside these architectural advancements, personalized generation has become a major focus. Techniques like Textual Inversion [Gal et al., 2022], DreamBooth [Ruiz et al., 2023], IP-Adapter [Ye et al., 2023], and especially LoRA [Hu et al., 2022] have enabled users to customize T2I models with specific concepts, objects, or styles without extensive retraining. LoRA has become the most widely adopted due to its efficiency.

   FreeFuse fits into this timeline by addressing a critical challenge—multi-subject generation with LoRAs—specifically within the context of the powerful and efficient DiT-based diffusion models, building upon the capabilities of LoRA while overcoming its inherent limitations in multi-concept scenarios.

3.4. Differentiation Analysis

FreeFuse differentiates itself from existing multi-LoRA multi-concept generation methods primarily through its emphasis on practicality, efficiency, and training-free operation, particularly for DiT models.

The following table, adapted from Table 1 in the paper, summarizes the key differentiators:

The following are the results from Table 1 of the original paper:

• Training-Free and No LoRA Modification: Unlike Mix-of-Show which requires retraining LoRAs or OMG which relies on specific LoRA redraw tendencies, FreeFuse operates entirely without retraining or modifying the existing LoRAs. This makes it highly portable and compatible with any pre-trained LoRA.

• No Auxiliary Models: OMG utilizes an auxiliary model for character localization, adding complexity and potential points of failure. FreeFuse generates its masks solely from the internal attention mechanisms of the diffusion model itself, avoiding external dependencies.

• No User-Defined Prompts/Regions: Mix-of-Show demands manual spatial constraints, and CLoRA requires template prompts to derive masks. FreeFuse automatically derives dynamic, context-aware subject masks without any explicit user input regarding regions or specific prompt structures, requiring only LoRA activation words.

• Cross-LoRA Awareness: Many methods, particularly those relying on static merging or rigid regional constraints, lack awareness of how different LoRAs might interact. FreeFuse implicitly handles this by dynamically generating masks based on attention maps, which reflect the interplay of LoRAs during inference.

• Efficiency: FreeFuse extracts subject masks from a single attention block at a single denoising step (e.g., 17th block at 6th step for FLUX.1-dev), offering significant efficiency gains over approaches that might repeatedly update attention maps or require more intensive computations throughout the inference process.

• Applicability to DiT Models: Most prior multi-LoRA methods were implemented on U-Net-based models. FreeFuse explicitly addresses and demonstrates its capabilities within the more advanced DiT-based models (e.g., FLUX.1-dev), which are becoming the new standard.

  In essence, FreeFuse provides a highly practical, efficient, and user-friendly solution that automates the complex task of multi-subject LoRA fusion by intelligently leveraging the intrinsic attention mechanisms of modern DiT models, distinguishing itself through its "training-free" and "no-user-intervention" philosophy.

4. Methodology

4.1. Principles

The core principle behind FreeFuse is to mitigate feature conflicts during multi-subject LoRA joint inference by ensuring each subject LoRA only influences its intended region within the generated image. The key insight is that context-aware dynamic subject masks can be automatically and efficiently derived from the cross-attention layer weights of the diffusion model.

The theoretical basis is that by applying a spatial mask to the output of a LoRA, the influence on the masked region is nearly identical to the scenario where that LoRA was used individually for that region. This approximation holds because:

1. LoRAs primarily inject semantic features into the feed-forward (FF) and value (V) layers of the Transformer.

2. LoRA outputs ($\Delta \mathbf{h}$) are generally much smaller in magnitude than the base model's outputs.

3. Attention mechanisms exhibit locality, meaning tokens within a target region primarily attend to each other, with minimal influence from outside the region.

   This allows FreeFuse to "localize" the effect of each LoRA without modifying the LoRA itself or retraining, effectively preventing feature bleeding and competition.

4.2. Core Methodology In-depth (Layer by Layer)

FreeFuse operates in a two-stage pipeline:

1. Stage 1: Subject Mask Calculation: Automatic derivation of high-quality subject masks from attention maps.

2. Stage 2: Mask Application: Repeatedly applying these masks to LoRA outputs during inference.

   The pipeline is illustrated in Figure 5 from the original paper:

   该图像是论文FreeFuse中的示意图，展示了多主体LoRA融合的工作流程。图中通过对不同主体LoRA输出应用自动生成的掩码，再与原始模块输出合并，最终进行去噪步骤生成融合结果。

Figure 5: Pipeline. Our pipeline consists of two stages: the first derives subject masks from attention maps, and the second applies these masks to LoRA outputs, ensuring that each LoRA only operates within its corresponding subject region.

4.2.1. Analysis of Feature Conflicts and Masking Justification

The paper first highlights the problem: when multiple subject LoRAs are applied, they compete in key regions (like faces), leading to feature conflicts and confusion, as shown in Figure 3a.

该图像是图表和插图，展示了不同LoRA层被禁用后的视觉效果和平均MSE损失对比。左侧展示了禁用Q和K层、V层、投影输出层以及前馈层对生成图像的影响，右侧柱状图显示前馈层与V层禁用导致MSE损失显著增加。

Figure 3: Conflicts Ananlysis

To address this, the proposed solution is to apply a spatial mask $\mathbf{M}$ to the LoRA output $\Delta \mathbf{h}$ for each subject.

The masked LoRA output, denoted as $\tilde{\Delta \mathbf{h}}$, is calculated as: $ \tilde{\Delta \mathbf{h}} = \mathbf{M} \odot \Delta \mathbf{h} $ where:

• $\tilde{\Delta \mathbf{h}}$ is the masked LoRA output.

• $\mathbf{M}$ is a binary spatial mask, where elements are 1 within the specified target region for a subject and 0 elsewhere.

• $\Delta \mathbf{h}$ is the original output of the LoRA (the change it proposes to the base model's hidden states).

• $\odot$ denotes the element-wise (Hadamard) product.

  The authors justify this by observing that LoRAs primarily modify the feed-forward (FF) and value (V) layers. This is supported by experiments shown in Figure 4, which demonstrates that disabling LoRA components in FF or $V$ layers causes significant semantic loss, indicating these are the primary injection points for semantic information.

  该图像是图表和插图，展示了不同LoRA层被禁用后的视觉效果和平均MSE损失对比。左侧展示了禁用Q和K层、V层、投影输出层以及前馈层对生成图像的影响，右侧柱状图显示前馈层与V层禁用导致MSE损失显著增加。

Figure 4: Left: Experiments show that removing LoRA from the feedforward (FF) and value (V) layers causes relatively significant semantic loss than removing it from other layers. Right: We randomly downloaded 45 FLUX-based LoRAs from Civitai and sampled 225 images. Results show that disabling the FF or V layers causes a large increase in L2 loss, while other layers have limited effect, indicating that semantic information is primarily injected through the V and FF layers.

Furthermore, attention mechanisms in DiT models exhibit locality. Tokens within a subject's region primarily attend to other tokens within that same region. The attention output of a LoRA in one attention block can be represented as: $ \mathrm{Attn}(\mathbf{Q}, \mathbf{K}, \mathbf{V} + \Delta \mathbf{V}) = \mathrm{softmax} \left( \frac{\mathbf{Q} \mathbf{K}^{\top}}{\sqrt{D}} \right) (\mathbf{V} + \Delta \mathbf{V}) $ where:

• $\mathrm{Attn}$ represents the attention mechanism.

• $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ are the Query, Key, and Value matrices from the base diffusion model.

• $\Delta \mathbf{V}$ captures the contribution of the LoRA (from both its FF layer in the previous block and its $V$ layer in the current block).

• $D$ is the token dimension (for scaling).

  The locality of attention means that for a mask $\mathbf{M}$, the sum of attention weights between tokens within the masked region is much greater than the sum of attention weights from tokens inside to tokens outside the region: $ \sum_{i : \mathbf{M}i = 1} \sum{j : \mathbf{M}j = 1} \alpha{ij} \gg \sum_{i : \mathbf{M}i = 1} \sum{j : \mathbf{M}j = 0} \alpha{ij} $ where $\alpha_{ij}$ is the attention weight from token $i$ to token $j$.

This strong locality implies that the semantic representation of target tokens is dominated by FF and value semantics from within the masked region. Therefore, masking the LoRA output ($\Delta \mathbf{h}$) leads to a result within the masked region that is approximately equivalent to using the full LoRA output: $ \mathbf{M} \odot f(\mathbf{h} + \Delta \mathbf{h}) \approx \mathbf{M} \odot f(\mathbf{h} + \tilde{\Delta \mathbf{h}}) $ where:

• $f$ is the diffusion generation model.

• $\mathbf{h}$ is the hidden state before LoRA application.

• $\Delta \mathbf{h}$ is the full LoRA output.

• $\tilde{\Delta \mathbf{h}}$ is the masked LoRA output.

  This mathematical justification supports the idea that mask-based LoRA output fusion is a valid and effective approach to address feature conflicts. The challenge then becomes how to generate these LoRA masks automatically and robustly.

4.2.2. Stage 1: Subject Mask Calculation

This stage automatically generates subject masks using cross-attention maps. This process happens once at a specific early denoising step (e.g., step 6 for FLUX.1-dev).

4.2.2.1. Cross-Attention Map Computation and Attention Sink Handling

First, cross-attention maps are computed between text queries and image keys. $ \mathbf{A}{\mathrm{cross}} = \mathrm{softmax}\left(\frac{\mathbf{Q}{\mathrm{text}} \mathbf{K}_{\mathrm{img}}^T}{\sqrt{D}}\right) $ where:

• $\mathbf{A}_{\mathrm{cross}}$ is the cross-attention map.

• $\mathbf{Q}_{\mathrm{text}} \in \mathbb{R}^{B \times N_{\mathrm{text}} \times D}$ is the Query matrix derived from text embeddings.

• $\mathbf{K}_{\mathrm{img}} \in \mathbb{R}^{B \times N_{\mathrm{img}} \times D}$ is the Key matrix derived from image tokens (latents).

• $B$ is the batch size.

• $N_{\mathrm{text}}$ is the sequence length of the text tokens.

• $N_{\mathrm{img}}$ is the sequence length of the image tokens.

• $D$ is the token dimension.

  Raw attention maps often suffer from the attention sink phenomenon, where boundary or uninformative pixels accumulate excessive attention. To counteract this, a heuristic filtering mechanism is applied, combining Top-K thresholding with spatial edge detection: $ M_{\mathrm{topk}}(i, j) = \mathbb{I}[{\mathbf{A}}(i, j) \geq \tau_k] $ $ \mathcal{M}{\mathrm{edge}}(i, j) = \mathbb{I}[(i, j) \in \mathcal{E}] $ $ \mathcal{M}{\mathrm{handle_sink}} = \neg (\mathcal{M}{\mathrm{topk}} \wedge \mathcal{M}{\mathrm{edge}}) $ where:

• $M_{\mathrm{topk}}(i, j)$ is a binary mask indicating pixels (i, j) whose attention value is above a Top-K threshold.

• $\mathbb{I}[\cdot]$ is the indicator function, which returns 1 if the condition is true and 0 otherwise.

• $\tau_k$ is the $k$-th largest attention value, with $k = \lfloor N_{\mathrm{img}} \times p \rfloor$. In practice, $p$ is chosen as 1% (meaning the top 1% of pixels with highest attention).

• $\mathcal{M}_{\mathrm{edge}}(i, j)$ is a binary mask indicating pixels (i, j) that belong to edge regions.

• $\mathcal{E}$ represents the edge pixel regions.

• $\mathcal{M}_{\mathrm{handle\_sink}}$ is the final attention sink handling mask. The operation $\neg (\mathcal{M}_{\mathrm{topk}} \wedge \mathcal{M}_{\mathrm{edge}})$ essentially masks out regions that are both Top-K and edge regions, preventing attention sinks at the boundaries.

  The filtered attention map is then normalized: $ \tilde{\mathbf{A}} = \frac{\mathbf{A} \odot \mathcal{M}{\mathrm{handle_sink}}}{\sum_j (\mathbf{A} \odot \mathcal{M}{\mathrm{handle_sink}})_{ij}} $ where:

• $\tilde{\mathbf{A}}$ is the normalized, attention sink-handled cross-attention map.

• The denominator ensures that the attention weights for each token sum to 1 after filtering.

4.2.2.2. LoRA Activation Word Attention Map Derivation

Given a set of LoRA activation words $\{ w_1, w_2, \ldots, w_L \}$ (e.g., "harry_potter", "daiyulin") and their corresponding token position sets $\{ \mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_L \}$ within the prompt, the cross-attention map for each LoRA is extracted by averaging over its token positions: $ \mathbf{M}_l = \frac{1}{|\mathcal{T}l|} \sum{idx \in \mathcal{T}_l} \tilde{\mathbf{A}}[idx, :] $ where:

• $\mathbf{M}_l$ is the preliminary attention map for the $l$-th LoRA.

• $|\mathcal{T}_l|$ is the number of tokens in the token position set for the $l$-th LoRA.

• $\tilde{\mathbf{A}}[idx, :]$ refers to the attention weights from the idx-th text token to all image tokens.

  However, these cross-attention maps (from $\mathbf{M}_l$) can still exhibit interference between LoRAs. To enhance locality and produce more cohesive attention patterns, self-attention maps are leveraged. First, the most salient regions are identified by selecting the top 1% pixels from the cross-attention map: $ \mathcal{T}_{1%} = \mathrm{TopK}(\mathbf{M}l, K = \lfloor N{\mathrm{img}} \times 0.01 \rfloor) $ where:

• $\mathcal{T}_{1\%}$ is the set of indices for the top 1% pixels.

• $\mathrm{TopK}$ function returns the indices of the top $K$ values.

  The final attention map for the $l$-th LoRA then uses the self-attention weights originating from these salient regions: $ \mathbf{M}l^{\mathrm{self_attn}} = \frac{1}{|\mathcal{T}{1%}|} \sum_{i \in \mathcal{T}{1%}} \mathbf{A}{\mathrm{self}}[i, :] $ where:

• $\mathbf{M}_l^{\mathrm{self\_attn}}$ is the enhanced spatial attention distribution for the $l$-th LoRA activation word.

• $\mathbf{A}_{\mathrm{self}} \in \mathbb{R}^{N_{\mathrm{img}} \times N_{\mathrm{img}}}$ is the self-attention map computed between image tokens. This map shows how much each image token attends to other image tokens.

• The average over self-attention from the salient cross-attention regions provides a more focused and localized map.

4.2.2.3. Superpixel-based Ensemble Masking

Pixel-wise competition between LoRA attention maps can lead to fragmented or "holey" masks. To create spatially coherent masks, a superpixel-based ensemble approach is used. At a designated denoising step (e.g., step 6), the predicted sample $\mathbf{x}_0$ (the image latent at that step) is used to generate superpixels via SLIC segmentation: $ \mathcal{R} = \mathrm{SLIC}(\mathbf{x}0, n{\mathrm{segments}}, \mathrm{compactness}, \sigma) $ where:

• $\mathcal{R}$ is the set of superpixel regions.

• $\mathbf{x}_0$ is the predicted image sample at the current denoising step.

• $n_{\mathrm{segments}}$ is the desired number of superpixels. In practice, it's set to the square root of the total image area (number of pixels).

• $\mathrm{compactness}$ (e.g., 10) controls the balance between color similarity and spatial proximity in superpixel formation.

• $\sigma$ is the standard deviation for Gaussian smoothing.

  For each superpixel region $r_j \in \mathcal{R}$, an aggregated attention score is computed for each LoRA: $ s_{l,j} = \sum_{(u, v) \in r_j} \mathbf{M}_l^{\mathrm{self_attn \uparrow}}(u, v) $ where:

• $s_{l,j}$ is the aggregated attention score for the $l$-th LoRA in superpixel region $r_j$.

• $\mathbf{M}_l^{\mathrm{self\_attn \uparrow}}$ is the enhanced self-attention map (upsampled to the image resolution if necessary) for the $l$-th LoRA.

• (u, v) represents pixel coordinates within the superpixel region.

  The ownership of superpixel region $r_j$ is then assigned to the LoRA with the maximum aggregated score: $ l^* = \arg\max_l s_{l,j} $ The final binary mask for the $l$-th LoRA is constructed as: $ \mathbf{F}_l(u, v) = \begin{cases} 1, & \mathrm{if ~} (u, v) \in r_j \mathrm{~ and ~} l^* = l \ 0, & \mathrm{otherwise} \end{cases} $ This superpixel-based voting ensures that masks are spatially coherent and have fewer artifacts compared to pixel-wise approaches.

4.2.3. Stage 2: Mask Application

After the subject masks $\mathbf{F}_l$ are generated in Stage 1 (e.g., at step 6 of a 28-step inference), they are repeatedly applied during the remaining denoising steps (e.g., steps 7-28). For each LoRA $l$, its output $\Delta \mathbf{h}_l$ is multiplied by its corresponding mask $\mathbf{F}_l$ before being added to the base model's hidden state. This ensures that each LoRA's influence is strictly confined to its designated region, effectively mitigating feature conflicts. The process is highly efficient as mask calculation is a one-time operation early in the inference.

The practical implementation notes that masks are extracted from the 17th Double Stream Block of the FLUX.1-dev model at the 6th denoising step, and n_segments for SLIC is the square root of total image pixels.

5. Experimental Setup

5.1. Datasets

The experiments focused on character-centric tasks as these often present the most severe LoRA conflicts. To ensure a fair comparison, the authors prepared identical sets of 5-character LoRAs for each method pipeline, resulting in a total of 20 LoRAs.

• Character LoRAs: Each of the 5 characters had a dedicated LoRA.
• Training Data: For each LoRA, 15 high-quality images covering multiple angles and diverse outfits were collected.
• Prompt Generation: Gemini-2.5 [Comanici et al., 2025] was used to generate the prompts for training the LoRAs.
• Training Process: Each LoRA was trained using the optimal training method recommended by the respective method's base pipeline until convergence, on the same datasets.
• Evaluation Prompts: A challenging set of 50 prompt sets was prepared (detailed in Appendix B). These prompts involved:
  • Close Character Interactions: e.g., hugging, kissing, caressing a face, whispering, tending a wound.
  • Complex Actions: e.g., pillow fights, arm wrestling, eating pizza.
  • Intricate Lighting Conditions: e.g., faces illuminated by a campfire or lantern. This challenging set was designed to thoroughly examine each method's performance on complex multi-subject generation tasks.

An example of a prompt used for evaluation (from Appendix B) is: harry-potter tucking a flower in daiyulin's hair, both smiling warmly face-to-face.

5.2. Evaluation Metrics

The paper employed four distinct evaluation metrics, plus a custom VLM Score, to comprehensively assess method performance. For all metrics, higher scores indicate better performance.

1. LVFace (Facial Similarity Scoring):

   • Conceptual Definition: Measures how well each method preserves the unique character-specific features (facial identity) of the subjects. It assesses the similarity between the generated faces and the original reference faces. This directly addresses evaluation objective (1): "Ability to best preserve subject characteristics in complex scenes."
   • Mathematical Formula: The paper refers to LVFace [You et al., 2025] as a state-of-the-art face recognition model for scoring, but does not provide its internal mathematical formula. LVFace likely outputs a similarity score (e.g., cosine similarity of face embeddings) between two given face images. If $f_E$ is the embedding function of LVFace, and $I_{gen}$ is a generated face image, $I_{ref}$ is its reference, the similarity score would generally be: $ \mathrm{Similarity}(I_{gen}, I_{ref}) = \mathrm{cosine_similarity}(f_E(I_{gen}), f_E(I_{ref})) $ where $\mathrm{cosine\_similarity}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}$.
   • Symbol Explanation:
     • $I_{gen}$: The face image generated by the model.
     • $I_{ref}$: The reference face image of the character.
     • $f_E(\cdot)$: The embedding function of the LVFace model that extracts a high-dimensional feature vector (embedding) from a face image.
     • $\mathrm{cosine\_similarity}(\cdot, \cdot)$: A common metric to measure the similarity between two non-zero vectors.

2. DINOv3 (Subject Region Feature Similarity):

   • Conceptual Definition: DINOv3 [Siméoni et al., 2025] is a self-supervised visual Transformer model capable of detecting subject regions and extracting feature embeddings. This metric evaluates how well the features of the generated subjects align with the features of the training images. It addresses evaluation objective (2): "Ability to generate images with quality closest to the pretraining data."
   • Mathematical Formula: The paper doesn't provide a specific formula for how DINOv3 is used to calculate this score. Typically, DINOv3 might be used to extract feature vectors for detected subjects in both generated and training images, and then a similarity metric (e.g., Euclidean distance or cosine similarity) is computed. Let F(I, R) be the DINOv3 feature extractor for region $R$ in image $I$. $ \mathrm{DINOv3_Score} = \mathrm{Similarity}(F(I_{gen}, R_{gen}), F(I_{train}, R_{train})) $
   • Symbol Explanation:
     • $I_{gen}$: The generated image.
     • $R_{gen}$: The detected subject region in the generated image.
     • $I_{train}$: A training image for the subject.
     • $R_{train}$: The subject region in the training image.
     • $\mathrm{Similarity}(\cdot, \cdot)$: A function to compare the feature embeddings.

3. DreamSim (Human Preference Alignment for Image Quality):

   • Conceptual Definition: DreamSim [Fu et al., 2023] is a metric designed to better align with human preferences for image quality compared to traditional metrics. It measures the aesthetic quality and realism of the generated images, thereby addressing evaluation objective (2).
   • Mathematical Formula: Not provided in the paper, but DreamSim typically uses a learned perceptual similarity metric derived from synthetic data, aiming to capture how humans perceive image similarity and quality. It would output a scalar score.

4. HPSv3 (Human Preference Score):

   • Conceptual Definition: HPSv3 [Ma et al., 2025] is a state-of-the-art human preference alignment model, proven effective in reinforcement learning [Xue et al., 2025]. It assesses both the overall image quality and how well the generated image follows the instructions in the prompt. This metric directly addresses evaluation objectives (3) "Robustness in adhering to complex prompts" and (4) "Alignment with human preference in terms of lighting, details, realism, and artifact-free generation."
   • Mathematical Formula: Not provided. HPSv3 is a learned model that predicts a score reflecting human preference given an image and a text prompt.

5. VLM Score (Vision Language Model Scoring):

   • Conceptual Definition: A custom metric defined in the paper, utilizing Gemini-2.5 [Comanici et al., 2025] (a Vision Language Model) as a scoring model. It evaluates image quality across three dimensions: character consistency, prompt consistency, and image clarity/quality. This holistic metric addresses multiple evaluation objectives.
   • Scoring Breakdown:
     • Character presence and clarity: 50 points
     • Prompt adherence: 25 points
     • Image clarity and quality: 25 points
     • Total score: 100 points
   • Prompt for VLM Scoring (from Appendix C):

     You are an image quality evaluator specializing in character generation and image quality assessment.
     
     Please evaluate the quality of the last image (the generated image) based on the following criteria:
     
     Reference images: The first {len(reference_images)} images show reference characters <A> and <B> that should appear in the generated image. Target image: The last image is the generated image that should be evaluated. Generation prompt: "{prompt_text}"
     
     Evaluation criteria (total 100 points):
     
     1. Character presence and clarity (50 points): Both characters from the reference images appear in the target image with clear and recognizable features.   
     2. Prompt adherence (25 points): The generated image follows the requirements described in the prompt.   
     3. Image clarity and quality (25 points): The image is clear, not blurry, and free of artifacts.
     
        Please provide:
     
     1. Detailed analysis for each criterion   
     2. Score for each criterion (out of the maximum points)   
     3. Total score (sum of all criteria scores)   
     4. Brief reasoning for the scores
     
        Format your response as:
     Character Analysis: [your analysis]   
     Character Score: [0-50]   
     Prompt Analysis: [your analysis]   
     Prompt Score: [0-25]   
     Clarity Analysis: [your analysis]   
     Clarity Score: [0-25]   
     Total Score: [0-100]   
     Reasoning: [brief explanation]
     

5.2.1. Experimental Protocol

• Character Pairs: The 5 prepared character LoRAs were paired pairwise to form 10 unique pairs.
• Seeds: For each prompt and character pair, results were generated using 10 different random seeds (ranging from 42 to 52).
• Total Images: Each method generated $50 \text{ prompts} \times 10 \text{ pairs} \times 10 \text{ seeds} = 5000$ images.
• Averaging: Both global averages and 10-Pass averages were calculated. The 10-Pass average takes the best result from the 10 outputs per prompt (likely based on IDA or a similar quality metric mentioned in qualitative comparisons) for averaging, reflecting the best achievable quality.

5.3. Baselines

The FreeFuse method was compared against a set of representative baselines and competing methods for multi-subject generation:

• Direct LoRA Joint Inference (LoRA Merge): This serves as the primary baseline. It represents the naive approach of simply applying multiple LoRAs simultaneously to the diffusion model without any specific conflict mitigation strategy. The paper refers to it as "direct LoRA joint inference" or "LoRA Merge".
• ZipLoRA [Shah et al., 2024]: A method that fuses multiple LoRAs before inference. While effective for style transfer, it's known to have limitations in multi-concept generation.
• OMG (Occlusion-friendly Multi-concept Generation) [Kong et al., 2024]: This method introduces an auxiliary model to localize character regions and uses noise blending to mitigate conflicts.
• Mix-of-Show [Gu et al., 2023]: This approach requires retraining LoRAs and manually specifying spatial constraints (e.g., bounding boxes) for each LoRA.
• CLoRA [Meral et al., 2024]: Leverages attention maps to derive concept masks but requires template prompts as a basis for mask extraction.

5.4. Implementation Details

• Base Model: FLUX.1-dev model, which is a DiT-based text-to-image model.
• Framework: Code built on Huggingface Diffusers [hunggingface, 2025].
• LoRA Training: The LoRAs used in experiments were trained with the Aitoolkit [ostris, 2025] framework.
• Mask Extraction Timing:
  • Standard 28-step inference process.
  • No intervention during the first 6 steps.
  • At step 6, the subject mask is extracted by computing the Attention Map from the 17th double_stream_block.
• Superpixel Parameters: For superpixel-level voting, n_segments was set to the square root of the total image pixels, and compactness was set to 10.
• Mask Application: During the remaining denoising steps (steps 7-28), each LoRA output is multiplied by its corresponding mask until inference completes.
• Hardware: Experiments were conducted on a single NVIDIA L20 GPU with 48GB VRAM.
• Inference Time: Achieved an average inference time of 37 seconds.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results validate that FreeFuse significantly outperforms existing approaches in multi-subject generation tasks across various quantitative metrics and qualitative assessments.

The following are the results from Table 2 of the original paper:

Quantitative Results Analysis (Table 2):

• VLM Score: FreeFuse achieves a significantly higher VLM Score of 74.03, compared to the next best Mix-of-Show at 57.74. This metric, derived from Gemini-2.5 evaluating character consistency, prompt adherence, and image quality, indicates FreeFuse's superior performance across these critical aspects. The large margin suggests that FreeFuse excels at producing images that are both visually high-quality and semantically aligned with complex multi-subject prompts.

• LVFace (Facial Similarity): For the LVFace-AVG metric (average facial similarity), FreeFuse's score (e.g., 0.5949 in the 10-Pass for cemm row, and 0.5946 for EAONI for LoRA Merge, though the specific rows for LVFace are a bit unclear in the presented table format, assuming the last two rows before VLM Score are the relevant ones). The paper notes that Mix-of-Show might slightly outperform FreeFuse in LVFace-AVG due to its reliance on user-specified rectangular regions, which might simplify face detection. However, FreeFuse shows superior performance in 10-Pass tasks, implying that when given multiple attempts, its adaptive masks (which can capture more complex interactions) lead to better results. For EASdH, FreeFuse achieves 12.25 (10-Pass), significantly higher than all others, indicating excellent subject characteristic preservation.

• DINOv3 and DreamSim (Image Quality and Feature Alignment): While DINOv3 can sometimes score high for artifact-heavy images, DreamSim better aligns with human preferences. The provided table seems to combine DINOv3 and DreamSim into EAONI, cemm, and Tee (likely referring to the characteristics of the LoRAs, as the evaluation metrics mentioned are DINOv3 and DreamSim in the intro paragraph for these objectives). FreeFuse consistently shows higher 10-Pass scores across these rows (e.g., 0.5949 for EAONI, 0.8052 for cemm, 0.4685 for Tee), indicating better alignment with training data features and higher perceived image quality according to human preferences.

• HPSv3 (Human Preference Score): The HPSv3 score for FreeFuse (e.g., 12.25 for EASdH 10-Pass, assuming it corresponds to HPSv3 from the text) is notably higher than competitors, demonstrating its superior ability in generating aesthetically pleasing images that follow instructions well.

  Overall, FreeFuse demonstrates clear advantages in image quality, character feature preservation, and alignment with human preferences and prompts, especially under challenging multi-subject interaction scenarios.

Qualitative Results Analysis (Figure 6):

该图像是图6的定性结果比较，展示了多主体LoRA融合中不同方法对视觉效果和自动生成掩码的影响。左侧为原始人物图像及描述，中间显示缺失关键步骤时的掩码效果，右侧为本文方法的完整融合结果，掩码更加精准。

Figure 6: Qualitative results

Figure 6 presents a qualitative comparison, showing that FreeFuse produces superior images compared to baseline and competing methods.

• Visual Fidelity: FreeFuse generates images with higher detail, better lighting, and greater realism.
• Character Interaction: It effectively handles complex interactions, such as physical contact, which other methods struggle with. The example shows FreeFuse successfully depicting harry-potter tucking a flower in daiyulin's hair with warm smiles and natural facial expressions, while other methods show merged, distorted, or incorrect features (e.g., Mix-of-Show struggles with interaction, CLoRA has poor character quality, OMG shows feature blending).
• Subject Fidelity: The characters generated by FreeFuse more closely resemble their original LoRA representations without feature bleeding or distortion.

6.2. Ablation Studies / Parameter Analysis

The paper includes an ablation study (Figure 7) to demonstrate the importance of each component of FreeFuse's mask generation pipeline.

该图像是多组多主体文本到图像生成结果对比的示意图，展示了不同方法下的生成效果，包括Mix-of-Show、CLoRA、ZipLoRA、OMG、LoRA Merge及FreeFuse。图中可见FreeFuse方法在融合多主体时呈现更自然和协调的图像风格。

Figure 7: Ablation studies demonstrate that each step of our method is essential for producing highly usable subject masks.

The study investigates the impact of removing three key components: attention sink handling, the use of self-attention maps, and block-level voting.

• Without Attention Sink Handling: If attention sink handling is omitted, the cross-attention maps become noisy. This often causes one LoRA to over-focus on trivial boundary elements (sinks), leaving most of the meaningful regions to be dominated by another LoRA. This leads to poor mask separation and reduced generation quality.

• Without Self-Attention Maps: Removing the use of self-attention maps (relying solely on cross-attention) results in masks with severe cross-intrusion. This means the masks for different subjects overlap significantly or are intermingled, leading to feature conflicts and confusion similar to basic LoRA merging. The self-attention component is crucial for enhancing the locality and cohesiveness of each subject's attention region.

• Without Block-Level Voting (Superpixel-based Ensemble Masking): Without superpixel-based ensemble masking (or block-level voting), the masks generated at a pixel-wise level contain numerous holes and are fragmented. This lack of spatial coherence in the masks translates into artifacts and inconsistencies in the generated image. The superpixel approach effectively smooths and consolidates these regions.

  The ablation study clearly demonstrates that each step—attention sink handling, self-attention map integration, and superpixel-based ensemble masking—is critical for generating the high-quality, usable subject masks that underpin FreeFuse's success in multi-subject generation.

7. Conclusion & Reflections

7.1. Conclusion Summary

FreeFuse presents a highly practical and efficient method for multi-concept generation by fusing multiple subject LoRAs in text-to-image diffusion models. The core insight is that feature conflicts during joint inference can be effectively mitigated by constraining each LoRA to its target region using automatically derived dynamic subject masks. The mathematical analysis confirms that masking LoRA outputs well approximates individual LoRA application within specified regions. FreeFuse achieves this training-free, requiring no LoRA modifications, auxiliary models, or user-defined prompt templates or regions. Instead, it intelligently leverages cross-attention layer weights, enhanced by attention sink handling, self-attention maps, and superpixel-based block voting, to generate high-quality masks from a single attention block at an early denoising step. Extensive experiments demonstrate that FreeFuse surpasses previous methods in subject fidelity, prompt adherence, and overall generation quality, particularly in complex character-centric tasks.

7.2. Limitations & Future Work

The authors acknowledge a key limitation: the effectiveness of FreeFuse's theoretical foundation—that applying subject masks approximates individual LoRA usage—gradually diminishes as the number of subject LoRAs increases. This occurs because each LoRA is assigned an increasingly smaller region, thereby creating more opportunities for features from other LoRAs to intrude into the target region. The authors identify addressing this feature intrusion with a higher number of subjects as a goal for future improvements.

7.3. Personal Insights & Critique

FreeFuse represents a significant step forward in the practicality and usability of multi-subject generation with LoRAs. Its "training-free" and "no auxiliary model" nature is a major advantage, drastically lowering the barrier to entry for users who want to combine multiple personalized concepts. The efficiency gains from extracting masks once, early in the inference process, are also noteworthy.

One key insight drawn from this paper is the power of leveraging the intrinsic attention mechanisms of Transformer-based models for semantic control. The ability to automatically derive masks from cross-attention without explicit training or external models is elegant and robust. The ablation study effectively demonstrates the synergistic effect of its components, particularly the attention sink handling and superpixel-based voting, which address common issues in raw attention maps.

Potential Issues/Areas for Improvement:

1. Generalization Beyond Characters: While the paper focuses on character-centric tasks, it would be interesting to see how FreeFuse performs with other types of subjects or concepts (e.g., specific objects, styles, or environments) in multi-concept scenarios. Do cross-attention maps for non-character LoRAs exhibit similar locality and separability?
2. Increased Subject Count: The acknowledged limitation regarding the increasing number of subject LoRAs is critical. If masks become too small or fragmented, feature intrusion will indeed become a bigger problem. Future work could explore more advanced mask refinement techniques or incorporate some form of soft masking or attention control that allows for more nuanced blending at boundaries without full intrusion.
3. Ambiguous Prompts: The reliance on LoRA activation words and cross-attention might face challenges with ambiguous prompts or LoRAs whose activation words overlap with common vocabulary, potentially leading to incorrect mask assignments.
4. DiT Model Dependency: The method is developed and tested on DiT-based models. While DiT is advanced, its direct applicability to U-Net-based models (which are still widely used) would need further validation. The specific attention block and denoising step chosen might also be sensitive to the base model.

Transferability/Applicability to Other Domains: The core idea of deriving semantic masks from attention mechanisms and using them to constrain modular components (LoRAs) could be applied to other areas:

• Local Editing: FreeFuse's mask generation could be adapted for local image editing, allowing users to precisely control which parts of an image are affected by an edit (e.g., changing color, style, or adding elements to a specific region).

• Style Transfer: When combining multiple style LoRAs, the same masking principle could ensure that different styles are applied to distinct regions of an image without bleeding into each other.

• Fine-Grained Control: This method offers a pathway towards more fine-grained control in generative models, allowing users to dictate how different elements of a prompt (or different LoRAs) manifest spatially without needing complex segmentation tools.

• Explainability: The generated attention-based masks can also provide a degree of explainability, showing which regions of the image correspond to which LoRA or prompt token, which is valuable for understanding model behavior.

  In conclusion, FreeFuse offers a pragmatic and powerful solution that simplifies the complex task of multi-subject generation, setting a high bar for efficiency and user-friendliness in the evolving landscape of personalized text-to-image synthesis.