1. Bibliographic Information

1.1. Title

Curriculum Conditioned Diffusion for Multimodal Recommendation

1.2. Authors

Yimeng Yang, Haokai Ma, Lei Meng, Shuo Xu, Ruobing Xie, Xiangxu Meng. Their affiliations include Shandong University (School of Software, Shandong Research Institute of Industrial Technology) and Tencent.

1.3. Journal/Conference

The paper does not explicitly state the journal or conference it was published in, but the publication date and format suggest it is likely a conference proceeding or a preprint. The abstract mentions "Published at (UTC): 2025-04-11T00:00:00.000Z", indicating a future publication date at the time of the abstract's writing, or a planned publication date.

1.4. Publication Year

2025 (as per the stated publication date in the abstract's metadata: 2025-04-11T00:00:00.000Z).

1.5. Abstract

This paper introduces Curriculum Conditioned Diffusion for Multimodal Recommendation (CCDRec), a novel framework designed to address data sparsity in multimodal recommendation (MMRec) by leveraging Diffusion Models (DMs). Traditional MMRec methods often neglect the impact of negative instances. CCDRec uniquely integrates the reverse phase of DMs into negative sampling to identify suitable negative instances in a curricular manner. It comprises three main modules:

1. Diffusion-controlled Multimodal Aligning (DMA): Aligns multimodal knowledge with collaborative signals by capturing fine-grained relationships among modalities in a probabilistic distribution space.
2. Negative-sensitive Diffusive Inferring (NDI): Progressively synthesizes a negative sample pool with diverse hardness using the step-by-step inference of DMs.
3. Curricular Negative Sampler (CNS): Adapts the curriculum learning paradigm with the DMA's reverse phase to sample gold-standard negative instances, enhancing optimization by gradually increasing training complexity. Extensive experiments on three datasets with four different recommendation backbones demonstrate the effectiveness and robustness of CCDRec. Visualization analyses further explain the mechanisms of DMA for multimodal representation alignment and CNS for curricular negative discovery.

1.6. Original Source Link

/files/papers/692faab1ab04788a90066006/paper.pdf The publication status is unclear, given the future publication date in the abstract metadata. It appears to be a preprint or submitted paper at the time of abstract generation.

2. Executive Summary

2.1. Background & Motivation

Multimodal recommendation (MMRec) is crucial for enhancing recommendation systems by integrating diverse data types like text, images, and audio to better understand user preferences. Current MMRec methods typically focus on deriving item representations from observed user behaviors within multimodal graphs, often using Graph Neural Networks (GNNs) or self-supervised learning. While these approaches have improved item representation learning, they frequently overlook the significant impact of negative instances on personalized preference understanding. In recommender systems, negative sampling is vital for learning from a sparse user-item interaction matrix, but traditional static strategies (e.g., uniform or popularity-based sampling) lack the flexibility to capture dynamic user preferences. More advanced "hard negative sampling" (HNS) techniques aim to select informative negative instances, but existing HNS methods in multimodal recommendation often directly apply collaborative filtering techniques, which may not be optimal for multimodal contexts.

The paper identifies two key challenges:

1. How to effectively leverage Diffusion Models (DMs) to integrate multimodal information with collaborative knowledge, leading to more accurate item-aligned representations.
2. How to develop a negative sampling strategy specifically for multimodal recommendation that seamlessly combines the step-by-step reverse process of DMs with dynamic negative sampling.

2.2. Main Contributions / Findings

The paper proposes CCDRec to address the aforementioned challenges, offering the following primary contributions:

1. Novel Framework for Multimodal Recommendation: CCDRec is introduced as the first framework to explore negative sampling strategies for multimodal recommendations using Diffusion Models. It skillfully combines the reverse phase of conditioned DMs with negative sampling to pinpoint optimal negative instances.
2. Three Model-Agnostic Modules:
   • Diffusion-controlled Multimodal Aligning (DMA): Leverages DMs to capture fine-grained relationships among modalities and align multimodal features with collaborative features, generating accurate item-aligned representations.
   • Negative-sensitive Diffusive Inferring (NDI): Constructs sample pools with diverse hardness by progressively synthesizing information-rich features during the DM inference process. This allows for flexible selection of negative instances with varying difficulty.
   • Curricular Negative Sampler (CNS): Dynamically models user preferences by introducing harder negative samples progressively throughout training, aligning with curriculum learning principles to improve generalization and convergence.
3. Extensive Experimental Validation: Demonstrates the effectiveness and universality of CCDRec through extensive experiments on three real-world datasets (Baby, Sports, Clothing) and four diverse multimodal recommendation backbones (LATTICE, FREEDOM, MCDRec, MG). CCDRec consistently achieves significant improvements over strong baselines.
4. In-depth Analysis: Provides visualization analyses to clarify the underlying mechanisms of DMA in multimodal representation alignment and CNS in curricular negative discovery, showing how DMA clusters related representations for the same user and how CNS progressively samples harder negatives.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Multimodal Recommendation (MMRec): Recommender systems that incorporate information from multiple modalities (e.g., visual, textual, audio, etc.) in addition to traditional collaborative filtering data (user-item interactions). The goal is to provide more accurate and personalized recommendations by leveraging the richer item content information, especially to mitigate the cold-start and data sparsity problems.
• Collaborative Filtering (CF): A common technique in recommender systems that makes recommendations based on the preferences or behaviors of other similar users (user-based CF) or similar items (item-based CF). It relies on historical interactions between users and items.
• Data Sparsity: A common challenge in recommender systems where the number of observed user-item interactions is very small compared to the total possible interactions. This makes it difficult to accurately learn user preferences and item characteristics.
• Graph Neural Networks (GNNs): Neural networks designed to operate on graph-structured data. They learn representations (embeddings) of nodes by aggregating information from their neighbors, making them suitable for modeling relationships in recommender systems (e.g., user-item interaction graphs, item-item similarity graphs).
• Self-Supervised Learning (SSL): A paradigm where a model learns representations from unlabeled data by solving a pretext task. In MMRec, SSL can be used to ensure coherence between different modalities of an item or to enrich latent item representations.
• Diffusion Models (DMs): A class of generative models that learn to reverse a gradual diffusion process. In the forward (diffusion) process, data is progressively perturbed by adding Gaussian noise until it becomes pure noise. In the reverse (denoising) process, the model learns to gradually remove this noise, transforming noisy data back into coherent samples (e.g., images, text, representations). DMs are known for their high-quality generation and step-by-step inference.
  • Forward Diffusion Process: Given an initial data point $x_0$, a sequence of noisy samples $x_1, \dots, x_T$ is generated by gradually adding Gaussian noise. The distribution of $x_t$ conditioned on $x_0$ is: $ q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t)\mathbf{I}) $ where $\alpha_t = 1 - \beta_t$ are variance parameters, and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$. This allows sampling $x_t$ directly from $x_0$ using the reparameterization trick: $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$, where $\epsilon \sim \mathcal{N}(0, \mathbf{I})$.
  • Reverse Denoising Process: The model learns to approximate the reverse conditional probability $p_\theta(x_{t-1} | x_t)$, which estimates the previous state $x_{t-1}$ from $x_t$. This is typically done by predicting the noise $\epsilon$ added at step $t$ or by predicting $x_0$.
• Negative Sampling: In recommender systems, particularly for implicit feedback, only positive user-item interactions are observed. Negative sampling involves selecting unobserved (negative) items to train the model, teaching it to distinguish between liked and disliked items.
  • Uniform Sampling: Randomly selecting items as negatives.
  • Popularity-based Sampling: Sampling less popular items as negatives, assuming popular items are more likely to be positive interactions.
  • Hard Negative Sampling (HNS): Selecting negative items that are "hard" for the model to distinguish from positive items. These are often items that the model incorrectly predicts as positive or items that are very similar to positive items but have not been interacted with. HNS aims to provide more informative gradients, but too hard negatives can hinder training stability.
• Curriculum Learning (CL): A training strategy inspired by how humans learn, where a model is trained on progressively harder examples. It starts with easier samples, builds a strong foundation, and then gradually introduces more complex ones, leading to better generalization and faster convergence.
• Bayesian Personalized Ranking (BPR): A pairwise ranking loss function widely used in recommender systems. It optimizes the model to rank observed items higher than unobserved items for each user. $ \mathcal{L}{BPR} = - \sum{(u, i, j) \in D_S} \log \sigma(\hat{x}{ui} - \hat{x}{uj}) $ where $D_S$ is the training set containing triplets (u, i, j) indicating user $u$ prefers item $i$ over item $j$, $\hat{x}_{ui}$ is the predicted score of item $i$ for user $u$, and $\sigma$ is the sigmoid function.

3.2. Previous Works

The paper categorizes related work into three areas:

• Multimodal Recommendation:

  • Early studies (He and McAuley 2016; Chen et al. 2019) used pre-extracted visual features to enrich item representations.
  • GNN-based methods (Wang et al. 2021; Zhang et al. 2021; Zhou and Shen 2023) leverage GNNs to extract user-specific modal preferences and higher-order relationships.
  • Self-supervised learning (SSL) techniques (Tao et al. 2022; Zhou et al. 2023) were introduced for content coherence.
  • Generative models, including Variational Autoencoders and Diffusion Models (Bai et al. 2023; Yu et al. 2023; Ma et al. 2024c), have also been explored. Notably, MCDRec (Ma et al. 2024c) uses DMs to model multimodal and collaborative data.
  • Gap identified: Existing methods overlook the modeling of negative behaviors.

• Diffusion Models in Recommendation:

  • Inspired by DMs' success in computer vision (Rombach et al. 2022; Zheng et al. 2024; Yu et al. 2024), applications in recommendation include:
    • DiffRec (Wang et al. 2023a): Generates global yet personalized collaborative information via denoising.
    • PDRec (Ma et al. 2024a): Utilizes diffusion-based preferences across items with plug-in modules.
    • Other methods (Li, Sun, and Li 2023; Yang et al. 2024): Explore item space distribution and dynamics using DMs.
    • MCDRec (Ma et al. 2024c): Injects modality-aware uncertainty into item representations to mitigate biases.

• Negative Sampling in Recommendation:

  • Traditional methods use BPR (Rendle et al. 2012) and static negative sampling (Guo et al. 2017; Mikolov et al. 2013).
  • Hard Negative Sampling (HNS) methods, like DNS (Zhang et al. 2013) which oversamples high-score negatives, SRNS (Ding et al. 2020) using variance-based functions, and MixGCF (Huang et al. 2021) generating synthetic negatives, aim for more informative samples.
  • Gap identified: These methods primarily rely on collaborative filtering and graph representation learning, limiting their suitability for multimodal recommenders.

3.3. Technological Evolution

The field has evolved from basic CF models to incorporating rich side information (multimodal content) for item representation, then utilizing advanced graph-based methods (GNNs) and self-supervised learning for better representation learning. More recently, generative models, particularly Diffusion Models, have emerged as powerful tools for generating diverse and high-quality data, and their potential is now being explored in recommendation for data augmentation, uncertainty modeling, and representation learning. Concurrently, negative sampling strategies have progressed from simple random selection to sophisticated hard negative mining, recognizing the importance of informative negative examples for robust model training.

3.4. Differentiation Analysis

The core difference and innovation of CCDRec compared to prior work lie in its unique integration of Diffusion Models with negative sampling in a multimodal recommendation context:

1. Diffusion for Multimodal Alignment: Unlike previous multimodal methods that might use GNNs or simple fusion mechanisms, CCDRec explicitly uses a conditioned DM (DMA) to align multimodal features with collaborative signals in a probabilistic distribution space. This offers a more fine-grained and robust way to fuse information and handle inconsistencies.
2. DM for Dynamic Negative Sampling: While DMs have been used in recommendation for other purposes (e.g., item generation, preference modeling), CCDRec is the first to leverage the step-by-step reverse process of DMs to dynamically synthesize negative samples of varying hardness. This addresses the limitation of traditional HNS methods that are not tailored for multimodal contexts.
3. Curriculum Learning Integration: The Curricular Negative Sampler (CNS) further refines the negative sampling process by applying curriculum learning principles, ensuring that the model is exposed to progressively harder negatives. This contrasts with static or simpler HNS strategies that might introduce overly challenging negatives early in training, potentially hindering convergence.
4. Model-Agnostic Modules: The proposed DMA, NDI, and CNS are designed to be plug-and-play modules, demonstrating their effectiveness and generalizability across different multimodal recommendation backbones, rather than being tied to a specific architecture.

4. Methodology

The proposed Curriculum Conditioned Diffusion for Multimodal Recommendation (CCDRec) framework enhances multimodal item fusion using Diffusion Models and applies diffusion-generated knowledge for adaptive negative sampling. The overall architecture is illustrated in Figure 2.

The following are the results from Figure 2 of the original paper:

该图像是示意图，展示了课程条件扩散框架（CCDRec）在多模态推荐中的结构。图中包括用户和物品的协同信息、图谱与多模态信息的结合，以及扩散控制的多模态对齐模块（DMA）和负样本敏感扩散推理模块（NDI）。图示还揭示了课程负采样器（CNS）的功能，逐步调整训练复杂度以优化推荐效果。箭头指向不同模块，标明信号流动和信息处理的过程。

4.1. Task Formulation and Overall Framework

The objective of multimodal recommendation is to obtain more precise item representations for recommendations by leveraging additional multimodal information.

• User embeddings: $e_u$ for user $u \in \mathcal{U}$

• Item embeddings: $e_i$ for item $i \in \mathcal{I}$

• Multimodal features for each item $i$: visual feature $e_i^v$ and textual feature $e_i^t$.

  CCDRec comprises three main modules:

1. Diffusion-controlled Multimodal Aligning (DMA): Utilizes a Diffusion Model (DM) to capture fine-grained correlations across different modalities and align multimodal features with collaborative features, generating accurate item-aligned representations.
2. Negative-sensitive Diffusive Inferring (NDI): Constructs sample pools with diverse hardness using item-aligned features generated at various diffusion steps, facilitating knowledge-aware negative sampling.
3. Curricular Negative Sampler (CNS): Selects progressively harder negative samples throughout training by adapting curriculum learning, which improves generalization and convergence.

4.2. Base Multimodal Recommender

The paper uses FREEDOM (Zhou and Shen 2023) as the base multimodal recommender.

1. Modality-aware Item-Item Graphs: FREEDOM constructs modality-aware item-item graphs using raw features ($e_i^v$, $e_i^t$).
2. Graph Simplification: These graphs are simplified using KNN sparsification to form normalized adjacency matrices.
3. Unified Latent Item-Item Graph: The matrices are merged to create a unified latent item-item graph $S$.
4. Feature Aggregation: Graph convolutions are applied on $S$ for feature aggregation and information propagation to obtain $\hat{h}_i$.
5. User-Item Graph: In the user-item graph $\widehat{A}$, multiple convolutional operations (using LightGCN settings) derive ID embeddings for users ($\widetilde{h}_u$) and items ($\widetilde{h}_i$).
6. Final Representations: The final representations for users and items are:
   • Users: $\boldsymbol{h_u} = \boldsymbol{\tilde{h}_u}$
   • Items: $\boldsymbol{h_i} = \hat{\boldsymbol{h}}_i + \tilde{\boldsymbol{h}}_i$
7. Modality Projection: Multilayer Perceptrons (MLPs) project features of each modality: $ \boldsymbol{h}_i^m = \boldsymbol{e}_i^m \mathbf{W}_m + \boldsymbol{b}_m $ where $m \in \{v, t\}$ for visual and textual modalities, $\mathbf{W}_m$ is the weight matrix, and $\boldsymbol{b}_m$ is the bias vector.

4.3. Diffusion-controlled Multimodal Aligning (DMA)

The DMA module is inspired by Denoising Diffusion Probabilistic Models (DDPMs) and aims to capture probabilistic correlations between multiple modalities while aligning multimodal information with collaborative signals. It generates aligned multimodal fused features to address inconsistencies and capture deeper user preferences.

4.3.1. Learning Phase of DMA

1. Initial Item Embedding: An item ID embedding is denoted as $e_i$, which is initially $e_i^0$.
2. Forward Diffusion Process: Gaussian noise is gradually introduced into $e_i^0$ over $T$ steps. The noisy representation $e_i^t$ at step $t$ is sampled from a conditional Gaussian distribution: $ q \left( e _ { i } ^ { t } \mid e _ { i } ^ { 0 } \right) = \mathcal { N } \left( e _ { i } ^ { t } ; \sqrt { \bar { \alpha } _ { t } } e _ { i } ^ { 0 } , \left( 1 - \overline { { \alpha } } _ { t } \right) \mathrm { I } \right) $ where $t \sim \mathrm{Uniform}\{1, 2, \ldots, T\}$, $\mathcal{N}$ denotes the Gaussian distribution, $\bar{\alpha}_t$ is the product of $\alpha_s = 1 - \beta_s$ for $s=1$ to $t$, and $\mathbf{I}$ is the identity matrix.
3. Reparameterization: This allows $e_i^t$ to be expressed as: $ e _ { i } ^ { t } = \sqrt { { \bar { \alpha } } _ { t } } e _ { i } ^ { 0 } + \sqrt { 1 - { \bar { \alpha } } _ { t } } \epsilon $ where $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ is the standard Gaussian noise.
4. Reverse Denoising Process: The reverse process iteratively removes noise to generate the denoised representation $e_i^{t-1}$ from $e_i^t$: $ p _ { \theta } \left( e _ { i } ^ { t - 1 } \mid { e } _ { i } ^ { t } \right) = \mathcal { N } \left( e _ { i } ^ { t } ; f _ { \theta } \left( e _ { i } ^ { t } , t , h _ { i } ^ { v } , h _ { i } ^ { t } \right) , \Sigma _ { \theta } \left( e _ { i } ^ { t } , t \right) \right) $ Here, $p_\theta$ is the learned reverse distribution, $f_\theta$ is the conditional estimator, and $\Sigma_\theta$ is the variance. The variance is given by: $ \Sigma _ { \theta } \left( e _ { i } ^ { t } , t \right) = \sigma _ { t } ^ { 2 } \mathrm { I } = \frac { 1 - \bar { \alpha } _ { t - 1 } } { 1 - \bar { \alpha } _ { t } } \beta _ { t } \mathrm { I } $ While traditional DMs predict noise, this approach trains an estimator $f_\theta$ to directly approximate $e_i^0$. The mean $\mu_\theta$ for the reverse step is calculated as: $ \mu _ { \theta } \left( e _ { i } ^ { t } , t , h _ { i } ^ { v } , h _ { i } ^ { t } \right) = \frac { 1 } { \sqrt { \alpha _ { t } } } \left( e _ { i } ^ { t } - \frac { \beta _ { t } } { \sqrt { 1 - \bar { \alpha } _ { t } } } f _ { \theta } \left( e _ { i } ^ { t } , t , h _ _ { i } ^ { v } , h _ { i } ^ { t } \right) \right) $ where $f_\theta(.)$ is the tailored conditional estimator.
5. Loss Function: To train the conditional estimator $f_\theta$, the model minimizes a Mean-Squared Error (MSE) loss function, which is a simplified form of the variational bound (VLB): $ \mathcal { L } _ { d m } = E _ { e _ { i } ^ { 0 } , e _ { i } ^ { t } } \left[ \left. e _ { i } ^ { 0 } - f _ { \theta } \left( e _ { i } ^ { t } , t , \pmb { h } _ { i } ^ { v } , \pmb { h } _ { i } ^ { t } \right) \right. ^ { 2 } \right] $ Here, $e_i^0$ is the initial item embedding, and $f_\theta$ generates the estimated item representation $\hat{e}_i^0$.
6. Fused Representation: The higher-order item representation $\hat{h}_i$ from the base recommender is merged with the item-aligned representation $\tilde{e}_i^0$ (estimated $e_i^0$) to form the fused representation $\hat{e}_i^f$: $ \hat { e } _ { i } ^ { f } = ( 1 - \mu ) \hat { h } _ { i } + \mu \cdot \widetilde { e } _ { i } ^ { 0 } $ where $\mu$ is an adjustable parameter controlling the diffused weight.

4.3.2. Conditional Estimator

• The conditional estimator $f_\theta(.)$ is implemented using a Transformer architecture, similar to (Li, Sun, and Li 2023; Wang et al. 2024).
• Input: The input feature matrix $\mathbf{F} \in \mathbb{R}^{B \times M \times d}$ is formed by concatenating the noised item representation $e_i^t$, textual feature $\boldsymbol{h}_i^t$, visual feature $\boldsymbol{h}_i^v$, and a sinusoidal time step embedding $t_i$. $B$ is batch size, $M$ is number of modalities, $d$ is feature dimension.
• Mechanism: The Transformer's self-attention mechanism captures complex dependencies and relationships between the various modalities.
• Output: The aggregated attention output $\tilde{e}_i^0$ is obtained by averaging across modalities, which serves as the precise estimation of $\hat{e}_i^0$.

4.3.3. Inference Phase of DMA

• This phase is executed after each training epoch.
• Noise Addition: Starting with an item embedding $e_i^0$, noise is added to obtain $\hat{e}_i^t$ at the complete diffusion step $t=T$.
• Step-by-step Denoising: A step-by-step reverse denoising process is performed: $\hat{e}_i^T \to \hat{e}_i^{T-1} \to \dots \to \hat{e}_i^0$, to achieve the final $\hat{e}_i^{\bar{0}}$ representation that conforms to the collaborative representation distribution.
• Item-fuse Representation: The item-fuse representation is generated as: $ \hat { e } _ { i } ^ { f } = \left( 1 - \mu \right) \hat { \pmb { h } } _ { i } + \bar { \mu } \cdot \hat { e } _ { i } ^ { 0 } $

4.4. Negative-sensitive Diffusive Inferring (NDI)

NDI aims to synthesize negative sample pools with varying difficulty levels. Hard negative instances are valuable but can hinder early training if too challenging. NDI leverages the DM inference process to create a knowledge-aware negative sample pool using features from different diffusion steps.

• Execution: This process is executed once at the start of each epoch to minimize computational cost.
• Progressive Synthesis: The reverse process of DM starts from noise and gradually generates the final representation over $T$ steps. Samples at earlier reverse steps are noisier and simpler, while samples at later steps (closer to $t=0$) are less noisy and contain richer information, making them "harder" negatives.
• Sample Candidate Pools: Item representations are extracted after specific diffusion steps: $T/4, T/2, 3T/4,$ and $T$ steps (where $T$ represents the full noise state, and 0 represents the fully denoised state or $e_i^0$). These form four sample candidate pools: $ \hat { E } _ { | \mathcal { I } | } ^ { t } = \left[ \hat { e } _ { 0 } ^ { t } , \hat { e } _ { 1 } ^ { t } , \cdots , \hat { e } _ { | \mathcal { I } | } ^ { t } \right] \in \mathbb { R } ^ { | \mathcal { I } | \times d } , t \in \big { \frac { 3 T } { 4 } , \frac { T } { 2 } , \frac { T } { 4 } , 0 \big } . $ Here, $|\mathcal{I}|$ is the total number of items, $d$ is the feature dimension, and $t$ denotes the diffusion step. A larger $t$ means more noise (easier negative), and $t=0$ means fully denoised (hardest negative).

4.5. Curricular Negative Sampler (CNS)

CNS implements curriculum learning by progressively introducing harder negative samples during training to improve generalization and speed up convergence.

• Dynamic Sample Pool Selection: The specific sample pool $\hat{E}_{|\mathcal{I}|}^t$ to be used at epoch $n$ is determined by the formula: $ t = ( T / 4 ) \times \left( 3 - \operatorname* { m i n } \left( 3 , \lfloor n / \Delta \tau \rfloor \right) \right) . $
  • $T$: Total diffusion steps.
  • $\Delta \tau$: Controls the interval (in epochs) between curriculum stages.
  • $n$: Current epoch number.
  • $\lfloor n / \Delta \tau \rfloor$: Determines the current curriculum stage.
  • The $min(3, ...)$ ensures that the index for $T/4$ multiplication does not exceed 3, cycling through $3T/4, T/2, T/4, 0$. This means:
    • For $n < \Delta \tau$, $\lfloor n / \Delta \tau \rfloor = 0$, $t = (T/4) \times 3 = 3T/4$ (easiest negatives).
    • For $\Delta \tau \le n < 2\Delta \tau$, $\lfloor n / \Delta \tau \rfloor = 1$, $t = (T/4) \times 2 = T/2$ (medium-easy negatives).
    • For $2\Delta \tau \le n < 3\Delta \tau$, $\lfloor n / \Delta \tau \rfloor = 2$, $t = (T/4) \times 1 = T/4$ (medium-hard negatives).
    • For $n \ge 3\Delta \tau$, $\lfloor n / \Delta \tau \rfloor \ge 3$, $t = (T/4) \times 0 = 0$ (hardest negatives, fully denoised).
• Curriculum End: $\tau_{end}$ marks the ending epoch for the CL strategy. After $n > \tau_{end}$, the model always uses the final item representations (corresponding to $t=0$) as the sample pool, which are considered the hardest samples.
• Negative Sampling Process:
  1. Candidate Selection: $10\%$ of items are randomly sampled from the chosen sample pool $\hat{E}_{|\mathcal{I}|}^t$ as candidates.
  2. Similarity Calculation: For a given positive item $h_p$, its representation $\hat{e}_p^t$ at the specific diffusion step $t$ is retrieved from the selected sample pool. The similarity between $\hat{e}_p^t$ and all candidate items is computed.
  3. Top-k Selection: From the candidates, the top $k$ items with the highest similarity scores are selected, forming $\mathcal{C}_k$.
  4. Curricular Negative Item: An item $\hat{e}_c^{\bar{t}}$ is randomly resampled from $\mathcal{C}_k$. The index $c$ is then mapped back to its final representation $h_c$ as the curricular negative instance.
  5. Inclusion of Easy Negatives: To improve generalization and stability, a randomly selected easy negative item $h_r$ is also included for joint training.

4.6. Optimization Objectives

The model is optimized using a modified Bayesian Personalized Ranking (BPR) loss, incorporating both randomly sampled easy negatives ($h_r$) and curriculum-sampled hard negatives ($h_c$).

• BPR Loss for Random Negatives: $ \mathcal { L } _ { b p r } ^ { r } = \sum _ { \left( u , p , r \right) \in \mathcal { R } } \left( - \log \sigma \left( h _ { u } ^ { \top } h _ { p } - h _ { u } ^ { \top } h _ { r } \right) \right) $ where $\mathcal{R}$ denotes the set of observed user-positive item-random negative triplets, $h_u$ is user $u$'s embedding, $h_p$ is positive item $p$'s embedding, $h_r$ is random negative item $r$'s embedding, and $\sigma$ is the sigmoid function.
• BPR Loss for Curricular Negatives: $ \mathcal { L } _ { b p r } ^ { c } = \sum _ { \left( u , p , c \right) \in \mathcal { R } } \left( - \log \sigma \left( h _ { u } ^ { \top } h _ { p } - h _ { u } ^ { \top } h _ { c } \right) \right) $ where $h_c$ is the curriculum-sampled hard negative item $c$'s embedding.
• Overall Objective Function: The total loss $\mathcal{L}$ combines the two BPR losses and the diffusion model loss: $ \mathcal { L } = ( 1 - \omega ) \cdot \mathcal { L } _ { b p r } ^ { r } + \omega \cdot \mathcal { L } _ { b p r } ^ { c } + \lambda \cdot \mathcal { L } _ { d m } $ Here, $\omega$ controls the weight of the curricular negative BPR loss, and $\lambda$ controls the weight of the diffusion model loss.

5. Experimental Setup

5.1. Datasets

Experiments are conducted on three real-world multimodal datasets from the Amazon platform, commonly used in prior research (Zhou 2023).

• Data Pre-processing: A 5-core setting is applied to both items and users, meaning only users and items with at least 5 interactions are retained (as per He and McAuley 2016).
• Multimodal Features:

  • Visual Features: Pre-extracted features with a dimension of 4096 (following Zhou and Shen 2023).

  • Textual Features: Obtained using sentence-transformers (Reimers and Gurevych 2019) with 384-dimensional embeddings.

    The following are the results from Table 1 of the original paper:

    Dataset	#Users	#Items	#Interactions	Sparsity
    Baby	19,445	7,050	160,792	99.88%
    Sports	35,598	18,357	296,337	99.95%
    Clothing	39,387	23,033	237,488	99.97%

5.2. Evaluation Metrics

The effectiveness of the recommendation system is evaluated using two standard metrics:

1. Recall@k (R@k):
   • Conceptual Definition: Measures the proportion of relevant items successfully recommended within the top $k$ items. It indicates the model's ability to retrieve all relevant items for a user.
   • Mathematical Formula: $ \mathrm{Recall@k} = \frac{1}{|\mathcal{U}|} \sum_{u \in \mathcal{U}} \frac{|\mathrm{R}(u) \cap \mathrm{T}(u)|}{|\mathrm{T}(u)|} $
   • Symbol Explanation:
     • $|\mathcal{U}|$: Total number of users.
     • $\mathrm{R}(u)$: Set of top $k$ items recommended for user $u$.
     • $\mathrm{T}(u)$: Set of true relevant items (ground truth) for user $u$.
     • $|\cdot|$: Cardinality of a set.
2. Normalized Discounted Cumulative Gain at k (NDCG@k):
   • Conceptual Definition: A ranking-aware metric that considers the position of relevant items in the recommendation list. It assigns higher scores to relevant items that appear earlier in the list.
   • Mathematical Formula: $ \mathrm{NDCG@k} = \frac{1}{|\mathcal{U}|} \sum_{u \in \mathcal{U}} \frac{\mathrm{DCG@k}(u)}{\mathrm{IDCG@k}(u)} $ where $\mathrm{DCG@k}(u)$ is the Discounted Cumulative Gain for user $u$ at rank $k$: $ \mathrm{DCG@k}(u) = \sum_{j=1}^{k} \frac{2^{\mathrm{rel}j} - 1}{\log_2(j+1)} $ and $\mathrm{IDCG@k}(u)$ is the Ideal Discounted Cumulative Gain (DCG of the perfectly sorted list) for user $u$ at rank $k$: $ \mathrm{IDCG@k}(u) = \sum{j=1}^{|\mathrm{T}(u)|, j \le k} \frac{2^{\mathrm{rel}_j} - 1}{\log_2(j+1)} $
   • Symbol Explanation:
     • $|\mathcal{U}|$: Total number of users.
     • $\mathrm{rel}_j$: Relevance score of the item at position $j$ in the recommended list (typically 1 for relevant, 0 for non-relevant).
     • $\mathrm{T}(u)$: Set of true relevant items for user $u$.
     • $\log_2(j+1)$: Discount factor that reduces the contribution of items further down the list. The evaluation is performed for $k \in \{5, 10\}$.

5.3. Baselines

The CCDRec method is compared against two categories of baseline models:

• CF-based Recommenders:
  • BPR (Rendle et al. 2012): A foundational pairwise ranking method for implicit feedback.
  • LightGCN (He et al. 2020): A simplified and effective GCN model for recommendation.
• Multimodal Recommenders:
  • MMGCN (Wei et al. 2019): A Multi-modal Graph Convolution Network.
  • SLMRec (Tao et al. 2022): Self-supervised Learning for Multimedia Recommendation.
  • LATTICE (Zhang et al. 2021): Mines latent structures for multimedia recommendation.
  • BM3 (Zhou et al. 2023): Bootstrap Latent Representations for Multi-modal Recommendation.
  • FREEDOM (Zhou and Shen 2023): The base model used in CCDRec, focusing on freezing and denoising graph structures.
  • MG (Zhong et al. 2024): Mirror Gradient, focusing on robust multimodal recommender systems.
  • MCDRec (Ma et al. 2024c): Multimodal Conditioned Diffusion Model for Recommendation, which also uses DMs for item modeling.

5.4. Parameter Settings

• Embedding Size: Both user and item embedding sizes are set to 64 for all models, ensuring fair comparison.
• Negative Samples for Baselines: Results for other methods are presented using two random negative samples.
• Hyper-parameter Search: A comprehensive grid search is performed.
  • GCN Layers: Set to 2.
  • Loss Weights:
    • $\lambda$ (for $\mathcal{L}_{dm}$): $\{0.5, 1, 2\}$
    • $\omega$ (for $\mathcal{L}_{bpr}^c$): $\{0.5, 0.7, 0.8, 0.9\}$
  • Diffusion Process:
    • $T$ (number of steps): $\{5, 10, 20, 40, 100\}$
    • $\mu$ (diffused weight for fused representation): $\{0.3, 0.5, 0.8\}$
  • Curriculum Learning:
    • $\Delta \tau$ (interval for curriculum stages): $\{3, 5, 10, 15, 20\}$
    • $\tau_{end}$ (epoch to end CL strategy): $\{30, 50, 75, 100\}$
• Early Stopping: The early stopping strategy is adopted (following Zhou and Shen 2023) to prevent overfitting.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate the effectiveness and robustness of CCDRec across three real-world datasets (Baby, Sports, Clothing) when compared against various baselines.

The following are the results from Table 2 of the original paper:

Key Observations (RQ1):

1. Superior Performance: CCDRec consistently outperforms all baseline methods across all metrics (R@5, R@10, N@5, N@10) on all three datasets. This suggests that its approach of combining diffusion-enhanced item fusion and diffusion knowledge-guided negative sampling effectively leverages multimodal information to learn fine-grained user preferences.
2. Multimodal vs. CF-based: Multimodal recommendation methods generally show better performance than traditional CF-based recommenders (BPR-MF, LightGCN), highlighting the value of incorporating multimodal information. CCDRec further improves upon state-of-the-art multimodal recommenders.
3. Impact on Different Backbones: CCDRec shows significant improvements when integrated with various base models.
   • It achieves the most notable improvements on LATTICE, with Recall@10 improvements of up to 22.85% (Clothing dataset) and NDCG@10 up to 21.32% (Clothing dataset).
   • It achieves peak results across all datasets when combined with FREEDOM, showing improvements in Recall@10 up to 8.67% (Clothing dataset) and NDCG@10 up to 9.20% (Clothing dataset).
   • Improvements are also observed for MG, which does not inherently use a multimodal diffusion strategy. These results underscore CCDRec's ability to effectively model item multimodal fusion representations using its tailored Diffusion Models (DMA).
4. Improvements over Diffusion-based Baselines: CCDRec even improves upon MCDRec, which also incorporates diffusion-guided item modeling. This suggests two things:
   • Potential limitations of MCDRec's U-Net architecture for item feature reconstruction, possibly leading to the loss of crucial modal information.
   • The significant contribution of CCDRec's diffusion-guided negative sampling strategy in providing valuable guidance for model training.

6.2. Ablation Studies / Parameter Analysis

Ablation studies were conducted to examine the contribution of each component (DMA, NDI, CNS) to the overall performance of CCDRec. The notation FREEDOM + DMA + NDI refers to a version where negative samples are randomly selected from the final sample pool $\hat{E}_{|\mathcal{I}|}^0$ (the hardest samples) in NDI without the curricular strategy. CCDRec (FREEDOM) is equivalent to FREEDOM + DMA + NDI + CNS.

The following are the results from Figure 3 of the original paper:

Key Observations (RQ2 & RQ3):

1. Effectiveness of DMA: FREEDOM + DMA consistently outperforms FREEDOM across the datasets. This indicates that the DMA module effectively captures fine-grained modal preferences and generates more accurate item-aligned representations, contributing positively to recommendation performance.
2. Effectiveness of NDI: FREEDOM + DMA + NDI shows significant improvement over FREEDOM + DMA. This highlights that the NDI module effectively uncovers latent negative instances by leveraging item-aligned representations, providing a richer source of negative samples than simple random sampling.
3. Effectiveness of CNS: CCDRec (FREEDOM) (which includes CNS) further boosts performance compared to FREEDOM + DMA + NDI. This underscores the value of the curriculum-based dynamic negative sampling strategy in CNS, which adapts difficulty based on inference steps for optimal training.
4. Generalizability (RQ3): The consistent improvements observed across different variants and base models (LATTICE, FREEDOM, MG) confirm that the proposed components (DMA, NDI, CNS) are effective and generalizable across various multimodal recommendation architectures.

6.3. Performance against Other Hard Negative Sampling Methods

To further validate CCDRec's performance, it was compared with three other hard negative sampling (HNS) methods: DNS, MixGCF, and RealHNS. These methods were integrated into the three base models (LATTICE, FREEDOM, MG) under consistent experimental settings.

The following are the results from Table 3 of the original paper:

Key Observations (RQ4):

1. Superiority of CCDRec: CCDRec consistently outperforms other hard negative sampling methods (DNS, MixGCF, RealHNS) when integrated with different base models (LATTICE, FREEDOM, MG) on both Baby and Clothing datasets. This further validates the superiority of CCDRec's diffusion-guided curricular negative sampling strategy.
2. Value of Hard Negative Sampling: All HNS methods, including DNS, MixGCF, and RealHNS, generally improve performance over the base models alone. This substantiates the significant latent potential of negative information in multimodal recommendation. Judicious selection of negative sampling strategies (like CCDRec's) can indeed enhance a model's ability to learn and model user multimodal preferences.

6.4. In-depth Analyses of CCDRec

6.4.1. Estimation of Multimodal Alignment in DMA (RQ5)

To understand how DMA affects the distribution of multimodal item representations, t-SNE (Van der Maaten and Hinton 2008) is used to visualize embeddings of five randomly selected users and their interacted items on the Baby dataset at the initial and convergence states. The same color represents representations belonging to the same user.

The following are the results from Figure 4 of the original paper:

Observations:

• Initial State: At the initial stage of training, the related representations (user embedding, item embeddings, item-fused embeddings) of the same user are scattered throughout the low-dimensional space, showing no clear clustering.
• Convergence State: In contrast, at the convergence stage, these representations for the same user exhibit significant clustering. Crucially, the item-fused representation is observed to be the closest to the user's embedding. This indicates that DMA is effective in precisely capturing the fine-grained relationships among multi-modalities of the same item and aligning them with user preferences, leading to better-clustered and more coherent representations.

6.4.2. Estimation of Negative Inference in CNS (RQ6)

To investigate the effectiveness of the negative instances sampled by CNS, a visualization is performed using t-SNE. Two users with their positive samples are selected, and negative instances of diverse hardness, sampled by CNS from different inference steps of DMA, are visualized.

The following are the results from Figure 5 of the original paper:

Observations:

• Hardness Progression: The visualization clearly shows that the "hardness" of the sampled negative instances increases as the DMA inference phase progresses (i.e., as the diffusion step $t$ decreases from $3T/4$ to 0).
• Proximity to Positive Samples: Specifically, negative samples from earlier steps (e.g., $3T/4, T/2$) are noisier and simpler, appearing further away from the user and positive samples in the low-dimensional space. As the inference proceeds to later steps ($T/4, 0$), the negative instances become less noisy and progressively move closer to the representation of the corresponding user and positive samples. These closer negatives are "harder" because they are more similar to positive interactions, making them more challenging for the model to distinguish.
• Effectiveness of CNS: This phenomenon highlights that CNS successfully discovers and samples negative instances with diverse hardness levels in a curricular manner, providing an effective mechanism to boost recommender optimization by gradually increasing the learning difficulty.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces Curriculum Conditioned Diffusion for Multimodal Recommendation (CCDRec), a novel framework that ingeniously integrates the reverse process of Diffusion Models (DMs) with negative sampling to select optimal negative instances for multimodal recommendation. CCDRec comprises three key modules:

1. Diffusion-controlled Multimodal Aligning (DMA): Leverages DMs to achieve fine-grained alignment between multimodal item features and collaborative signals, generating accurate item-aligned representations.
2. Negative-sensitive Diffusive Inferring (NDI): Creates diverse negative sample pools by synthesizing features at various diffusion steps, allowing for flexible hardness selection.
3. Curricular Negative Sampler (CNS): Dynamically selects progressively harder negative samples throughout training, following curriculum learning principles, thereby enhancing model optimization and generalization. Extensive experiments across three real-world datasets and four diverse base models robustly demonstrate CCDRec's superior effectiveness and universality compared to existing state-of-the-art methods. Visualization analyses further confirmed the underlying mechanisms of DMA in multimodal alignment and CNS in curricular negative discovery.

7.2. Limitations & Future Work

The authors highlight the following future research directions:

• Continued exploration of the untapped potential of DMs in negative sampling.
• Investigation of CCDRec's effectiveness in other more challenging recommendation scenarios, such as multimodal cross-domain recommendation or sequential recommendation.

7.3. Personal Insights & Critique

The CCDRec framework presents a highly innovative approach by explicitly connecting the generative capabilities and step-by-step inference of Diffusion Models to the critical task of negative sampling in multimodal recommendation. The integration of curriculum learning to manage the hardness of these diffusion-generated negatives is particularly elegant, addressing a well-known challenge in hard negative mining where overly difficult samples can destabilize early training.

Inspirations and Applications:

• Beyond Recommendation: The core idea of using DMs to generate "hard" yet "learnable" negative examples, and then introducing them in a curricular fashion, could be transferable to other machine learning tasks where informative negative examples are crucial but difficult to obtain. Examples include contrastive learning for representation learning, or even anomaly detection where synthetic "near-miss" anomalies could be generated.
• Personalized Content Generation: The DMA module's ability to align multimodal features with collaborative signals could inspire personalized content generation. Instead of just recommending existing items, a DM could be conditioned on user preferences and multimodal inputs to generate novel items (e.g., product designs, advertisements) tailored to individual tastes.
• Robustness to Data Sparsity: The generative nature of DMs inherently helps address data sparsity by enriching item representations and creating diverse negative samples, making the model less reliant on direct observed interactions.

Potential Issues, Unverified Assumptions, or Areas for Improvement:

• Computational Cost: While the authors state that NDI is executed only once per epoch to minimize cost, Diffusion Models are generally computationally intensive, especially during the reverse inference process to generate diverse negative samples. The paper should provide a more detailed analysis of the computational overhead compared to traditional negative sampling methods, particularly for very large item catalogs or frequent model retraining.

• Parameter Sensitivity: Diffusion models and curriculum learning strategies typically involve many hyper-parameters ($T, \mu, \lambda, \omega, \Delta \tau, \tau_{end}$). The robustness of CCDRec to different parameter settings and the ease of tuning these parameters in real-world large-scale systems would be a practical concern.

• Interpretation of "Hardness": While the visualization intuitively shows negatives getting "harder" (closer to positives), the precise definition of "hardness" in the context of multimodal representations and how it universally translates to optimal learning across diverse users and items could be further explored. Does "closer in embedding space" always equate to optimally "hard" for learning?

• Cold-Start Scenarios: While MMRec generally helps cold-start, the DMA relies on an initial item ID embedding ($e_i^0$) and base recommender outputs ($\hat{h}_i, \tilde{h}_i$). For truly cold-start items with very few interactions, how robust these initial embeddings are, and how effectively the diffusion process can enrich them, would be an interesting area to analyze. The current framework seems to enrich existing items rather than purely generating for unseen ones.

• Multimodality Beyond Visual/Textual: The current work primarily focuses on visual and textual modalities. Extending CCDRec to incorporate other modalities like audio, video, or even temporal sequences (e.g., in sequential recommendation) would be a natural next step, potentially requiring more complex conditional estimators or diffusion processes.

• Human Evaluation: While quantitative metrics are strong, a human evaluation study, perhaps on the "quality" or "perceived difficulty" of the sampled negative items, could provide valuable qualitative insights into CNS's effectiveness.

  Overall, CCDRec is a well-designed and empirically validated framework that pushes the boundaries of multimodal recommendation by creatively integrating Diffusion Models and curriculum learning into the negative sampling process.