1. Bibliographic Information

1.1. Title

The title of the paper is M2oE: Multimodal Collaborative Expert Peptide Model. It indicates the development of a novel model for peptide prediction that leverages multimodal information (likely sequence and structure) and a collaborative expert system.

1.2. Authors

• Zengzhu Guo: From the School of Information Sciences, Guangdong University of Finance and Economics, Guangzhou, China. Email: gzz3383@163.com.
• Zhiqi Ma: From the School of Medicine, The Chinese University of Hong Kong ShenZhen (CUHK-ShenZhen). Email: zhiqima@link.cuhk.edu.cn.

1.3. Journal/Conference

The paper does not explicitly state the journal or conference of publication. However, the Published at (UTC): 2024-12-03T00:00:00.000Z timestamp suggests it might be a preprint or a paper submitted for publication, as the date is in the future relative to the current analysis time. Without a specific venue name, its reputation and influence cannot be commented upon.

1.4. Publication Year

The publication year, based on the provided Published at (UTC) timestamp, is 2024.

1.5. Abstract

Peptides are crucial biomolecules composed of amino acids, holding significant importance in biological processes and drug development. Peptide prediction tasks are essential for identifying functional peptides. Traditional models typically encode peptides using their primary sequence and structural information. However, current research often focuses on single-modal prediction (either structure or sequence), overlooking multi-modal approaches. This leads to single-modal models struggling with datasets where information for their specific modality is sparse. To address this, the paper introduces the M2oE (Multimodal Collaborative Expert Peptide Model). This model integrates sequence and spatial structural information, utilizing an expert model and a Cross-Attention Mechanism to balance and enhance its capabilities. Experimental results demonstrate that M2oE achieves excellent performance in complex prediction tasks.

1.6. Original Source Link

The original source link is /files/papers/6921c121d8097f0bc1d013e4/paper.pdf. Given the provided link structure and the future publication date, it is likely a link to a preprint version of the paper. Its publication status is likely "preprint" or "submitted/under review".

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the limitation of single-modal peptide prediction models. Peptides, as biomolecules, possess both a primary amino acid sequence and a spatial structure, both of which are crucial for defining their properties and functions. While previous models have made strides in peptide prediction, most studies predominantly focus on single-modality data (either sequence or structure). The authors identify a significant challenge: single-modal models perform poorly when the dataset has less information or is biased against their particular modality. For instance, a sequence-based model might struggle with a dataset where structural motifs are more discriminative, and vice versa. This is important because peptides are increasingly vital in drug design (e.g., antimicrobial and anticancer agents), and accurate prediction accelerates discovery. The paper's entry point is to propose a truly multi-modal approach that effectively integrates both sequence and structural information, thereby overcoming the weaknesses of single-modal models and achieving a more balanced and robust predictive capability.

2.2. Main Contributions / Findings

The paper presents the M2oE multi-modal collaborative expert peptide model with the following primary contributions:

1. Sequence-Structure Mixing Expert Model: The authors propose a novel sequence-structure mixing expert model that specifically addresses the challenge of expert allocation in multimodal contexts. This moves beyond simple concatenation or fixed-weight fusion.
2. Interactive Attention Networks for Enhanced Representation: The model leverages multimodal characteristics to improve mixed expert representation through interactive attention networks (specifically Cross-Attention). This mechanism allows experts to learn from different modalities while also aligning similar characteristics and differentiating distinct ones.
3. Learnable Weights for Modality Significance: The model incorporates learnable weights ($\alpha$) to dynamically evaluate the importance of sequence and spatial information across varying data distribution scenarios. This adaptive weighting helps the model adjust its reliance on each modality based on the specific data.
4. Excellent Performance in Complex Tasks: The experimental results demonstrate that the M2oE model performs excellently in complex task predictions (e.g., Antimicrobial Peptide (AMP) classification and Aggregation Propensity (AP) regression), surpassing baseline single-modality and existing mixture models. It effectively addresses the limitations of single-modal models by achieving balanced improvements across different datasets.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand the M2oE model, a reader should be familiar with the following concepts:

• Peptides and Amino Acids:

  • Peptides: These are short chains of amino acids linked by peptide bonds. They are biomolecules that play vital roles in biological processes, distinct from proteins which are typically longer chains. Their functions can range from hormones to antimicrobial agents.
  • Amino Acids: The basic building blocks of peptides and proteins. There are 20 common types, each with a unique side chain that dictates its chemical properties. The order of amino acids in a peptide forms its primary sequence.

• Deep Learning (DL): A subfield of machine learning that uses artificial neural networks with multiple layers (hence "deep") to learn complex patterns from data.

  • Recurrent Neural Networks (RNNs): A class of neural networks where connections between nodes form a directed graph along a temporal sequence. This allows them to exhibit temporal dynamic behavior, making them suitable for sequence data.
  • Long Short-Term Memory (LSTM) Networks: A specialized type of RNN capable of learning long-term dependencies. They mitigate the vanishing gradient problem often encountered in standard RNNs through the use of gates (input, forget, output) that control information flow.
  • Bidirectional LSTM (BiLSTM): An extension of LSTM that processes sequence data in both forward and backward directions, allowing it to capture dependencies from both past and future contexts.
  • Transformer Architecture: A deep learning model introduced in 2017, entirely relying on attention mechanisms rather than recurrent or convolutional layers. It excels at handling sequential data by processing all parts of the input simultaneously, making it highly parallelizable and efficient for tasks like natural language processing and, as seen here, peptide sequence encoding.
  • Graph Neural Networks (GNNs): A class of neural networks designed to operate directly on graph-structured data. They learn node embeddings by aggregating information from a node's neighbors, making them suitable for modeling molecular structures where atoms are nodes and bonds are edges.
  • Graph Convolutional Networks (GCNs): A specific type of GNN that generalizes convolutional operations to graph data. They learn node features by performing localized spectral convolutions or message passing operations on the graph.

• Multimodal Learning: An area of machine learning that aims to build models capable of processing and relating information from multiple modalities (e.g., text, image, audio, sequence, structure). The challenge lies in effectively fusing and integrating these diverse data types.

• Mixture of Experts (MoE) Models: A neural network architecture that divides the input space among multiple expert networks, each specializing in a different part of the problem. A gating network learns to select or weight the outputs of these experts for a given input. This allows MoE models to scale to very large numbers of parameters while keeping computational costs manageable by only activating a subset of experts for each input.

• Cross-Attention Mechanism: An attention mechanism used in multimodal models to allow information from one modality (e.g., sequence) to influence the processing of another modality (e.g., structure), and vice versa. It enables the model to align and relate features across different data types, facilitating multimodal fusion.

3.2. Previous Works

The paper builds upon a rich history of deep learning models applied to various data types, specifically drawing from advances in sequence modeling and graph modeling.

• Sequence Models:

  • RNNs [8], LSTMs [9], BiLSTMs [10]: These recurrent neural networks were early pioneers in processing sequential data. They capture dependencies within a sequence by maintaining a hidden state that is updated at each step. LSTMs and BiLSTMs improved upon RNNs by addressing issues like vanishing gradients and enabling bidirectional context. In peptide prediction, they would process the amino acid sequence to learn features.
  • Transformers [11]: Represent a paradigm shift in sequence modeling, replacing recurrent and convolutional layers with self-attention mechanisms. They allow all elements in a sequence to be processed in parallel, capturing long-range dependencies more effectively. Transformers are cited as being particularly effective for peptide sequences.
  • SwitchTransformer [16]: An MoE variant of the Transformer architecture. It introduces sparsity by routing tokens to a subset of expert networks within the Feed-Forward Network (FFN) layers. This enables scaling to trillion-parameter models while maintaining computational efficiency. The paper uses SwitchTransformer as a strong baseline sequence model.

• Graph Models:

  • Graph Neural Networks (GNNs) [12]: Used for graph-structured data, crucial for representing peptide molecules where atoms are nodes and bonds are edges. They learn node embeddings by aggregating information from neighbors.
  • Graph Convolutional Networks (GCNs) [12]: A specific type of GNN that generalizes convolutional operations to graph data. They are instrumental in capturing molecular spatial information. The paper uses GCN as a foundational graph encoder.
  • GMoE [15]: A Graph Mixture of Experts model. It applies the MoE concept to graph learning, allowing different experts to specialize in processing different parts or characteristics of a graph. It is used as a mixture model baseline.

• Multimodal Models and Mixture of Experts:

  • GITFormer [14]: An example of advanced multimodal fusion in AI4Science, integrating graphical, imaging, and textual information. This shows the growing trend and potential of multimodal approaches in scientific domains.
  • MoE Models (e.g., GMoE [15], SwitchTransformer [16]): These models optimize token allocation to improve adaptability and scalability. The paper builds on the success of these MoE architectures.
  • Previous MoE applications [17][18][19][20]: The paper references various Mixture of Experts applications in areas like outbreak detection, emotion prediction, biomedical question answering, and image fusion, demonstrating the versatility of the MoE paradigm.

3.3. Technological Evolution

The field of peptide prediction has evolved significantly:

1. Traditional Methods: Initially relying on physicochemical properties and statistical models.
2. Early Deep Learning for Sequences: Adoption of RNNs, LSTMs, and BiLSTMs for peptide sequence analysis, treating peptides as linear sequences of amino acids.
3. Attention Mechanisms and Transformers: The Transformer architecture revolutionized sequence modeling, enabling more efficient capture of long-range dependencies and parallel processing, leading to higher performance for sequence-based peptide prediction.
4. Graph-based Deep Learning for Structures: Recognition of the importance of spatial structure led to the integration of GNNs (like GCN, GAT, GraphSAGE) to model peptides as graphs, capturing molecular topology.
5. Initial Multimodal Attempts: Some studies attempted multimodal approaches, but often lacked genuine integration, sometimes using contrastive learning or simple concatenation without deep interaction.
6. Advanced Multimodal and MoE Systems: The current work, M2oE, represents the cutting edge by combining multimodal fusion with Mixture of Experts and Cross-Attention. It addresses the limitations of prior multimodal attempts by focusing on sophisticated fusion methods and dynamic expert allocation.

3.4. Differentiation Analysis

Compared to existing methods, M2oE introduces several core differences and innovations:

• Beyond Single-Modality: Unlike most previous studies that predominantly focus on either sequence or structure (Transformer, SwitchTransformer for sequence; GCN, GAT, GraphSAGE for structure), M2oE explicitly integrates both. This directly addresses the problem that single-modal models struggle with datasets where their specific modality is information-poor.
• Genuine Multimodal Integration: The paper highlights that even some contrastive learning techniques often lack genuine integration. M2oE achieves deeper integration through its Sparse Cross Mixture of Experts (SCMoE) fusion module and Cross-Attention Mechanism, which allows for interactive attention between modalities, aligning similar characteristics and drawing away different characteristics.
• Adaptive Modality Weighting: The introduction of learnable weights ($\alpha$) in the fusion module is a significant innovation. Traditional multimodal fusion often uses fixed weights or simple concatenation. M2oE dynamically assesses the significance of sequence and spatial information under different data distribution scenarios, leading to more robust predictions.
• Enhanced Expert Allocation: While Mixture of Experts (MoE) models (GMoE, SwitchTransformer) are known for token allocation and adaptability, M2oE proposes a specific sequence-structure mixing expert model that addresses the challenge of expert allocation in a multimodal context, combined with Cross-Attention to ensure balanced improvements across modalities. This prevents MoE from only enhancing performance in one modality without benefiting others.
• Robustness to Data Bias: The central motivation of M2oE is to overcome the single-modal models' inability to handle datasets with less information in that particular modality. By collaboratively leveraging both modalities and dynamically adjusting their importance, M2oE demonstrates superior performance across diverse datasets, even when one modality might be less informative.

4. Methodology

4.1. Principles

The core idea behind M2oE is to combine the strengths of both peptide sequence and spatial structural information through a sophisticated multimodal fusion framework. It hypothesizes that neither modality alone is sufficient for robust peptide prediction across all scenarios, especially when one modality might contain limited information. The model leverages Mixture of Experts (MoE) to allow specialized processing of features and introduces Cross-Attention to facilitate deep interaction and alignment between the modalities. Furthermore, it incorporates learnable weights to dynamically adjust the contribution of each modality based on the specific data distribution, ensuring balanced and improved performance. The theoretical basis lies in the complementary nature of sequence and structural data for peptide function and the ability of MoE to scale and specialize, combined with attention mechanisms for effective inter-modal communication.

4.2. Core Methodology In-depth (Layer by Layer)

The M2oE model integrates sequence and graph encoders with a Sparse Cross Mixture of Experts (SCMoE) fusion module and a final fusion module for prediction. The overall architecture is depicted in Figure 1.

The following figure (images/1.jpg) shows the M2oE model architecture.

该图像是示意图，展示了M2oE模型的结构，包括序列编码器、图编码器和解码器，以及稀疏交叉专家机制。图中采用了多头自注意力和聚合机制，旨在整合氨基酸序列与分子图的信息，进行复杂任务的预测。

VLM Description: The image is a schematic diagram illustrating the structure of the M2oE model, including the sequential encoder, graph encoder, and decoder, as well as the sparse cross mixture of experts mechanism. The diagram employs multi-head attention and aggregation mechanisms to integrate information from the amino acid sequence and molecular graph for predicting complex tasks.

4.2.1. Benchmark Dataset

The M2oE model is evaluated on a benchmark dataset derived from Liu et al. [21]. This dataset is categorized into classification and regression tasks.

• Classification Task: Involves Antimicrobial Peptides (AMPs) [22]. The goal is to classify whether a peptide is antimicrobial (label 1) or non-antimicrobial (label 0).

• Regression Task: Involves Aggregation Propensity (AP) [13]. The goal is to predict a continuous value representing the peptide's aggregation tendency.

  Both datasets are partitioned into training, validation, and test sets with an 8:1:1 ratio.

The following are the results from Table I of the original paper:

4.2.2. Sequence and Graph Encodings

4.2.2.1. Sequence Encoder

The peptide sequence $S \subseteq \mathbf{R}^M$ is treated similarly to sentence data, requiring word-base embedding and positional identification for input. However, unlike natural language, peptide sequences are divided based on amino acids, simplifying the tokenization process.

The core component of the sequence encoder is the Transformer architecture, specifically utilizing Multihead Self Attention (MSA) and Feed Forward (FFN) layers.

• Multihead Self Attention (MSA): This mechanism scores the context of amino acids within the sequence and captures various dependencies. It allows the model to weigh the importance of different amino acids for understanding each amino acid in the sequence.

• Feed Forward (FFN): Combined with a nonlinear activation function and additional trainable parameters, FFN further captures non-linear relationships between amino acids and maps them to a higher-dimensional space.

  The output of the sequence encoder is represented as the amino acid feature $s \in S^{M \times d}$, where $M$ is the length of the peptide sequence (number of amino acids), and $d$ is the feature hidden dimension for each amino acid.

4.2.2.2. Graph Encoder

The peptide molecule is defined as a graph $\mathcal{G} = (\nu, \varepsilon)$.

• $\nu = \{\nu_i\}_{i=1}^N$: Represents the set of nodes (e.g., atoms or amino acid residues) in the graph.

• $\varepsilon \subseteq \nu \times \nu$: Represents the set of edges, indicating the existence of chemical bonds or interactions between nodes.

  An Adjacency matrix $A \in \{0, 1\}^{N \times N}$ describes the relationships between nodes. $A_{ij} = 1$ if there is an edge between node $i$ and node $j$, otherwise $A_{ij} = 0$.

Graph Convolutional Networks (GCN) [12] are employed to leverage relative edges and node attributes to learn latent representations of the nodes. A single layer of the graph convolutional encoder is represented by the following formula:

$ X^{(l+1)} = f_{GCN}(A, X^{(l)}; W^{(l)}) = \sigma (\hat{A} X^{(l)} W^{(l)}) $

Where:

• $X^{(l)}$: The feature matrix at layer $l$, where each row corresponds to a node's feature vector.

• $X^{(l+1)}$: The feature matrix at layer $l+1$, representing the updated node features.

• $f_{GCN}$: The GCN encoder function.

• $A$: The adjacency matrix of the graph.

• $W^{(l)}$: The learnable weight matrix for layer $l$ of the model.

• $\sigma$: A non-linear activation function, specifically Leaky ReLU, applied element-wise.

• $\hat{A}$: The normalized adjacency matrix. It is derived from $\widetilde{A} = A + I$, where $I$ is the identity matrix. Adding $I$ ensures that each node retains its own information during aggregation (self-loops). Then, $\hat{A} = D^{-\frac{1}{2}} \widetilde{A} D^{-\frac{1}{2}}$, where $D$ is the degree matrix of $\widetilde{A}$ (a diagonal matrix where $D_{ii} = \sum_j \widetilde{A}_{ij}$). This normalization helps prevent numerical instabilities and scales the feature vectors.

  The initial values $X^{(0)}$ are randomly initialized using a normal distribution. The final output by GCN is represented as $X \in \mathbf{R}^{N \times D}$, where $N$ is the number of nodes and $D$ denotes each node's embedding dimension.

4.2.3. Sparse Cross Mixture of Experts (SCMoE)

The SCMoE module is designed to fuse information from the parallel Transformer (sequence) and SAGEGraph (graph) encoders, which capture primary peptide sequence information and secondary molecular structure information, respectively. The premise is that sequence and structural information can represent and complement each other.

The SCMoE model contains C sequence mixing experts and graph mixing experts. These experts learn from tokens routed by different types of data through an expert network. The interactive attention network enables the mixing experts to focus on different modalities directly, thereby enhancing their representational capabilities through a multimodal alignment approach.

4.2.3.1. Routing Network

The routing network is controlled by a learnable matrix $W^{\bar{r}} \in \mathbf{R}^{d \times C}$, where $d$ is the feature dimension and $C$ is the number of experts. This matrix calculates the similarity between each token and the mixing experts and assigns tokens to the topk most similar experts.

The allocation method is given by the following formula:

$ Router(X_i) = Topk(\alpha_j X_{ij} + N(0, 1) \cdot Softplus(X_{ij} W_{noise})) \ \alpha_j = \frac{X_{ij} W^r}{\sum_{j=0}^{topk} X_{ij} W^r} $

Where:

• $Router(X_i)$: The output of the router for the $i$-th token $X_i$, indicating its assignment to experts.

• $Topk(\cdot)$: A function that selects the top k values or experts based on their scores.

• $X_i$: The $i$-th token (either from sequence or graph modality).

• $X_{ij}$: The component of the $i$-th token $X_i$ relevant to the $j$-th expert.

• $\alpha_j$: The normalized gating weight for the $j$-th expert, indicating its importance for token $X_i$.

• $W^r$: A learnable parameter matrix used in calculating $alpha_j$, essentially scoring the token-expert similarity.

• $N(0, 1)$: A stochastic variable sampled from the standard normal distribution. This noise term is added to the routing decision to mitigate the problem of some tokens never being assigned to experts when using Topk alone [23]. It provides a chance for tokens ranked after the initial top k to also be allocated.

• $W_{noise} \in \mathbf{R}^{d \times C}$: Learnable parameters associated with the noise component.

• $Softplus(\cdot)$: A nonlinear activation function ($Softplus(x) = \ln(1 + e^x)$) that helps prevent vanishing gradients and ensures the noise scaling factor is positive.

  The peptide sequence is composed of amino acid symbols, where each character can be considered a local feature. The combination of these local features assigned to mixed experts implicitly expresses certain characteristics of the peptide sequence.

4.2.3.2. Cross-Attention (CRA)

To improve the MoE and address the challenge of single-modal information making it difficult to directly learn implicit peptide characteristics, Cross-Attention (CRA) is proposed. It aims to align similar characteristics between modalities while also allowing different characteristics to emerge.

The Cross-Attention mechanism, as described, involves exchanging queries between modalities for spatial interaction. Let $F_{seq}$ and $F_{gra}$ denote the features from the sequence encoder and graph encoder, respectively. These features are transformed into Query (Q), Key (K), and Value (V) vectors. The Cross-Attention computations are as follows:

$ F_{fgra} = \mathrm{Softmax}\left(\frac{Q_{seq} K_{gra}^{\top}}{d_k}\right) V_{gra} \ F_{fseq} = \mathrm{Softmax}\left(\frac{Q_{gra} K_{seq}^{\top}}{d_k}\right) V_{seq} $

Where:

• $F_{fgra}$: Fused graph features derived from sequence query attending to graph keys and values. This means the sequence is "looking at" the graph features.

• $F_{fseq}$: Fused sequence features derived from graph query attending to sequence keys and values. This means the graph is "looking at" the sequence features.

• $Q_{seq}, K_{seq}, V_{seq}$: Query, Key, and Value matrices derived from the sequence features $F_{seq}$.

• $Q_{gra}, K_{gra}, V_{gra}$: Query, Key, and Value matrices derived from the graph features $F_{gra}$.

• $d_k$: A scaling factor (typically the square root of the dimension of the key vectors) used to prevent the dot products from growing too large, which could push the softmax function into regions with tiny gradients.

• $\mathrm{Softmax}(\cdot)$: The softmax function, which normalizes the attention scores into a probability distribution.

  These cross-attention matrices ($F_{fgra}$ and $F_{fseq}$) are then transformed and updated. The updated interactive features are then integrated into the routing network. For instance, the updated sequence features can be formed by concatenating the original sequence features with the fused sequence features: $F_{seq}^{new} = \mathrm{Concat}(F_{seq}, F_{fseq})$. This enhanced feature representation then participates in the expert allocation via the Router mechanism (similar to formula 2).

4.2.4. Fusion Module And Loss

The fusion module combines the enhanced embeddings from both modalities to make final predictions. Given that the SCMoE module already ensures characteristic expression for each modality and enhances potential features through inter-modal information, the fusion module primarily uses the nonlinear capability of an MLP to capture correlations between features and map them to the classification space (for AMP) or regression space (for AP).

A key innovation here is the use of learnable weights ($\alpha$) to dynamically assess the importance of sequence and spatial information under different data distribution scenarios, which is an improvement over traditional fixed-weight fusion methods.

The prediction $\hat{y}$ is calculated as:

$ \hat{y} = \sigma (\alpha MLP_1 (Z_{seq}) + (1-\alpha) MLP_2 (Z_{gra})) $

Where:

• $\hat{y}$: The predicted output (e.g., probability for AMP classification, or a regression value for AP).
• $\sigma$: The Sigmoid function ($Sigmoid(x) = \frac{1}{1 + e^{-x}}$), which maps the predictive data into a probability space (for classification tasks). For regression, it might be omitted or replaced with a linear activation.
• $\alpha$: A learnable weight (scalar between 0 and 1) that determines the contribution of the sequence encoder's output.
• $(1-\alpha)$: The complementary weight, determining the contribution of the graph encoder's output.
• $MLP_1$: A Multi-Layer Perceptron that processes the sequence embedding $Z_{seq}$.
• $MLP_2$: A Multi-Layer Perceptron that processes the graph embedding $Z_{gra}$.
• $Z_{seq}$: The embedding output from the sequence encoder after SCMoE processing.
• $Z_{gra}$: The embedding output from the graph encoder after SCMoE processing.

4.2.4.1. Loss Functions

The routing network in MoE can suffer from load imbalance issues, where some experts receive most tokens, effectively degrading the model into a single-expert model. To counteract this, a load balancing loss is introduced to ensure equal probability of selection for each expert.

The load balancing loss $L_{Load}$ is given by:

$ L_{load} = \sum_{i=1}^{C} \left( \frac{n_i}{\sum_{j=1}^{C} n_j} - \frac{1}{C} \right)^2 $

Where:

• $L_{load}$: The load balancing loss.

• $C$: The total number of experts.

• $n_i$: The number of tokens assigned to the $i$-th expert.

• $\frac{n_i}{\sum_{j=1}^{C} n_j}$: The proportion of tokens assigned to the $i$-th expert.

• $\frac{1}{C}$: The ideal uniform proportion of tokens if load balancing were perfect.

  Additionally, the routing network might preferentially allocate tokens to a few stronger experts, leaving others idle. To address this expert importance imbalance, an importance balancing loss is used.

The expert importance balancing loss $L_{importance}$ is given by:

$ L_{importance} = \omega_{imp} \cdot CV(\sum_{x \in X} Router(x)) \ CV(X) = \frac{\sigma_x}{\mu_x} $

Where:

• $L_{importance}$: The expert importance balancing loss.

• $\omega_{imp}$: A fixed hyperparameter that controls the weighting of this loss, helping to enforce similar abilities among different experts.

• $CV(\cdot)$: The coefficient of variation function, which measures the degree of discreteness or relative variability of the expert importance. It is the ratio of the standard deviation to the mean.

• $\sum_{x \in X} Router(x)$: Represents the aggregated routing decisions or importance scores for all tokens $x$ in the dataset $X$.

• $\sigma_x$: The standard deviation of the data $X$.

• $\mu_x$: The mean of the data $X$.

  Finally, the total optimization objective $L$ combines the prediction error (calculated using Binary Cross Entropy for classification) with these two balancing losses:

$ L = BCE(y, \hat{y}) + L_{Load} + L_{importance} $

Where:

• $L$: The total loss to be minimized during training.
• $BCE(y, \hat{y})$: The Binary Cross Entropy loss between the true labels $y$ and the predicted probabilities $\hat{y}$. This is a common loss function for binary classification tasks.
• $y$: The true label (e.g., 0 or 1 for AMP classification).
• $\hat{y}$: The predicted probability of the positive class.

5. Experimental Setup

5.1. Datasets

The study utilizes two benchmark datasets from Liu et al. [21] to evaluate the model's performance on different types of peptide prediction tasks:

• Antimicrobial Peptides (AMPs) Dataset [22]: This dataset is used for a classification task. The goal is to predict whether a peptide exhibits antimicrobial properties.
  • Characteristics: Peptides are labeled as AMP (label 1) or non-AMP (label 0).
  • Scale: The total dataset contains 9321 samples, split into 5437 AMP and 2019 non-AMP for training, 679 AMP and 252 non-AMP for validation, and 681 AMP and 253 non-AMP for testing.
• Aggregation Propensity (AP) Dataset [13]: This dataset is used for a regression task. The goal is to predict the aggregation propensity of peptides, which is a continuous value.

  • Characteristics: Each peptide is associated with a numerical value representing its aggregation propensity.

  • Scale: The total dataset contains 62159 samples, with 54159 for training, 4000 for validation, and 4000 for testing.

    Both datasets are partitioned into training, validation, and test sets with an 8:1:1 ratio. These datasets are standard in peptide prediction research and are effective for validating the multimodal approach due to their distinct task types (classification vs. regression) and underlying biological relevance, which may rely differently on sequence versus structural features.

The following are the results from Table I of the original paper:

5.2. Evaluation Metrics

The performance of the models is evaluated using different metrics for classification and regression tasks.

5.2.1. For Classification (AMPs Dataset)

• Accuracy (ACC)
  • Conceptual Definition: Accuracy measures the proportion of true results (both true positives and true negatives) among the total number of cases examined. It indicates how often the model makes a correct prediction.
  • Mathematical Formula: $ ACC = \frac{TP + TN}{TP + TN + FP + FN} $
  • Symbol Explanation:
    • TP: True Positives, the number of positive instances correctly predicted as positive.
    • TN: True Negatives, the number of negative instances correctly predicted as negative.
    • FP: False Positives, the number of negative instances incorrectly predicted as positive.
    • FN: False Negatives, the number of positive instances incorrectly predicted as negative.

5.2.2. For Regression (AP Dataset)

• Mean Absolute Error (MAE)

  • Conceptual Definition: MAE is a measure of errors between paired observations expressing the same phenomenon. It is the average of the absolute differences between predictions and true values. MAE is robust to outliers compared to MSE because it does not square the errors.
  • Mathematical Formula: $ MAE = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i| $
  • Symbol Explanation:
    • $N$: The total number of data points.
    • $y_i$: The true value for the $i$-th data point.
    • $\hat{y}_i$: The predicted value for the $i$-th data point.
    • $|\cdot|$: The absolute value function.

• Mean Squared Error (MSE)

  • Conceptual Definition: MSE measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value. MSE gives more weight to larger errors, making it sensitive to outliers.
  • Mathematical Formula: $ MSE = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2 $
  • Symbol Explanation:
    • $N$: The total number of data points.
    • $y_i$: The true value for the $i$-th data point.
    • $\hat{y}_i$: The predicted value for the $i$-th data point.

• R-squared ($R^2$)

  • Conceptual Definition: R-squared, also known as the coefficient of determination, indicates the proportion of the variance in the dependent variable that can be predicted from the independent variable(s). It represents how well the regression model fits the observed data. A higher R-squared value (closer to 1) indicates a better fit.
  • Mathematical Formula: $ R^2 = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}i)^2}{\sum{i=1}^{N} (y_i - \bar{y})^2} $
  • Symbol Explanation:
    • $N$: The total number of data points.
    • $y_i$: The true value for the $i$-th data point.
    • $\hat{y}_i$: The predicted value for the $i$-th data point.
    • $\bar{y}$: The mean of the true values ($\bar{y} = \frac{1}{N} \sum_{i=1}^{N} y_i$).
    • $\sum_{i=1}^{N} (y_i - \hat{y}_i)^2$: The sum of squared residuals (unexplained variance).
    • $\sum_{i=1}^{N} (y_i - \bar{y})^2$: The total sum of squares (total variance).

5.3. Baselines

The M2oE model is compared against a range of state-of-the-art methods, categorized into sequence models, graph models, and mixture models, to demonstrate its effectiveness.

• Sequence Models: These models primarily process the peptide's amino acid sequence.
  • Transformer: A foundational attention-based model for sequence data.
  • SwitchTransformer [16]: An MoE-based Transformer that routes tokens to different experts, showcasing advanced sequence modeling with sparsity.
• Graph Models: These models primarily process the peptide's spatial structural information represented as a graph.
  • GCN: Graph Convolutional Networks, a fundamental GNN for node feature learning.
  • GAT: Graph Attention Networks, which apply attention mechanisms to graph structures.
  • GraphSAGE: Graph Sample and Aggregate, a framework for generating embeddings by sampling and aggregating features from a node's local neighborhood.
• Mixture Models: These models attempt some form of multimodal fusion or utilize Mixture of Experts in a different context.

  • GMoE [15]: Graph Mixture of Experts, applying the MoE concept to graph-structured data.

  • Repcon(Avg): Likely a representation concatenation method with averaging, representing a simpler multimodal fusion approach.

  • M2oE(WS): A variant of M2oE using weighted summation without the full SCMoE and Cross-Attention mechanisms. This serves as an ablation baseline for the full M2oE.

  • M2oE(Concat): A variant of M2oE using concatenation for fusion, again likely without the full SCMoE and Cross-Attention mechanisms, serving as another ablation baseline.

  • M2oE(Parallel): This refers to the full M2oE model, where sequence and graph encoders operate in parallel and their features are fused through the SCMoE and Cross-Attention module. This is the model being proposed and evaluated.

    These baselines are representative as they cover both single-modal (sequence and graph) and various multimodal/MoE approaches, allowing for a comprehensive comparison of M2oE's performance and the specific contributions of its cross-attention and expert fusion mechanisms.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results highlight the M2oE model's ability to effectively balance and integrate sequence and structural features for peptide prediction tasks, outperforming both single-modality and existing mixture models.

The paper initially points out a key observation:

• On the AP dataset (regression), sequence models generally perform better. For instance, SwitchTransformer achieves an impressive $R^2$ of 0.951.

• Conversely, on the AMP dataset (classification), graph models show superior performance, with GraphSAGE leading with an accuracy of 0.847.

  This observation supports the paper's initial motivation: single-modality models excel when the dataset inherently favors that specific modality but encounter challenges when the data bias shifts. This underscores the need for a multimodal approach that can adapt.

The M2oE model effectively addresses this issue by synergistically combining the strengths of both modalities.

• For the AP dataset, M2oE achieves an $R^2$ of 0.951, matching the best single-modality SwitchTransformer, while also demonstrating competitive MAE (3.68E-2) and MSE (2.21E-3) values. This indicates that M2oE can perform at the level of the best specialized single-modal model even for a task biased towards sequence information.

• For the AMP dataset, M2oE achieves an accuracy of 0.862. This is a significant improvement, surpassing the best graph model (GraphSAGE at 0.847) and all other baselines. This demonstrates M2oE's ability to leverage sequence information to enhance graph-driven tasks and achieve a new state-of-the-art.

  The results suggest that M2oE doesn't just average the performance of two modalities; it actively improves upon them by intelligently fusing information. The paper states that MoE enhances performance in single modalities independently without benefiting other modalities directly, which is why Cross-Attention was implemented to ensure balanced improvements across modalities.

6.2. Data Presentation (Tables)

The following are the results from Table II of the original paper:

Ablation Study Analysis (Table II): Table II presents an ablation study conducted on the AP dataset to evaluate the contribution of the Cross-Attention (CRA) and Mixture of Experts (MoE) components within the M2oE model.

• M2oE without CRA nor MoE: This variant, representing a basic multimodal fusion without the key proposed mechanisms, shows the lowest performance with MAE of 3.96E-2, MSE of 2.57E-3, and $R^2$ of 0.942. This serves as a baseline for the impact of CRA and MoE.

• M2oE without CRA: Removing the Cross-Attention mechanism leads to a drop in performance compared to the full M2oE, with MAE of 3.74E-2, MSE of 2.27E-3, and $R^2$ of 0.949. This suggests that Cross-Attention plays a crucial role in enhancing the model's ability to integrate and align information across modalities.

• M2oE without MoE: Removing the Mixture of Experts component also results in a performance decrease, showing MAE of 3.79E-2, MSE of 2.38E-3, and $R^2$ of 0.947. This indicates that the MoE framework is effective in allowing specialized processing and improving overall representation.

• M2oE (Full Model): The complete M2oE model achieves the best results with MAE of 3.68E-2, MSE of 2.21E-3, and $R^2$ of 0.951.

  The ablation results clearly demonstrate the effectiveness of both the Cross-Attention and Mixture of Experts modules, as their removal individually or jointly leads to a degradation in performance. The full M2oE model consistently outperforms its ablated versions, confirming that these components are essential for its superior performance, contributing to a 0.9% improvement over the baseline (likely referring to the M2oE without CRA nor MoE variant or a simple concatenation approach).

The following are the results from Table III of the original paper:

Comparison with State-of-the-Art Methods (Table III): Table III provides a comprehensive comparison of M2oE against various sequence, graph, and mixture models on both AP (regression) and AMP (classification) datasets.

• Sequence Models:
  • Transformer and SwitchTransformer perform very well on AP ($R^2$ of 0.947 and 0.951 respectively), confirming that AP is largely a sequence-dependent task. However, their performance on AMP (ACC of 0.813 and 0.808) is notably lower than graph-based or multimodal models.
• Graph Models:
  • GCN, GAT, and GraphSAGE generally show weaker performance on AP compared to sequence models (e.g., GraphSAGE $R^2$ of 0.947 is good but MAE/MSE are higher than SwitchTransformer). Crucially, they perform significantly better on AMP (ACC up to 0.847 for GraphSAGE), indicating the importance of structural information for antimicrobial peptide prediction.
• Mixture Models (Baselines):
  • GMoE shows mixed results, good on AP ($R^2$ 0.947) but slightly lower on AMP (ACC 0.837) compared to GraphSAGE.
  • Repcon(Avg), M2oE(WS), and M2oE(Concat) represent simpler fusion strategies or M2oE variants without full CRA and MoE. They generally improve upon basic single-modal performance but do not reach the peak performance of the best single-modal models on their preferred task, nor do they achieve the highest overall score on AMP. For example, M2oE(Concat) has an $R^2$ of 0.949 for AP and ACC of 0.824 for AMP, which is better than Transformer on AMP but worse than GraphSAGE.
• M2oE(Parallel) (Full M2oE Model):

  • AP Dataset: Achieves an $R^2$ of 0.951, which is tied with the best single-modal SwitchTransformer, while having very competitive MAE (3.68E-2) and MSE (2.21E-3). This shows it retains the strong performance of sequence models on sequence-biased tasks.

  • AMP Dataset: Achieves the highest accuracy of 0.862. This significantly surpasses all single-modal models (e.g., GraphSAGE at 0.847, SwitchTransformer at 0.808) and all other mixture baselines.

    The results clearly validate the effectiveness of the proposed M2oE model. It demonstrates the ability to either match or exceed the performance of specialized single-modal models and significantly outperforms simpler multimodal fusion approaches. Its superior performance on AMP specifically highlights the power of its integrated sequence-structure approach, where the Cross-Attention and MoE mechanisms effectively combine complementary information for improved prediction.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper successfully proposes the M2oE (Multimodal Collaborative Expert Peptide Model), which represents a significant advancement in peptide prediction by effectively integrating sequence and spatial structural information. The model's key innovations include a sparse mixed expert model for specialized processing and a Cross-Attention Mechanism to facilitate deep interaction and alignment between modalities. Furthermore, the M2oE model incorporates learnable weights ($\alpha$) to dynamically assess and adapt to the relative importance of each modality across different data distribution scenarios. Experimental results on both Antimicrobial Peptide (AMP) classification and Aggregation Propensity (AP) regression tasks confirm that M2oE achieves excellent performance, consistently matching or surpassing state-of-the-art single-modal and existing mixture models. The ablation studies further demonstrate the individual contributions and effectiveness of both the Cross-Attention and Mixture of Experts components. This research effectively addresses the limitations of unimodal scenarios by providing a robust and balanced multimodal solution.

7.2. Limitations & Future Work

The authors point out a specific direction for future work:

• Connecting to More Complex Tasks: The paper suggests that the multimodal expert model could be extended to more complex tasks beyond prediction, such as peptide generation tasks. This implies that the rich, fused multimodal representations learned by M2oE could serve as a powerful foundation for generative models, opening avenues for de novo peptide design based on desired sequence and structural properties.

  While not explicitly stated as limitations in the conclusion, the paper's discussion of load imbalance and expert importance issues within MoE models implies that managing and optimizing these aspects is an ongoing challenge that the proposed loss functions aim to mitigate. The complexity of integrating multiple encoders, SCMoE, and Cross-Attention also suggests potential challenges in terms of computational resources and model interpretability.

7.3. Personal Insights & Critique

The M2oE model presents a highly intuitive and well-motivated approach to peptide prediction. The core idea that single-modal models falter when their specific modality is information-poor is a critical observation, and the proposed multimodal collaborative expert system is a logical and effective response.

Key Strengths:

• Holistic Data Utilization: The seamless integration of both sequence and structural information is a significant step towards more comprehensive peptide analysis.
• Adaptive Fusion: The learnable weights ($\alpha$) for balancing modality contributions are a clever design choice, allowing the model to dynamically adjust its reliance on sequence versus structure based on the specific data distribution. This makes the model more robust and generalizable.
• Intelligent Expert Routing: The Sparse Cross Mixture of Experts (SCMoE) with its enhanced routing network and balancing losses effectively addresses known challenges in MoE models, ensuring that experts are utilized efficiently and collaboratively.
• Cross-Attention for Deeper Interaction: The inclusion of Cross-Attention is crucial. It moves beyond simple concatenation by enabling a deeper, interactive understanding between modalities, aligning their complementary features and enhancing their respective representations.
• Empirical Validation: The comprehensive experimental results and ablation studies rigorously validate the model's performance and the contribution of its individual components, making the claims credible.

Potential Areas for Further Exploration/Critique:

• Computational Cost & Scalability: While MoE can improve scalability by activating only a subset of experts, the parallel encoders, cross-attention, and multiple MLP layers still contribute to a relatively complex architecture. A deeper analysis of the computational overhead (e.g., training time, inference time) compared to simpler baselines would be valuable.

• Interpretability: With multiple experts and cross-attention, understanding exactly why M2oE makes a particular prediction can become challenging. Future work could focus on interpretability methods to shed light on which amino acid motifs, structural elements, or expert pathways are most influential for specific predictions.

• Definition of "Less Information": The paper states that single-modal models struggle with "less information in that particular modality." While exemplified by dataset performance, a more formal definition or quantitative measure of "information content" per modality in a dataset could strengthen this claim and guide future dataset design.

• Hyperparameter Sensitivity: The model has several hyperparameters, including topk for expert allocation, omega_imp for importance loss, and potentially the number of experts and attention heads. A sensitivity analysis for these parameters could provide further insights into the model's robustness.

  Overall, M2oE provides a compelling framework for multimodal peptide modeling. Its principles could certainly be transferred to other biomolecular prediction tasks (e.g., protein-ligand binding, RNA folding) or even multimodal learning problems in other AI4Science domains where sequence and structural/graphical information are both critical and complementary. The adaptive weighting and cross-modal attention mechanisms are particularly insightful and could inspire similar architectures in other complex multimodal data fusion scenarios.