1. Bibliographic Information

1.1. Title

The central topic of the paper is "Multi-Peptide: Multimodality Leveraged Language-Graph Learning of Peptide Properties".

1.2. Authors

The authors of this study are Srivathsan Badrinarayanan, Chakradhar Guntuboina, Parisa Mollaei, and Amir Barati Farimani. Their affiliations are primarily with Carnegie Mellon University, across departments including Chemical Engineering, Electrical and Computer Engineering, Mechanical Engineering, Biomedical Engineering, and Machine Learning. Amir Barati Farimani is the corresponding author.

1.3. Journal/Conference

This paper is published as part of the "Journal of Chemical Information and Modeling special issue 'Harnessing the Power of Large Language Model-Based Chatbots for Scientific Discovery'". The Journal of Chemical Information and Modeling (JCIM) is a highly reputable peer-reviewed scientific journal published by the American Chemical Society (ACS). It focuses on new methods and approaches in chemical informatics and computational chemistry, making it a well-regarded venue for research on computational prediction of molecular properties, especially those involving machine learning and bioinformatics.

1.4. Publication Year

The paper is published in 2025.

1.5. Abstract

The research addresses the critical need for improved prediction of peptide properties due to their significance in biological and therapeutic contexts. It introduces Multi-Peptide, an innovative approach that integrates transformer-based language models with graph neural networks (GNNs). Specifically, Multi-Peptide combines PeptideBERT, a transformer model tailored for peptide property prediction, with a GNN encoder to simultaneously capture both sequence-based and structural features of peptides. The methodology employs a contrastive loss framework to align the embeddings from these two distinct modalities (language for sequence, graph for structure) into a shared latent space. This alignment aims to enhance the predictive accuracy of the transformer model. Evaluations on hemolysis and nonfouling datasets demonstrate the robustness of Multi-Peptide, achieving a state-of-the-art accuracy of 88.057% in hemolysis prediction. The study concludes by highlighting the significant potential of multimodal learning in bioinformatics, paving the way for more accurate and reliable predictions in peptide-related research and applications.

1.6. Original Source Link

The original source link is /files/papers/6921cd82d8097f0bc1d013f0/paper.pdf. Given the provided context "Published at (UTC): 2024-12-19T00:00:00.000Z", this indicates that the paper has been officially published.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the accurate and efficient prediction of peptide properties. Peptides, which are short chains of amino acids, are fundamental to numerous biological processes and hold immense promise in therapeutic applications and biomaterial development. Key properties like hemolysis (the rupture of red blood cells, critical for drug safety) and nonfouling behavior (resistance to unwanted interactions, vital for biomedical devices) significantly influence their utility.

Historically, computational tools like quantitative structure-activity relationship (QSAR) models have been used to link peptide sequences to their properties. However, these methods face challenges in scalability and computational efficiency as the volume of available peptide sequence data rapidly expands. This limitation underscores a critical gap: existing computational approaches struggle to decipher the complex, non-linear relationships between protein sequences and their diverse properties efficiently.

More recently, machine learning and deep learning techniques, particularly transformers and large language models (LLMs), have shown great promise in protein sequence analysis. Models like AlphaFold have revolutionized protein structure prediction from sequences. While these sequence-based models excel at capturing long-range dependencies, they inherently lack the ability to directly incorporate spatial arrangements and local interactions among amino acids that are crucial for many peptide properties. This means they might miss vital structural information.

The paper's entry point and innovative idea lie in addressing this limitation by integrating multimodality. It proposes that by combining sequence information (processed by language models) with structural information (processed by graph neural networks), a more comprehensive understanding of peptide properties can be achieved. This synergistic approach aims to overcome the "blind spot" of purely sequence-based models regarding spatial and structural features, thereby enhancing predictive capabilities.

2.2. Main Contributions / Findings

The paper makes several primary contributions:

• Introduction of Multi-Peptide: The study proposes Multi-Peptide, an innovative multimodal learning framework that combines transformer-based language models (specifically PeptideBERT) with Graph Neural Networks (GNNs) for peptide property prediction. This is a novel integration designed to leverage both sequence and structural information.

• Leveraging Multimodality through Contrastive Learning: Multi-Peptide employs a contrastive loss framework, inspired by CLIP (Contrastive Language-Image Pretraining), to align embeddings from text (sequence) and graph (structure) modalities into a shared latent space. This alignment process allows the PeptideBERT model to learn from the structural context provided by the GNN, effectively transferring knowledge between modalities.

• Enhanced Transformer Model Accuracy: The core finding is that this multimodal training enhances the predictive accuracy of the transformer model. The PeptideBERT component, after being fine-tuned within the Multi-Peptide framework, achieves improved performance.

• State-of-the-Art Performance in Hemolysis Prediction: Multi-Peptide demonstrates robust performance, achieving a state-of-the-art (SOTA) accuracy of 88.057% on the hemolysis dataset, outperforming previous models including fine-tuned PeptideBERT and HAPPENN.

• Demonstration of Multimodal Potential in Bioinformatics: The study highlights the significant potential of multimodal learning in bioinformatics, providing a methodology for more accurate and reliable predictions in peptide-based research by comprehensively utilizing both sequence and derived structural data.

• Qualitative Analysis of Embedding Spaces: Through t-SNE visualizations, the study provides insights into how different models (PeptideBERT, GNN, and Multi-Peptide) capture and separate classes in their embedding spaces, offering a qualitative understanding of the multimodal learning process, particularly noting challenges with the nonfouling dataset.

  These findings contribute to solving the problem of comprehensive peptide property prediction by addressing the limitations of single-modality approaches, paving the way for more effective peptide design and therapeutic development.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully grasp the Multi-Peptide approach, an understanding of several foundational concepts is essential.

• Peptides: Peptides are short chains of amino acids, typically consisting of 2 to 50 amino acids, linked by peptide bonds. They are smaller than proteins (which are generally over 50 amino acids) but perform a vast array of biological functions, acting as hormones, antibiotics, enzymes, and more. Their specific sequence and three-dimensional structure dictate their function and interaction with other molecules.

• Hemolysis: Hemolysis is the process by which red blood cells are destroyed, releasing hemoglobin into the surrounding fluid. In the context of peptide therapeutics, hemolytic activity is a critical safety concern, as peptides designed for drug delivery or antimicrobial purposes must not cause significant red blood cell damage. Predicting whether a peptide is hemolytic or non-hemolytic is therefore crucial for drug development.

• Nonfouling: Nonfouling refers to the ability of a material or surface to resist the adsorption of proteins, cells, and other biological molecules. Nonfouling peptides are highly desirable in biomedical applications, such as biosensors, implants, and drug delivery systems, where preventing unwanted interactions (fouling) is essential for maintaining functionality and biocompatibility.

• Transformer Models: Transformer models are a type of neural network architecture introduced in 2017, primarily for natural language processing (NLP) tasks. They are characterized by their self-attention mechanism, which allows the model to weigh the importance of different parts of an input sequence when processing each element. Unlike recurrent neural networks (RNNs), transformers can process all input elements in parallel, making them highly efficient for long sequences and capable of capturing long-range dependencies effectively.

  • Self-Attention Mechanism: At the core of transformers is the self-attention mechanism. For each word (or token) in an input sequence, self-attention calculates how much attention it should pay to other words in the same sequence. It computes three vectors for each token: a Query (Q), Key (K), and Value (V). The attention score is then calculated as the dot product of the Query with all Key vectors, scaled by $\sqrt{d_k}$ (where $d_k$ is the dimension of the Key vectors), and then passed through a softmax function to get attention weights. These weights are then multiplied by the Value vectors and summed to produce the output for that token. $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ Here, $Q$ represents the query matrix, $K$ represents the key matrix, and $V$ represents the value matrix. $K^T$ is the transpose of the key matrix. The softmax function normalizes the scores, ensuring they sum to 1, effectively producing a probability distribution. The scaling factor $\sqrt{d_k}$ is used to prevent the dot products from becoming too large, which can push the softmax function into regions with very small gradients.

• BERT (Bidirectional Encoder Representations from Transformers): BERT is a powerful transformer-based language model developed by Google. Its key innovation is bidirectional training, meaning it considers the context from both the left and right sides of a word during pre-training. This allows it to learn deep contextual representations of words. BERT is typically pre-trained on large text corpora using tasks like Masked Language Model (MLM) (predicting masked words) and Next Sentence Prediction (NSP), and then fine-tuned for specific downstream tasks.

• Graph Neural Networks (GNNs): GNNs are a class of neural networks designed to operate directly on graph-structured data. Unlike traditional neural networks that process grid-like data (images) or sequences (text), GNNs can learn representations of nodes and edges in a graph by aggregating information from their neighbors. This makes them ideal for modeling molecular structures, where atoms are nodes and chemical bonds are edges, or in this case, representing the 3D structure of peptides where atoms are nodes and their spatial relationships define edges.

  • SAGEConv (GraphSAGE Convolution): GraphSAGE (Graph Sample and Aggregate) is a prominent GNN architecture. SAGEConv is a specific convolutional layer within GraphSAGE that learns to aggregate feature information from a node's local neighborhood. Instead of learning an embedding for each node (which doesn't generalize to unseen nodes), GraphSAGE learns a function that generates embeddings by sampling and aggregating features from a node's local neighbors. This allows it to generalize to unseen nodes and entire graphs.

• AlphaFold: AlphaFold (and its successor AlphaFold2) is an artificial intelligence program developed by Google DeepMind that predicts 3D protein structures from their amino acid sequences with unprecedented accuracy. It has significantly advanced structural biology by bridging the sequence-to-structure knowledge gap. AlphaFold generates Protein Data Bank (PDB) files, which are standard formats for storing atomic coordinates and other structural information of macromolecules.

• Contrastive Learning: Contrastive learning is a machine learning paradigm where the model learns representations by pushing "positive" pairs (e.g., different augmented views of the same image, or a text and its corresponding image) closer together in the embedding space while simultaneously pushing "negative" pairs (unrelated images and texts) farther apart. This encourages the model to learn what makes similar things similar and dissimilar things dissimilar, leading to robust and discriminative embeddings.

• CLIP (Contrastive Language-Image Pretraining): CLIP, pioneered by OpenAI, is a specific instance of contrastive learning designed to learn visual concepts from natural language supervision. It trains a text encoder and an image encoder to predict which image-text pairs are a match in a dataset. The model is trained to maximize the cosine similarity between the embeddings of paired (image, text) data and minimize it for unpaired data. Multi-Peptide adapts this concept to language (peptide sequence) and graph (peptide structure) modalities.

3.2. Previous Works

The paper builds upon a foundation of research in computational biology, machine learning, and specifically protein/peptide property prediction.

• Quantitative Structure-Activity Relationship (QSAR) Models: Historically, QSAR models have been a cornerstone for predicting molecular properties. These models establish mathematical relationships between chemical structure (e.g., peptide sequence features) and biological activity. While effective, the paper notes their limitations in scalability and computational efficiency when dealing with exponentially growing peptide databases, necessitating more advanced methods.

• Machine Learning in Bioinformatics: The broader field has seen a significant transformation with the advent of machine learning techniques. These methods are better equipped to handle large biological datasets and uncover complex patterns. The paper references several works showcasing the application of ML to GPCR activity prediction and protein property prediction.

• Protein Structure Prediction (AlphaFold): The development of AlphaFold by Google DeepMind (Jumper et al., 2021) is a critical precursor. AlphaFold enabled highly accurate prediction of protein 3D structures from amino acid sequences, effectively bridging the sequence-to-structure gap. This technological breakthrough is directly leveraged by Multi-Peptide to generate the structural PDB files that serve as input for the GNN module. Without AlphaFold's capability to generate reliable structural data, the multimodal approach would be significantly hampered.

• PeptideBERT: PeptideBERT (Guntuboina et al., 2023) is a direct predecessor and a core component of Multi-Peptide. PeptideBERT is a transformer-based language model specifically fine-tuned for peptide property prediction based solely on textual peptide sequences. It builds upon ProtBERT (Elnagga et al., 2021), which is a BERT model pre-trained on massive protein sequence datasets to learn general protein language representations. PeptideBERT attaches a tunable head to ProtBERT to adapt it to specific peptide prediction tasks. Multi-Peptide takes PeptideBERT as its base language model and aims to enhance its predictive capabilities by incorporating structural information.

• Multimodal Learning in Other Domains: The paper acknowledges that multimodal architectures (combining different types of data like text, image, graph) have proven successful in improving predictive accuracy across various domains. It cites examples in heat transport, student engagement prediction, and cross-modal retrieval, setting the stage for its application in bioinformatics. The CLIP framework (Radford et al., 2021) is a direct inspiration for the contrastive learning component, adapting its success in aligning image and text embeddings to the peptide sequence and structure domain.

3.3. Technological Evolution

The technological evolution leading to Multi-Peptide can be traced through several stages:

1. Early Computational Chemistry (QSAR): Initial attempts to predict properties relied on deriving mathematical relationships from chemical structures. These methods were often limited by their reliance on expert-defined features and computational costs for large datasets.
2. Machine Learning Wave: With increasing computational power and data availability, machine learning algorithms offered more flexible and data-driven approaches, handling larger datasets and discovering more complex patterns.
3. Deep Learning and Transformers: The advent of deep learning, particularly transformer architectures and LLMs, revolutionized sequence-based tasks. Models like ProtBERT and PeptideBERT demonstrated the power of self-supervised pre-training on vast protein sequence data, enabling contextual understanding of amino acid sequences.
4. Structure Prediction Breakthroughs: AlphaFold marked a significant leap, providing accurate 3D protein structures from sequences, which was previously a major bottleneck. This made structural information readily available for computational models.
5. Multimodal Integration: The current paper represents the next step: integrating these advancements. Recognizing the limitations of purely sequence-based models (missing spatial context) and purely structure-based models (potentially less semantic context), Multi-Peptide combines the strengths of transformer-based language models (for sequence context) and GNNs (for structural context) through a contrastive learning framework. This moves beyond single-modality learning to a more holistic understanding.

3.4. Differentiation Analysis

Compared to the main methods in related work, Multi-Peptide introduces several core differences and innovations:

• Integration of Modalities: The primary innovation is the explicit integration of two distinct data modalities—peptide sequences (textual) and peptide structures (graphical)—into a single learning framework. Prior works, especially PeptideBERT, focused solely on sequence data. While AlphaFold generates structural data, Multi-Peptide is unique in using this structural data alongside sequence data for property prediction in a unified model.
• Leveraging Contrastive Loss for Modality Alignment: Multi-Peptide uses a CLIP-inspired contrastive loss to align the embeddings from the PeptideBERT encoder and the GNN encoder into a shared latent space. This is a sophisticated mechanism for knowledge transfer and synergistic learning between heterogeneous data types, which is not present in single-modality models.
• Enhancing Transformer Accuracy through Structural Context: Instead of just concatenating features, Multi-Peptide uses the structural information learned by the GNN to improve the weights of the PeptideBERT transformer. The GNN is used during training to transfer knowledge, but can be discarded at inference time, making the final inference model a more robust transformer that implicitly understands structural nuances without direct structural input at prediction. This is a key difference from traditional multimodal fusion where both modalities are required at inference.
• Addressing Limitations of Sequence-Only Models: By incorporating structural data, Multi-Peptide directly addresses the PeptideBERT's limitation in capturing spatial arrangements and local interactions among amino acids. This leads to a more comprehensive representation of peptide properties.
• State-of-the-Art Performance: The ability to achieve state-of-the-art accuracy on challenging datasets like hemolysis prediction validates the effectiveness of this multimodal, contrastive learning approach over existing methods, including fine-tuned PeptideBERT and other specialized predictors like HAPPENN.

4. Methodology

4.1. Principles

The core idea behind Multi-Peptide is to enhance the prediction accuracy of transformer-based language models for specific peptide properties by leveraging information from a different, complementary modality: protein structure. The theoretical basis is that peptide properties are influenced by both their linear amino acid sequence and their three-dimensional structural arrangement. A language model (like PeptideBERT) excels at understanding sequence context and long-range dependencies, while a Graph Neural Network (GNN) is adept at capturing local interactions and spatial relationships within a molecular structure. By combining these two modalities and aligning their learned representations through a contrastive loss framework, the model can gain a more holistic and robust understanding of peptide features, leading to improved predictive capabilities. The intuition is that if a sequence and its corresponding structure represent the same peptide, their learned embeddings should be similar in a shared latent space, while embeddings from unrelated sequences and structures should be dissimilar.

4.2. Core Methodology In-depth (Layer by Layer)

The Multi-Peptide framework comprises three main components: a pretrained language model (PeptideBERT), a Graph Neural Network (GNN), and a shared latent space where contrastive loss is computed. The overall process involves data preparation, individual model pretraining, and then a contrastive learning phase to fine-tune the PeptideBERT model using the GNN's structural insights.

4.2.1. Data Sets

The study utilizes two primary datasets for evaluating peptide properties: hemolysis and nonfouling behavior.

• Data Sources:

  • Hemolysis Data: Computational techniques are used to predict hemolytic properties based on data from the Database of Antimicrobial Activity and Structure of Peptides (DBAASPv3). After removing duplicates with conflicting labels, the dataset consists of 845 positively marked sequences (15.065%) and 4764 negatively marked sequences (84.935%).
  • Nonfouling Data: Information for forecasting resistance against non-specific interactions (nonfouling) is gathered from a study by Barre et al. (2018). This dataset initially contains 3,600 positively marked sequences and 13,585 negatively marked sequences. After removing 7 duplicate sequences and 3 sequences for which AlphaFold2 failed to generate PDB files, it results in 3596 positively marked sequences (20.937%) and 13579 negatively marked sequences (79.063%). Negative examples are derived from insoluble and hemolytic peptides, along with scrambled negatives of similar length.

• Peptide Representation:

  • Textual Representation: Peptide sequences are represented as text, using letter sequences of various lengths. The following figure (Figure 3 from the original paper) shows the distribution of peptide sequence lengths and the number of atoms:

    该图像是四个频率分布图，展示了肽序列长度和原子数量的统计特征。图a和c分别为肽序列长度的分布，图b和d显示了肽中原子数量的分布。各图标注了最小值和最大值。

  • Structural Representation: For each protein sequence, AlphaFold2 is used to generate Protein Data Bank (PDB) files. These PDB files contain detailed atomic coordinates and other information, providing insights into the 3D structural arrangement of the peptides. To mitigate noise, 5 PDB files are generated for each sequence, and the one with the highest confidence score (pLDDT) is selected.

• Preprocessing:

  • Amino Acid Encoding: Each of the 20 amino acids is initially represented by its corresponding index in a predefined array.
  • PeptideBERT Compatibility: For PeptideBERT (built on ProtBERT), the integer-encoded sequences are converted back to letter characters and then re-encoded using ProtBERT's specific encoding scheme.
  • GNN Compatibility: For the GNN, the input data consists of features extracted directly from the AlphaFold2-generated PDB files. These features for each constituent atom serve as nodes in the graph and include:
    • Atom coordinates ($\bar{\Psi}(x, y, z)$)
    • Atomic number
    • Atomic mass
    • Atomic radius
    • Indication of whether the atom is part of a side chain or backbone
    • Residue index
    • Number of atoms in the residue
    • Residue sequence number Edges in the graph represent the relationships (e.g., spatial proximity, covalent bonds) between these atomic nodes.

• Addressing Data Imbalance: Both datasets exhibit class imbalance, with a higher number of negative examples. To counter this bias, an oversampling technique is applied, where positive examples are duplicated to balance the class distribution.

• Data Splitting: Each balanced dataset is split into a training set (80%) and a test set (20%).

4.2.2. Model Architecture

The overall model framework for Multi-Peptide integrates a transformer, a GNN, and a shared latent space for contrastive learning. The following figure (Figure 2 from the original paper) illustrates this architecture:

该图像是一个示意图，展示了Multi-Peptide方法如何结合PeptideBERT与GNN编码器进行肽属性预测的过程。通过使用Clip矩阵，模型获取序列和结构特征，从而预测肽的非粘附性和溶血性属性。

4.2.2.1. PeptideBERT Transformer

The base language model is PeptideBERT, which is a transformer-based model fine-tuned for peptide property prediction. It was originally fine-tuned from ProtBERT, a BERT model pre-trained on protein sequences. PeptideBERT processes protein sequences and their corresponding attention masks to generate contextual text embeddings through its underlying encoder. While excellent at capturing long-range dependencies and global context within sequences, it inherently struggles with capturing local structural features and other spatial dependencies.

4.2.2.2. GNN Module

To address the structural information, a Graph Neural Network (GNN) module is employed.

• Input: The GNN takes as input the graph representation derived from AlphaFold2-generated PDB files. Each atom in the peptide corresponds to a node, and the 11 features mentioned previously (coordinates, atomic number, etc.) constitute the node features. Edges represent inter-atomic relationships.
• Architecture: The GNN module leverages PyTorch Geometric's SAGEConv layer to perform graph convolutions. SAGEConv iteratively aggregates information from neighboring nodes, enabling the model to learn localized structural patterns.
• Output: The SAGEConv layers are followed by a fully connected neural network incorporating Rectified Linear Unit (ReLU) activation functions and a sigmoid layer. This converts the aggregated graph information into suitable graph embeddings, which capture the structural characteristics of the peptide.

4.2.2.3. Shared Latent Space and Pretraining

• Projection Heads: To enable contrastive learning between the two modalities, projection heads are used. These consist of linear projection layers with Gaussian Error Linear Unit (GELU) activation, dropout, and layer normalization. Their purpose is to map the high-dimensional graph embeddings from the GNN and the text embeddings from PeptideBERT into a unified shared latent space. This shared space facilitates the joint learning of properties from both structural and textual representations.
• Individual Pretraining: Prior to multimodal integration, both the PeptideBERT model and the GNN model are pretrained individually on their respective modalities (sequence data for PeptideBERT, PDB graph data for GNN) for each protein property dataset. This step ensures that each model develops a foundational understanding of its specific data type before attempting to combine their learnings.
• Knowledge Transfer: The key aspect of Multi-Peptide is to improve the PeptideBERT model. During the subsequent contrastive learning phase, the pretrained GNN model's weights are frozen. This allows the knowledge learned by the GNN about structural features to be transferred to the PeptideBERT model through the shared latent space alignment, effectively updating PeptideBERT's weights to incorporate structural insights.

4.2.3. Contrastive Learning

The integration and knowledge transfer between the GNN and PeptideBERT are achieved through a contrastive loss framework, inspired by Contrastive Language-Image Pretraining (CLIP).

• Mechanism: Contrastive learning trains the framework to link related pairs of inputs (a peptide sequence and its corresponding structural graph) while distinguishing them from unrelated ones. In the embedding space, this means bringing embeddings of genuine sequence-structure matches closer together and pushing embeddings of non-matching pairs farther apart.

• Embedding Generation: For a given protein, $p$, the graph embedding ($e_g$) and text embedding ($e_t$) are generated by their respective encoders: $ e_g = \mathrm{GNN}(\mathrm{structure}(p)) $ $ e_t = \mathrm{PeptideBERT}(\mathrm{sequence}(p)) $ Here, $\mathrm{GNN}(\mathrm{structure}(p))$ refers to the embedding generated by the GNN from the structural representation of protein $p$, and $\mathrm{PeptideBERT}(\mathrm{sequence}(p))$ refers to the embedding generated by PeptideBERT from the sequence representation of protein $p$.

• Similarity Measurement: The similarity between two vectors (embeddings from different modalities), $x$ and $y$, is measured using the dot product between their normalized embeddings. This reflects the cosine similarity. Let $\mathrm{sim}(x, y)$ denote this similarity: $ \mathrm{sim}(x, y) = \frac{x \cdot y}{|x| |y|} $ Where $x \cdot y$ is the dot product of vectors $x$ and $y$, and $\|x\|$ and $\|y\|$ are their respective Euclidean norms. In the context of CLIP, it's often the dot product of normalized embeddings.

• Loss Function: The contrastive loss uses cross-entropy loss to compare predicted similarities between embeddings and their target values. The softmax operation normalizes similarity scores, adjusted by a temperature parameter ($\tau$), to guide learning. This aims to align embeddings from both modalities meaningfully. The overall symmetric loss, $L(\mathbf{g}, \mathbf{t})$, between the graph ($\mathbf{g}$) and text ($\mathbf{t}$) modalities is given by: $ L ( { \bf g } , { \bf t } ) = \frac { 1 } { 2 } [ l ( { \bf g } , { \bf t } ) + l ( { \bf t } , { \bf g } ) ] $ This symmetric loss ensures that the model learns to align both from text to graph and from graph to text. The component loss $l(\mathbf{g}, \mathbf{t})$ (loss for aligning graph to text) is defined as: $ l ( \mathbf { g } , \mathbf { t } ) = - \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \log \left( \frac { e ^ { \sin (e _ { g } ^ { i } , e _ { t } ^ { i } ) / \tau } } { \sum _ { j = 1 } ^ { N } e ^ { \sin (e _ { g } ^ { i } , e _ { t } ^ { j } ) / \tau } } \right) $ Where:

  • $N$ is the batch size (number of protein samples).

  • $i$ indexes the current protein sample in the batch.

  • $j$ iterates over all protein samples in the batch.

  • $e_g^i$ is the graph embedding for the $i$-th protein.

  • $e_t^i$ is the text embedding for the $i$-th protein.

  • $\mathrm{sim}(e_g^i, e_t^i)$ is the similarity score between the graph and text embeddings for the same protein $i$. This is the positive pair.

  • $\mathrm{sim}(e_g^i, e_t^j)$ is the similarity score between the graph embedding of protein $i$ and the text embedding of protein $j$. If $i \neq j$, this represents a negative pair.

  • $\tau$ (tau) is the temperature parameter. This hyperparameter controls the scaling of the similarity scores before the softmax function. A smaller $\tau$ makes the softmax output sharper, emphasizing the highest similarity scores more strongly, while a larger $\tau$ smooths the distribution. It plays a crucial role in regulating the learning dynamics of contrastive objectives.

  • The numerator $e^{\mathrm{sim}(e_g^i, e_t^i) / \tau}$ represents the similarity of the positive pair.

  • The denominator $\sum_{j=1}^N e^{\mathrm{sim}(e_g^i, e_t^j) / \tau}$ sums the similarities of the current graph embedding $e_g^i$ with all text embeddings in the batch, including the positive one and all negative ones.

  • The term inside the logarithm is essentially the probability that $e_g^i$ correctly identifies its matching $e_t^i$ among all other N-1 negative text embeddings in the batch.

  • The negative logarithm is used because cross-entropy loss minimizes this quantity, effectively maximizing the probability of correct matches.

    The objective of this contrastive loss for binary classification is to enable the model to learn discriminative representations. It encourages distinct representations for positive and negative examples, facilitating better separation between the two classes by pushing positive pairs together and negative pairs apart in the shared latent space.

4.2.4. Weight Updates and Inference

• Training Phase: After individual pretraining, the models are combined. During the contrastive learning phase, backpropagation occurs through the contrastive loss matrix. Crucially, the weights of the pretrained GNN model are frozen during this step. This design choice means that only the weights of the PeptideBERT model are updated. The GNN acts as a static source of structural knowledge, guiding the PeptideBERT to learn better representations that implicitly account for structural information.
• Inference Phase: For inference on unseen peptide sequences, only the updated PeptideBERT model is used as a zero-shot classifier. The GNN model is discarded at inference time. This means that while Multi-Peptide leverages multimodality during training to enhance PeptideBERT's understanding, the final deployed model for prediction is a more robust PeptideBERT that no longer requires structural input. This design makes the inference process more efficient as it only relies on the sequence data.

5. Experimental Setup

5.1. Datasets

The study evaluates Multi-Peptide on two distinct peptide property prediction tasks: hemolysis and nonfouling behavior.

• Hemolysis Dataset:

  • Source: Database of Antimicrobial Activity and Structure of Peptides (DBAASPv3).
  • Characteristics: This dataset is used to predict whether a peptide causes red blood cell lysis. It is initially imbalanced.
  • Scale (after deduplication): 845 positively marked sequences (15.065%) and 4764 negatively marked sequences (84.935%).
  • Purpose: To assess the model's ability to identify peptides that are safe for therapeutic applications with respect to red blood cell integrity.

• Nonfouling Dataset:

  • Source: A study by Barre et al. (2018) focused on classifying multifunctional peptides.
  • Characteristics: This dataset is used to predict whether a peptide resists non-specific molecular interactions. It is also initially imbalanced. Negative examples are derived from insoluble and hemolytic peptides, plus scrambled negatives.
  • Scale (after deduplication and PDB generation): 3596 positively marked sequences (20.937%) and 13579 negatively marked sequences (79.063%).
  • Purpose: To evaluate the model's performance in identifying peptides suitable for applications requiring resistance to biofouling.

• Dataset Preprocessing:

  • Amino Acid Encoding: Each of the 20 standard amino acids is represented by a unique index.
  • PeptideBERT Encoding: For PeptideBERT, sequences are converted from integer indices back to character strings and then re-encoded using ProtBERT's specific tokenization scheme.
  • GNN Feature Extraction: For the GNN, AlphaFold2 is used to generate Protein Data Bank (PDB) files for each peptide sequence. To improve structural data quality, 5 PDB files are generated for each sequence, and the one with the highest pLDDT (per-residue confidence score) is selected. Node features for the GNN are extracted from these PDB files and include: atom coordinates ($\bar{\Psi}(x, y, z)$), atomic number, atomic mass, atomic radius, indicator for side chain/backbone, residue index, number of atoms in the residue, and residue sequence number.
  • Class Balancing: An oversampling technique is applied to both datasets to address their inherent class imbalance (more negative examples than positive). This involves duplicating positive examples to prevent the model from being biased towards the majority class.
  • Splitting: Each balanced dataset is split into 80% for training and 20% for testing.

5.2. Evaluation Metrics

The primary evaluation metric used in this study is Accuracy (%).

• Accuracy:
  1. Conceptual Definition: Accuracy is a measure of how often the model correctly predicts the outcome. In binary classification (like hemolysis or nonfouling prediction), it represents the proportion of true results (both true positives and true negatives) among the total number of cases examined. It assesses the overall correctness of the model's predictions.
  2. Mathematical Formula: For a binary classification task, accuracy is calculated as: $ \mathrm{Accuracy} = \frac{\mathrm{TP} + \mathrm{TN}}{\mathrm{TP} + \mathrm{TN} + \mathrm{FP} + \mathrm{FN}} $
  3. Symbol Explanation:
     • $\mathrm{TP}$: True Positives, the number of correctly predicted positive instances.
     • $\mathrm{TN}$: True Negatives, the number of correctly predicted negative instances.
     • $\mathrm{FP}$: False Positives, the number of incorrectly predicted positive instances (actual negatives predicted as positives).
     • $\mathrm{FN}$: False Negatives, the number of incorrectly predicted negative instances (actual positives predicted as negatives).

5.3. Baselines

The paper compares the Multi-Peptide framework against several baseline models to contextualize its performance:

• Individual Components:

  • Pretrained PeptideBERT: This refers to the PeptideBERT model that has been individually pretrained on peptide sequence data but not subjected to the multimodal contrastive learning framework. It serves as a direct comparison for the language model's performance without structural insights.
  • Pretrained GNN: This refers to the GNN model that has been individually pretrained on the structural graph data, also without the multimodal contrastive learning. It indicates the performance of solely relying on structural information.

• Other State-of-the-Art and Traditional Models:

  • Fine-tuned PeptideBERT: This is a PeptideBERT model that has been fine-tuned directly on the specific peptide property prediction task, without the Multi-Peptide's GNN-assisted contrastive training. This is a strong baseline, as PeptideBERT itself is a powerful sequence-based model.

  • HAPPENN: This is a novel tool (Hemolytic Activity Peptide Predictor Employing Neural Networks) specifically designed for hemolytic activity prediction of therapeutic peptides (Timmons & Hewage, 2020).

  • Embedding + LSTM: A model that uses embeddings (likely sequence-based) combined with a Long Short-Term Memory (LSTM) network, a type of Recurrent Neural Network (RNN) often used for sequence modeling. This represents an older, but still relevant, deep learning approach for sequence data.

  • One-hots + RNN: A model that uses one-hot encoding for amino acids combined with an RNN. This is a more basic representation and model compared to transformer-based approaches or those using embeddings.

    These baselines are representative because they cover different levels of complexity and types of input data (sequence-only, structure-only, and various deep learning architectures), allowing for a comprehensive evaluation of Multi-Peptide's advancements.

6. Results & Analysis

6.1. Core Results Analysis

The Multi-Peptide framework was evaluated on hemolysis and nonfouling datasets, demonstrating its effectiveness, particularly in hemolysis prediction, while revealing challenges for nonfouling.

The following are the results from Table 1 of the original paper:

Analysis of Table 1:

• Hemolysis Dataset: Multi-Peptide's BERT achieved an accuracy of 88.057%. This is a notable improvement over the Pretrained PeptideBERT (85.981%) and the Pretrained GNN (83.24%) when used individually. This result strongly validates the paper's hypothesis that integrating multimodality through contrastive loss can enhance the performance of a transformer model. The synergistic learning from both sequence and structural features leads to a more robust PeptideBERT.

• Nonfouling Dataset: For nonfouling prediction, Multi-Peptide's BERT achieved 83.847%. While this is an improvement over the Pretrained GNN (79.42%), it is lower than the Pretrained PeptideBERT's accuracy of 88.150%. This discrepancy indicates that the multimodal integration, while beneficial for hemolysis, did not yield an improvement for nonfouling and, in fact, slightly degraded the performance compared to a purely sequence-based PeptideBERT.

  The following are the results from Table 2 of the original paper:

  Data set	Model	Accuracy (%)
  Hemolysis	Multi-Peptide's BERT (this study)	88.057
  	Fine-tuned PeptideBERT26	86.051
  	HAPPENN43	85.7
  Nonfouling	Multi-Peptide's BERT (this study)	83.847
  	Fine-tuned PeptideBERT26	88.365
  	Embedding + LSTM26	82.0
  	One-hots + RNN44	76.0

Analysis of Table 2:

• Hemolysis Dataset: Multi-Peptide's BERT achieves the state-of-the-art (SOTA) accuracy of 88.057%, surpassing Fine-tuned PeptideBERT (86.051%) and HAPPENN (85.7%). This clearly demonstrates the advantage of introducing and leveraging an additional mode of data (structural information) through Multi-Peptide for this specific property.
• Nonfouling Dataset: Multi-Peptide's BERT (83.847%) performs better than Embedding + LSTM (82.0%) and One-hots + RNN (76.0%), indicating its superiority over simpler deep learning and traditional methods. However, it still falls short of the Fine-tuned PeptideBERT (88.365%).

Reasons for Nonfouling Discrepancy (as discussed by the authors):

1. GNN's Comparative Performance: As seen in Table 1, the Pretrained GNN for nonfouling (79.42%) performs significantly worse than Pretrained PeptideBERT (88.150%). This suggests that the structural representations captured by the GNN might not be as strongly correlated with the nonfouling property as sequence information is. It implies that predicting nonfouling behavior primarily from structure might be inherently more difficult or less informative than from sequence, which contradicts some general understandings of protein property prediction.
2. Data Quality and Noise: The effectiveness of the GNN heavily relies on the accuracy of the AlphaFold generated protein structures. While AlphaFold is highly accurate, structural data can be noisier than sequence data. Regions with pLDDT scores below 70 (indicating lower confidence) might introduce inaccuracies. Even though the highest confidence PDB file is chosen, residual noise could degrade overall model performance.
3. Challenges in Modality Alignment: Combining sequence-based features from the transformer with structure-based features from the GNN presents inherent challenges in effectively aligning these different feature types. If the features are not well-aligned or complementary for a specific property like nonfouling, the contrastive loss framework might struggle to learn useful representations, potentially leading to suboptimal performance or even degradation compared to a single-modality model.
4. Increased Complexity: Introducing a GNN significantly increases the overall model complexity. This necessitates more extensive and precise data for effective training. If the nonfouling dataset, despite oversampling, does not provide sufficient high-quality structural diversity or clear structural signals, the added complexity might not translate into performance gains.

6.2. Data Presentation (Tables)

(Tables are transcribed in section 6.1)

6.3. Ablation Studies / Parameter Analysis

The paper details the training strategy and provides qualitative insights into the embedding spaces to understand model behavior.

Training Strategy:

• Individual Pretraining: Both PeptideBERT and GNN models were initially trained individually on their respective datasets for 50 epochs.
• Contrastive Learning Phase: The contrastive loss-implemented ensemble (with GNN weights frozen) was trained for 100 epochs. This longer fine-tuning period aimed to ensure sufficient learning time given the model complexity and data dependencies.
• Learning Rate: A higher learning rate of 6.0e-5 was used during contrastive training (compared to the individual pretraining stage of the transformer). This is because contrastive training aims for quick learning of distinguishable features.
• Learning Rate Scheduler: A Learning Rate on Plateau scheduler was employed, reducing the learning rate by a factor of 0.4 for a period of 5 epochs when the validation loss plateaued.
• Optimizer: AdamW optimizer was used.
• Loss Function: Binary cross-entropy loss was employed within the larger contrastive loss framework.
• Batch Size: A batch size of 20 was used for each task.
• Hardware: Training was performed on four NVIDIA GeForce RTX 2080Ti GPUs with 11GiB of memory each.

Embedding Space Analysis for the Nonfouling Data Set (using t-SNE): To qualitatively understand the model's performance, especially the behavior on the nonfouling dataset where Multi-Peptide did not surpass PeptideBERT, t-distributed stochastic neighbor embedding (t-SNE) was used. t-SNE is a dimensionality reduction technique that visualizes high-dimensional data in a 2D or 3D space, preserving local neighborhoods.

The following figure (Figure 4 from the original paper) shows the t-SNE plots:

该图像是四幅示意图，展示了Multi-Peptide模型在不同维度下对肽类属性的分类结果。图a、b和c为2D投影，显示了两个类别（0和1）的分布情况；图d为3D投影，提供了更全面的视角。这些图反映了模型在对肽类属性进行预测时的表现。

• PeptideBERT Embeddings (e.g., Figures a, b, c): The t-SNE plot for PeptideBERT embeddings revealed a large, central cluster of negatively marked sequences. This cluster was surrounded by smaller, more dispersed groups containing both negatively and positively marked sequences.

  • Analysis: This indicates that PeptideBERT successfully captures semantic similarities within each class. However, the overlap between classes, especially the presence of mixed groups, highlights the challenge of achieving clear class separation when relying solely on sequence information.

• GNN Embeddings (not explicitly visualized but described): The paper describes GNN embeddings as showing a more segregated distribution but still with minimal segregation for positives mirroring the negatives. While a significant pattern of negatively marked sequences occupied most of the plot, the overlap between classes indicated that the GNN struggled to achieve clear class separation.

  • Analysis: This aligns with the lower accuracy of the Pretrained GNN for nonfouling. It suggests that while the GNN captures structural nuances, its ability to form distinct, class-specific clusters for nonfouling peptides was limited, implying structural features alone might not be sufficiently discriminative for this property.

• Post-Contrastive Loss Embeddings (e.g., Figure d for 3D): Following the multimodal contrastive learning (graph-assisted pretraining), the t-SNE plots showed a prominent central cluster of negatively marked sequences. Crucially, the mixed patches (containing both positive and negative classes) appeared smaller and more concentrated compared to the PeptideBERT embeddings. The 3D t-SNE plot (Figure d) further showed some separation within the smaller mixed clusters.

  • Analysis: This suggests that multimodal pretraining was effective in contrasting the classes in the reduced-dimensional space. The contrastive loss mechanism helped to refine the embedding space, attempting to improve class discrimination. However, the remaining overlap, especially for the nonfouling dataset's overall lower accuracy compared to Fine-tuned PeptideBERT, implies that even with this refinement, the GNN's initial weaker separation (as noted earlier) prevented the contrastive framework from fully achieving superior class separation for nonfouling. If the GNN had provided a better initial separation of classes, the contrastive framework would likely have amplified this separation even further.

    Overall Interpretation from t-SNE: The t-SNE visualizations provide valuable qualitative insights. PeptideBERT demonstrated good semantic understanding, the GNN struggled with discriminative structural features for nonfouling, and the contrastive framework did attempt to improve class separation by leveraging both. These visualizations underscore the potential of such multimodal approaches, while also highlighting the importance of each modality's individual informativeness for a given prediction task.

7. Conclusion & Reflections

7.1. Conclusion Summary

In this study, the authors introduced Multi-Peptide, a novel approach designed to enhance the accuracy of peptide property prediction by integrating multimodality into machine learning models. The framework synergistically combines a transformer-based language model (PeptideBERT) with a Graph Neural Network (GNN) to capture both sequence-based and structural features of peptides. A key innovation of Multi-Peptide is its use of a Contrastive Language-Image Pretraining (CLIP) variant, which aligns embeddings from these two distinct modalities into a shared latent space, thereby facilitating more robust and informed predictions.

The experimental results demonstrated the significant potential of Multi-Peptide. Specifically, the model achieved state-of-the-art accuracy of 88.057% on the hemolysis dataset, outperforming previous benchmarks, including a standalone fine-tuned PeptideBERT. This success highlights the robustness of the approach in handling complex data structures and effectively extracting meaningful features from both sequence and structural information. While the performance on the nonfouling dataset did not surpass that of the fine-tuned PeptideBERT, the overall study validates the power of integrating diverse data modalities through advanced machine learning techniques. Multi-Peptide represents a substantial step forward in leveraging multimodality for peptide property prediction, offering a deeper understanding of peptide characteristics in bioinformatics.

7.2. Limitations & Future Work

The authors acknowledge specific limitations and suggest future research directions:

• Nonfouling Performance: The primary limitation is the observation that Multi-Peptide's accuracy for the nonfouling dataset did not surpass that of the fine-tuned PeptideBERT. This was attributed to several factors:
  • The GNN's individual accuracy for nonfouling was lower than PeptideBERT's, suggesting structural representations might be less correlated with nonfouling properties than sequence data.
  • Potential noise in AlphaFold2-generated PDB files, especially in regions with lower pLDDT scores, could degrade overall model performance.
  • Challenges in effectively aligning features from different modalities if they are not inherently complementary or well-aligned for a specific property.
  • Increased model complexity due to GNN integration, requiring more extensive and precise data.
• Model Integration Refinement: The study suggests that combining sequence-based features from the transformer with structure-based features from the GNN presents challenges in effectively aligning these different feature types.
• Future Work: The authors propose focusing on:
  • Refining the integration of modalities: This implies exploring more sophisticated methods for fusing or aligning the information from the sequence and graph encoders beyond the current CLIP-inspired approach.
  • Further optimizing the model architecture: This could involve exploring different GNN architectures, PeptideBERT variants, or the design of the projection heads and the contrastive loss function itself, to better harness the complementary strengths of sequence-based and structural features.
  • Potentially investigating the quality of AlphaFold predictions specifically for properties like nonfouling and how to mitigate the impact of less confident structural regions.

7.3. Personal Insights & Critique

This paper presents a compelling argument for the value of multimodal learning in bioinformatics, particularly for complex biological entities like peptides. The Multi-Peptide framework is a well-conceived approach that directly addresses a known limitation of purely sequence-based models – their inability to inherently capture 3D structural information. The adoption of a CLIP-like contrastive learning paradigm is a clever way to align diverse data types without requiring direct concatenation, allowing for knowledge transfer and a more robust final PeptideBERT model that can still perform inference solely on sequence.

Inspirations:

• The Power of Multimodality: The study clearly demonstrates that for certain properties (hemolysis), combining modalities can yield significant performance gains, highlighting that a more complete picture often emerges from integrating different data perspectives. This concept is broadly applicable; many scientific domains rely on heterogeneous data (e.g., textual reports, image diagnostics, time-series sensor data), and multimodal learning offers a powerful paradigm for synthesis.
• Knowledge Transfer through Contrastive Learning: The freezing of the GNN weights during the contrastive learning phase, allowing PeptideBERT's weights to be updated, is an elegant solution. It enables the benefits of structural insights to be imbued into the language model without increasing inference complexity, which is crucial for practical applications. This "teacher-student" like relationship facilitated by CLIP is a valuable design pattern.
• Leveraging Foundational Models: The paper's reliance on ProtBERT, PeptideBERT, and AlphaFold2 underscores the importance of foundational models and pre-trained components in accelerating research. Rather than building everything from scratch, leveraging state-of-the-art tools for specific sub-tasks (sequence encoding, structure prediction) allows researchers to focus on novel integration strategies.

Potential Issues, Unverified Assumptions, or Areas for Improvement:

• The "Black Box" of GNN for Nonfouling: The discrepancy in performance for the nonfouling dataset is a significant point. While the authors speculate about GNN's weaker signal or noisy structural data, a deeper analysis of why structural features might be less discriminative for nonfouling compared to sequence features would be insightful. Is it possible that nonfouling is more governed by surface properties or charge distributions that AlphaFold structures might not perfectly capture, or that the GNN architecture was not optimal for extracting these specific features? Further analysis could involve:

  • Feature Importance Analysis: Identifying which structural features (atom coordinates, atomic radius, etc.) are most influential for nonfouling prediction.
  • Error Analysis: Examining cases where PeptideBERT succeeded for nonfouling but Multi-Peptide failed (or vice-versa) to pinpoint specific types of peptides or structural motifs that are misclassified.

• Generalizability of pLDDT: While selecting the highest pLDDT PDB file is a good heuristic, the assumption that AlphaFold's confidence score directly correlates with the usefulness of the structure for property prediction could be further investigated. Some properties might be robust to local structural inaccuracies, while others might be highly sensitive.

• Computational Cost for Training: While GNN is discarded at inference, the training phase requires significant computational resources (four RTX 2080Ti GPUs) and a longer training schedule (100 epochs for contrastive learning). For even larger datasets or more complex peptides, this could become a bottleneck. Exploring more efficient GNN architectures or contrastive learning strategies could be beneficial.

• Hyperparameter Sensitivity: The paper mentions iterative experimentation for hyperparameter tuning. A more detailed ablation study on the temperature parameter (\tau) and learning rates, especially for the contrastive loss, could provide valuable insights into its sensitivity and optimal ranges for different peptide properties.

  Overall, Multi-Peptide is a valuable contribution, showcasing a robust methodology for leveraging multimodal information in peptide research. Its strengths lie in its innovative integration strategy and demonstrated success in hemolysis prediction, while its limitations point towards exciting avenues for future research in refining multimodal architectures and understanding property-specific data requirements.