CoGANPPIS: A Coevolution-enhanced Global Attention Neural Network for
  Protein-Protein Interaction Site Prediction
Protein-protein interactions are of great importance in biochemical
processes. Accurate prediction of protein-protein interaction sites (PPIs) is
crucial for our understanding of biological mechanism. Although numerous
approaches have been developed recently and achieved gratifying results, there
are still two limitations: (1) Most existing models have excavated a number of
useful input features, but failed to take coevolutionary features into account,
which could provide clues for inter-residue relationships; (2) The
attention-based models only allocate attention weights for neighboring
residues, instead of doing it globally, which may limit the model's prediction
performance since some residues being far away from the target residues might
also matter.
  We propose a coevolution-enhanced global attention neural network, a
sequence-based deep learning model for PPIs prediction, called CoGANPPIS.
Specifically, CoGANPPIS utilizes three layers in parallel for feature
extraction: (1) Local-level representation aggregation layer, which aggregates
the neighboring residues' features as the local feature representation; (2)
Global-level representation learning layer, which employs a novel
coevolution-enhanced global attention mechanism to allocate attention weights
to all residues on the same protein sequences; (3) Coevolutionary information
learning layer, which applies CNN & pooling to coevolutionary information to
obtain the coevolutionary profile representation. Then, the three outputs are
concatenated and passed into several fully connected layers for the final
prediction. Extensive experiments on two benchmark datasets have been
conducted, demonstrating that our proposed model achieves the state-of-the-art
performance.
Jiaxing Guo, Xuening Zhu, Zixin Hu, Xiaoxi Hu
CoGANPPIS: Coevolution-enhanced Global Attention Neural Network for Protein-Protein Interaction Site Prediction

###### Abstract

Protein-protein interactions are of great importance in biochemical processes. Accurate prediction of the protein-protein interaction sites (PPIs) from protein sequences deepens our understanding of biological mechanism and is crucial for new drug design. However, conventional experimental methods for PPIs prediction are costly and time-consuming so that many computational approaches, especially ML-based approaches, have been developed recently. Although these approaches have achieved gratifying results, there are still two limitations: (1) Most existing models have excavated a number of useful input features, but failed to take coevolutionary features into account, which could provide clues for inter-residue relationships and could be helpful for PPIs prediction; (2) The attention-based models only allocate attention weights for neighboring residues, instead of doing it globally, which may limit the model's prediction performance since some residues being far away from the target residues in the protein sequence might also matter.

We propose a coevolution-enhanced global attention neural network, a sequence-based deep learning model for PPIs prediction, called CoGANPPIS. Specifically, CoGANPPIS utilizes three layers in parallel for feature extraction: (1) Local-level representation aggregation layer, which aggregates the neighboring residues' features as the local feature representation similar to previous studies; (2) Global-level representation learning layer, which employs a novel coevolution-enhanced global attention mechanism to allocate attention weights to all the residues on the same protein sequences; (3) Coevolutionary information learning layer, which applies CNN & pooling to coevolutionary information to obtain the coevolutionary profile representation. Then, the three outputs are concatenated and passed into several fully connected layers for the final prediction. Extensive experiments on two benchmark datasets have been conducted, demonstrating that our proposed model achieves the state-of-the-art performance. The source code is publicly available at [https://github.com/Slam1423/CoGANPPIS_source_code](https://github.com/Slam1423/CoGANPPIS_source_code).

## 1 Introduction

Proteins participate in a variety of biological processes in organisms. They rarely act alone, instead, they usually carry out various functions by interacting with different kinds of molecules, such as DNA, lipids, carbohydrates, and other proteins (Branden and Tooze, 2012; Murray et al., 2009; Ardejani et al., 2017). The process of establishing physical contacts of high specificity between two or more protein molecules is known as protein-protein interaction, which plays an important role in many biochemical processes including immune response, muscle contraction, and signal transduction. Considering the high practical and research values of PPIs prediction, many approaches have been proposed so far. There are some conventional experimental
methods, such as two-hybrid screening, relationship purification, and intragenic complementation, being commonly applied to identify PPIs (Westermarck et al., 2013; Terentive et al., 2009; Brettner and Masel, 2012; Smallwood et al., 2002). However, these experimental methods suffer from being costly and time-consuming so that more accurate and efficient computational predictors for PPIs are of great value for biologists.

With the rapid development of computer science, a lot of computational approaches, especially ML-based approaches, have been developed, which take protein sequences or structures as input and be known as sequence-based and structure-based respectively (Hou et al., 2016). Although Structure-based methods have achieved some promising progress in recent years (Gainza et al., 2020; Yuan et al., 2022; Huang et al., 2023), they may cause problems for biological researchers since the number of proteins with available structures is limited. For example, AlphaFold2 has shown promising performance in protein structure prediction, but its effectiveness on some newly-discovered proteins and some exotic proteins still remains to be tested. At the same time, its requirement for computational resources could be too high for most researchers (Jumper et al., 2021). In contrast, sequence-based methods are more practical since protein sequences are easier to be obtained with the noticeable development of high-throughput techniques.

Sequence-based methods could be classified as partner-specific and non partner-specific sequence-based PPIs prediction (Casadio et al., 2022), and in this paper we focus on the later one. The partner-specific sequence-based PPIs prediction aims to identify the interaction residue pairs of two given proteins, which has not been covered in our present work.

Sequence-based methods can be further classified into 2 categories: traditional machine learning approaches and deep learning approaches. The commonly-used traditional machine learning approaches include SVM (Yan et al., 2004; Wang et al., 2006; Chen and Li, 2010; Chen et al., 2012; Porollo and Meller, 2007), Naive Bayes (Yan et al., 2004; Murakami and Mizuguchi, 2010), shallow neural network (Ofran and Rost, 2003, 2007), random forest (Chen and Jeong, 2009; Northey et al., 2018; Wang et al., 2019), and logistic regression (Dhole et al., 2014; Zhang and Kurgan, 2019). However, these methods cannot capture the relationships among different residues located on the same protein sequences since they treat every residue as an independent sample.

In recent years, due to the great success of deep learning in many fields such as computer vision, speech recognition and natural language processing, the models based on deep learning have also been used in PPIs prediction. Among these, DELPHI, a fine-tuned ensemble model combining recurrent neural networks and convolutional neural networks, showed significant improvement in PPIs prediction compared with traditional machine learning models (Li et al., 2021). DeepPPISP, a convolutional neural network-based model, achieved good outcomes by combining both local and global sequence contexts as input and processing them respectively (Zeng et al., 2020). On the basis of DeepPPISP, ACNN, an attention-based convolutional neural network, made a better understanding of the local environment of the target residues by giving different attention weights to the neighboring residues (Lu et al., 2021). HANPPIS improved the performance and interpretability by using a double-layer attention mechanism (Tang et al., 2021). Besides, ensnet_p, an ensemble model combining several neural net architectures, achieved stable and peak prediction accuracy (Stringer et al., 2022).

Conventional sequence-based input features can be roughly classified into 4 categories: raw sequence features, evolutionary information, residue physiochemical properties, and predicted structural features (Casadio et al., 2022). Raw sequence features refer to the features that can be straightly obtained by the protein sequences, the most commonly used of which are amino acid types. Evolutionary information usually refers to the position-specific scoring matrices (PSSM) as well as other conservative scores of the proteins, which are mainly calculated from multiple sequence alignments (MSA) and very informative for protein-related prediction tasks. Residue physiochemical properties (such as residue's charge and polarity) have been applied in many models in recent years, which can be obtained from databases or some specific predictors. Besides, in the absence of protein structure, the predicted structural features (such as hydrophobicity, secondary structure, disorder) can also be helpful.

Recently, another sequence-based feature, coevolutionary information based feature, has been applied to another important protein-related problem, protein contact-map prediction, and brings about significant performance improvement (Wang et al., 2017; Hanson et al., 2018; Li et al., 2021). These features are mainly obtained by direct coupling analysis (DCA) and could quantify the inter-residue coevolutionary relationships. Intuitively, for a residue, its properties and behaviours should be more similar to the residues closely related
to it, which inspires us to introduce it into our model.

To evaluate the performance of our model, we compare it with other seven sequence-based models (PSIVER, ISIS, SPRINGS, DELPHI, DeepPPISP, ACNN and ensnet_p) on two benchmark datasets Dset422 and Dset448. The experimental results show that our model achieves state-of-the-art performance for PPIs prediction. The main contributions of this paper are as follows:

(1) To the best of our knowledge, this is the first time to introduce coevolutionary information as input features into deep learning based PPIs prediction model and we verify its usefulness in PPIs prediction by ablation analysis.

(2) We propose a novel coevolution-enhanced global attention mechanism for global-level representation learning, which allocates attention weights based on a better understanding of the whole protein sequences and the coevolutionary relationships among the residues.

(3) We conduct extensive experiments on two benchmark datasets, the results of which demonstrate that CoGANPPIS achieves state-of-the-art performance.

The rest of this paper is organized as follows: Section 2 introduces the datasets and the input features. Section 3 presents the architecture of our model, the experimental results and experiment analysis. In the end, section 4 summarizes the discussion of this paper.

## 2 Materials and Methods

### Datasets

In this study, two benchmark datasets, Dset422 and Dset448, are used in experiments. Dset422 consists of Dset72 (Murakami and Mizuguchi, 2010), Dset186, and Dset164 (Singh et al., 2014), whose protein sequences were collected from Protein Data Bank (Sussman et al., 1999). The protein sequence homology is less than 25% and if an amino acid has absolute solvent proximity less than 1 A\({}^{2}\) before and after binding with other proteins, it will be defined as an interaction site (Zeng et al., 2020). Dset448 is sourced from the BioLip database, where residues are defined as interaction sites if the distance between an atom of this residue and an atom of a given protein-partner less than 0.5 A plus the sum of the Van der Waal's radii of the two atoms (Zhang and Kurgan, 2019). First, the protein sequences were mapped into Uniprot databases to collect binding residues across different complexes. Then, they were clustered by Blastclust at 25% similarity, after which one protein was selected from each cluster to ensure that proteins in Dset448 share similarities less than 25%. Besides, a dataset of 4392 protein sequences was constructed in the paper of PIPENN (Stringer et al., 2022). Here, we name it Dset4392. The data of Dset4392 is from the BioLip Database, where binding sites are defined similar to Dset448. We utilize it for pretraining.

For Dset422 and Dset448, we randomly divided them into training set (about 83% of randomly selected proteins), validation set (about 5% of randomly selected proteins), and test set (the remaining proteins) respectively. Consequently, for Dset422, there are 352 proteins in the training set, 21 proteins in the validation set, and 49 proteins in the test set. And for Dset448, there are 373 proteins in the training set, 22 proteins in the validation set, and 53 proteins in the test set.

We count the distribution of sequence lengths of the three datasets. As shown in Table 1, only a small proportion of protein sequences in the two datasets are longer than 500. Hence, for the convenience of model training, we preprocess the protein sequences by unifying their lengths to 500, that is, if a protein sequence is longer than 500, we truncate it to 500; if a protein sequence is shorter than 500, we pad their length to 500 with zeros.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Length range & 1-100 & 101-200 & 201-300 & 301-400 & 400-500 & 500+ & total \\ \hline Dset422 & 85(20.1\%) & 176(41.7\%) & 68(16.1\%) & 56(13.3\%) & 23(5.5\%) & 14(3.3\%) & 422(100.0\%) \\ Dset448 & 34(7.6\%) & 127(28.3\%) & 134(29.9\%) & 95(21.2\%) & 38(8.5\%) & 20(4.5\%) & 448(100.0\%) \\ Dset4392 & 345(7.9\%) & 1266(28.8\%) & 969(22.1\%) & 716(16.3\%) & 511(11.6\%) & 585(13.3\%) & 4392(100.0\%) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of protein sequence lengths
### Input Features

The features commonly used in previous research, including raw sequence features, the position-specific scoring matrix, and predicted secondary structures, are applied in our model. Besides, we also introduce coevolutionary information into our model, which, to the best of our knowledge, is the first time to be used as input features in deep learning based PPIs prediction. These features are described in detail as follows.

#### 2.2.1 Raw sequence features

In this study, we include two raw features, amino acid type and sequence length into our model. Most proteins consist of 20 different amino acids. Hence, we encode each amino acid residue as a 20D one-hot vector representing the amino acid type at this position. Besides, we utilize an integer to represent the length of the sequence for each residue.

#### 2.2.2 Position-specific scoring matrix

Position-specific scoring matrix (PSSM) contains evolutionary information, which has been shown effective for PPIs prediction (cheol Jeong et al., 2010). We perform the PSI-BLAST algorithm on each input sequence against NCBI's non-redundant sequence database with three iterations and an E-value threshold of 0.001 to obtain its PSSM, where every amino acid residue on the underlying sequence is encoded as a vector with 20 elements.

#### 2.2.3 Predicted secondary structures

NetSurfP-3.0 is a tool for predicting secondary structures from protein sequences (Hoe et al., 2022). Here we utilize it to predict relative solvent accessibility (RSA), access surface area (ASA), 3-state secondary structure as well as 8-state secondary structure for each residue. Each amino acid residue is encoded as a vector with 13 elements, which represents the predicted RSA, predicted ASA, and the probabilities of being the corresponding secondary structure states at the position.

#### 2.2.4 Coevolutionary information

Coevolutionary relationships between amino acid residues refers to the interdependent changes that occur in pairs of residues on the same protein sequences, which help maintain proteins' stability, function, and folding (De Juan et al., 2013). As mentioned earlier, coevolutionary information based input features brought about great performance improvement in protein-contact map prediction, inspiring us to apply it in this study.

Direct-coupling analysis (DCA) is one of the main computational approaches to capture proteins' coevolutionary information. The key idea of DCA is to disentangle direct pairwise couplings of each two amino acid residues on the same protein sequences. For each protein, DCA takes its multiple sequence alignments (MSA) as the input, which is obtained by BLASTP, and returns a \(N\times N\) matrix, where \(N\) refers to the length of this protein sequence. The \((i,j)\) element of this matrix refers to the direct coupling degree of the \(i\)th residue and the \(j\)th residue on this protein sequence. The larger this value is, the higher the coevolutionary relationship exists between these two residues. In our study of predicting whether an amino acid residue is an interaction site or not, the corresponding column of the target amino acid residue in the DCA matrix has been extracted as its coevolutionary information feature.

There are three usual DCA algorithms, mpDCA (Weigt et al., 2009), mfDCA (Morcos et al., 2011), and plmDCA (Ekeberg et al., 2013). The mpDCA uses a semi-heuristic message-passing approach, and its slow computation speed makes it difficult to be applied to a large-scale dataset. The mfDCA uses a mean-field approach based on the maximum-entropy model, which greatly improves the computational speed. On the basis of mfDCA, the plmDCA applies pseudo-likelihood to the Potts model and achieves higher accuracy than mfDCA. Based on the above comparison, we utilize plmDCA to generate DCA matrices in this study.

### Model Architecture

Figure 1 is an overview of the proposed framework. First, we extract the local features, global features, and coevolutionary features of the primary protein sequence with different components respectively, which, in
this paper, are referred to as local-level representation aggregation layer, global-level representation learning layer, and coevolutionary information learning layer. Each layer outputs a feature representation vector. Then we concatenate three feature representation vectors as the output of the feature extraction and pass it into the prediction layer consisting of four fully connected layers for the final prediction about whether the target amino acid residue is an interaction site or not. Now we introduce the three layers in feature extraction in detail.

#### 2.3.1 Local-level representation aggregation layer

For each target residue, a sliding window of length (\(2n+1\)) is used to aggregate the features of itself and its neighboring \(2n\) residues. For the \(i\)th residue on the protein sequence, we denote its local feature representation as \(h_{i}^{local}\).

#### 2.3.2 Global-level representation learning layer

It has been shown that the global features of protein sequences are critical for PPIs prediction (Zeng et al., 2020). Also, the coevolutionary information could quantify the inter-residue coevolutionary relationships (Wang et al., 2017; Hanson et al., 2018; Li et al., 2021). Hence, we consider to utilize a coevolution-enhanced global attention mechanism to distinguish the importance of residues. Assuming that the \(i\)th residue is the predicting target, all residues on the same sequence are linearly added according to the attention scores,

\[h_{i}^{p}=\sum_{j=1}^{N}\alpha_{ij}h_{j}, \tag{1}\]

where \(h_{j}\) refers to the PSSM, predicted secondary structure, and raw protein sequence features of residue \(j\). \(\alpha_{ij}\) refers to the attention score, which estimates the importance weight of residue \(j\) on target residue \(i\). Intuitively, different residues on the same protein sequence should have different importance for the target residue, and those which match the characteristics of the whole protein and share a close relationship with the target residue should be paid more attention to. Therefore, \(\alpha_{ij}\) is calculated as follows:

\[\alpha_{ij}=\text{softmax}(\pi(i,j))=\frac{\exp{(\pi(i,j))}}{\sum_{k=1}^{N} \exp(\pi(i,k))}, \tag{2}\]

\[\pi(i,j)=q_{1}^{\text{T}}\text{LeakyRelu}([W_{1}(p\odot h_{j})\|W_{2}(h_{i} \odot h_{j})]\|w_{ij}). \tag{3}\]

Figure 1: The model architecture of CoGANPPIS. (A) An overall illustration of the proposed model. The feature extraction consists of three layers: local-level representation aggregation layer, global-level representation learning layer and coevolutionary information learning layer, whose outputs are passed into the prediction layer for the final prediction. (B) The structure of CNN & pooling layer in coevolutionary information learning layer. We use batchnormalization, multiple convolution kernels, activation as well as maxpooling to capture the coevolutionary information from DCA. (C) The structure of the prediction layer. We concatenate the outputs from feature extraction and utilize four fully-connected layers to predict whether the residue is an interaction site or not.
Here, we use LeakyRelu as the activation function. \(\odot\) indicates element-wise product, \(\|\) indicates concatenation operation, and \(w_{ij}\in\mathbb{R}^{1}\) refers to the \((i,j)\) element of DCA matrix of the current protein, which provides us some clues for the coevolutionary relationship between the residue \(i\) and residue \(j\). \(W_{1}\in\mathbb{R}^{d\times d}\), \(W_{2}\in\mathbb{R}^{d\times d}\) and \(q_{1}\in\mathbb{R}^{2d+1}\) are trainable parameters. \(p\) can be seen as the feature representation of the whole protein sequence, which is obtained by mean-pooling on all the residues' features on this protein,

\[p=\frac{1}{N}\sum_{j=1}^{N}h_{j}. \tag{4}\]

Our approach makes the attention weights between the target residue and other residues dependent on not only the whole protein sequence feature representation but also the coevolutionary relationships between them, suggesting that those residues which match the characteristics of the whole protein and are closely related to the target residue will be attached more importance.

Then we concatenate the global representation of the target residue \(h_{i}^{p}\) and its original feature \(h_{i}\),

\[h_{i}^{global}=h_{i}^{p}\|h_{i}, \tag{5}\]

where \(h_{i}^{global}\) is the result of the global-level representation learning layer.

#### 2.3.3 Coevolutionary information learning layer

We have introduced coevolutionary information into the attention mechanism to exploit the relationship among residues as above. Now we further utilize the coevolutionary information on a larger scale, i.e, on the whole protein sequence level. Suppose we are predicting the \(i\)th residue on the protein sequence. First, we take its corresponding column in the DCA matrix as its coevolutionary information. Then we pass it into the CNN & pooling layer as shown in Figure 1:

\[h_{i,k}^{dca}=\text{Relu}(\text{conv1d}^{(k)}(\text{BN}(\text{ DCA}[:,i]))),k\in[1,K], \tag{6}\]

\[h_{i}^{dca}=\|_{k=1}^{K}h_{i,k}^{dca}, \tag{7}\]

where \(\text{DCA}[:,i]\) is the \(i\)th column of the DCA matrix of the underlying protein. BN refers to the Batch-Normalization operation and the 1D convolution operation conv1d extracts the normalized coevolutionary features. We use Relu as the activation. Here we use \(K\) different convolution kernels for a better extraction of coevolutionary features. Finally, we obtain the coevolutionary representation \(h_{i}^{dca}\) by concatenating all \(K\) results linearly.

#### 2.3.4 Prediction layer

To predict whether an amino acid residue is an interaction site or not, first we concatenate three former feature extraction results to obtain the final representation:

\[h_{i}^{pred}=h_{i}^{local}\|h_{i}^{global}\|h_{i}^{dca}, \tag{8}\]

where \(h_{i}^{local}\), \(h_{i}^{global}\) and \(h_{i}^{dca}\) are the results of local-level representation aggregation layer, global-level representation learning layer, and coevolutionary information learning layer. \(h_{i}^{pred}\) is the final representation of the residue, which will be passed into fully connected layers:

\[x^{(t)}=\text{Relu}(W^{(t)}x^{(t-1)}+b^{(t)}),t\in[1,T], \tag{9}\]

where \(x^{(t)}\) and \(x^{(t-1)}\) refer to the input vector and output vector of the \(t\)th fully connected layer, respectively. Here, \(x^{(0)}=h_{i}^{pred}\). \(W^{(t)}\) denotes the weight matrix and \(b^{(t)}\) denotes the bias. Besides, ReLU and dropout are utilized in each layer except the last one. After the last layer, a Sigmoid function is used to generate the final prediction:

\[\hat{y}=\frac{1}{1+e^{-x^{(T)}}}, \tag{10}\]
where \(\hat{y}\) denotes the predicted probability of the residue being the interaction site. And \(1-\hat{y}\) is the predicted probability of the residue being the non-interaction site.

### Evaluation Metrics

PPIs prediction can be seen as a binary classification problem for identifying whether an amino acid residue is an interaction site or not. Consequently, there could be four types of results based on the residue's true category and predicted category, i.e., true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Here, TP and TN refer to the correctly predicted interaction sites and non-interaction sites respectively; FP and FN refer to the incorrectly predicted interaction sites and non-interaction sites respectively.

We select six evaluation metrics to comprehensively evaluate the predictive performance, including area under the precision-recall curve (AUPRC), accuracy (ACC), recall, precision, F-measure (F\({}_{1}\)), and Matthews correlation coefficient (MCC). Considering that our dataset is imbalanced with more non-interaction sites than interaction sites, F\({}_{1}\) and MCC indices deserve more attention. The formulas for calculating these metrics are as follows:

\[\text{ACC}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{ FN}}, \tag{11}\]

\[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}, \tag{12}\]

\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}, \tag{13}\]

\[\text{F}_{1}=\frac{2\times\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}}, \tag{14}\]

\[\text{MCC}=\frac{\text{TP}\times\text{TN}-\text{FP}\times\text{FN}}{\sqrt{( \text{TP}+\text{FP})(\text{TP}+\text{FN})(\text{TN}+\text{FP})(\text{TN}+ \text{FN})}}. \tag{15}\]

## 3 Results and Discussion

### Implementation

In the feature extraction part, the sliding window length in the local-level representation aggregation layer is set as 7. We use three convolution kernels in the CNN & pooling layer and the sizes are set as 3, 5 and 7, respectively. In the classification part, we conducted four fully connected layers of 1024, 256, 8, and 1 node, with the first three fully connected layers accompanied by a dropout ratio of 0.1. We use weighted cross-entropy loss as the loss function:

\[L=-\frac{1}{m}\sum_{i=1}^{m}(wy_{i}log(\hat{y_{i}})+(1-y_{i})log(1-\hat{y_{i}} )), \tag{16}\]

where \(m\) is the number of training samples. \(w\) refers to the weight and is set to 4. Interaction site is labeled as 1 (\(y_{i}=1\)) and non-interaction site is labeled as 0 (\(y_{i}=0\)). \(\hat{y}_{i}\) is the predicted probability of being interaction site of the sample \(i\). Besides, we utilize Adaptive Momentum (Adam) as the optimizer with a learning rate of 0.0001. The batch size is set to 256. The model is implemented by PyTorch and trained on NVIDIA GTX 1080 Ti.

Fine tuning is used in this model. Before training on Dset422 and Dset448, we first trained our model on Dset4392. In the epoch of achieving the best performance on Dset4392, the parameters of the model are saved to files. When training on Dset422 and Dset448, we loaded the saved weights into the feature extraction part from the file and froze the weights so that during the process of training, the parameters in feature extraction stayed unchanged. Training and validation data are used only to train the fully connected layers in the prediction layer.
### Comparison with competing methods

To evaluate the predictive performance of our model (CoGANPPIS), we compare it with seven popular sequence-based competing methods (PSIVER, ISIS, SPRINGS, DELPHI, DeepPPISP, ACNN and ensnet_p). Specifically, PSIVER (Murakami and Mizuguchi, 2010) utilizes Naive Bayes Classifier and kernel density estimation method to predict PPIs based on sequence features. ISIS (Ofran and Rost, 2007) combines predicted structural features with evolutionary information to predict PPIs based on shallow neural networks. SPRINGS utilizes an artificial neural network to generate PPIs predictions (Singh et al., 2014). DELPHI (Li et al., 2021) employs a fine-tuned ensemble model by combining several recurrent neural networks and convolutional neural networks. DeepPPISP (Zeng et al., 2020) considers both local contextual and global information and applies a convolutional neural network to predict PPIs. ACNN (Lu et al., 2021) employs a local attention mechanism to make PPIs prediction. And ensnet_p is an ensemble model combining different neural net models (Stringer et al., 2022). In words, all these traditional approaches do not apply coevolutionary information or global attention mechanism.

Table 2 presents the experimental results of seven sequence-based competitive PPIs prediction models and our proposed model. It can be observed that CoGANPPIS achieves the best performance across both two datasets in terms of all six metrics consistently, which ascertains its effectiveness. The ROC and PR curves of CoGANPPIS and other competing methods on Dset422 and Dset448 are shown in Figure 2. It demonstrates that AUPRC and AUC of CoGANPPIS are higher than that of other competing methods.

### Ablation analysis

In this part, we test the usefulness of introducing coevolutionary information into PPIs prediction. First, we evaluate the performance of the model with coevolutionary information included. Then we remove coevolutionary information from the model and train the model again to measure the comparing performance. The model without coevolutionary information is named as CoGANPPIS\({}^{\odot}\). Table 3 demonstrates the performance of CoGANPPIS and CoGANPPIS\({}^{\odot}\). We can find that CoGANPPIS outperforms CoGANPPIS\({}^{\odot}\) on

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline Dataset & \multicolumn{6}{c}{Dset422} & \multicolumn{6}{c}{Dset448} \\ \cline{2-13} Method & AUPRC & ACC & Recall & Precision & F\({}_{1}\) & MCC & AUPRC & ACC & Recall & Precision & F\({}_{1}\) & MCC \\ \hline PSIVER & 0.230 & 0.553 & 0.426 & 0.204 & 0.276 & 0.086 & 0.238 & 0.631 & 0.337 & 0.225 & 0.270 & 0.131 \\ ISIS & 0.284 & 0.671 & 0.532 & 0.249 & 0.339 & 0.198 & 0.316 & 0.624 & 0.514 & 0.293 & 0.373 & 0.255 \\ SPRINGS & 0.279 & 0.547 & 0.772 & 0.189 & 0.303 & 0.120 & 0.305 & 0.601 & 0.546 & 0.190 & 0.282 & 0.134 \\ DELPHI & 0.311 & 0.628 & 0.533 & 0.288 & 0.371 & 0.232 & 0.320 & 0.633 & 0.588 & 0.280 & 0.379 & 0.234 \\ DeepPPISP & 0.320 & 0.655 & 0.577 & 0.303 & 0.397 & 0.206 & 0.351 & 0.661 & 0.588 & 0.307 & 0.404 & 0.276 \\ ACNN & 0.306 & 0.689 & 0.598 & 0.254 & 0.356 & 0.224 & 0.301 & 0.693 & 0.603 & 0.259 & 0.362 & 0.232 \\ ensnet_p & 0.377 & 0.696 & 0.600 & 0.291 & 0.392 & 0.246 & 0.385 & 0.745 & 0.580 & 0.302 & 0.397 & 0.259 \\ CoGANPPIS (not pretrained) & 0.364 & **0.753** & 0.431 & **0.382** & 0.405 & 0.251 & 0.355 & 0.674 & 0.551 & **0.324** & 0.408 & 0.252 \\ CoGANPPIS & **0.392** & 0.702 & **0.625** & 0.323 & **0.425** & **0.304** & **0.393** & **0.746** & **0.630** & **0.324** & **0.428** & **0.307** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison

Figure 2: ROC and PR plots of CoGANPPIS and other seven competing models. (a) ROC plot on Dset422, (b) PR plot on Dset422, (c) ROC plot on Dset448, (d) PR plot on Dset448. CoGANPPIS clearly outperforms other models
both two datasets, which indicates that introducing coevolutionary information could help improve predictive accuracy and thus validates its effectiveness in PPIs prediction.

### Model performance on proteins of different lengths

Considering that protein sequence length varies greatly from each other, it could be necessary to study the predictive performance on proteins of different length. To answer this question, we plot the experimental results of CoGANPPIS, CoGANPPIS\({}^{\ominus}\) as well as ensnet_p in terms of \(F_{1}\) and MCC under proteins of different lengths on the two datasets in Figure 3.

Through the results, we have the following observations: First, we can observe that with the protein sequence length increasing, the performance of three models on two datasets all show an overall downward trend. This can be explained as the protein structure and function become more complex with the increase of length, making PPIs more difficult to be predicted. Second, it is interesting that the performance improvement of CoGANPPIS and CoGANPPIS\({}^{\ominus}\) compared with ensnet_p increases as the increase of the protein sequence length. Take the \(F_{1}\) on Dset422 in Figure 3 as an example, when the length is less than 100, the \(F_{1}\) of

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline Dataset & \multicolumn{8}{c}{Dset422} & \multicolumn{8}{c}{Dset448} \\ \cline{2-13} Method & AUPRC & ACC & Recall & Precision & F\({}_{1}\) & MCC & AUPRC & ACC & Recall & Precision & F\({}_{1}\) & MCC \\ \hline CoGANPPIS\({}^{\ominus}\) & 0.383 & 0.647 & 0.613 & 0.316 & 0.417 & 0.293 & 0.388 & 0.670 & **0.637** & 0.316 & 0.422 & 0.300 \\ CoGANPPIS & **0.392** & **0.702** & **0.625** & **0.323** & **0.425** & **0.304** & **0.393** & **0.746** & 0.630 & **0.324** & **0.428** & **0.307** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation analysis

Figure 3: Models’ performance under various protein lengths. (a) \(F_{1}\) on Dset422, (b) \(F_{1}\) on Dset448, (c) MCC on Dset422, (d) MCC on Dset448
CoGANNPPIS and CoGANNPPIS\({}^{\odot}\) are 0.450 and 0.443 respectively, which are only 0.023 and 0.016 higher than that of ensnet_p (0.427). When the protein length is between 200 and 300, the improvements increase to 0.029 and 0.019 (0.414 vs 0.385 and 0.404 vs 0.385). When the protein length is greater than 500, the gaps further increase to 0.031 and 0.02 (0.403 vs 0.372 and 0.392 vs 0.372). This clearly shows that the longer the protein sequence is, the more PPIs prediction relies on global information extraction, which can be better captured by our global attention mechanism (even without coevolutionary information). Third, we pay attention to the comparison between CoGANNPPIS and CoGANNPPIS\({}^{\odot}\). The two metrics \(F_{1}\) and MCC of CoGANNPPIS on both two datasets are better than the ones of CoGANNPPIS\({}^{\odot}\), which also verifies the effectiveness of coevolutionary information in PPIs prediction and confirms the conclusion that we obtained in ablation analysis.

### Impact of coevolution-enhanced global attention mechanism

We have verified the effectiveness of coevolutionary information and global attention mechanism. Now let's further study how the coevolution-enhanced global attention mechanism works. First, for each pair of residues, we examine the relationships between its direct coupling degree and labels. As shown in Figure 4(a), the larger the direct coupling degree, the higher the probability that the pairs of residues have the same label. Further, let's take protein Q2T3W4 in Dset448 as an example. We first extract the attention weights of the first residue on Q2T3W4 in the training process. Then we plot a scatter diagram and fit a trend line of the points with attention weights larger than 0.001 as shown in Figure 4(b). We find that the slope of the trend line is positive, which implies that in general, the larger the direct coupling degrees, the higher the attention weights. Hence, the target residue could be paid more attention to the residues with high correlation during training process, which is an noticeable advantage of this attention mechanism.

### Visualization

As mentioned above, the attention weights between two residues show a positive correlation with their direct coupling degrees. It prompts us to explore to which extent our attention mechanism enhances compared with allocating weights purely according to DCA. For a protein sequence with \(N\) amino acid residues, we sort residue pairs according to their attention weights and direct coupling degrees respectively, and keep the \(5N\), \(10N\) as well as \(20N\) highest-ranking pairs. Figure 5 shows an example of the protein 7cei_A from Dset422. The three rows refer to the \(5N\), \(10N\) and \(20N\) situations in turn (Only the corresponding number of selected residue pairs are colored). The pairs with same labels are indicated in red and the pairs with different labels are

Figure 4: Analysis of coevolution-enhanced global attention mechanism. (a) We plot the probabilities of residue pairs having the same label (orange) and the probabilities of having different labels (green) under different direct coupling degrees in our dataset, (b) We take protein Q2T3W4 as an example and plot the scatter diagram of its attention weights under different direct coupling degrees. We also paint the trend line of the scattered points with attention weights larger than 0.001, which shows the positive correlation between the two variables
shown in green. It is evident that our attention mechanism works consistently better than pure DCA because of the obviously higher proportion of red points in all three situations. To become more quantitative, we have binned the predicted pairs according to their separation along the protein sequence as shown in the third column in Figure 5. We observe that our attention mechanism captures the residues with same labels more accurately than pure DCA. Also, our attention mechanism could attach more attention weights to distant residues, whereas pure DCA tends to pay more attention to neighboring aggregated residues, which could be attributed to the consideration of the whole protein feature representation of our attention mechanism.

Figure 5: Visualization of attention weights of protein 7cei_A using our attention mechanism (a, d and g) and pure DCA (b, e and h). Resistive pairs are sorted according to the attention weights and the direct coupling degrees respectively, and kept the SN (a-c), 10N (d-f) as well as 20N (g-i) highest-ranking pairs. These pairs with same labels are indicated in red and the pairs with different labels are shown in green. The right-most panels (c, f and i) bin weights of residue pairs according to their separation along the protein sequence. The overall bars count all predictions, the shaded parts refer to the TPs. Note that our attention mechanism captures the residues with same labels more accurately than pure DCA. Besides, the residues paid attention to by our attention mechanism are more evenly distributed than pure DCA
Conclusion

The aim of this paper is to improve the PPIs prediction performance solely based on protein sequences, which is important for understanding the biological mechanism of proteins both experimentally and theoretically. A dozen of sequence-based PPIs predictors have been developed in recent years. However, most of these works just utilize some commonly used features without considering coevolutionary information which provides rich clues for inter-residue relationships. Also, they are not good at predicting PPIs of long-length proteins.

Here, we propose a coevolution-enhanced global attention neural network (CoGANPPIS). Specifically, we employ a coevolution-enhanced global attention mechanism both for better inter-residue relationship capture and for better prediction of long-length proteins. We further aggregate the local residue features and apply a CNN & pooling layer to the coevolutionary information features as a supplement. Then we utilize several fully connected layers to generate the final prediction. Extensive experiments of CoGANPPIS and other seven popular methods on two standardized datasets show that our proposed model CoGANPPIS achieves the state-of-the-art performance.

Further experimental analysis shows that: (1) Coevolutionary information can improve the performance of PPIs prediction. (2) CoGANPPIS can bring more performance improvement compared with previous methods as the protein sequence becomes longer, implying that CoGANPPIS has a better understanding of the whole protein sequences. (3) Compared with allocating attention weights according to pure DCA, the proposed coevolution-enhanced global attention mechanism pays more attention to the residues with same labels and shows a more evenly distributed attention weights instead of locally aggregated attention weights.

Although CoGANPPIS shows advantages over previous methods, it has some limitations: First, CoGANPPIS takes a lot of computation time due to its usage of multiple sequence alignments and direct coupling analysis to generate coevolutionary information. In addition, DCA's accuracy depends on the number of homologs, making it difficult for those proteins with little homologs. In the future, we would be commited to find useful, practical and time-saving features to make prediction faster and more accurate.

## Funding

This study was supported by funding from the National Natural Science Foundation of China (72222009, 71991472 to X.Z), the National Natural Science Foundation of China (3210040426 to Z.H.), the Shanghai Rising-Star Program (21QB1400900 to Z.H.), and was also partly supported by a grant from the major project of Study on Pathogenesis and Epidemic Prevention Technology System (2021YFC2302500) by the Ministry of Science and Technology of China.
