Before discussing the results, it is necessary to state some of the abbreviations given in the tables and figures as follows:
-
All-onesSim: The value obtained for the matrix where every element is equal to one.
-
MeanRandoms: The average value obtained for random matrices.
-
BestRandom: The best value obtained for the random matrices.
-
WorstRandom: The worst value obtained for the random matrices.
-
CosinePF: The value obtained for the matrix calculated by cosine similarity for the PubChem fingerprint.
-
DicePF: The value obtained for the matrix calculated by Dice similarity for the PubChem fingerprint.
-
TanimotoPF: The value obtained for the matrix calculated by Tanimoto similarity for the PubChem fingerprint.
The evaluation results on Enzyme, GPCR, Ion Channel and Nuclear Receptors datasets are shown in Tables 2, 3, 4 and 5, respectively. It is worth noting that the best parameters for each algorithm are obtained in [3], and we have used these parameters here as well.
Table 2
Comparing different drug-drug similarities on Enzyme dataset
|
|
Method
|
All-onesSim
|
MeanRandoms
|
BestRandom
|
WorstRandom
|
CosinePF
|
DicePF
|
TanimotoPF
|
SIMCOMP
|
AUC
|
NRLMF
|
0.971239
|
0.968016
|
0.969477
|
0.966555
|
0.976691
|
0.97665
|
0.975809
|
0.97632
|
BLM-NII
|
0.977584
|
0.7542
|
0.80648
|
0.718349
|
0.977368
|
0.977764
|
0.976215
|
0.969431
|
NetLapRLS
|
0.959789
|
0.96367
|
0.964636
|
0.962813
|
0.966335
|
0.966613
|
0.968903
|
0.972169
|
WNN-GIP
|
0.938578
|
0.515036
|
0.524567
|
0.507309
|
0.914265
|
0.897733
|
0.875283
|
0.964062
|
AUPR
|
NRLMF
|
0.84053
|
0.841717
|
0.845043
|
0.839261
|
0.870242
|
0.870329
|
0.870117
|
0.875611
|
BLM-NII
|
0.592729
|
0.023396
|
0.034385
|
0.019237
|
0.605514
|
0.60798
|
0.535238
|
0.703746
|
NetLapRLS
|
0.784019
|
0.787323
|
0.787526
|
0.787082
|
0.789326
|
0.789748
|
0.791864
|
0.794216
|
WNN-GIP
|
0.476497
|
0.011065
|
0.01174
|
0.010693
|
0.256565
|
0.281493
|
0.243454
|
0.69719
|
Table 3
Comparing different drug-drug similarities on GPCR dataset
|
|
Method
|
All-onesSim
|
MeanRandoms
|
BestRandom
|
WorstRandom
|
CosinePF
|
DicePF
|
TanimotoPF
|
SIMCOMP
|
AUC
|
NRLMF
|
0.932221
|
0.922694
|
0.929836
|
0.917277
|
0.956879
|
0.957188
|
0.95682
|
0.960355
|
BLM-NII
|
0.94386
|
0.671366
|
0.692179
|
0.647643
|
0.934594
|
0.928518
|
0.879454
|
0.943664
|
NetLapRLS
|
0.902196
|
0.90289
|
0.905388
|
0.896996
|
0.910593
|
0.910846
|
0.91363
|
0.914909
|
WNN-GIP
|
0.872255
|
0.528443
|
0.540304
|
0.517898
|
0.804141
|
0.787323
|
0.901193
|
0.933079
|
AUPR
|
NRLMF
|
0.570196
|
0.62361
|
0.642159
|
0.60265
|
0.69302
|
0.689631
|
0.688301
|
0.702622
|
BLM-NII
|
0.373081
|
0.054418
|
0.062578
|
0.046693
|
0.342311
|
0.33531
|
0.324491
|
0.514827
|
NetLapRLS
|
0.606391
|
0.611795
|
0.612422
|
0.611115
|
0.613065
|
0.613264
|
0.615776
|
0.615446
|
WNN-GIP
|
0.278136
|
0.033394
|
0.035729
|
0.031326
|
0.2326
|
0.230504
|
0.428247
|
0.466361
|
Table 4
Comparing different drug-drug similarities on Ion Channels dataset
|
|
Method
|
All-onesSim
|
MeanRandoms
|
BestRandom
|
WorstRandom
|
CosinePF
|
DicePF
|
TanimotoPF
|
SIMCOMP
|
AUC
|
NRLMF
|
0.979234
|
0.975785
|
0.977846
|
0.973896
|
0.981475
|
0.980925
|
0.980701
|
0.983564
|
BLM-NII
|
0.974675
|
0.702874
|
0.744834
|
0.672745
|
0.96044
|
0.958388
|
0.944077
|
0.981287
|
NetLapRLS
|
0.958158
|
0.95734
|
0.957955
|
0.956605
|
0.959433
|
0.959498
|
0.959527
|
0.959882
|
WNN-GIP
|
0.861103
|
0.525954
|
0.535912
|
0.516046
|
0.930855
|
0.919196
|
0.944477
|
0.956789
|
AUPR
|
NRLMF
|
0.865326
|
0.856477
|
0.863683
|
0.847016
|
0.864683
|
0.85956
|
0.858608
|
0.863386
|
BLM-NII
|
0.521158
|
0.058516
|
0.068181
|
0.051707
|
0.484567
|
0.482101
|
0.636176
|
0.821476
|
NetLapRLS
|
0.81846
|
0.820111
|
0.820284
|
0.819911
|
0.821028
|
0.821095
|
0.821819
|
0.823003
|
WNN-GIP
|
0.34916
|
0.038653
|
0.04019
|
0.037466
|
0.53947
|
0.524961
|
0.594643
|
0.667893
|
In each row of tables, the best similarity matrix for each algorithm is bolded. The best AUC and AUPR are also marked with underlines. The first point about these tables is that the use of random matrices has degraded the efficiency of the methods. In fact, what the first four columns of the tables show is that ignoring the drug-drug similarities yields far better results than using inaccurate drug-drug similarities. It should be noted that the NRLMF has less tolerance than other methods in this case. Although the purpose of this study is not to identify a better method, but in most cases, the performance of NRLMF is better than other methods. Of course, this performance is due to its many parameters. In the Enzyme dataset (Table 2), the AUPR value for all methods and the AUC value for NetLapRLS and WNN-GIP methods are the best values when SIMCOMP similarity is considered. The NRLMF and BLM-NII methods obtain the best AUC value if they use the CosinePF and DicePF similarities, respectively. In the GPCR dataset (Table 3), the AUC for BLM-NII and the AUPR for NetLapRLS are the best values if they use the All-onesSim and TanimotoPF similarities, respectively. Except for these two cases, according to Table 3, the use of SIMCOMP has given the best results in all cases. Table 4 shows that, in the Ion Channels dataset, using All-onesSim for the NRLMF method leads to a better AUPR. In all other cases, it is clear that SIMCOMP is the best.
Table 5
Comparing different drug-drug similarities on Nuclear Receptors dataset
|
Method
|
All-onesSim
|
MeanRandoms
|
BestRandom
|
WorstRandom
|
CosinePF
|
DicePF
|
TanimotoPF
|
SIMCOMP
|
AUC
|
NRLMF
|
0.889655
|
0.864416
|
0.887016
|
0.832639
|
0.937526
|
0.945632
|
0.945968
|
0.948522
|
BLM-NII
|
0.775846
|
0.580958
|
0.613693
|
0.537734
|
0.797103
|
0.803759
|
0.896945
|
0.905075
|
NetLapRLS
|
0.79702
|
0.802193
|
0.819461
|
0.77738
|
0.823197
|
0.824621
|
0.835049
|
0.849627
|
WNN-GIP
|
0.810938
|
0.541304
|
0.591313
|
0.504229
|
0.900681
|
0.898618
|
0.90459
|
0.90394
|
AUPR
|
NRLMF
|
0.515368
|
0.499308
|
0.564758
|
0.427307
|
0.720545
|
0.728063
|
0.726034
|
0.722834
|
BLM-NII
|
0.391072
|
0.123816
|
0.178884
|
0.087529
|
0.485113
|
0.495231
|
0.63054
|
0.659326
|
NetLapRLS
|
0.428803
|
0.430737
|
0.437371
|
0.421767
|
0.444117
|
0.445235
|
0.454609
|
0.464816
|
WNN-GIP
|
0.317686
|
0.095114
|
0.118977
|
0.079856
|
0.581542
|
0.584819
|
0.590779
|
0.582391
|
In the Nuclear Receptors dataset (Table 5), the SIMCOMP gives both the best AUC and AUPR for NetLapRLS and BLM-NII methods. The same thing happens with TanimotoPF and WNN-GIP. The AUC and AUPR values for NRLMF are the best if it uses the SIMCOMP and DicePF similarities, respectively. In summary, these tables show that in almost 94% of experiments, the use of drug-drug similarities has led to better results.
So far we have seen that drug-drug similarities can increase the accuracy of DTI predictions. But which method of calculating drug-drug similarities is more appropriate for the DTI predictions problem? The answer shown in Tables 2–5 is clearly SIMCOMP. But the results shown in these tables are obtained by parameters tuned for SIMCOMP [3]. Therefore, we randomly selected a dataset for each method and tuned the parameters of that method for all drug-drug similarities except random similarities. Nuclear Receptors, GPCR, Ion Channel and Enzyme datasets were considered for NRLMF, NetLapRLS, WNN-GIP and BLM-NII methods respectively. The results of these experiments are illustrated in Fig. 1. The use of SIMCOMP for NetLapRLS and WNN-GIP methods gives the best AUC in GPCR and Ion Channel datasets, respectively. The AUCs and AUPRs calculated in the rest of the experiments, i.e. 75% of them, show that TanimotoPF gave better results than the rest of the similarities. In general, it can be concluded that for these datasets and these methods, TanimotoPF and SIMCOMP are more appropriate than other similarities in the DTI prediction problem.
To investigate the effect of the type and size of the datasets on the values obtained in the experiments, we check the values in Tables 2 to 5 in a different way. Figures 2 to 5 are given for this purpose. In each figure, we considered a method and illustrated the values of AUC and AUPR obtained for that method over all datasets. The results for the NRLMF, BLM-NII, NetLapRLS and WNN-GIP methods are shown in Figs. 2 to 5, respectively.
The results of Figs. 2 to 5 can be summarized as follows:
-
By replacing the similarities, the change in the value of AUPR is greater than that of AUC.
-
Ion Channel and Enzyme datasets seem to be less dependent on similarity matrices replacement.
-
In almost all figures, when the similarity matrix is replaced, the amount of AUC and AUPR changes for the Nuclear Receptors dataset is greater than what happens for other datasets. This has sometimes happened with less tolerance for the GPCR dataset.
-
Compared to other methods, the NRLMF and NetLapRLS methods are less dependent on similarities and by replacing the matrices, their AUC and AUPR values change slightly.
In addition to the more changes that occur in the results on Nuclear Receptors and GPCR datasets, all methods perform worse on these two data, compared to other data. If we review Table 1 again, we find that these two datasets are smaller than the Ion Channel and Enzyme datasets, and the difference between the \(A{D}_{T}\) and \(A{T}_{D}\) criteria in these two data is a larger number. Also, the \({D}_{1T}\) criterion has a larger value for these two data, especially for the Nuclear Receptors dataset. Probably, these factors have caused that the different methods cannot have better performance and less tolerance on these two datasets.