ACS Publications. Most Trusted. Most Cited. Most Read
My Activity
CONTENT TYPES

Figure 1Loading Img

Beware of Simple Methods for Structure-Based Virtual Screening: The Critical Importance of Broader Comparisons

Cite this: J. Chem. Inf. Model. 2023, 63, 5, 1401–1405
Publication Date (Web):February 27, 2023
https://doi.org/10.1021/acs.jcim.3c00218

Copyright © 2023 The Authors. Published by American Chemical Society. This publication is licensed under

CC-BY 4.0.
  • Open Access

Article Views

3850

Altmetric

-

Citations

LEARN ABOUT THESE METRICS
PDF (2 MB)

Abstract

We discuss how data unbiasing and simple methods such as protein-ligand Interaction FingerPrint (IFP) can overestimate virtual screening performance. We also show that IFP is strongly outperformed by target-specific machine-learning scoring functions, which were not considered in a recent report concluding that simple methods were better than machine-learning scoring functions at virtual screening.

This publication is licensed under

CC-BY 4.0.
  • cc licence
  • by licence

It is now well-established that machine learning (ML) scoring functions (SFs) excel at predicting the binding affinity of protein–ligand complexes from their crystal structures. This was originally shown by comparing the first version of RF-Score with 16 classical SFs, (1) and this performance gap has been expanding ever since fueled by the growing availability of data and code to reproduce these results and build upon them. (2−7) However, to excel at virtual screening as well, an ML SF needs to be trained on data sets with a large proportion of negative instances, (8,9) typically assumed and called decoys. Using the same method to generate decoys, both for the training set and the test set, results in artificially improving virtual screening performance. (8) Learning such decoy bias can be avoided by simply selecting decoys differently in each data set. (10) The prospective yield of the model can also be overestimated by using test molecules that are easy to discriminate given the training set composition. Asymmetric Validation Embedding (AVE) (11) benchmarks have been proposed to improve this situation, (11) and simple models have been claimed to perform better than ML models. (12) Here we assess the importance of comparing beyond AVE and just two generic ML SFs.

AVE is an innovative method that aims at designing unbiased data sets for ligand-based classification problems such as those in virtual screening. In particular, AVE has been used for training and evaluating ML SFs. Starting from a set of molecules labeled as either active or inactive for a given target, AVE generates a benchmark by partitioning this set into four subsets (training actives, training inactives, test actives, test inactives) such that the test molecules are not too similar to the training molecules, and the training actives/inactives are not too different from the test inactives/actives, according to their ECFP4 fingerprints. (13) AVE was claimed to prevent molecular property prediction models, including ML SFs, from overfitting the benchmarks it generates. (11)

We select acetylcholinesterase (ACHE) and HMG-CoA reductase (HMGR) to illustrate the discussion. In another study, these targets were selected for reasons unrelated to the performance of the investigated SFs (two generic and six target-specific). (14) As these targets obtained neither the best nor the worst performance on their corresponding DEKOIS2.0 test sets, (15) we also select them for this study. Furthermore, these targets were cocrystallized in high resolution, with at least a ligand having the same bioactivity as those reported on PubChem (16) and ChEMBL, (17) and have never been assayed in any high-throughput screening campaign on PubChem (as of January 2023).

The experimental design comprised the following steps: retrieving experimental data (true actives and true inactives) of the two targets from PubChem and ChEMBL, downloading the benchmarking sets ACHE and HMGR from DEKOIS2.0, (15) generating property-matched decoys with DeepCoy, (18) a deep learning method for designing diverse and likely inactive molecules by tailoring their chemical features to those of an input active, from PubChem/ChEMBL true actives, and molecular docking with Smina v2019-10-15. (19) Initial SMILES strings (PubChem/ChEMBL/DeepCoy) and sdf structures (DEKOIS2.0) of the ligands were converted into three-dimensional (3D) multi-mol2 files using Open Babel v2.3.1 (20) with the locations of hydrogen atoms being assigned at the physiological pH as usual. The PDB IDs 1EVE and 1HW8 were, respectively, used as target templates for ACHE and HMGR whenever DEKOIS2.0 ligands were involved (Table 1), whereas two other PDB structures 1E66 (ACHE) and 3CCW (HMGR), as proposed by DUD-E (https://www.dude.docking.org/targets) authors, were employed in other circumstances (i.e., for docking PubChem/ChEMBL/DeepCoy ligands), as we would like to use different receptor conformations to train and test our ML models. All target structures and their cocrystallized ligands were prepared with Chimera v1.15 (Dock Prep tool), (21) according to parameters recommended by dos Santos et al. (2018). (22) Docking was carried out within a search space of 27 000 Å3 (30 Å on each dimension) whose center was placed on the X-ray ligand of each protein; all other parameters were set following Smina recommendations. (19)

Table 1. Information on the Training-Test Data Splits Prepared for the Targets ACHE and HMGR
 ACHE dataHMGR data
SplitDEKOIS2.0AVEDEKOIS2.0AVE
Training set
SourcePubChem/ChEMBLPubChem/ChEMBL and DeepCoyPubChem/ChEMBLPubChem/ChEMBL and DeepCoy
Number of actives166125113110
Number of inactives19965209236238
Nature of inactivesTrue inactivesTrue inactives and DeepCoy decoysTrue inactivesTrue inactives and DeepCoy decoys
Test set
SourceDEKOIS2.0PubChem/ChEMBL and DeepCoyDEKOIS2.0PubChem/ChEMBL and DeepCoy
Number of actives40374031
Number of inactives1200217012002077
Nature of inactivesProperty-matched decoys from ZINCTrue inactives and DeepCoy decoysProperty-matched decoys from ZINCTrue inactives and DeepCoy decoys

Two different training-test partitions were prepared for each target, one with the ACHE/HMGR data set of DEKOIS2.0 as test set and the PubChem/ChEMBL data (excluding any molecules already in the test set) as training set for target-specific ML SFs. We call this split DEKOIS2.0, which was effectively made at random with training inactives being selected differently from test inactives to avoid decoy bias. (10) The other partition, called AVE, was issued by the AVE script (training-to-test ratio = 3, other parameters set as default (11)), splitting the same population of ACHE/HMGR true actives and true inactives retrieved from PubChem/ChEMBL, plus DeepCoy-generated decoys property-matched to these true actives. These partitions are summarized in Table 1.

Five target-specific ML SFs were trained using the training sets of the aforementioned partitions. They used the following five algorithms: random forest (RF), (8,23) extreme gradient boosting (XGB), (24,25) support vector machine (SVM), (26,27) artificial neural network (ANN), (28,29) and deep neural network (DNN). (30,31) Protein–Ligand extended connectivity (PLEC) fingerprints (32) were used as features to describe ligand–receptor complexes after docking. Each of these ML SFs was then evaluated on the test set corresponding to its training set (hereinafter referred to as ACHE-DEKOIS2.0, ACHE-AVE, HMGR-DEKOIS2.0, and HMGR-AVE). Note that none of the training and test set pairs have any molecule in common by construction (in AVE splits, molecules in one set are, in addition, dissimilar to those in the other set). On the other hand, four generic SFs: Smina, (19) IFP, (12) CNN-Score, (33) and RF-Score-VS v2 (8) were also tested on the same four test sets as those of the target-specific ones. Table 2 summarizes the performance of nine SFs mentioned above, in terms of area under the precision-recall curve (PR-AUC). The PR-AUC of each target-specific ML SF on each test set is the median value obtained after 10 training-test runs. We used PR-AUC as the performance metric, as it is more informative than ROC-AUC in strongly class-imbalanced test sets (34) such as those in virtual screening.

Table 2. PR-AUCs of Nine SFs on Four Test Sets: ACHE-DEKOIS2.0, ACHE-AVE, HMGR-DEKOIS2.0, HMGR-AVEa
Test setACHE-DEKOIS2.0ACHE-AVEHMGR-DEKOIS2.0HMGR-AVE
Generic SF
Smina0.1950.0830.0800.020
IFP0.1670.0210.1250.084
CNN-Score0.0550.0300.1710.026
RF-Score-VS v20.0830.0500.7630.034
Target-specific ML SF
RF0.518 [0.511, 0.528]0.525 [0.524, 0.532]0.968 [0.963, 0.972]0.645 [0.641, 0.650]
XGB0.165 [0.165, 0.165]0.501 [0.501, 0.501]0.886 [0.886, 0.886]0.688 [0.688, 0.688]
SVM0.566 [0.566, 0.566]0.438 [0.438, 0.438]0.961 [0.961, 0.961]0.675 [0.675, 0.675]
ANN0.094 [0.082, 0.114]0.458 [0.452, 0.462]0.946 [0.941, 0.947]0.671 [0.667, 0.672]
DNN0.091 [0.086, 0.127]0.416 [0.392, 0.416]0.791 [0.772, 0.849]0.717 [0.682, 0.717]
Expected at random
P/(P+N)0.0320.0170.0320.015
a

Underlined are the cases where an AVE test set led to better virtual screening performance than the corresponding DEKOIS2.0 test set using the same SF. For target-specific ML SFs, the PR-AUCs outside square brackets are median values after 10 training-test runs, while those inside square brackets are the 1st (lower) and 3rd (upper) quartiles of the 10 runs. For each test set, the PR-AUC expected by a random-guessing model is P/(P+N), where P and N are the numbers of positives (actives) and negatives (inactives) (34).

Table 2 shows that an SF performed better on an AVE test set than on the corresponding DEKOIS2.0 one in three of the 36 cases (8.3%). All of these cases involve the ACHE target and three target-specific ML SFs (XGB, ANN, DNN). The four generic SFs (two non-ML SFs and two ML SFs) gave better performance on DEKOIS2.0 test sets in all instances. Therefore, as expected, test sets by AVE are generally more challenging than their DEKOIS2.0 counterparts (Figure 1).

Figure 1

Figure 1. PR-AUCs of nine SFs on the two test sets (DEKOIS2.0 and AVE) of each target (ACHE, HMGR). The PR-AUC of each target-specific ML SF is the median value obtained after 10 training-test runs. Most SFs achieved higher PR-AUCs on DEKOIS2.0 benchmarks than on the corresponding AVE test sets. There are only three cases where an AVE-issued test set led to much better virtual screening performance by an SF than DEKOIS2.0: these cases are represented by three squares inside the red dotted circle in the ACHE plot and correspond to three target-specific ML SFs employing either XGB, ANN, or DNN as learning algorithm.

However, these three cases demonstrate that AVE benchmarks can sometimes be much easier than those generated effectively at random. For example, the bias of ACHE-DEKOIS2.0 in Table 2 is substantially larger than the bias of ACHE-AVE (0.387 vs 0.002), yet PR-AUCs for the XGB model are much larger when the ACHE-AVE partition is used (0.165 vs 0.501). This is due to the AVE bias not always being well-correlated with how challenging a data set is in practice, which could already be seen in the AVE study. Indeed, a careful inspection of that paper will identify data sets with practically no bias according to AVE but ROC-AUCs of up to 0.9 (e.g., some targets in Figure 11 (11)).

We agree that AVE data splits are suited to compare ML and non-ML methods for virtual screening in a distribution-shift scenario. However, caution should be taken when one interprets AVE retrospective performance in absolute terms. For example, despite HMGR-AVE having a bias of just 0.008, the DNN model obtained a median PR-AUC as high as 0.717 with a 100% hit rate within the 21 highest-ranked molecules. While many studies achieve excellent prospective hit rates, (23,24,27,35−41) almost never 100% is obtained, and it is safe to assume that there are many more unpublished prospective experiments due to not yielding any discovery. Therefore, we would certainly expect the prospective performance of these SFs to be considerably worse than that obtained with AVE splits. We do not recommend either to use AVE-trained models for prospective purposes, since its debiasing process has been found to reduce the information available to train a model, impairing its ability to generalize. (42) Training with the most relevant data for the test set is a more promising approach. (43)

Regardless of partitioning data to have minimal AVE bias (11) or differently generated decoys, (10) it is clear that simple non-ML methods such as IFP are not the most suitable for structure-based virtual screening. This is particularly noticeable when using target-specific ML SFs (Figure 1). Furthermore, CNN-score strongly outperformed IFP in a PD-L1 benchmark. (44) Markedly better performance than IFP has also been reported with ML SFs using the AVE-split HTS-based LIT-PCBA benchmark. (45) None of these ML SFs were included in a study concluding that IFP was the best choice for structure-based virtual screening. (12) Moreover, it is important to note that IFP is prone to overfitting retrospective benchmarks, in that one has to select the pose of a molecule bound to the target as the search template with knowledge of whether that 3D conformation is well-represented among test-set actives. As shown elsewhere, (46) if this template is well-represented in the test set, then virtual screening performance will tend to be high (otherwise performance will be low), but this information is not available in prospective scenarios. Taken together, these counterexamples show the critical importance of broader comparisons to reach robust conclusions in this research topic.

Accession Codes

The Python scripts and data sets related to this paper are freely available at https://github.com/vktrannguyen/AVE-vs-DEKOIS.

Author Information

ARTICLE SECTIONS
Jump To

  • Corresponding Author
  • Author
  • Funding

    The French Association for Cancer Research (ARC) for funding the postdoctoral position of V-K.T-N. The Wolfson Foundation and the Royal Society for a Royal Society Wolfson Fellowship awarded to P.J.B.

  • Notes
    The authors declare no competing financial interest.

References

ARTICLE SECTIONS
Jump To

This article references 46 other publications.

  1. 1
    Ballester, P. J.; Mitchell, J. B. O. A Machine Learning Approach to Predicting Protein-Ligand Binding Affinity with Applications to Molecular Docking. Bioinformatics 2010, 26, 11691175,  DOI: 10.1093/bioinformatics/btq112
  2. 2
    Ain, Q. U.; Aleksandrova, A.; Roessler, F. D.; Ballester, P. J. Machine-Learning Scoring Functions to Improve Structure-Based Binding Affinity Prediction and Virtual Screening. WIREs Comput. Mol. Sci. 2015, 5, 405424,  DOI: 10.1002/wcms.1225
  3. 3
    Li, H.; Sze, K.-H.; Lu, G.; Ballester, P. J. Machine-Learning Scoring Functions for Structure-Based Drug Lead Optimization. Wiley Interdiscip. Rev. Comput. Mol. Sci. 2020, 10, e1465  DOI: 10.1002/wcms.1465
  4. 4
    Shen, C.; Ding, J.; Wang, Z.; Cao, D.; Ding, X.; Hou, T. From Machine Learning to Deep Learning: Advances in Scoring Functions for Protein-Ligand Docking. Wiley Interdiscip. Rev. Comput. Mol. Sci. 2020, 10, e1429  DOI: 10.1002/wcms.1429
  5. 5
    Nguyen, D. D.; Gao, K.; Wang, M.; Wei, G. W. MathDL: Mathematical Deep Learning for D3R Grand Challenge 4. J. Comput. Aided. Mol. Des. 2020, 34, 131147,  DOI: 10.1007/s10822-019-00237-5
  6. 6
    Meng, Z.; Xia, K. Persistent Spectral-Based Machine Learning (PerSpect ML) for Protein-Ligand Binding Affinity Prediction. Sci. Adv. 2021, 7, 7,  DOI: 10.1126/sciadv.abc5329
  7. 7
    Li, H.; Peng, J.; Sidorov, P.; Leung, Y.; Leung, K. S.; Wong, M. H.; Lu, G.; Ballester, P. J. Classical Scoring Functions for Docking Are Unable to Exploit Large Volumes of Structural and Interaction Data. Bioinformatics 2019, 35, 39893995,  DOI: 10.1093/bioinformatics/btz183
  8. 8
    Wójcikowski, M.; Ballester, P. J.; Siedlecki, P. Performance of Machine-Learning Scoring Functions in Structure-Based Virtual Screening. Sci. Rep. 2017, 7, 46710,  DOI: 10.1038/srep46710
  9. 9
    Li, H.; Sze, K.-H.; Lu, G.; Ballester, P. J. Machine-Learning Scoring Functions for Structure-Based Virtual Screening. WIREs Comput. Mol. Sci. 2021, 11, e1478  DOI: 10.1002/wcms.1478
  10. 10
    Ballester, P. J. Selecting Machine-Learning Scoring Functions for Structure-Based Virtual Screening. Drug Discovery Today: Technologies 2019, 32, 8187,  DOI: 10.1016/j.ddtec.2020.09.001
  11. 11
    Wallach, I.; Heifets, A. Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization. J. Chem. Inf. Model. 2018, 58, 916932,  DOI: 10.1021/acs.jcim.7b00403
  12. 12
    Tran-Nguyen, V. K.; Bret, G.; Rognan, D. True Accuracy of Fast Scoring Functions to Predict High-Throughput Screening Data from Docking Poses: The Simpler the Better. J. Chem. Inf. Model. 2021, 61, 27882797,  DOI: 10.1021/acs.jcim.1c00292
  13. 13
    Rogers, D.; Hahn, M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. 2010, 50, 742754,  DOI: 10.1021/ci100050t
  14. 14
    Fresnais, L.; Ballester, P. J. The Impact of Compound Library Size on the Performance of Scoring Functions for Structure-Based Virtual Screening. Brief. Bioinform. 2021, 22, bbaa095  DOI: 10.1093/bib/bbaa095
  15. 15
    Bauer, M. R.; Ibrahim, T. M.; Vogel, S. M.; Boeckler, F. M. Evaluation and Optimization of Virtual Screening Workflows with DEKOIS 2.0 - A Public Library of Challenging Docking Benchmark Sets. J. Chem. Inf. Model. 2013, 53, 14471462,  DOI: 10.1021/ci400115b
  16. 16
    Kim, S.; Chen, J.; Cheng, T.; Gindulyte, A.; He, J.; He, S.; Li, Q.; Shoemaker, B. A.; Thiessen, P. A.; Yu, B.; Zaslavsky, L.; Zhang, J.; Bolton, E. E. PubChem in 2021: New Data Content and Improved Web Interfaces. Nucleic Acids Res. 2021, 49, D1388D1395,  DOI: 10.1093/nar/gkaa971
  17. 17
    Mendez, D.; Gaulton, A.; Bento, A. P.; Chambers, J.; De Veij, M.; Félix, E.; Magariños, M. P.; Mosquera, J. F.; Mutowo, P.; Nowotka, M.; Gordillo-Marañón, M.; Hunter, F.; Junco, L.; Mugumbate, G.; Rodriguez-Lopez, M.; Atkinson, F.; Bosc, N.; Radoux, C. J.; Segura-Cabrera, A.; Hersey, A.; Leach, A. R. ChEMBL: Towards Direct Deposition of Bioassay Data. Nucleic Acids Res. 2019, 47, D930D940,  DOI: 10.1093/nar/gky1075
  18. 18
    Imrie, F.; Bradley, A. R.; Deane, C. M. Generating Property-Matched Decoy Molecules Using Deep Learning. Bioinformatics 2021, 37, btab080  DOI: 10.1093/bioinformatics/btab080
  19. 19
    Koes, D. R.; Baumgartner, M. P.; Camacho, C. J. Lessons Learned in Empirical Scoring with Smina from the CSAR 2011 Benchmarking Exercise. J. Chem. Inf. Model. 2013, 53, 18931904,  DOI: 10.1021/ci300604z
  20. 20
    O’Boyle, N. M.; Banck, M.; James, C. A.; Morley, C.; Vandermeersch, T.; Hutchison, G. R. Open Babel: An Open Chemical Toolbox. J. Cheminform. 2011, 3, 33,  DOI: 10.1186/1758-2946-3-33
  21. 21
    Pettersen, E. F.; Goddard, T. D.; Huang, C. C.; Couch, G. S.; Greenblatt, D. M.; Meng, E. C.; Ferrin, T. E. UCSF Chimera─a Visualization System for Exploratory Research and Analysis. J. Comput. Chem. 2004, 25, 16051612,  DOI: 10.1002/jcc.20084
  22. 22
    dos Santos, R. N.; Ferreira, L. G.; Andricopulo, A. D. Practices in Molecular Docking and Structure-Based Virtual Screening. In Computational Drug Discovery and Design; Methods in Molecular Biology Series; Humana Press Inc., 2018; Vol. 1762, pp 3150.  DOI: 10.1007/978-1-4939-7756-7_3 .
  23. 23
    Ballester, P. J.; Mangold, M.; Howard, N. I.; Robinson, R. L. M.; Abell, C.; Blumberger, J.; Mitchell, J. B. O. Hierarchical Virtual Screening for the Discovery of New Molecular Scaffolds in Antibacterial Hit Identification. J. R. Soc. Interface 2012, 9, 31963207,  DOI: 10.1098/rsif.2012.0569
  24. 24
    Adeshina, Y. O.; Deeds, E. J.; Karanicolas, J. Machine Learning Classification Can Reduce False Positives in Structure-Based Virtual Screening. Proc. Natl. Acad. Sci. U. S. A. 2020, 117, 1847718488,  DOI: 10.1073/pnas.2000585117
  25. 25
    Xiong, G.-L.; Ye, W.-L.; Shen, C.; Lu, A.-P.; Hou, T.-J.; Cao, D.-S. Improving Structure-Based Virtual Screening Performance via Learning from Scoring Function Components. Brief. Bioinform. 2021, 22, bbaa094  DOI: 10.1093/bib/bbaa094
  26. 26
    Li, L.; Khanna, M.; Jo, I.; Wang, F.; Ashpole, N. M.; Hudmon, A.; Meroueh, S. O. Target-Specific Support Vector Machine Scoring in Structure-Based Virtual Screening: Computational Validation, In Vitro Testing in Kinases, and Effects on Lung Cancer Cell Proliferation. J. Chem. Inf. Model. 2011, 51, 755759,  DOI: 10.1021/ci100490w
  27. 27
    Sun, H.; Pan, P.; Tian, S.; Xu, L.; Kong, X.; Li, Y.; Dan, L.; Hou, T. Constructing and Validating High-Performance MIEC-SVM Models in Virtual Screening for Kinases: A Better Way for Actives Discovery. Sci. Rep. 2016, 6, 24817,  DOI: 10.1038/srep24817
  28. 28
    Durrant, J. D.; McCammon, J. A. NNScore 2.0: A Neural-Network Receptor-Ligand Scoring Function. J. Chem. Inf. Model. 2011, 51, 28972903,  DOI: 10.1021/ci2003889
  29. 29
    Durrant, J. D.; McCammon, J. A. NNScore: A Neural-Network-Based Scoring Function for the Characterization of Protein-Ligand Complexes. J. Chem. Inf. Model. 2010, 50, 18651871,  DOI: 10.1021/ci100244v
  30. 30
    Wang, D.; Cui, C.; Ding, X.; Xiong, Z.; Zheng, M.; Luo, X.; Jiang, H.; Chen, K. Improving the Virtual Screening Ability of Target-Specific Scoring Functions Using Deep Learning Methods. Front. Pharmacol. 2019, 10, 924,  DOI: 10.3389/fphar.2019.00924
  31. 31
    Ashtawy, H. M.; Mahapatra, N. R. Task-Specific Scoring Functions for Predicting Ligand Binding Poses and Affinity and for Screening Enrichment. J. Chem. Inf. Model. 2018, 58, 119133,  DOI: 10.1021/acs.jcim.7b00309
  32. 32
    Wójcikowski, M.; Kukiełka, M.; Stepniewska-Dziubinska, M. M.; Siedlecki, P. Development of a Protein-Ligand Extended Connectivity (PLEC) Fingerprint and Its Application for Binding Affinity Predictions. Bioinformatics 2019, 35, 13341341,  DOI: 10.1093/bioinformatics/bty757
  33. 33
    Ragoza, M.; Hochuli, J.; Idrobo, E.; Sunseri, J.; Koes, D. R. Protein-Ligand Scoring with Convolutional Neural Networks. J. Chem. Inf. Model. 2017, 57, 942957,  DOI: 10.1021/acs.jcim.6b00740
  34. 34
    Saito, T.; Rehmsmeier, M. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS One 2015, 10, e0118432  DOI: 10.1371/journal.pone.0118432
  35. 35
    Durrant, J. D.; Carlson, K. E.; Martin, T. A.; Offutt, T. L.; Mayne, C. G.; Katzenellenbogen, J. A.; Amaro, R. E. Neural-Network Scoring Functions Identify Structurally Novel Estrogen-Receptor Ligands. J. Chem. Inf. Model. 2015, 55, 19531961,  DOI: 10.1021/acs.jcim.5b00241
  36. 36
    Zhan, W.; Li, D.; Che, J.; Zhang, L.; Yang, B.; Hu, Y.; Liu, T.; Dong, X. Integrating Docking Scores, Interaction Profiles and Molecular Descriptors to Improve the Accuracy of Molecular Docking: Toward the Discovery of Novel Akt1 Inhibitors. Eur. J. Med. Chem. 2014, 75, 1120,  DOI: 10.1016/j.ejmech.2014.01.019
  37. 37
    Ghislat, G.; Rahman, T.; Ballester, P. J. Identification and Validation of Carbonic Anhydrase Ii as the First Target of the Anti-Inflammatory Drug Actarit. Biomolecules 2020, 10, 111,  DOI: 10.3390/biom10111570
  38. 38
    Ariey-Bonnet, J.; Carrasco, K.; Le Grand, M.; Hoffer, L.; Betzi, S.; Feracci, M.; Tsvetkov, P.; Devred, F.; Collette, Y.; Morelli, X.; Ballester, P.; Pasquier, E. In Silico Molecular Target Prediction Unveils Mebendazole as a Potent MAPK14 Inhibitor. Mol. Oncol. 2020, 14, 30833099,  DOI: 10.1002/1878-0261.12810
  39. 39
    Ballester, P. J.; Westwood, I.; Laurieri, N.; Sim, E.; Richards, W. G. Prospective Virtual Screening with Ultrafast Shape Recognition: The Identification of Novel Inhibitors of Arylamine N-Acetyltransferases. J. R. Soc. Interface R. Soc. 2010, 7, 335342,  DOI: 10.1098/rsif.2009.0170
  40. 40
    Stein, R. M.; Kang, H. J.; McCorvy, J. D.; Glatfelter, G. C.; Jones, A. J.; Che, T.; Slocum, S.; Huang, X. P.; Savych, O.; Moroz, Y. S.; Stauch, B.; Johansson, L. C.; Cherezov, V.; Kenakin, T.; Irwin, J. J.; Shoichet, B. K.; Roth, B. L.; Dubocovich, M. L. Virtual Discovery of Melatonin Receptor Ligands to Modulate Circadian Rhythms. Nature 2020, 579, 609614,  DOI: 10.1038/s41586-020-2027-0
  41. 41
    Ghislat, G.; Rahman, T.; Ballester, P. J. Recent Progress on the Prospective Application of Machine Learning to Structure-Based Virtual Screening. Current Opinion in Chemical Biology 2021, 65, 2834,  DOI: 10.1016/j.cbpa.2021.04.009
  42. 42
    Sundar, V.; Colwell, L. The Effect of Debiasing Protein-Ligand Binding Data on Generalization. J. Chem. Inf. Model. 2020, 60, 5662,  DOI: 10.1021/acs.jcim.9b00415
  43. 43
    Liu, S.; Alnammi, M.; Ericksen, S. S.; Voter, A. F.; Ananiev, G. E.; Keck, J. L.; Hoffmann, F. M.; Wildman, S. A.; Gitter, A. Practical Model Selection for Prospective Virtual Screening. J. Chem. Inf. Model. 2019, 59, 282293,  DOI: 10.1021/acs.jcim.8b00363
  44. 44
    Tran-Nguyen, V. K.; Simeon, S.; Junaid, M.; Ballester, P. J. Structure-Based Virtual Screening for PDL1 Dimerizers: Evaluating Generic Scoring Functions. Curr. Res. Struct. Biol. 2022, 4, 206210,  DOI: 10.1016/j.crstbi.2022.06.002
  45. 45
    Shen, C.; Weng, G.; Zhang, X.; Leung, E. L. H.; Yao, X.; Pang, J.; Chai, X.; Li, D.; Wang, E.; Cao, D.; Hou, T. Accuracy or Novelty: What Can We Gain from Target-Specific Machine-Learning-Based Scoring Functions in Virtual Screening? Brief. Bioinform. 2021, 22.  DOI: 10.1093/bib/bbaa410 .
  46. 46
    Ballester, P. J.; Finn, P. W.; Richards, W. G. Ultrafast Shape Recognition: Evaluating a New Ligand-Based Virtual Screening Technology. J. Mol. Graph. Model. 2009, 27, 836845,  DOI: 10.1016/j.jmgm.2009.01.001

Cited By

ARTICLE SECTIONS
Jump To

This article is cited by 1 publications.

  1. Saiveth Hernández-Hernández, Pedro J. Ballester. On the Best Way to Cluster NCI-60 Molecules. Biomolecules 2023, 13 (3) , 498. https://doi.org/10.3390/biom13030498
  • Abstract

    Figure 1

    Figure 1. PR-AUCs of nine SFs on the two test sets (DEKOIS2.0 and AVE) of each target (ACHE, HMGR). The PR-AUC of each target-specific ML SF is the median value obtained after 10 training-test runs. Most SFs achieved higher PR-AUCs on DEKOIS2.0 benchmarks than on the corresponding AVE test sets. There are only three cases where an AVE-issued test set led to much better virtual screening performance by an SF than DEKOIS2.0: these cases are represented by three squares inside the red dotted circle in the ACHE plot and correspond to three target-specific ML SFs employing either XGB, ANN, or DNN as learning algorithm.

  • References

    ARTICLE SECTIONS
    Jump To

    This article references 46 other publications.

    1. 1
      Ballester, P. J.; Mitchell, J. B. O. A Machine Learning Approach to Predicting Protein-Ligand Binding Affinity with Applications to Molecular Docking. Bioinformatics 2010, 26, 11691175,  DOI: 10.1093/bioinformatics/btq112
    2. 2
      Ain, Q. U.; Aleksandrova, A.; Roessler, F. D.; Ballester, P. J. Machine-Learning Scoring Functions to Improve Structure-Based Binding Affinity Prediction and Virtual Screening. WIREs Comput. Mol. Sci. 2015, 5, 405424,  DOI: 10.1002/wcms.1225
    3. 3
      Li, H.; Sze, K.-H.; Lu, G.; Ballester, P. J. Machine-Learning Scoring Functions for Structure-Based Drug Lead Optimization. Wiley Interdiscip. Rev. Comput. Mol. Sci. 2020, 10, e1465  DOI: 10.1002/wcms.1465
    4. 4
      Shen, C.; Ding, J.; Wang, Z.; Cao, D.; Ding, X.; Hou, T. From Machine Learning to Deep Learning: Advances in Scoring Functions for Protein-Ligand Docking. Wiley Interdiscip. Rev. Comput. Mol. Sci. 2020, 10, e1429  DOI: 10.1002/wcms.1429
    5. 5
      Nguyen, D. D.; Gao, K.; Wang, M.; Wei, G. W. MathDL: Mathematical Deep Learning for D3R Grand Challenge 4. J. Comput. Aided. Mol. Des. 2020, 34, 131147,  DOI: 10.1007/s10822-019-00237-5
    6. 6
      Meng, Z.; Xia, K. Persistent Spectral-Based Machine Learning (PerSpect ML) for Protein-Ligand Binding Affinity Prediction. Sci. Adv. 2021, 7, 7,  DOI: 10.1126/sciadv.abc5329
    7. 7
      Li, H.; Peng, J.; Sidorov, P.; Leung, Y.; Leung, K. S.; Wong, M. H.; Lu, G.; Ballester, P. J. Classical Scoring Functions for Docking Are Unable to Exploit Large Volumes of Structural and Interaction Data. Bioinformatics 2019, 35, 39893995,  DOI: 10.1093/bioinformatics/btz183
    8. 8
      Wójcikowski, M.; Ballester, P. J.; Siedlecki, P. Performance of Machine-Learning Scoring Functions in Structure-Based Virtual Screening. Sci. Rep. 2017, 7, 46710,  DOI: 10.1038/srep46710
    9. 9
      Li, H.; Sze, K.-H.; Lu, G.; Ballester, P. J. Machine-Learning Scoring Functions for Structure-Based Virtual Screening. WIREs Comput. Mol. Sci. 2021, 11, e1478  DOI: 10.1002/wcms.1478
    10. 10
      Ballester, P. J. Selecting Machine-Learning Scoring Functions for Structure-Based Virtual Screening. Drug Discovery Today: Technologies 2019, 32, 8187,  DOI: 10.1016/j.ddtec.2020.09.001
    11. 11
      Wallach, I.; Heifets, A. Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization. J. Chem. Inf. Model. 2018, 58, 916932,  DOI: 10.1021/acs.jcim.7b00403
    12. 12
      Tran-Nguyen, V. K.; Bret, G.; Rognan, D. True Accuracy of Fast Scoring Functions to Predict High-Throughput Screening Data from Docking Poses: The Simpler the Better. J. Chem. Inf. Model. 2021, 61, 27882797,  DOI: 10.1021/acs.jcim.1c00292
    13. 13
      Rogers, D.; Hahn, M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. 2010, 50, 742754,  DOI: 10.1021/ci100050t
    14. 14
      Fresnais, L.; Ballester, P. J. The Impact of Compound Library Size on the Performance of Scoring Functions for Structure-Based Virtual Screening. Brief. Bioinform. 2021, 22, bbaa095  DOI: 10.1093/bib/bbaa095
    15. 15
      Bauer, M. R.; Ibrahim, T. M.; Vogel, S. M.; Boeckler, F. M. Evaluation and Optimization of Virtual Screening Workflows with DEKOIS 2.0 - A Public Library of Challenging Docking Benchmark Sets. J. Chem. Inf. Model. 2013, 53, 14471462,  DOI: 10.1021/ci400115b
    16. 16
      Kim, S.; Chen, J.; Cheng, T.; Gindulyte, A.; He, J.; He, S.; Li, Q.; Shoemaker, B. A.; Thiessen, P. A.; Yu, B.; Zaslavsky, L.; Zhang, J.; Bolton, E. E. PubChem in 2021: New Data Content and Improved Web Interfaces. Nucleic Acids Res. 2021, 49, D1388D1395,  DOI: 10.1093/nar/gkaa971
    17. 17
      Mendez, D.; Gaulton, A.; Bento, A. P.; Chambers, J.; De Veij, M.; Félix, E.; Magariños, M. P.; Mosquera, J. F.; Mutowo, P.; Nowotka, M.; Gordillo-Marañón, M.; Hunter, F.; Junco, L.; Mugumbate, G.; Rodriguez-Lopez, M.; Atkinson, F.; Bosc, N.; Radoux, C. J.; Segura-Cabrera, A.; Hersey, A.; Leach, A. R. ChEMBL: Towards Direct Deposition of Bioassay Data. Nucleic Acids Res. 2019, 47, D930D940,  DOI: 10.1093/nar/gky1075
    18. 18
      Imrie, F.; Bradley, A. R.; Deane, C. M. Generating Property-Matched Decoy Molecules Using Deep Learning. Bioinformatics 2021, 37, btab080  DOI: 10.1093/bioinformatics/btab080
    19. 19
      Koes, D. R.; Baumgartner, M. P.; Camacho, C. J. Lessons Learned in Empirical Scoring with Smina from the CSAR 2011 Benchmarking Exercise. J. Chem. Inf. Model. 2013, 53, 18931904,  DOI: 10.1021/ci300604z
    20. 20
      O’Boyle, N. M.; Banck, M.; James, C. A.; Morley, C.; Vandermeersch, T.; Hutchison, G. R. Open Babel: An Open Chemical Toolbox. J. Cheminform. 2011, 3, 33,  DOI: 10.1186/1758-2946-3-33
    21. 21
      Pettersen, E. F.; Goddard, T. D.; Huang, C. C.; Couch, G. S.; Greenblatt, D. M.; Meng, E. C.; Ferrin, T. E. UCSF Chimera─a Visualization System for Exploratory Research and Analysis. J. Comput. Chem. 2004, 25, 16051612,  DOI: 10.1002/jcc.20084
    22. 22
      dos Santos, R. N.; Ferreira, L. G.; Andricopulo, A. D. Practices in Molecular Docking and Structure-Based Virtual Screening. In Computational Drug Discovery and Design; Methods in Molecular Biology Series; Humana Press Inc., 2018; Vol. 1762, pp 3150.  DOI: 10.1007/978-1-4939-7756-7_3 .
    23. 23
      Ballester, P. J.; Mangold, M.; Howard, N. I.; Robinson, R. L. M.; Abell, C.; Blumberger, J.; Mitchell, J. B. O. Hierarchical Virtual Screening for the Discovery of New Molecular Scaffolds in Antibacterial Hit Identification. J. R. Soc. Interface 2012, 9, 31963207,  DOI: 10.1098/rsif.2012.0569
    24. 24
      Adeshina, Y. O.; Deeds, E. J.; Karanicolas, J. Machine Learning Classification Can Reduce False Positives in Structure-Based Virtual Screening. Proc. Natl. Acad. Sci. U. S. A. 2020, 117, 1847718488,  DOI: 10.1073/pnas.2000585117
    25. 25
      Xiong, G.-L.; Ye, W.-L.; Shen, C.; Lu, A.-P.; Hou, T.-J.; Cao, D.-S. Improving Structure-Based Virtual Screening Performance via Learning from Scoring Function Components. Brief. Bioinform. 2021, 22, bbaa094  DOI: 10.1093/bib/bbaa094
    26. 26
      Li, L.; Khanna, M.; Jo, I.; Wang, F.; Ashpole, N. M.; Hudmon, A.; Meroueh, S. O. Target-Specific Support Vector Machine Scoring in Structure-Based Virtual Screening: Computational Validation, In Vitro Testing in Kinases, and Effects on Lung Cancer Cell Proliferation. J. Chem. Inf. Model. 2011, 51, 755759,  DOI: 10.1021/ci100490w
    27. 27
      Sun, H.; Pan, P.; Tian, S.; Xu, L.; Kong, X.; Li, Y.; Dan, L.; Hou, T. Constructing and Validating High-Performance MIEC-SVM Models in Virtual Screening for Kinases: A Better Way for Actives Discovery. Sci. Rep. 2016, 6, 24817,  DOI: 10.1038/srep24817
    28. 28
      Durrant, J. D.; McCammon, J. A. NNScore 2.0: A Neural-Network Receptor-Ligand Scoring Function. J. Chem. Inf. Model. 2011, 51, 28972903,  DOI: 10.1021/ci2003889
    29. 29
      Durrant, J. D.; McCammon, J. A. NNScore: A Neural-Network-Based Scoring Function for the Characterization of Protein-Ligand Complexes. J. Chem. Inf. Model. 2010, 50, 18651871,  DOI: 10.1021/ci100244v
    30. 30
      Wang, D.; Cui, C.; Ding, X.; Xiong, Z.; Zheng, M.; Luo, X.; Jiang, H.; Chen, K. Improving the Virtual Screening Ability of Target-Specific Scoring Functions Using Deep Learning Methods. Front. Pharmacol. 2019, 10, 924,  DOI: 10.3389/fphar.2019.00924
    31. 31
      Ashtawy, H. M.; Mahapatra, N. R. Task-Specific Scoring Functions for Predicting Ligand Binding Poses and Affinity and for Screening Enrichment. J. Chem. Inf. Model. 2018, 58, 119133,  DOI: 10.1021/acs.jcim.7b00309
    32. 32
      Wójcikowski, M.; Kukiełka, M.; Stepniewska-Dziubinska, M. M.; Siedlecki, P. Development of a Protein-Ligand Extended Connectivity (PLEC) Fingerprint and Its Application for Binding Affinity Predictions. Bioinformatics 2019, 35, 13341341,  DOI: 10.1093/bioinformatics/bty757
    33. 33
      Ragoza, M.; Hochuli, J.; Idrobo, E.; Sunseri, J.; Koes, D. R. Protein-Ligand Scoring with Convolutional Neural Networks. J. Chem. Inf. Model. 2017, 57, 942957,  DOI: 10.1021/acs.jcim.6b00740
    34. 34
      Saito, T.; Rehmsmeier, M. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS One 2015, 10, e0118432  DOI: 10.1371/journal.pone.0118432
    35. 35
      Durrant, J. D.; Carlson, K. E.; Martin, T. A.; Offutt, T. L.; Mayne, C. G.; Katzenellenbogen, J. A.; Amaro, R. E. Neural-Network Scoring Functions Identify Structurally Novel Estrogen-Receptor Ligands. J. Chem. Inf. Model. 2015, 55, 19531961,  DOI: 10.1021/acs.jcim.5b00241
    36. 36
      Zhan, W.; Li, D.; Che, J.; Zhang, L.; Yang, B.; Hu, Y.; Liu, T.; Dong, X. Integrating Docking Scores, Interaction Profiles and Molecular Descriptors to Improve the Accuracy of Molecular Docking: Toward the Discovery of Novel Akt1 Inhibitors. Eur. J. Med. Chem. 2014, 75, 1120,  DOI: 10.1016/j.ejmech.2014.01.019
    37. 37
      Ghislat, G.; Rahman, T.; Ballester, P. J. Identification and Validation of Carbonic Anhydrase Ii as the First Target of the Anti-Inflammatory Drug Actarit. Biomolecules 2020, 10, 111,  DOI: 10.3390/biom10111570
    38. 38
      Ariey-Bonnet, J.; Carrasco, K.; Le Grand, M.; Hoffer, L.; Betzi, S.; Feracci, M.; Tsvetkov, P.; Devred, F.; Collette, Y.; Morelli, X.; Ballester, P.; Pasquier, E. In Silico Molecular Target Prediction Unveils Mebendazole as a Potent MAPK14 Inhibitor. Mol. Oncol. 2020, 14, 30833099,  DOI: 10.1002/1878-0261.12810
    39. 39
      Ballester, P. J.; Westwood, I.; Laurieri, N.; Sim, E.; Richards, W. G. Prospective Virtual Screening with Ultrafast Shape Recognition: The Identification of Novel Inhibitors of Arylamine N-Acetyltransferases. J. R. Soc. Interface R. Soc. 2010, 7, 335342,  DOI: 10.1098/rsif.2009.0170
    40. 40
      Stein, R. M.; Kang, H. J.; McCorvy, J. D.; Glatfelter, G. C.; Jones, A. J.; Che, T.; Slocum, S.; Huang, X. P.; Savych, O.; Moroz, Y. S.; Stauch, B.; Johansson, L. C.; Cherezov, V.; Kenakin, T.; Irwin, J. J.; Shoichet, B. K.; Roth, B. L.; Dubocovich, M. L. Virtual Discovery of Melatonin Receptor Ligands to Modulate Circadian Rhythms. Nature 2020, 579, 609614,  DOI: 10.1038/s41586-020-2027-0
    41. 41
      Ghislat, G.; Rahman, T.; Ballester, P. J. Recent Progress on the Prospective Application of Machine Learning to Structure-Based Virtual Screening. Current Opinion in Chemical Biology 2021, 65, 2834,  DOI: 10.1016/j.cbpa.2021.04.009
    42. 42
      Sundar, V.; Colwell, L. The Effect of Debiasing Protein-Ligand Binding Data on Generalization. J. Chem. Inf. Model. 2020, 60, 5662,  DOI: 10.1021/acs.jcim.9b00415
    43. 43
      Liu, S.; Alnammi, M.; Ericksen, S. S.; Voter, A. F.; Ananiev, G. E.; Keck, J. L.; Hoffmann, F. M.; Wildman, S. A.; Gitter, A. Practical Model Selection for Prospective Virtual Screening. J. Chem. Inf. Model. 2019, 59, 282293,  DOI: 10.1021/acs.jcim.8b00363
    44. 44
      Tran-Nguyen, V. K.; Simeon, S.; Junaid, M.; Ballester, P. J. Structure-Based Virtual Screening for PDL1 Dimerizers: Evaluating Generic Scoring Functions. Curr. Res. Struct. Biol. 2022, 4, 206210,  DOI: 10.1016/j.crstbi.2022.06.002
    45. 45
      Shen, C.; Weng, G.; Zhang, X.; Leung, E. L. H.; Yao, X.; Pang, J.; Chai, X.; Li, D.; Wang, E.; Cao, D.; Hou, T. Accuracy or Novelty: What Can We Gain from Target-Specific Machine-Learning-Based Scoring Functions in Virtual Screening? Brief. Bioinform. 2021, 22.  DOI: 10.1093/bib/bbaa410 .
    46. 46
      Ballester, P. J.; Finn, P. W.; Richards, W. G. Ultrafast Shape Recognition: Evaluating a New Ligand-Based Virtual Screening Technology. J. Mol. Graph. Model. 2009, 27, 836845,  DOI: 10.1016/j.jmgm.2009.01.001

Pair your accounts.

Export articles to Mendeley

Get article recommendations from ACS based on references in your Mendeley library.

Pair your accounts.

Export articles to Mendeley

Get article recommendations from ACS based on references in your Mendeley library.

You’ve supercharged your research process with ACS and Mendeley!

STEP 1:
Click to create an ACS ID

Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

MENDELEY PAIRING EXPIRED
Your Mendeley pairing has expired. Please reconnect