Cart
Article
Evaluating Virtual Screening Methods: Good and Bad Metrics for the “Early Recognition” Problem
Full Text HTML
Hi-Res PDFPurchase the full-text
- PDF/HTML,
figures/images,
references and tables,
(where available)
Section:Abstract
Many metrics are currently used to evaluate the performance of ranking methods in virtual screening (VS), for instance, the area under the receiver operating characteristic curve (ROC), the area under the accumulation curve (AUAC), the average rank of actives, the enrichment factor (EF), and the robust initial enhancement (RIE) proposed by Sheridan et al. In this work, we show that the ROC, the AUAC, and the average rank metrics have the same inappropriate behaviors that make them poor metrics for comparing VS methods whose purpose is to rank actives early in an ordered list (the “early recognition problem”). In doing so, we derive mathematical formulas that relate those metrics together. Moreover, we show that the EF metric is not sensitive to ranking performance before and after the cutoff. Instead, we formally generalize the ROC metric to the early recognition problem which leads us to propose a novel metric called the Boltzmann-enhanced discrimination of receiver operating characteristic that turns out to contain the discrimination power of the RIE metric but incorporates the statistical significance from ROC and its well-behaved boundaries. Finally, two major sources of errors, namely, the statistical error and the “saturation effects”, are examined. This leads to practical recommendations for the number of actives, the number of inactives, and the “early recognition” importance parameter that one should use when comparing ranking methods. Although this work is applied specifically to VS, it is general and can be used to analyze any method that needs to segregate actives toward the front of a rank-ordered list.
Citing Articles
Citation data is made available by participants in CrossRef's Cited-by Linking service. For a more comprehensive list of citations to this article, users are encouraged to perform a search in SciFinder.
This article has been cited by 47 ACS Journal articles (5 most recent appear below).
Drug Effect Prediction by Polypharmacology-Based Interaction Profiling
Zoltán Simon, Ágnes Peragovics, Margit Vigh-Smeller, Gábor Csukly, László Tombor, Zhenhui Yang, Gergely Zahoránszky-Kőhalmi, László Végner, Balázs Jelinek, Péter Hári, Csaba Hetényi, István Bitter, Pál Czobor, and András Málnási-CsizmadiaJournal of Chemical Information and Modeling2012 52 (1), 134-145Drug Effect Prediction by Polypharmacology-Based Interaction Profiling
Zoltán Simon, Ágnes Peragovics, Margit Vigh-Smeller, Gábor Csukly, László Tombor, Zhenhui Yang, Gergely Zahoránszky-Kőhalmi, László Végner, Balázs Jelinek, Péter Hári, Csaba Hetényi, István Bitter, Pál Czobor, and András Málnási-CsizmadiaJournal of Chemical Information and Modeling2012 52 (1), 134-145Most drugs exert their effects via multitarget interactions, as hypothesized by polypharmacology. While these multitarget interactions are responsible for the clinical effect profiles of drugs, current methods have failed to uncover the complex ...
PLS-DA - Docking Optimized Combined Energetic Terms (PLSDA-DOCET) Protocol: A Brief Evaluation
Sorin Avram, Liliana M. Pacureanu, Edward Seclaman, Alina Bora, and Ludovic KuruncziJournal of Chemical Information and Modeling2011 51 (12), 3169-3179PLS-DA - Docking Optimized Combined Energetic Terms (PLSDA-DOCET) Protocol: A Brief Evaluation
Sorin Avram, Liliana M. Pacureanu, Edward Seclaman, Alina Bora, and Ludovic KuruncziJournal of Chemical Information and Modeling2011 51 (12), 3169-3179Docking studies have become popular approaches in drug design, where the binding energy of the ligand in the active site of the protein is estimated by a scoring function. Many promising techniques were developed to enhance the performance of scoring ...
Jointly Handling Potency and Toxicity of Antimicrobial Peptidomimetics by Simple Rules from Desirability Theory and Chemoinformatics
Maykel Cruz-Monteagudo, Fernanda Borges, and M. Natália D. S. CordeiroJournal of Chemical Information and Modeling2011 51 (12), 3060-3077Jointly Handling Potency and Toxicity of Antimicrobial Peptidomimetics by Simple Rules from Desirability Theory and Chemoinformatics
Maykel Cruz-Monteagudo, Fernanda Borges, and M. Natália D. S. CordeiroJournal of Chemical Information and Modeling2011 51 (12), 3060-3077Today, emerging and increasing resistance to antibiotics has become a threat to public health worldwide. Antimicrobial peptides have unique action mechanisms making them an attractive therapeutic prospect to be applied against resistant bacteria. However, ...
The Ensemble Performance Index: An Improved Measure for Assessing Ensemble Pose Prediction Performance
Oliver Korb, Patrick McCabe, and Jason ColeJournal of Chemical Information and Modeling2011 51 (11), 2915-2919The Ensemble Performance Index: An Improved Measure for Assessing Ensemble Pose Prediction Performance
Oliver Korb, Patrick McCabe, and Jason ColeJournal of Chemical Information and Modeling2011 51 (11), 2915-2919We present a theoretical study on the performance of ensemble docking methodologies considering multiple protein structures. We perform a theoretical analysis of pose prediction experiments which is completely unbiased, as we make no assumptions about ...
Rapid Shape-Based Ligand Alignment and Virtual Screening Method Based on Atom/Feature-Pair Similarities and Volume Overlap Scoring
G. Madhavi Sastry, Steven L. Dixon, and Woody ShermanJournal of Chemical Information and Modeling2011 51 (10), 2455-2466Rapid Shape-Based Ligand Alignment and Virtual Screening Method Based on Atom/Feature-Pair Similarities and Volume Overlap Scoring
G. Madhavi Sastry, Steven L. Dixon, and Woody ShermanJournal of Chemical Information and Modeling2011 51 (10), 2455-2466Shape-based methods for aligning and scoring ligands have proven to be valuable in the field of computer-aided drug design. Here, we describe a new shape-based flexible ligand superposition and virtual screening method, Phase Shape, which is shown to ...
Tools
-
Add to Favorites
-
Download Citation
-
Email a Colleague -
Permalink
Order Reprints
Rights & Permissions
Citation Alerts
History
- Published In Issue March 26, 2007
- Received October 2, 2006
ACS
Network
Facebook
Tweet This
CiteULike
Newsvine
Digg This
Delicious
