ACS Publications. Most Trusted. Most Cited. Most Read
My Activity
CONTENT TYPES

Figure 1Loading Img

Applicability Domain Analysis (ADAN): A Robust Method for Assessing the Reliability of Drug Property Predictions

View Author Information
Research Programme on Biomedical Informatics (GRIB), Department of Experimental and Health Sciences, Universitat Pompeu Fabra, IMIM (Hospital del Mar Medical Research Institute), Dr. Aiguader, 88, E-08003 Barcelona, Spain
§ Department of Medicinal Chemistry, University of Vienna, Althanstrasse 14, 1090 Vienna, Austria
Cite this: J. Chem. Inf. Model. 2014, 54, 5, 1500–1511
Publication Date (Web):May 1, 2014
https://doi.org/10.1021/ci500172z
Copyright © 2014 American Chemical Society

    Article Views

    1279

    Altmetric

    -

    Citations

    LEARN ABOUT THESE METRICS
    Other access options
    Supporting Info (1)»

    Abstract

    Abstract Image

    We report a novel method called ADAN (Applicability Domain ANalysis) for assessing the reliability of drug property predictions obtained by in silico methods. The assessment provided by ADAN is based on the comparison of the query compound with the training set, using six diverse similarity criteria. For every criterion, the query compound is considered out of range when the similarity value obtained is larger than the 95th percentile of the values obtained for the training set. The final outcome is a number in the range of 0–6 that expresses the number of unmet similarity criteria and allows classifying the query compound within seven reliability categories. Such categories can be further exploited to assign simpler reliability classes using a traffic light schema, to assign approximate confidence intervals or to mark the predictions as unreliable. The entire methodology has been validated simulating realistic conditions, where query compounds are structurally diverse from those in the training set. The validation exercise involved the construction of more than 1000 models. These models were built using a combination of training set, molecular descriptors, and modeling methods representative of the real predictive tasks performed in the eTOX project (a project whose objective is to predict in vivo toxicological end points in drug development). Validation results confirm the robustness of the proposed assessment methodology, which compares favorably with other classical methods based solely on the structural similarity of the compounds. ADAN characteristics make the method well-suited for estimate the quality of drug predictions obtained in extremely unfavorable conditions, like the prediction of drug toxicity end points.

    Read this article

    To access this article, please review the available access options below.

    Get instant access

    Purchase Access

    Read this article for 48 hours. Check out below using your ACS ID or as a guest.

    Recommended

    Access through Your Institution

    You may have access to this article through your institution.

    Your institution does not have access to this content. You can change your affiliated institution below.

    Supporting Information

    ARTICLE SECTIONS
    Jump To

    Activity values centered to the mean (Figure S1); compounds inside the defined CI for RF modeling (Figures S2 and S3); count of compounds per ADAN category with PLS modeling (Figure S4); count of compounds per ADAN category with RF modeling (Figure S5); prediction error for PLS and RF (Figure S6); ADAN and prediction error example (Figure S7); phi coefficients between ADAN criteria pairs for all studied series (PLS modeling) (Figure S8); phi coefficients between ADAN criteria pairs for all studied series (RF modeling) (Figure S9). Average predictive quality of the PLS models used for the ADAN validation grouped by datasets and molecular descriptors (Table S1). Average predictive quality of the RF models used for the ADAN validation grouped by datasets and molecular descriptors (Table S2). This material is available free of charge via the Internet at http://pubs.acs.org.

    Terms & Conditions

    Most electronic Supporting Information files are available without a subscription to ACS Web Editions. Such files may be downloaded by article for research use (if there is a public use license linked to the relevant article, that license may permit other uses). Permission may be obtained from ACS for other uses through requests via the RightsLink permission system: http://pubs.acs.org/page/copyright/permissions.html.

    Cited By

    This article is cited by 46 publications.

    1. Benjamin P. Brown, Jeffrey Mendenhall, Alexander R. Geanes, Jens Meiler. General Purpose Structure-Based Drug Discovery Neural Network Score Functions with Human-Interpretable Pharmacophore Maps. Journal of Chemical Information and Modeling 2021, 61 (2) , 603-620. https://doi.org/10.1021/acs.jcim.0c01001
    2. Anke Wilm, Ulf Norinder, M. Isabel Agea, Christina de Bruyn Kops, Conrad Stork, Jochen Kühnl, Johannes Kirchmair. Skin Doctor CP: Conformal Prediction of the Skin Sensitization Potential of Small Organic Molecules. Chemical Research in Toxicology 2021, 34 (2) , 330-344. https://doi.org/10.1021/acs.chemrestox.0c00253
    3. Eni Minerali, Daniel H. Foil, Kimberley M. Zorn, Thomas R. Lane, Sean Ekins. Comparing Machine Learning Algorithms for Predicting Drug-Induced Liver Injury (DILI). Molecular Pharmaceutics 2020, 17 (7) , 2628-2637. https://doi.org/10.1021/acs.molpharmaceut.0c00326
    4. Alejandro Speck-Planche and M. Natália Dias Soeiro Cordeiro . Speeding up Early Drug Discovery in Antiviral Research: A Fragment-Based in Silico Approach for the Design of Virtual Anti-Hepatitis C Leads. ACS Combinatorial Science 2017, 19 (8) , 501-512. https://doi.org/10.1021/acscombsci.7b00039
    5. Robert P. Sheridan . The Relative Importance of Domain Applicability Metrics for Estimating Prediction Errors in QSAR Varies with Training Set Diversity. Journal of Chemical Information and Modeling 2015, 55 (6) , 1098-1107. https://doi.org/10.1021/acs.jcim.5b00110
    6. Kajjana Boonpalit, Jiramet Kinchagawat, Supawadee Namuangruk. Expanding the Applicability Domain of Machine Learning Model for Advancements in Electrochemical Material Discovery. ChemElectroChem 2024, 11 (10) https://doi.org/10.1002/celc.202300681
    7. Viet-Khoa Tran-Nguyen, Muhammad Junaid, Saw Simeon, Pedro J. Ballester. A practical guide to machine-learning scoring for structure-based virtual screening. Nature Protocols 2023, 18 (11) , 3460-3511. https://doi.org/10.1038/s41596-023-00885-w
    8. Jie Yu, Dingyan Wang, Mingyue Zheng. Uncertainty quantification: Can we trust artificial intelligence in drug discovery?. iScience 2022, 25 (8) , 104814. https://doi.org/10.1016/j.isci.2022.104814
    9. Zakari Ya'u Ibrahim, Adamu Uzairu, Gideon Adamu Shallangwa, Stephen Eyije Abechi, Sulaiman Isyaku. Quantitative Structure-Activity Relationship, Structure-Based Design, And ADMET Studies of Pyrimethamine and Cycloguanil Analogues Inhibitors of Plasmodium Falciparum Dihydrofolate Reductase-Thymidylate Synthase (Pf-DHFR-TS). SSRN Electronic Journal 2022, 2 https://doi.org/10.2139/ssrn.4153714
    10. David Z Huang, J. Christian Baber, Sogole Sami Bahmanyar. The challenges of generalizability in artificial intelligence for ADME/Tox endpoint and activity prediction. Expert Opinion on Drug Discovery 2021, 16 (9) , 1045-1056. https://doi.org/10.1080/17460441.2021.1901685
    11. Lewis H. Mervin, Simon Johansson, Elizaveta Semenova, Kathryn A. Giblin, Ola Engkvist. Uncertainty quantification in drug design. Drug Discovery Today 2021, 26 (2) , 474-489. https://doi.org/10.1016/j.drudis.2020.11.027
    12. Tom M. Nolte, Thomas Nauser, Lorenz Gubler, A. Jan Hendriks, Willie J. G. M. Peijnenburg. Thermochemical unification of molecular descriptors to predict radical hydrogen abstraction with low computational cost. Physical Chemistry Chemical Physics 2020, 22 (40) , 23215-23225. https://doi.org/10.1039/D0CP03750H
    13. Nicola Gilmour, Petra S. Kern, Nathalie Alépée, Fanny Boislève, Dagmar Bury, Elodie Clouet, Morihiko Hirota, Sebastian Hoffmann, Jochen Kühnl, Jon F. Lalko, Karsten Mewes, Masaaki Miyazawa, Hayato Nishida, Anne Osmani, Dirk Petersohn, Shuichi Sekine, Erwin van Vliet, Martina Klaric. Development of a next generation risk assessment framework for the evaluation of skin sensitisation of cosmetic ingredients. Regulatory Toxicology and Pharmacology 2020, 116 , 104721. https://doi.org/10.1016/j.yrtph.2020.104721
    14. Eko Aditya Rifai, Marc van Dijk, Daan P. Geerke. Recent Developments in Linear Interaction Energy Based Binding Free Energy Calculations. Frontiers in Molecular Biosciences 2020, 7 https://doi.org/10.3389/fmolb.2020.00114
    15. T. M. Nolte, W. J. G. M. Peijnenburg, T. J. H. M. van Bergen, A. J. Hendriks. Transition-state rate theory sheds light on ‘black-box’ biodegradation algorithms. Green Chemistry 2020, 22 (11) , 3558-3571. https://doi.org/10.1039/D0GC00337A
    16. Stanley E Lazic, Dominic P Williams. Improving drug safety predictions by reducing poor analytical practices. Toxicology Research and Application 2020, 4 , 239784732097863. https://doi.org/10.1177/2397847320978633
    17. Alexander Amberg, Lennart T Anger, Joel Bercu, David Bower, Kevin P Cross, Laura Custer, James S Harvey, Catrin Hasselgren, Masamitsu Honma, Candice Johnson, Robert Jolly, Michelle O Kenyon, Naomi L Kruhlak, Penny Leavitt, Donald P Quigley, Scott Miller, David Snodin, Lidiya Stavitskaya, Andrew Teasdale, Alejandra Trejo-Martin, Angela T White, Joerg Wichard, Glenn J Myatt. Extending (Q)SARs to incorporate proprietary knowledge for regulatory purposes: is aromatic N -oxide a structural alert for predicting DNA-reactive mutagenicity?*. Mutagenesis 2019, 34 (1) , 67-82. https://doi.org/10.1093/mutage/gey020
    18. Alexander Amberg, Roxanne V. Andaya, Lennart T. Anger, Chris Barber, Lisa Beilke, Joel Bercu, Dave Bower, Alessandro Brigo, Zoryanna Cammerer, Kevin P. Cross, Laura Custer, Krista Dobo, Helga Gerets, Véronique Gervais, Susanne Glowienke, Stephen Gomez, Jacky Van Gompel, James Harvey, Catrin Hasselgren, Masamitsu Honma, Candice Johnson, Robert Jolly, Raymond Kemper, Michelle Kenyon, Naomi Kruhlak, Penny Leavitt, Scott Miller, Wolfgang Muster, Russell Naven, John Nicolette, Alexis Parenty, Mark Powley, Donald P. Quigley, M. Vijayaraj Reddy, Jennifer C. Sasaki, Lidiya Stavitskaya, Andrew Teasdale, Alejandra Trejo-Martin, Sandy Weiner, Dennie S. Welch, Angela White, Joerg Wichard, David Woolley, Glenn J. Myatt. Principles and procedures for handling out-of-domain and indeterminate results as part of ICH M7 recommended (Q)SAR analyses. Regulatory Toxicology and Pharmacology 2019, 102 , 53-64. https://doi.org/10.1016/j.yrtph.2018.12.007
    19. Sankalp Jain, Gerhard F. Ecker. In Silico Approaches to Predict Drug-Transporter Interaction Profiles: Data Mining, Model Generation, and Link to Cholestasis. 2019, 383-396. https://doi.org/10.1007/978-1-4939-9420-5_26
    20. Thierry Hanser, Chris Barber, Sébastien Guesné, Jean François Marchaland, Stéphane Werner. Applicability Domain: Towards a More Formal Framework to Express the Applicability of a Model and the Confidence in Individual Predictions. 2019, 215-232. https://doi.org/10.1007/978-3-030-16443-0_11
    21. Manuel Pastor, Jordi Quintana, Ferran Sanz. Development of an Infrastructure for the Prediction of Biological Endpoints in Industrial Environments. Lessons Learned at the eTOX Project. Frontiers in Pharmacology 2018, 9 https://doi.org/10.3389/fphar.2018.01147
    22. Glenn J. Myatt, Ernst Ahlberg, Yumi Akahori, David Allen, Alexander Amberg, Lennart T. Anger, Aynur Aptula, Scott Auerbach, Lisa Beilke, Phillip Bellion, Romualdo Benigni, Joel Bercu, Ewan D. Booth, Dave Bower, Alessandro Brigo, Natalie Burden, Zoryana Cammerer, Mark T.D. Cronin, Kevin P. Cross, Laura Custer, Magdalena Dettwiler, Krista Dobo, Kevin A. Ford, Marie C. Fortin, Samantha E. Gad-McDonald, Nichola Gellatly, Véronique Gervais, Kyle P. Glover, Susanne Glowienke, Jacky Van Gompel, Steve Gutsell, Barry Hardy, James S. Harvey, Jedd Hillegass, Masamitsu Honma, Jui-Hua Hsieh, Chia-Wen Hsu, Kathy Hughes, Candice Johnson, Robert Jolly, David Jones, Ray Kemper, Michelle O. Kenyon, Marlene T. Kim, Naomi L. Kruhlak, Sunil A. Kulkarni, Klaus Kümmerer, Penny Leavitt, Bernhard Majer, Scott Masten, Scott Miller, Janet Moser, Moiz Mumtaz, Wolfgang Muster, Louise Neilson, Tudor I. Oprea, Grace Patlewicz, Alexandre Paulino, Elena Lo Piparo, Mark Powley, Donald P. Quigley, M. Vijayaraj Reddy, Andrea-Nicole Richarz, Patricia Ruiz, Benoit Schilter, Rositsa Serafimova, Wendy Simpson, Lidiya Stavitskaya, Reinhard Stidl, Diana Suarez-Rodriguez, David T. Szabo, Andrew Teasdale, Alejandra Trejo-Martin, Jean-Pierre Valentin, Anna Vuorinen, Brian A. Wall, Pete Watts, Angela T. White, Joerg Wichard, Kristine L. Witt, Adam Woolley, David Woolley, Craig Zwickl, Catrin Hasselgren. In silico toxicology protocols. Regulatory Toxicology and Pharmacology 2018, 96 , 1-17. https://doi.org/10.1016/j.yrtph.2018.04.014
    23. Oriol López-Massaguer, Kevin Pinto-Gil, Ferran Sanz, Alexander Amberg, Lennart T Anger, Manuela Stolte, Carlo Ravagli, Philippe Marc, Manuel Pastor. Generating Modeling Data From Repeat-Dose Toxicity Reports. Toxicological Sciences 2018, 162 (1) , 287-300. https://doi.org/10.1093/toxsci/kfx254
    24. Eko Aditya Rifai, Marc van Dijk, Nico P. E. Vermeulen, Daan P. Geerke. Binding free energy predictions of farnesoid X receptor (FXR) agonists using a linear interaction energy (LIE) approach with reliability estimation: application to the D3R Grand Challenge 2. Journal of Computer-Aided Molecular Design 2018, 32 (1) , 239-249. https://doi.org/10.1007/s10822-017-0055-0
    25. Tom M. Nolte, Kevin Pinto-Gil, A. Jan Hendriks, Ad M. J. Ragas, Manuel Pastor. Quantitative structure–activity relationships for primary aerobic biodegradation of organic chemicals in pristine surface waters: starting points for predicting biodegradation under acclimatization. Environmental Science: Processes & Impacts 2018, 20 (1) , 157-170. https://doi.org/10.1039/C7EM00375G
    26. , Ferran Sanz, François Pognan, Thomas Steger-Hartmann, Carlos Díaz, Montserrat Cases, Manuel Pastor, Philippe Marc, Joerg Wichard, Katharine Briggs, David K. Watson, Thomas Kleinöder, Chihae Yang, Alexander Amberg, Maria Beaumont, Anthony J. Brookes, Søren Brunak, Mark T. D. Cronin, Gerhard F. Ecker, Sylvia Escher, Nigel Greene, Antonio Guzmán, Anne Hersey, Pascale Jacques, Lieve Lammens, Jordi Mestres, Wolfgang Muster, Helle Northeved, Marc Pinches, Javier Saiz, Nicolas Sajot, Alfonso Valencia, Johan van der Lei, Nico P. E. Vermeulen, Esther Vock, Gerhard Wolber, Ismael Zamora. Legacy data sharing to improve drug safety assessment: the eTOX project. Nature Reviews Drug Discovery 2017, 16 (12) , 811-812. https://doi.org/10.1038/nrd.2017.177
    27. Dong‐Sheng Cao, Zhen‐Ke Deng, Min‐Feng Zhu, Zhi‐Jiang Yao, Jie Dong, Rui‐Gang Zhao. Ensemble partial least squares regression for descriptor selection, outlier detection, applicability domain assessment, and ensemble modeling in QSAR/QSPR modeling. Journal of Chemometrics 2017, 31 (11) https://doi.org/10.1002/cem.2922
    28. Alejandro Speck-Planche, M. Natália D. S. Cordeiro. De novo computational design of compounds virtually displaying potent antibacterial activity and desirable in vitro ADMET profiles. Medicinal Chemistry Research 2017, 26 (10) , 2345-2356. https://doi.org/10.1007/s00044-017-1936-4
    29. Floriane Montanari, Barbara Zdrazil. How Open Data Shapes In Silico Transporter Modeling. Molecules 2017, 22 (3) , 422. https://doi.org/10.3390/molecules22030422
    30. V.V. Kleandrova, A. Speck-Planche. Multitasking Model for Computer-Aided Design and Virtual Screening of Compounds With High Anti-HIV Activity and Desirable ADMET Properties. 2017, 55-81. https://doi.org/10.1016/B978-0-08-101129-4.00003-5
    31. R.A. Lewis, F. Sirockin. 2/3D Pharmacophore Definitions and Their Application. 2017, 189-220. https://doi.org/10.1016/B978-0-12-409547-2.12346-7
    32. Tom M. Nolte, Ad M. J. Ragas. A review of quantitative structure–property relationships for the fate of ionizable organic chemicals in water matrices and identification of knowledge gaps. Environmental Science: Processes & Impacts 2017, 19 (3) , 221-246. https://doi.org/10.1039/C7EM00034K
    33. Natália Aniceto, Alex A. Freitas, Andreas Bender, Taravat Ghafourian. A novel applicability domain technique for mapping predictive reliability across the chemical space of a QSAR: reliability-density neighbourhood. Journal of Cheminformatics 2016, 8 (1) https://doi.org/10.1186/s13321-016-0182-y
    34. T. Hanser, C. Barber, J. F. Marchaland, S. Werner. Applicability domain: towards a more formal definition. SAR and QSAR in Environmental Research 2016, 27 (11) , 865-881. https://doi.org/10.1080/1062936X.2016.1250229
    35. Vishwesh Venkatraman, Mayuri Gupta, Marco Foscato, Hallvard F. Svendsen, Vidar R. Jensen, Bjørn K. Alsberg. Computer-aided molecular design of imidazole-based absorbents for CO 2 capture. International Journal of Greenhouse Gas Control 2016, 49 , 55-63. https://doi.org/10.1016/j.ijggc.2016.02.023
    36. Sailesh Abburu, Vishwesh Venkatraman, Bjørn K. Alsberg. TD-DFT based fine-tuning of molecular excitation energies using evolutionary algorithms. RSC Advances 2016, 6 (5) , 3661-3670. https://doi.org/10.1039/C5RA22800J
    37. Ravi Kumar, Kumardeep Chaudhary, Jagat Singh Chauhan, Gandharva Nagpal, Rahul Kumar, Minakshi Sharma, Gajendra P.S. Raghava. An in silico platform for predicting, screening and designing of antihypertensive peptides. Scientific Reports 2015, 5 (1) https://doi.org/10.1038/srep12512
    38. Peter Gedeck, Christian Kramer, Richard Lewis. In Silico Approaches to Address Compound Attrition. 2015, 264-286. https://doi.org/10.1002/9781118819586.ch10
    39. Pau Carrió, Oriol López, Ferran Sanz, Manuel Pastor. eTOXlab, an open source modeling framework for implementing predictive models in production environments. Journal of Cheminformatics 2015, 7 (1) https://doi.org/10.1186/s13321-015-0058-6
    40. Luigi Capoferri, Marlies C. A. Verkade-Vreeker, Danny Buitenhuis, Jan N. M. Commandeur, Manuel Pastor, Nico P. E. Vermeulen, Daan P. Geerke, . Linear Interaction Energy Based Prediction of Cytochrome P450 1A2 Binding Affinities with Reliability Estimation. PLOS ONE 2015, 10 (11) , e0142232. https://doi.org/10.1371/journal.pone.0142232
    41. Min Su, Jianjun Tan, Chun-Yuan Lin. Development of HIV-1 integrase inhibitors: recent molecular modeling perspectives. Drug Discovery Today 2015, 20 (11) , 1337-1348. https://doi.org/10.1016/j.drudis.2015.07.012
    42. Venkatesh Botu, Rampi Ramprasad. Adaptive machine learning framework to accelerate ab initio molecular dynamics. International Journal of Quantum Chemistry 2015, 115 (16) , 1074-1083. https://doi.org/10.1002/qua.24836
    43. Mark Hewitt, Claire M. Ellison, Mark T.D. Cronin, Manuel Pastor, Thomas Steger-Hartmann, Jordi Munoz-Muriendas, Francois Pognan, Judith C. Madden. Ensuring confidence in predictions: A scheme to assess the scientific validity of in silico models. Advanced Drug Delivery Reviews 2015, 86 , 101-111. https://doi.org/10.1016/j.addr.2015.03.005
    44. Ferran Sanz, Pau Carrió, Oriol López, Luigi Capoferri, Derk P. Kooi, Nico P. E. Vermeulen, Daan P. Geerke, Floriane Montanari, Gerhard F. Ecker, Christof H. Schwab, Thomas Kleinöder, Tomasz Magdziarz, Manuel Pastor. Integrative Modeling Strategies for Predicting Drug Toxicities at the eTOX Project. Molecular Informatics 2015, 34 (6-7) , 477-484. https://doi.org/10.1002/minf.201400193
    45. Vishwesh Venkatraman, Marco Foscato, Vidar R. Jensen, Bjørn Kåre Alsberg. Evolutionary de novo design of phenothiazine derivatives for dye-sensitized solar cells. Journal of Materials Chemistry A 2015, 3 (18) , 9851-9860. https://doi.org/10.1039/C5TA00625B
    46. Montserrat Cases, Katharine Briggs, Thomas Steger-Hartmann, François Pognan, Philippe Marc, Thomas Kleinöder, Christof Schwab, Manuel Pastor, Jörg Wichard, Ferran Sanz. The eTOX Data-Sharing Project to Advance in Silico Drug-Induced Toxicity Prediction. International Journal of Molecular Sciences 2014, 15 (11) , 21136-21154. https://doi.org/10.3390/ijms151121136

    Pair your accounts.

    Export articles to Mendeley

    Get article recommendations from ACS based on references in your Mendeley library.

    Pair your accounts.

    Export articles to Mendeley

    Get article recommendations from ACS based on references in your Mendeley library.

    You’ve supercharged your research process with ACS and Mendeley!

    STEP 1:
    Click to create an ACS ID

    Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

    Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

    Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

    MENDELEY PAIRING EXPIRED
    Your Mendeley pairing has expired. Please reconnect