Pair your accounts.

Export articles to Mendeley

Get article recommendations from ACS based on references in your Mendeley library.

Pair your accounts.

Export articles to Mendeley

Get article recommendations from ACS based on references in your Mendeley library.

You’ve supercharged your research process with ACS and Mendeley!

STEP 1:
Click to create an ACS ID

Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

Please note: If you switch to a different device, you may be asked to login again with only your ACS ID.

MENDELEY PAIRING EXPIRED
Your Mendeley pairing has expired. Please reconnect
ACS Publications. Most Trusted. Most Cited. Most Read
My Activity
CONTENT TYPES

Deep Learning-Based Annotation Transfer between Molecular Imaging Modalities: An Automated Workflow for Multimodal Data Integration

  • Alan M. Race*
    Alan M. Race
    Imaging and AI, Clinical Pharmacology and Safety Sciences, BioPharmaceuticals R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
    *Email: [email protected]. Phone: +49 (0)6421 28-65797. Fax: +49 (0)6421 28-68921.
    More by Alan M. Race
  • Daniel Sutton
    Daniel Sutton
    Imaging and AI, Clinical Pharmacology and Safety Sciences, BioPharmaceuticals R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
  • Gregory Hamm
    Gregory Hamm
    Imaging and AI, Clinical Pharmacology and Safety Sciences, BioPharmaceuticals R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
    More by Gregory Hamm
  • Gareth Maglennon
    Gareth Maglennon
    Oncology Safety, Clinical Pharmacology and Safety Sciences, BioPharmaceuticals R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
  • Jennifer P. Morton
    Jennifer P. Morton
    Cancer Research UK Beatson Institute, Garscube Estate, Switchback Road, Glasgow G61 1BD, U.K.
    Institute of Cancer Sciences, University of Glasgow, Garscube Estate, Switchback Road, Glasgow G61 1QH, U.K.
  • Nicole Strittmatter
    Nicole Strittmatter
    Imaging and AI, Clinical Pharmacology and Safety Sciences, BioPharmaceuticals R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
  • Andrew Campbell
    Andrew Campbell
    Cancer Research UK Beatson Institute, Garscube Estate, Switchback Road, Glasgow G61 1BD, U.K.
  • Owen J. Sansom
    Owen J. Sansom
    Cancer Research UK Beatson Institute, Garscube Estate, Switchback Road, Glasgow G61 1BD, U.K.
    Institute of Cancer Sciences, University of Glasgow, Garscube Estate, Switchback Road, Glasgow G61 1QH, U.K.
  • Yinhai Wang
    Yinhai Wang
    Discovery Sciences, R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
    More by Yinhai Wang
  • Simon T. Barry
    Simon T. Barry
    Bioscience, Early Oncology, AstraZeneca, Cambridge CB4 0WG, U.K.
  • Zoltan Takáts
    Zoltan Takáts
    Department of Surgery and Cancer, Imperial College London, London SW7 2AZ, U.K.
  • Richard J. A. Goodwin
    Richard J. A. Goodwin
    Imaging and AI, Clinical Pharmacology and Safety Sciences, BioPharmaceuticals R&D, AstraZeneca, Cambridge CB4 0WG, U.K.
    Institute of Infection, Immunity and Inflammation, College of Medical, Veterinary and Life Sciences, University of Glasgow, Glasgow G12 8QQ, U.K.
  • , and 
  • Josephine Bunch
    Josephine Bunch
    Department of Surgery and Cancer, Imperial College London, London SW7 2AZ, U.K.
    National Centre of Excellence in Mass Spectrometry Imaging (NiCE-MSI), National Physical Laboratory, Teddington TW11 0LW, U.K.
Cite this: Anal. Chem. 2021, 93, 6, 3061–3071
Publication Date (Web):February 3, 2021
https://doi.org/10.1021/acs.analchem.0c02726
Copyright © 2021 American Chemical Society

    Article Views

    2867

    Altmetric

    -

    Citations

    LEARN ABOUT THESE METRICS
    Other access options
    Supporting Info (1)»

    Abstract

    Abstract Image

    An ever-increasing array of imaging technologies are being used in the study of complex biological samples, each of which provides complementary, occasionally overlapping information at different length scales and spatial resolutions. It is important to understand the information provided by one technique in the context of the other to achieve a more holistic overview of such complex samples. One way to achieve this is to use annotations from one modality to investigate additional modalities. For microscopy-based techniques, these annotations could be manually generated using digital pathology software or automatically generated by machine learning (including deep learning) methods. Here, we present a generic method for using annotations from one microscopy modality to extract information from complementary modalities. We also present a fast, general, multimodal registration workflow [evaluated on multiple mass spectrometry imaging (MSI) modalities, matrix-assisted laser desorption/ionization, desorption electrospray ionization, and rapid evaporative ionization mass spectrometry] for automatic alignment of complex data sets, demonstrating an order of magnitude speed-up compared to previously published work. To demonstrate the power of the annotation transfer and multimodal registration workflows, we combine MSI, histological staining (such as hematoxylin and eosin), and deep learning (automatic annotation of histology images) to investigate a pancreatic cancer mouse model. Neoplastic pancreatic tissue regions, which were histologically indistinguishable from one another, were observed to be metabolically different. We demonstrate the use of the proposed methods to better understand tumor heterogeneity and the tumor microenvironment by transferring machine learning results freely between the two modalities.

    Read this article

    To access this article, please review the available access options below.

    Get instant access

    Purchase Access

    Read this article for 48 hours. Check out below using your ACS ID or as a guest.

    Recommended

    Access through Your Institution

    You may have access to this article through your institution.

    Your institution does not have access to this content. You can change your affiliated institution below.

    Supporting Information

    ARTICLE SECTIONS
    Jump To

    The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.analchem.0c02726.

    • Annotation transfer, H&E images for each classification and cluster, ion images and corresponding tentative identifications for top 20 ion images which correlate with each cluster, and validation of the registration process through comparison to literature (PDF)

    Terms & Conditions

    Most electronic Supporting Information files are available without a subscription to ACS Web Editions. Such files may be downloaded by article for research use (if there is a public use license linked to the relevant article, that license may permit other uses). Permission may be obtained from ACS for other uses through requests via the RightsLink permission system: http://pubs.acs.org/page/copyright/permissions.html.

    Cited By

    This article is cited by 30 publications.

    1. Bo-Jhang Lin, Tien-Chueh Kuo, Hsin-Hsiang Chung, Ying-Chen Huang, Ming-Yang Wang, Cheng-Chih Hsu, Po-Yang Yao, Yufeng Jane Tseng. MSIr: Automatic Registration Service for Mass Spectrometry Imaging and Histology. Analytical Chemistry 2023, 95 (6) , 3317-3324. https://doi.org/10.1021/acs.analchem.2c04360
    2. Zixuan Wang, Yaxin Zhang, Runtao Tian, Zhigang Luo, Ruiping Zhang, Xin Li, Zeper Abliz. Data-Driven Deciphering of Latent Lesions in Heterogeneous Tissue Using Function-Directed t-SNE of Mass Spectrometry Imaging Data. Analytical Chemistry 2022, 94 (40) , 13927-13935. https://doi.org/10.1021/acs.analchem.2c02990
    3. Wenyong Wu, Jinjun Hou, Zijia Zhang, Feifei Li, Rong Zhang, Lei Gao, Hui Ni, Tengqian Zhang, Huali Long, Min Lei, Bing Shen, Jun Yan, Ruimin Huang, Zhongda Zeng, Wanying Wu. Information Entropy-Based Strategy for the Quantitative Evaluation of Extensive Hyperspectral Images to Better Unveil Spatial Heterogeneity in Mass Spectrometry Imaging. Analytical Chemistry 2022, 94 (29) , 10355-10366. https://doi.org/10.1021/acs.analchem.2c00370
    4. Julia Kokesch-Himmelreich, Axel Treu, Alan M. Race, Kerstin Walter, Christoph Hölscher, Andreas Römpp. Do Anti-tuberculosis Drugs Reach Their Target?─High-Resolution Matrix-Assisted Laser Desorption/Ionization Mass Spectrometry Imaging Provides Information on Drug Penetration into Necrotic Granulomas. Analytical Chemistry 2022, 94 (14) , 5483-5492. https://doi.org/10.1021/acs.analchem.1c03462
    5. Stefania-Alexandra Iakab, Gerard Baquer, Marta Lafuente, Maria Pilar Pina, José Luis Ramírez, Pere Ràfols, Xavier Correig-Blanchar, María García-Altares. SALDI-MS and SERS Multimodal Imaging: One Nanostructured Substrate to Rule Them Both. Analytical Chemistry 2022, 94 (6) , 2785-2793. https://doi.org/10.1021/acs.analchem.1c04118
    6. Nicole Strittmatter, Frances M. Richards, Alan M. Race, Stephanie Ling, Daniel Sutton, Anna Nilsson, Yann Wallez, Jennifer Barnes, Gareth Maglennon, Aarthi Gopinathan, Rebecca Brais, Edmond Wong, Maria Paola Serra, James Atkinson, Aaron Smith, Joanne Wilson, Gregory Hamm, Timothy I. Johnson, Charles R. Dunlop, Brajesh P. Kaistha, Josephine Bunch, Owen J. Sansom, Zoltan Takats, Per E. Andrén, Alan Lau, Simon T. Barry, Richard J. A. Goodwin, Duncan I. Jodrell. Method To Visualize the Intratumor Distribution and Impact of Gemcitabine in Pancreatic Ductal Adenocarcinoma by Multimodal Imaging. Analytical Chemistry 2022, 94 (3) , 1795-1803. https://doi.org/10.1021/acs.analchem.1c04579
    7. Nicole Strittmatter, Panchali Kanvatirth, Paolo Inglese, Alan M. Race, Anna Nilsson, Andreas Dannhorn, Hiromi Kudo, Robert D. Goldin, Stephanie Ling, Edmond Wong, Frank Seeliger, Maria Paola Serra, Scott Hoffmann, Gareth Maglennon, Gregory Hamm, James Atkinson, Stewart Jones, Josephine Bunch, Per E. Andrén, Zoltan Takats, Richard J. A. Goodwin, Pietro Mastroeni. Holistic Characterization of a Salmonella Typhimurium Infection Model Using Integrated Molecular Imaging. Journal of the American Society for Mass Spectrometry 2021, 32 (12) , 2791-2802. https://doi.org/10.1021/jasms.1c00240
    8. Ang Guo, Zhiyu Chen, Yinzhong Ma, Yueguang Lv, Huanhuan Yan, Fang Li, Yao Xing, Qian Luo, Hairong Zheng. SOmicsFusion: Multimodal coregistration and fusion between spatial metabolomics and biomedical imaging. Artificial Intelligence Chemistry 2024, 2 (1) , 100058. https://doi.org/10.1016/j.aichem.2024.100058
    9. Andreas Dannhorn, Emine Kazanc, Lucy Flint, Fei Guo, Alfie Carter, Andrew R. Hall, Stewart A. Jones, George Poulogiannis, Simon T. Barry, Owen J. Sansom, Josephine Bunch, Zoltan Takats, Richard J. A. Goodwin. Morphological and molecular preservation through universal preparation of fresh-frozen tissue samples for multimodal imaging workflows. Nature Protocols 2024, 6 https://doi.org/10.1038/s41596-024-00987-z
    10. Guang Xu, Shengfeng Gan, Bo Guo, Li Yang. Application of clustering strategy for automatic segmentation of tissue regions in mass spectrometry imaging. Rapid Communications in Mass Spectrometry 2024, 38 (8) https://doi.org/10.1002/rcm.9717
    11. Kyle D Duncan, Helena Pětrošová, Julian J Lum, David R Goodlett. Mass spectrometry imaging methods for visualizing tumor heterogeneity. Current Opinion in Biotechnology 2024, 86 , 103068. https://doi.org/10.1016/j.copbio.2024.103068
    12. Yuzhong Zhou, Zhengping Lin, Zhengrong Wu, Zifeng Zhang. Knowledge fusion method of power grid model based on Seq2seq half pointer and half label method. Journal of Intelligent & Fuzzy Systems 2024, 46 (3) , 6939-6950. https://doi.org/10.3233/JIFS-236465
    13. Gerard Baquer, Lluc Sementé, Pere Ràfols, Lucía Martín-Saiz, Christoph Bookmeyer, José A. Fernández, Xavier Correig, María García-Altares. rMSIfragment: improving MALDI-MSI lipidomics through automated in-source fragment annotation. Journal of Cheminformatics 2023, 15 (1) https://doi.org/10.1186/s13321-023-00756-2
    14. Katerina V. Djambazova, Jacqueline M. Van Ardenne, Jeffrey M. Spraggins. Advances in imaging mass spectrometry for biomedical and clinical research. TrAC Trends in Analytical Chemistry 2023, 169 , 117344. https://doi.org/10.1016/j.trac.2023.117344
    15. Hsin‐Hsiang Chung, Penghsuan Huang, Chih‐Lin Chen, Chuping Lee, Cheng‐Chih Hsu. Next‐generation pathology practices with mass spectrometry imaging. Mass Spectrometry Reviews 2023, 42 (6) , 2446-2465. https://doi.org/10.1002/mas.21795
    16. Behnaz Akbari, Bertrand Russell Huber, Janet Hope Sherman. Unlocking the Hidden Depths: Multi-Modal Integration of Imaging Mass Spectrometry-Based and Molecular Imaging Techniques. Critical Reviews in Analytical Chemistry 2023, 17 , 1-30. https://doi.org/10.1080/10408347.2023.2266838
    17. Gerard Baquer, Lluc Sementé, Toufik Mahamdi, Xavier Correig, Pere Ràfols, María García‐Altares. What are we imaging? Software tools and experimental strategies for annotation and identification of small molecules in mass spectrometry imaging. Mass Spectrometry Reviews 2023, 42 (5) , 1927-1964. https://doi.org/10.1002/mas.21794
    18. Emre M. Isin. Unusual Biotransformation Reactions of Drugs and Drug Candidates. Drug Metabolism and Disposition 2023, 51 (4) , 413-426. https://doi.org/10.1124/dmd.121.000744
    19. Hang Hu, Julia Laskin. Emerging Computational Methods in Mass Spectrometry Imaging. Advanced Science 2022, 9 (34) https://doi.org/10.1002/advs.202203339
    20. Jyoti Dhanke, M. Pradeepa, R. Karthik, Veeresh Rampur, I. Poonguzhali, Hemanand Chittapragada. Heterogeneous sensor data fusion acquisition model for medical applications. Measurement: Sensors 2022, 24 , 100552. https://doi.org/10.1016/j.measen.2022.100552
    21. Yixuan Qiao, Lianhe Zhao, Chunlong Luo, Yufan Luo, Yang Wu, Shengtong Li, Dechao Bu, Yi Zhao. Multi-modality artificial intelligence in digital pathology. Briefings in Bioinformatics 2022, 23 (6) https://doi.org/10.1093/bib/bbac367
    22. Alex Dexter, Dimitrios Tsikritsis, Natalie A. Belsey, Spencer A. Thomas, Jenny Venton, Josephine Bunch, Marina Romanchikova. Next Generation Digital Pathology: Emerging Trends and Measurement Challenges for Molecular Pathology. Journal of Molecular Pathology 2022, 3 (3) , 168-181. https://doi.org/10.3390/jmp3030014
    23. Yiqin Wang, Qiong Wen, Luhua Jin, Wei Chen. Artificial Intelligence-Assisted Renal Pathology: Advances and Prospects. Journal of Clinical Medicine 2022, 11 (16) , 4918. https://doi.org/10.3390/jcm11164918
    24. Akhila Ajith, Phillip J. Milnes, Giles N. Johnson, Nicholas P. Lockyer. Mass Spectrometry Imaging for Spatial Chemical Profiling of Vegetative Parts of Plants. Plants 2022, 11 (9) , 1234. https://doi.org/10.3390/plants11091234
    25. Walid M Abdelmoula, Sylwia A Stopka, Elizabeth C Randall, Michael Regan, Jeffrey N Agar, Jann N Sarkaria, William M Wells, Tina Kapur, Nathalie Y R Agar, . massNet: integrated processing and classification of spatially resolved mass spectrometry data using deep learning for rapid tumor delineation. Bioinformatics 2022, 38 (7) , 2015-2021. https://doi.org/10.1093/bioinformatics/btac032
    26. Graham F. Smith. Artificial Intelligence in Drug Safety and Metabolism. 2022, 483-501. https://doi.org/10.1007/978-1-0716-1787-8_22
    27. Benjamin Balluff, Ron M.A. Heeren, Alan M. Race. An overview of image registration for aligning mass spectrometry imaging with clinically relevant imaging modalities. Journal of Mass Spectrometry and Advances in the Clinical Lab 2022, 23 , 26-38. https://doi.org/10.1016/j.jmsacl.2021.12.006
    28. Bruno Debus, Hadi Parastar, Peter Harrington, Dmitry Kirsanov. Deep learning in analytical chemistry. TrAC Trends in Analytical Chemistry 2021, 145 , 116459. https://doi.org/10.1016/j.trac.2021.116459
    29. G. Guo, M. Papanicolaou, N. J. Demarais, Z. Wang, K. L. Schey, P. Timpson, T. R. Cox, A. C. Grey. Automated annotation and visualisation of high-resolution spatial proteomic mass spectrometry imaging data using HIT-MAP. Nature Communications 2021, 12 (1) https://doi.org/10.1038/s41467-021-23461-w
    30. Peggi M. Angel, Denys Rujchanarong, Sarah Pippin, Laura Spruill, Richard Drake. Mass Spectrometry Imaging of Fibroblasts: Promise and Challenge. Expert Review of Proteomics 2021, 18 (6) , 423-436. https://doi.org/10.1080/14789450.2021.1941893