Multitask and Multimodal Neural Network Model for Interpretable Analysis of X-ray Images

Ivan Rodin, Irina Fedulova, Artem Shelmanov, Dmitry V. Dylov

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    2 Citations (Scopus)

    Abstract

    The quality and interpretability of the state-of-the-art methods for automatic analysis of chest X-ray images is still not sufficient. We address this problem by presenting a model that combines the analysis of frontal chest X-ray scans with structured patient information contained within radiology records. The proposed model generates a short textual summary with essential information on the found pathologies along with their location and severity; and the 2D heatmaps localizing each pathology on the original X-ray images. We test the proposed model on the MIMIC-CXR dataset. It achieves the state-of-the-art performance for image labelling and captioning (78.5% of correctly generated sentences) and defeats other similar solutions that dismiss the additional patient data (by 5.2% of correctly generated sentences). We also propose an automatic approach to label mining that leverages multimodal data: the X-ray images, related textual reports, patients' age and sex.

    Original languageEnglish
    Title of host publicationProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
    EditorsIllhoi Yoo, Jinbo Bi, Xiaohua Tony Hu
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages1601-1604
    Number of pages4
    ISBN (Electronic)9781728118673
    DOIs
    Publication statusPublished - Nov 2019
    Event2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019 - San Diego, United States
    Duration: 18 Nov 201921 Nov 2019

    Publication series

    NameProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

    Conference

    Conference2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
    Country/TerritoryUnited States
    CitySan Diego
    Period18/11/1921/11/19

    Keywords

    • chest X-ray
    • image captioning
    • localization map

    Fingerprint

    Dive into the research topics of 'Multitask and Multimodal Neural Network Model for Interpretable Analysis of X-ray Images'. Together they form a unique fingerprint.

    Cite this