Multitask and Multimodal Neural Network Model for Interpretable Analysis of X-ray Images

Ivan Rodin, Irina Fedulova, Artem Shelmanov, Dmitry V. Dylov

    Результат исследований: Глава в книге, отчете, сборнике статейМатериалы для конференциирецензирование

    3 Цитирования (Scopus)

    Аннотация

    The quality and interpretability of the state-of-the-art methods for automatic analysis of chest X-ray images is still not sufficient. We address this problem by presenting a model that combines the analysis of frontal chest X-ray scans with structured patient information contained within radiology records. The proposed model generates a short textual summary with essential information on the found pathologies along with their location and severity; and the 2D heatmaps localizing each pathology on the original X-ray images. We test the proposed model on the MIMIC-CXR dataset. It achieves the state-of-the-art performance for image labelling and captioning (78.5% of correctly generated sentences) and defeats other similar solutions that dismiss the additional patient data (by 5.2% of correctly generated sentences). We also propose an automatic approach to label mining that leverages multimodal data: the X-ray images, related textual reports, patients' age and sex.

    Язык оригиналаАнглийский
    Название основной публикацииProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
    РедакторыIllhoi Yoo, Jinbo Bi, Xiaohua Tony Hu
    ИздательInstitute of Electrical and Electronics Engineers Inc.
    Страницы1601-1604
    Число страниц4
    ISBN (электронное издание)9781728118673
    DOI
    СостояниеОпубликовано - нояб. 2019
    Событие2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019 - San Diego, Соединенные Штаты Америки
    Продолжительность: 18 нояб. 201921 нояб. 2019

    Серия публикаций

    НазваниеProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

    Конференция

    Конференция2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
    Страна/TерриторияСоединенные Штаты Америки
    ГородSan Diego
    Период18/11/1921/11/19

    Fingerprint

    Подробные сведения о темах исследования «Multitask and Multimodal Neural Network Model for Interpretable Analysis of X-ray Images». Вместе они формируют уникальный семантический отпечаток (fingerprint).

    Цитировать