Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Tao Tan, Bipul Das, Ravi Soni, Mate Fejes, Sohan Ranjan, Daniel Attila Szabo, Vikram Melapudi, K S Shriram, Utkarsh Agrawal, László Ruskó, Zita Herczeg, Barbara Darazs, Pal Tegzes, Lehel Ferenczi, Rakesh Mullick, Gopal Avinash

Abstract

The COVID-19 pandemic continues to spread and impact the well-being of the global population. The front-line modalities including computed tomography (CT) and X-ray play an important role for triaging COVID-19 patients. Considering the limited access of resources (both hardware and trained personnel) and decontamination, CT may not be ideal for triaging suspected subjects. Artificial intelligence (AI) assisted X-ray based applications for triaging and monitoring COVID-19 patients in a timely manner and to further delineate the disease region boundary are seen as a promising solution. Our proposed solution differs from existing solutions by industry and academic communities, and demonstrates a functional AI model to triage by inferencing using a single X-ray image, while the deep-learning model is trained using both X-ray and CT data. We report on how such a multi-modal training improves the solution compared to X-ray only training. The multi-modal solution increases the AUC (area under the receiver operating characteristic curve) from 0.89 to 0.93 and also positively impacts the Dice coefficient(0.59 to 0.62) for localizing the pathology.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_31

SharedIt: https://rdcu.be/cyl8r

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

In this paper, a X-ray based triaging and monitoring COVID-19 approach is proposed. CT data is employed to generate the synthetic x-ray SXR to augment the training data. the AUC score has been improved from 0.89 to 0.93 for the triage classification task. Comparing to CT, portable X-ray units are cost and time effective.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

A x-ray based triaging approach COVID-19 is introduced. With the augmented straining samples from the CT scan, the x-ray based performance has been improve.
It shows a potential for the practical value for COVID screening.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Writing needs improvements abstract, the multi-modal solution increases AUC and dice coefficient for what? Please clarify the statements.

sec2.1 HU terms should be clarified.
sec2.3 what is “image level classification” . What do you mean the image level? Isn’t x-ray an image?

The fig.1 may be problematic, why there are two lines from CT to SXR generation? Isn’t it redundant.

For segmentation evaluation, which one is used as the ground truth?

minor: page 3, “are still are added in the training pool.” ?
sec2.2, To alleviate the problem of the during image registration, sec4.2, “likeli-hood of COVID-10”.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It is not clear if the in-house COVID-19 data will be released. Code no release.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

See weakness .
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This works presents a x-ray based COVID-19 triaging approach. It potential to employ X-ray for time and cost efficiency.
The writing is not in high quality with multiple typos. Also, some concerns need to be addressed.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper
Following has been demonstrated:
1. Multi-model training to inform about pathology in X-ray images
2. Synthetic X-ray generation to bridge the information translation from CT images to X-ray.
3. Training with multiple modality data but inference with the easier modality.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper proposes a strong method that could be theoretically used in triaging cases based on pathology evidence.
2. The training setup is designed to use information from both chest CT and X-ray images.
3. High clinical feasibility
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

No major weakness The method relies heavily on how well the synthetic X-ray generation. So, a stronger valdation for the generated images is necessary.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducible with the information provided. Pristine annotations usage requires adjustments which are not discussed further and this might hinder reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The validation of synthetic data generation could be stronger.
2. Not sure how 48 hours was chosen as time-window.
3. The transferred annotations on XR are further adjusted manually reviewed by trained sta to build our pristine mask ground-truth/annotations (see 2.2). - There is no information about these adjustments in 2.2 or elsewhere. How many required annotations and is this linked to the choice of 48 hours. Couple of sentences regarding why and what would be helpful in reproducing.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Novelty clearly visible in approach and target application as well.
2. Demonstration of how training using information from multi modality is demonstrated which can be translated to different problems.
3. Reproducible and the idea is feasible to be realized in clinical setup with more development
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors created a multi-modal framework for COVID-19 diagnosis and segmentation that learns from both high-dimensional modality CT and X-ray images but infers from low-dimensional mono-modality X-ray images. The provided approach (multi-modal training) allowed authors to increase the AUC ROC metric from 0.89 to 0.93 and the Dice coefficient from 0.59 to 0.62.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors demonstrated not trivial training procedure and the proposed approach turned out to be pretty effective as it increased accuracy. The authors validated their idea not only for the classification task but also for the segmentation task.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The authors does not provide any description of how realistic synthetic X-ray images look like and what is the limitation of synthetic generation.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The trining procedure clearly described. Paper seems reproducibly despite using in-house datasets.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1) The authors presented an interesting approach and good results were achieved using mult-modal training. 2) It would be useful to describe synthetic X-ray generation more detailed and specify features, limitations and differences from real X-ray images.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Relevance and novelty.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes X-ray-based triage and monitoring of covid-19, wherein CT images are used to simulate x-ray images for augmentation and training. Rich information from CT could benefit the network of X-Ray. The application and method are strong. The experimental results are sufficient to support the conclusion of the paper.

Weakness needs to be addressed: 1) pls improve the writing and figures (make the lines straight); 2) make it clear about image-level annotation and ROI annotation and how they are used; 3) justify how well synthetic X-rays are.

Additionally, suggest to address other comments raised by reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

The authors would like to thank the chair and all the reviewers for the precious comments and suggestions. The authors find the comments very insightful and have provided great feedback to improve both the readability and the interpretability of the presented work. We have taken the inputs very constructively to enhance the text and add any other meaningful content to ensure good comprehension of the content for possible reproduction or clinical experimentation.

We have fixed the multiple-line issue of Fig 1, improved the writing and clarified regarding image-level annotations which are classification labels (COVID19, regular pneumonia and negative) and notation of ROI annotations to be pixel-level labels/annotations/markings of disease regions in this revision.

We have also added details of the parameters used for generating realistic synthetic X-ray from the corresponding subject’s CT data. It should be noted that synthetic X-rays generated for our training have a lower spatial resolution compared to the corresponding X-rays (CR/DX) given the limitation that the original slice thickness of CT images ranges from 3mm to 6mm. Given the clinical nature of the data we have consciously adopted not to interpolate this data, yet maximally use the multi-modality content to enhance the model.

Regarding paired data, X-rays and CTs were typically acquired within a small time-window (duration of patient hospital stay). We opted to use an empirical threshold of 48 hours to ensure that the clinical indication of pathology does not change significantly between the time of the X-ray and CT images. Since COVID-19 is a rapidly changing disease, including additional CTs or subjects for ground-truth (GT) transfer with a wider gap between the X-ray and CT images would misrepresent the training pool which aims to predict/triage the study and correlate to the RTPCR result.

Although we have aimed to create an approximate spatial match between SXR and XR, we cannot directly use transferred GT from 3D CT onto 2D and consider that to be GT for XR. The disease can rapidly change even within 48 hours and so in our study the transferred GT can only be used as a directional guidance by a human expert to then precisely refine the annotation on the 2D X-rays. The general instruction to the annotators when adjusting the ROI annotation, to remove the CT-mapped regions where lesions are totally invisible on X-rays. On the other hand those regions which minimally overlap with clearly visible pathology on the X-ray are preserved. Similar complex situations were observed in the heart/diaphragm regions due to lesion masking by overlapping structure/organs in the projection image. We clarified these issues in the attached revision.

Hope all changes improve the clarify of this paper.

back to top

Pristine annotations-based multi-modal trained artificial intelligence solution to triage chest X-Ray for COVID19