Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Amogh Hiremath, Lei Yuan, Rakesh Shiradkar, Kaustav Bera, Vidya Sankar Viswanathan, Pranjal Vaidya, Jennifer Furin, Keith Armitage, Robert Gilkeson, Mengyao Ji, Pingfu Fu, Amit Gupta, Cheng Lu, Anant Madabhushi

Abstract

Although, recently convolutional neural networks (CNNs) based prognostic models have been developed for COVID-19 severity prediction, most of these studies have analyzed characteristics of lung infiltrates (ground-glass opacities and consolidations) on chest radiographs or CT. However, none of the studies have explored the possible lung deformations due to the disease. Our hypothesis is that more severe disease results in more pronounced deformation. The key contributions of this work are three-fold: (1) A new lung deformation based biomarker analyzing regions of differential distensions between COVID-19 patients with mild and severe disease. (2) Integrating 3D-CNN characterization of lung deformation regions and lung infiltrates on lung CT into a novel framework (LuMiRa) for prognosticating COVID-19 severity. (3) Validating LuMiRa on one of the largest multi-institutional cohort till date (N=948 patients). We found that majority of the shape deformations were observed in the mediastinal surface of both the lungs and in left interior lobe. On a testing cohort based on two institutions, Av (N=419) and Bv (N=113), LuMiRa yielded an area under the receiver operating characteristic curve (AUC) of 0.89 and 0.77 respectively showing significant improvement over a 3D-CNN trained over just lung infiltrates (AUC=0.85 (p<0.001), AUC=0.75 (p=0.01)). Additionally, LuMiRa performed significantly better than machine learning models trained on clinical and radiomic features (0.82, 0.78 and 0.72, 0.72 on Av and Bv respectively).

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_35

SharedIt: https://rdcu.be/cyl8v

Link to the code repository

https://github.com/amogh3892/LuMiRa-An-Integrated-Lung-Deformation-Atlas-and-3D-CNN-model-of-Infiltrates-for-COVID-19-Prognosis.git

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This work characterizes lung deformation using a combination of quantitative features measuring distention along with deep features extracted using a 3D convolutional neural network. The authors hypothesize that the magnitude of lung deformation can be used as a marker of severity for CVOID-19. They achieved high performance, demonstrated on a large dataset from two institutions consisting of 948 cases.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Addresses a timely public health need. Assessing the severity of the disease and the probability of the need for ventilation is still a clinically relevant problem.
- Using lung deformation as a measure of disease severity is a novel perspective.
- The dataset utilizes cases from two institutions with large sample size.
- The evaluation included multiple experiments, including sensitivity analysis, interpretation of model features, and comparisons to baseline models.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- None noted
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Dataset is not available, but the authors have committed to releasing their source code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Can the authors comment on the accuracy of the registration to the template and the implications of the registration error? Given that lung is highly deformable and that scans may occur at different levels of inspiration/expiration (particularly in this diseased cohort), the registration task is quite a challenge.
2. Was there feature selection that was performed prior to fusing the pretrained models (Section 2.4)?
3. Details about the network architecture and hyperparameters would be informative.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed LuMiRa system tackles the issue of identifying COVID-19 patients who are at the highest risk of needing ventilation. The approach combines feature extraction and registration to a template to characterize lung deformation as a marker for disease severity. The approach is novel, and the dataset consists of cases from two institutions with an adequate sample size. Multiple experiments are performed, providing an assessment of model performance and interpretation of predictive features.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

In this paper, the authors introduce a CT-based biomarker to predict COVID-19 prognosis by combining atlas-bsed difference and abnormality segmentation. The overall framework is basically an ensemble model from different preprocessed features.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Interesting clinical problem with a multi-center data.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. A key issue is that this paper is not technically sound. Many details or necessary ablation are missing. a1) How the data is obtained? There are too much details missing. Especially, the prognosis label (target variable). Besides, how the registeration is performed (no citation)? How the manual segmentation is performed (no tool mentioned)? What are the clinical features (no supplementary materials)? What is the radiomics features used (no citation)? a2) The method is basically an ensemble model from different preprocessed features. It uses atlas-bsed difference and abnormality segmentation as channel to feed into a two-stream model. However, many possible methods are not explored to convince me the superiority of the proposed method. For instance, what about using the segmentation label with multi-task learning? Or using a 3-channel (CT+M1/M2’s channels) single model? Do we really need something so heavy?
2. The clinical motivation is not clearly stated. Is this problem clinically important? Is it time-consuming or challenging? Is the performance good enough for clinical use? How far is the prognosis label is obtained from the current CT scan?
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Poor. No code. No open data. Key details are missing.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

See weakness.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I feel pity about this paper. It could be a good paper, but it lacks key details and ablation. I suggestion reject this paper at this current form, but I will not be upset if meta-reviewer decides to accept this paper after rebuttal if he believe this paper could be refined in final version.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The study proposes a new biomarker (lung distention – shape deformation) to help the covid prognosis prediction (ventilator need). The study hypothesizes that severe cases have severe lung distention. The study then proposes a computation method to predict the covid-19 disease prognosis by training two parallel 3D-CNN with lung distention, and lung infiltrates biomarkers. The output of CNNs fused at a final node. The decision of framework is ventilation need. The framework extracts lung infiltrates biomarkers using U-net algorithm. The segmented regions are then fed into one of the 3D-CNN.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

If there is a clinical base that lung distention is one of the manifestations of Covid-19, and lung shape has a correlation with the severity of the disease, then using it as a separate biomarker and computing this biomarker through atlas registration are novel.

The study is validated on a multi-institutional relatively large cohort, instead of a limited size of cohort obtained from a single institution.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Table 1 and Table 2 show that combining shape info with lung infiltrates improves the model prediction performance. However, the study does not have any clinical reference, which shows that lung dissemination as one of the covid-19 manifestations.

A difference atlas (DA) is computed from the registered volumes of patients who needed a ventilator (V+) and patients who did not need a ventilator (V-). However, the next step, how 3D-CNN characterization of shape deformation needs more explanation. The explanation for shape prior usage for severity prediction of target CT is not clear. How is the difference atlas (DA) used for prediction? If shape prior (the thresholded binary mask of DA) and target CT (Xi) are input to the 3D-CNN, does the 3D CNN learn the distance between the target lung shape and the shape prior?
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The study says that the model is trained and validated on a large multi-institutional cohort. The data is not publicly available. There is no link or detailed info for the data. The parameters are missing. There is not any indication in the manuscript regarding sharing the data or code. The study is not reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Abstract. “None of the studies have explored the possible lung deformations due to the disease. Our hypothesis is that more severe disease results in more pronounced deformation.” …
What is the clinical support of this claim? Ground-class opacities and consolidations are manifestations of covid, and there are clinical papers to support this claim. Is there a clinical paper that shows the correlation between covid severity and stronger lung deformation?
1. Introduction. Page.1. The literature contains several AI-based prognostic models predicting the severity, as well as ventilator need. Please check the following articles:
a. Combining Initial Radiographs and Clinical Variables Improves Deep Learning Prognostication in Patients with COVID-19 from the Emergency Department Young Joon (Fred) Kwon et al. Radiology AI, 2021. b. Automated Assessment and Tracking of COVID-19 Pulmonary Disease Severity on Chest Radiographs using Convolutional Siamese Neural Networks, Matthew D. Li et al. Radiology AI, 2021. c. BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset, A Signoroni et al., Elsevier, Medical Image Analysis, 2021. (Brixia… First, a regional division is proposed as severity metric… it follows a deep-learning-based algorithm for severity prediction.) d. Clinical and chest radiography features determine patient outcomes in young and middle-aged adults with covid-19, D.Toussie et al., Radiology, 2020. e. Combining initial radiographs and clinical variables improves deep learning prognostication of patients with covid-19 from the emergency department, Y.J. Kwon et al, Radiology Artificial Intelligence, 2020.
1. Researchers used U-net to segment the lung infiltrate regions, but the watershed algorithm for lung lobe regions. The U-net could also be used for lobe segmentation as well due to its success in biomedical segmentation.
2. There is not any info or link regarding the large cohort of multi-institutional data.
3. It is not clear to me what the CNN learns from shape prior and CT inputs (M1).
4. Instead of preprint copies, please cite the peer-reviewed or conference version of the references (if available).
5. References 27 and 28 are same.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Researchers seems to spent good amount of time to come up idea, build framework and do the experiments. However, there seems to be missing parts: Clincal base for the new biomarker, clinical motivation in the background, how to use the shape prior, data info, parameters.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #4

Please describe the contribution of the paper

This work proposed a lung deformation-based biomarker for COVID-19 prognosis based on the assumption that more severe COVID-19 result in more pronounced deformation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This work demonstrate a novel method to combine the deformation information and lesion infiltration information for lung CT classification. Methodology seems correct and has been described with sufficient technique details. Experiments are comprehensive. Overall it is a well-written paper, which is easy to follow; in particular the illustrations are very helpful.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

It is not clear why pre-trained M1 and M2 models were used for decision fusion. Does it make more sense to train a single model with both shape and infiltration priors? I understand that, it might be easier to compare M1 and M2 with each other; but I think a single model might be a bit more elegant and also could be useful to reveal actual saliency areas in the images when both shape and infiltration priors are used.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Possibly reproducible if data and code are made available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Details of the dataset is missing.
1. How many cases are positive (need a ventilator) or negative?
2. What’s the distribution of positive and negative samples in the training, validation and testing sets?
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It is a good work, with a clearly defined hypothesis and a valid method to verify it. The methodology can be improved.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This work combines the deformation information and lesion infiltration information for lung CT classification. Reviewers recognized the overall design and experiment. However, details are needed for multiple critical components, including clinical support for key idea, registration method (and its potential variabilities), and how shape information is encoded. Please note, the aim of rebuttal is to clarify misunderstandings / rationale behind method and experiment settings. Promise of extra experiments will not be considered.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

Summary: We notice that the reviewers were not convinced with the clinical basis/ relevance of the atlas-based approach used in this study. Also, reviewers have a few questions about the architectural design of the network and about the missing information on the dataset, registration details etc.

1) What is the clinical support of this claim? A number of recent studies have shown that severe COVID-19 disease causes lung damage. Bussani et al on postmortem samples of 41 patients showed extensive damage, persistent distortion of normal lung structure. Tonelli et. al and Dimbath et. al have shown the link between increasing strain and mechanical deformation of lungs in COVID-19 patients with severe disease. These findings can be explained by severe lung damage that causes intense disruption of the normal lung parenchyma and interstitium, with fibrosis as a sequela. Our hypothesis is that distensions induced in the lung by COVID-19 are a reflection of more advanced lung fibrosis or Interstitial Lung changes induced by more severe disease. Early characterization of these physiological based changes can portend the possible long-term sequela of COVID-19 and help in better prognosis and treatment planning for COVID-19 patients. We intend to add a few more lines to the manuscript citing these studies, making the clinical relevance/ motivation clearer.

2) Does it learn the distance between target…shape prior: It is not clear to me what the CNN learns from shape prior and CT inputs (M1).

For the model M1, we are not directly extracting shape-based features. However, we first identify regions of shape distensions between mild and severe COVID-19 patients and extract lung CT based features based on of these regions similar to extracting lung CT based features from infiltrate regions with M2. Through the help of activation maps (Fig. 3) we illustrate that the network is extracting features at regions with possible lung distensions (surface of mediastinal surface of both the lungs) to characterize COVID-19 patients and make decisions. The advantage of including lung deformation based information into CNN characterization is illustrated by comparing with and without (Section 3.5, paragraph 2) the use of lung deformation atlas (AUC=0.87 and AUC=0.80 respectively (P < 0.05)). Note these prediction results are higher compared to results reported previously using radiomics and clinical model (Section 3.5)

3) Missing details (registration, dataset): Brief details of registration were provided in the section 2.2. The registration method used in this study was based on parameter file database of elastix toolbox (Klein & Staring), specifically established for lung CT (interpatient; affine + B-spline transformation). Further details will be shared in the supplementary section and parameter file will be included along with the code. Additionally, we have shown that the registration accuracy was good (Dice=0.91) (Section 3.3), similar to accuracies obtained in previous studies (Kaiwen Xu et. al). Additional details of the dataset (institutions, number of patients with mild and severe disease), CT acquisition parameters, and list of clinical features, radiomic features used in Section 3.5 will be included as separate tables in the supplementary section.

4) Does it make more sense to train a single model … infiltration priors?: The binary masks used in this study as an additional input channels to both M1 and M2 aids in setting attention regions on lung CT for networks to extract features from. When two binary masks are used within the network framework, the attention is based on the intersection of the binary masks and it can potentially filter out too much needed signal due to the low overlap of the masks. As a result, maximally leveraging the existing domain knowledge in two separate branches provides an easier, more intuitive implementation of the network.

5) Reproducibility: Code will be shared upon acceptance

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Reviewers generally find the topic and proposed method interesting. The major questions lie on clinical and methodological motivations: how well can deformation be used as a marker for ventilator prediction, and how well can registration / other features be extracted. Especially, the performance relies heavily on the specific registration scheme used, and clinical relevance is unclear. The mask from deformation may be regarded as a “softer” version covering potentially more areas than “infiltrate regions” from M2. Rebuttal partially answered the questions being raised, but the motivation and clinical relevance is still unclear to me. Theoretically, if M2 is trained with better labels, covering more disease patterns than just infiltrate, it should be able to catch similar / more accurate regions than M1. Without proper ablations, I am not fully convinced by the current statement.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The overall structure of the paper is clear: two channels of CNN and then combine together. Although the rebuttal clarifies some concerns, after reading the paper, I am confused about how exactly D_a and shape prior S_p are computed. If it is deformation then I doubt the results because different people have different lung size; if it is the t-test prior after warping onto the atlas, I think it makes sense. However, the authors mentioned shape deformation/distension etc here and there in the paper – this makes me confused. But overall, the idea and paper look novel. Besides the rebuttal, this should be cleared in the final version if accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors addressed most important concerns in their rebuttal and the paper addresses a (for now) clinically relevant with sound methodology and strong evaluation.
I therefore recommend acceptance of the paper and highly recommend authors to include the missing details mentioned.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

back to top

LuMiRa: An Integrated Lung Deformation Atlas and 3D-CNN model of Infiltrates for COVID-19 Prognosis