Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Seong Tae Kim, Leili Goli, Magdalini Paschali, Ashkan Khakzar, Matthias Keicher, Tobias Czempiel, Egon Burian, Rickmer Braren, Nassir Navab, Thomas Wendler

Abstract

Chest computed tomography (CT) has played an essential diagnostic role in assessing patients with COVID-19 by showing disease-specific image features such as ground-glass opacity and consolidation. Image segmentation methods have proven to help quantify the disease burden and even help predict the outcome. The availability of longitudinal CT series may also result in an efficient and effective method to reliably assess the progression of COVID-19, monitor the healing process and the response to different therapeutic strategies. In this paper, we propose a new framework to identify infection at a voxel level (identification of healthy lung, consolidation, and ground-glass opacity) and visualize the progression of COVID-19 using sequential low-dose non-contrast CT scans. In particular, we devise a longitudinal segmentation network that utilizes the reference scan information to improve the performance of disease identification. Experimental results on a clinical longitudinal dataset collected in our institution show the effectiveness of the proposed method compared to the static deep neural networks for disease quantification.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_26

SharedIt: https://rdcu.be/cyl8m

Link to the code repository

https://github.com/lilygoli/longitudinalCOVID

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors address a very interesting topic, which is quantitative assessment of Covid infection in CT images using deep learning. They propose a longitudinal segmentation to automatic evaluation of COVID19 illness progression over time, which gives the possibility to evaluate different therapy approaches over time.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A novel use of state-of-the-art deep learning to evaluate COVID progression in patients over time
- Four different classes of lung pathologies are taken into consideration, so good level of information is provided by the results.
- Presented good qualitative and quantitative results are presented
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The clinical usability of the method remains still as a future step.
- The method depends of a cropping step that needs manually segmented lung masks. This seems to be an obstacle for fully automatizing the method.
- In the description it remains unclear how the NN architecture is used: the authors mention a DenseNet but show an Encoder-Decoder structure in Fig. 1
- According to Table 1 a very small benefit is obtained from the progression lost, which seems to be one of the contributions of the paper.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors claim that the source code will be available after publication. It may be difficult to reproduce the results of the paper without the source code and only using paper descriptions.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- It would be more insightful to know the units used to measure error, for instance in Table 3: is this cm, pixel size…?
- It would be useful and motivating to the reader to have a more clear description on what a longitudinal segmentation is.
- Labels (a) (b) (c) are missing on Fig. 1. Signaling these parts in the figure would make the method more clear to the reader.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- A very interesting topic under analysis: COVID progression study.
- Use of deep learning to perform the study.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The study investigates the use of longitudinal CT scans of COVID-19 patients to develop neural networks to segment the pathologies and quantify the progression of diseases.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The introduction of longitudinal data helps to better segment the pathologies and quantify their differences in sequential images to help track disease progression. The evaluation is comprehensive with multiple ablative experiments.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The overall accuracy of the proposed network seems somewhat limited (Table 2, Table 3).
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

NA
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors could potentially try to add augmentations and tune the hyperparameters of the network to see if the accuracy of the results can be further improved.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The study is quite relevant in a world currently hit hard by the pandemic. The evaluation is comprehensive with many comparative studies.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper for the first time studied the longitudinal segmentation of CTs of COVID-19 patients. By using a deep neural network with two longitudinal scans as inputs as well as a progression loss in addition to segmentation loss, the paper achieved better result compared to the baseline method (one scan as input and no progression loss).
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper proposed a framework for the longitudinal segmentation of CTs of COVID-19 patients, which seems not be studied before.
2. The paper proposed a progression loss which improved segmentation results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Longitudinal segmentation of medical images is not new (e.g. Birenbaum, A., & Greenspan, H. (2016). Longitudinal Multiple Sclerosis Lesion Segmentation Using Multi-view Convolutional Neural Networks. Lecture Notes in Computer Science, 58–67. doi:10.1007/978-3-319-46976-8_7). There is not much significant innovation (such as a new neural network architecture design, a new data augmentation method, etc.) in this paper, except for the progression loss. The progression loss seems to have resulted in little improvement (Table 1).
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The source codes are made available, though the data are not.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. There may not be many studies of longitudinal segmentation of COVID-19 CTs but longitudinal segmentation of other medical images are widely studied. This paper should benefit from including those methods for comparison and discussion.
2. Though an existing architecture is used, the paper would be more readable if more details of the architecture are listed.
3. How does the difference between the predicted segmentation and ground truth affect diagnosis? Some discussion may be helpful.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Though the longitudinal segmentation of CTs of COVID-19 patients seems to be a new application, there is not much method innovation in this paper. Nor is there comparison between ample baseline methods such as different neural network architectures etc.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

6
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a longitudinal segmentation method for COVID-19 lesions. A progression loss is proposed, but the experiment shows that improvement is incremental. Moreover, only DenseNet is used as a baseline, which may be not convincing to illustrate the effectiveness of the method. The presentation of Encoder-Decoder network is really confusing in Fig. 1. Finally, The sample size is too small (38 patients), although the longitudinal COVID-19 data is valuable. I would suggest “Rebuttal”. Please clarify the concerns of all the reviewers.

May need also to address the following comments:
1. not exactly sure which part was improved. how well the images are aligned after deformable registration; in general it looks that progression loss promote consistency of segmentation, thus if the infection area changes a lot, why the Dice for each time-point can still be improved; basically, can both segmentation and progression always improve together? please make this point clear by addressing reviewers’ comments also.
2. table column names can be improved for better understanding without referring to the text.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Author Feedback

We thank the reviewers for their positive and constructive feedback. We appreciate that reviewers (R1,R2,R3) acknowledged the novelty (R1,R2), the comprehensive experiments (R1,R2), the importance of our work (R1,R2,R3) and the clarity and organization of our paper.

1) Meta Reviewer (MR1), R1, and R3 commented that the performance improvement from the progression loss is incremental. According to Table 3, the previous longitudinal approach [3] has a 2.1% (36.8% to 38.9%) increase in Dice compared to the static approach (static segmentation and deformable registration for progression analysis) [15-Eur. Radiol. 2021]. Our longitudinal architecture has a 3.4% increase with respect to the static counterpart (36.8% to 40.2%) and the model with the progression loss has a 4.8% increase with respect to static (36.8% to 41.6%). As in Table 3, the progression loss had statistically significant improvement for the recovery and average progression prediction compared to the longitudinal model without the progression loss (p<0.05). Furthermore, the model using the progression loss significantly outperformed the baseline methods [3,15] (p<0.05). Therefore, we believe our results are not incremental and highlight the usefulness of the progression loss for progression analysis.

2) MR1 commented on the dataset size. In this study, we used 86 longitudinal CT volumes from 38 patients. A well-known public longitudinal segmentation dataset (on brain MRI) [A1] contains 19 patients in total among them only 5 patients are available for training. Furthermore, [15] used a private longitudinal COVID-19 dataset which consisted of 24 patients. Considering the new application of COVID-19 and the difficulty of collecting longitudinal data, the dataset used in this paper is comparatively large and the results are meaningful.

[A1] Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage 2017.

3) MR1 commented on how segmentation and progression loss interact. The progression loss does not minimize the distance between the two segmentations. Instead it explicitly uses the structural changes of pathologies through times as cues for modeling the optimization. In other words, if there are large changes over time, the progression loss encourages the model to predict those large changes also in the segmentation, instead of segmenting similar maps overtime.

4) Clinical impact (R3): This paper can quantify the progression of different pathologic changes and could be used to evaluate therapy strategies (as highlighted by R1). This evaluation can also compute the differences in disease progression of different patient subgroups, such as different COVID-19 variants under the same therapy.

5) The major strength and novelty of our study are performing COVID-19 progression analysis from longitudinal CT scans using a newly proposed progression loss, as acknowledged by R1 and R2. Proposing a new network architecture or a data augmentation technique is out of the scope of this work.

6) MR1, R1, R3 required further details about the segmentation network. Fully convolutional (FC) DenseNet [10] was used as a base segmentation model as introduced on page 4. As in [4, 10, 22], FC DenseNet consists of a downsampling path with 5 Transitions Down blocks, each with 4 layers and an upsampling path with 5 Transitions Up blocks, each with 4 layers. We denote the downsampling path as Encoder and the upsampling path as Decoder in Fig. 1. Our code including detailed implementation will become publicly available and further clarifications will be added.

We will further clarify minor comments including table names in the camera-ready version. Our paper offers a new COVID-19 progression analysis from longitudinal CT scans. Our novel application, its impact and comprehensive evaluation were acknowledged by R1 and R2. Our method is general and applicable to various medical applications, thus we believe it is a valuable contribution to the MICCAI community.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Strengths: the longitudinal assessment of CTs for COVID-19 or the progression loss are new to some degree. The paper has its merit about such innovation or new applications.

Weakness: DenseNet is used as the baseline, and the sample size is small (38 patients), although the longitudinal COVID-19 data is valuable.

Comments: The rebuttal addressed quantitative comparison and significance of the paper, as well as the segmentation/registration interaction. I agree with the justification of the progression loss, which does not enforce two time-points to be segmented similarly. The paper is mostly clear and the contribution is sufficient for MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposed a new longitudinal segmentation and progression analysis network model for assessing COVID-19 disease over time. Overall the topic is important and relatively new, and the method seems novel and effective.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The value of longitudinal study is recognized by reviewers, major concern is the novelty. Neither positive reviews nor negative one provides much detail. In rebuttal, authors provided more details, which I think is relevant to clarify the questions raised. One thing I would like to point out is that the performance gain is not quite significant, but more importantly the Dice number is low, so maybe other metrics would be more suitable to show the capability of the proposed framework, e. g. FROC curve. Overall, my opinion is that although novelty is not very significant, this is still a valid study with decent contribution, and therefore I suggest acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

back to top

Longitudinal Quantitative Assessment of COVID-19 Infection Progression from Chest CTs