Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Zijie Chen, Cheng Li, Junjun He, Jin Ye, Diping Song, Shanshan Wang, Lixu Gu, Yu Qiao

Abstract

Radiation therapy (RT) is widely employed in the clinic for the treatment of head and neck (HaN) cancers. An essential step of RT planning is the accurate segmentation of various organs-at-risks (OARs) in HaN CT images. Nevertheless, segmenting OARs manually is time-consuming, tedious, and error-prone considering that typical HaN CT images contain tens to hundreds of slices. Automated segmentation algorithms are urgently required. Recently, convolutional neural networks (CNNs) have been extensively investigated on this task. Particularly, 3D CNNs are frequently adopted to process 3D HaN CT images. There are two issues with naïve 3D CNNs. First, the depth resolution of 3D CT images is usually several times lower than the in-plane resolution. Direct employment of 3D CNNs without distinguishing this difference can lead to the extraction of distorted image features and influence the final segmentation performance. Second, a severe class imbalance problem exists, and large organs can be orders of times larger than small organs. It is difficult to simultaneously achieve accurate segmentation for all the organs. To address these issues, we propose a novel hybrid CNN that fuses 2D and 3D convolutions to combat the different spatial resolutions and extract effective edge and semantic features from 3D HaN CT images. To accommodate large and small organs, our final model, named OrganNet2.5D, consists of only two instead of the classic four downsampling operations, and hybrid dilated convolutions are introduced to maintain the respective field. Experiments on the MICCAI 2015 challenge dataset demonstrate that OrganNet2.5D achieves promising performance compared to state-of-the-art methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87193-2_54

SharedIt: https://rdcu.be/cyhMx

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a 2.5D U-Net architecture for organ-at-risk segmentation from head and neck CT images. The proposed model consists of several different 2D and 3D convolution layers to extract both 2D detailed features and 3D semantic features.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The major strength is that the proposed 2.5D U-net architecture is novel and fairly good results are achieved in MICCAI 2015 challenge dataset. Experiments were also conducted on a more larger dataset.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

One weakness of the work is the experiments on the first dataset (the the collected public dataset) is simple. Segmentation results on all organs should be provided instead of only the small organs. Comparison to existing methods on this dataset should also be done.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

good
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

More experiments should be done on the first dataset, such as the comparison to existing methods. The results of all organs in this dataset should be provided. More efforts of experiment should be put on this dataset since it contains more data and organs. Experiments on combing the two datasets would also be helpful. Experiment of training on one dataset and testing on the other one will be also valuable. It is useful to show if the propose method can be easily transferred to new dataset.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed model is novel and the experiments are acceptable.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The author presents a network that combines 2D and 3D feature extraction: OrganNet2.5D, which is used for the segmentation of OARs in 3D HaN CT images. In order to solve the difference in the resolution and feature learning of in-plane and depth images, the author first uses 2D convolution to better extract edge image features, then 3D convolution is used to extract semantic features, and finally, multi-scale hybrid dilated convolutions are used to improve the receptive field and extract multi-scale features. The author conducted experiments on two datasets, and the experimental results proved the effectiveness of the proposed model and modules.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Novelty: The author considers the difference in resolution of different dimensions and puts 2D feature extraction and 3D feature extraction into the same network to improve the effect of feature extraction. Dataset: The author conducted experiments on two datasets, and the experimental results basically proved the effectiveness of the proposed algorithm.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Expression: There are certain expression, tense and grammatical errors in the article: such as firt, decreased by 1/10; inconsistent tense: the tense is inconsistent in the 2.4 Implementation details and 3 Experimental results. Experimental setting: Why does AnatomyNet use additional data for training but other networks do not? This setting loses the fairness of comparison. The comparison algorithms used are all proposed in 2019 or before, why not use the updated algorithm? Why is there no other comparison algorithm for comparison on the first dataset, and there is no ablation experiment result on the second dataset? Experimental analysis: The experiment result section does not indicate which results have statistical significance; and the figure does not show the results of other comparison algorithms, as well as does not show the improvement of the segmentation results brought by the module design for solving different problems, including the improvement of edge segmentation result and multi-scale segmentation result.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

According to the author’s description, the dataset and code can be obtained, and the experimental results can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The author proposes a novel network for the segmentation of multi-organ in the head and neck, but the expression of the article needs to be further revised, and the experimental part needs to be redesigned, including the consideration of adding new comparison algorithms, as well as ensure that all comparison experiments have the equal setting, as well as same experiments, need to be conducted in the two datasets. And at the same time, the corresponding improvements in results need to be displayed for the motivations proposed by different modules.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novellty, writing, experiment
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

A novel hybrid CNN is proposed to improve the performance of OARs in HaN CT images. The network uses 2D convolution blocks and 3D convolution blocks for extraction of low-level edge features and high-level semantics features. The method achieves superior performance on two datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) Paper is well organized and easily understandable. (2) Organ segmentation in 3D head and neck CT images is interesting. (3) The performance is good.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The novelty is limited. Modification on network architecture is trivial. Loss function is not new. (2) Some typos, e.g. ‘OrgainNet’ in Section 2.2 (3) Experiment setting is not reasonable. On the first datasets, the experiments are ablation studies. On the second datasets, the proposed network is compared with some state-of-the-art methods. (4) Some details need to be improved. In Table3, Mean DSCs of MICCAI2015 and AnatomyNet and FocusNet are single mean value, whereas the other two are mean and variance. (5) More other methods should also be compared.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The network architecture and experimental settings are detailed. But the code is not open-sourced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) The issues in the developed approaches should be analyzed more. For example, what’s the drawback of current 3D CNNs and 2D CNNs? Can some examples illustrates the issues? (2) Ablation studies and comparison with state-of-the-art methods should be conducted more sufficiently. (3) Details should be focused. Typos and missing data should be fixed and supplemented.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novelty is limited. Experiments are not arranged reasonably. Some details are missing. Typos.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

A 2.5D U-Net was proposed for multi-organ segmentation from head and neck CT images and it was validated on two datasets. The experiments seem to be inadequate. The reviewers have mixed comments on this paper. The authors are expected to clarify the following points in the rebuttal: 1) Fairness of comparison in the experiments, and why not comparing with more recent methods later than 2019? 2) Comparison with other methods on the first dataset and ablation study on the second dataset. 3) Novelty of this method. 4) analyze the drawback of current 3D and 2D networks. 5) details of the data and results in the first dataset.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

Dear Area Chair,

Thanks for organizing the review process and sending us the reviewers’ comments. We appreciate a lot that you gave us the opportunity to do a rebuttal. Thanks also to the three reviewers for their valuable comments and suggestions. As suggested, five major comments are addressed in this rebuttal.

Comment 1: Fairness of comparison in the experiments, and why not comparing with more recent methods later than 2019. Response: Addressing the first question, we listed the results of the comparison methods as they were published in the original papers. We did not try to reimplement these methods as we might not achieve the optimal results as the papers did. This is a common practice for papers working on organ segmentation in HaN CT images. Anyway, our results are better even though additional training data were utilized by AnatomyNet. For the second question, the results were not compared to more recent methods for two reasons: 1) The project was mainly conducted in 2019. 2) The three comparison methods in Table 3 are the most prevalent methods in the field. AnotomyNet has been compared in almost all relevant studies since its publication. Nevertheless, in our final paper, we will include several recently published representative methods. Right now, we are thinking of adding the neural network searching-based method (Guo et al. CVPR 2020) and SCAA (also combing 2D and 3D operators, Tang et al. WACV 2021). We will discuss the advantages as well as limitations of our proposed method when compared to these models. One observation is that despite the simple architecture, our method still achieves the best result on the segmentation of the hardest target (optic chiasm).

Comment 2: Comparison with other methods on the first dataset and ablation study on the second dataset. Response: Sorry for the confusion caused by the experimental design. As we stated in the response of Comment 1, we do not think it is appropriate to reimplement the published methods and make the comparisons. Basically, we utilized the two datasets for two purposes: Dataset 1 (collected public dataset) – Validate that each module of our proposed model is effective; Dataset 2 (MICCAI 2015 challenge dataset) – Confirm that our model is applicable when compared to state-of-the-art methods. In fact, this design follows the FocusNet paper. We will make it clear in our final paper.

Comment 3: Novelty of the method. Response: As recognized by Reviewer 1 and 2, the novelty of the method is mainly the proposed OrganNet2.5D model. We cannot agree with Reviewer 3 that modifying network architecture is trivial. Network architecture is very important, and many efforts have been devoted to optimizing network architecture for various applications. Besides, our proposed OrganNet2.5D model is not limited to organ segmentation in HaN CT images. The issue we are addressing is the different in-plane and out-plane resolutions, and it is quite common for medical imaging. Therefore, it is expected that OrganNet2.5D can serve as a good baseline for medical image segmentation, or at least, it can remind the following studies of this important issue.

Comment 4: Analyze the drawbacks of current 3D and 2D networks. Response: Thanks to the reviewer for this important suggestion. Due to the page limit, we only briefly mentioned in our paper that the main drawback of 3D networks is the ignorance of the different in-plane and out-plane resolutions and 2D networks cannot exploit the 3D spatial information. We will try to detail the relevant discussions in our final paper. In our opinion, 3D networks also suffer from the increased parameters and computational burden. As a result, methods tend to employ shallow networks or perform network training with image patches. Both limit the network performance.

Comment 5: Details of the data and results in the first dataset. Response: Sorry for the missing information. We will give the details in our final paper or in a supplementary file.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors proposed a novel network structure to deal with different in-plane and out-plane resolutions for medical image segmentation, which was usually ignored by previous works. It achieved better performance than the existing AnatomyNet and FocusNet. In the rebuttal, the author’s reply helped to clarify some points listed by the reviewers, such as the experiment setup and some details. However, in my opinion, this paper is not the first one to use 2.5D networks to deal with the large inter-slice spacing. They should consider to mention some related works, such as the following “3D anisotropic hybrid network: Transferring convolutional features from 2D images to 3D anisotropic volumes”, MICCAI 2018
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The reviews have highlighted the fact that this paper proposes a valuable contribution in terms of architecture design with good performance

There were some concerns mainly regarding the fairness of comparison in the experiments. In their rebuttal, the authors have provided explanations regarding this issue, therefore I recommend acceptance for this paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a network for the segmentation of multi-organs in the head and neck from 3D HaN CT images. The combination of 2D and 3D features extracted from 2D convolution blocks and 3D convolution blocks respectively is interesting, as low-level edge features and high-level semantics features can be used together to improve the performance of segmentation. The experimental results proved the efficiency of the proposed network, although some comparison experiments need to be improved. My proposition is “accept”.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

back to top

A Novel Hybrid Convolutional Neural Network for Accurate Organ Segmentation in 3D Head and Neck CT Images