Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Pengfei Gu, Hao Zheng, Yizhe Zhang, Chaoli Wang, Danny Z. Chen

Abstract

Accurate and automatic medical image segmentation is challenging due to significant size and shape variations of objects (e.g., in multi-scales) and missing/blurring object borders. In this paper, we propose a new deeply supervised k-complete-bipartite network with asymmetric convolutions (kCBAC-Net) to exploit multi-scale features and improve the capability of standard convolutions for segmentation. (1) We leverage a generalized complete bipartite network to reuse multi-scale features, consolidate feature hierarchies at different scales, and preserve maximum information flow between encoder and decoder layers. (2) To further capture multi-scale information, we sequentially connect k complete bipartite network modules together to facilitate their processing in different image scales. (3) We replace the standard convolution by asymmetric convolution block to strengthen the central skeleton parts of standard convolution, enhancing the model’s robustness on exploiting more discriminative features. (4) We employ auxiliary deep supervisions to boost information flow in the network and extract highly discriminative features. We evaluate our kCBAC-Net on three datasets (ultrasound lymph node segmentation (2D), 2017 ISIC Skin Lesion segmentation (2D), and MM-WHS CT (3D)), achieving state-of-the-art performance.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87193-2_32

SharedIt: https://rdcu.be/cyhMa

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper presents a medical image segmentation method named kCBAC-Net, which is an organic combination of 1) kU-Net, 2)CB-Net, 3) asymmetric convolution and 4) deep supervision. The proposed method is evaluated on both 2D (ultrasound lymph node segmentation and ISIC’17 skin lesion segmentation) and 3D (MM-WHS) datasets, and has achieved state-of-the-art performance. Ablation studies have been conducted to demonstrate the efficacy of each employed component.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The combination of kU-Net and CB-Net is interesting and yields excellent performance.
2. Employing asymmetric convolution and DS further improves the final accuracy.
3. Multiple datasets with different imaging modality have been used for evaluation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The improvements on all datasets are small, only around 1 percent in terms of Dice/IoU.
2. The three employed datasets are all quite small, especially the ultrasound dataset (only 100 2D images for test) and MM-WHS (only 4 3D scans is used for testing).
3. Considering the small data size and the subtle improvement magnitude, and no significance analysis is provided (neither the standard deviation), I cannot be fully convinced that the proposed method is really better than the compared ones. For example, on the MM-WHS dataset, only 4 scans are used for evaluation and the final method only outperforms the HFA-Net by 1 point. It is probable that the two methods have no statistical difference in Dice.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility is good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Given the small size of the first and third datasets, I believe a k-fold cross-validation is more appropriate for this paper, especially when the improvement achieved by the proposed method is very subtle.
2. In Table 1, what does the CB-Net w/o CBC mean? In section 2.1, it is mentioned that CB-Net uses the idea of CBC, so it is called complete bipartite network (CB-Net). So what is a complete bipartite network without complete bipartite connections?
3. As far as I know, in segmentation scenarios, IoU is equivalent to Jaccard Index and F1 score is the same as Dice. Why IoU and F1 are used in Table 1 while Jaccard and Dice are used in other tables? If they are the same, I suggest a unified usage to avoid confusing the readers.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall I hold a positive opinion towards this paper, because it has demonstrated a SOTA performance on three datasets (although the improvement may not be significant), and the idea of combining kU-Net with CB-Net is interesting to me. However, I cannot be fully convinced by its experiment results due to the relatively small improvement magnitude and the very small test sets. Together, I recommend a borderline acceptance.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposed a deep model called kCBAC-Net for medical image segmentation. The authors used asymmetric convolutions, densely skip connections, k-cascaded model and deep supervisions to improve the backbone. On two public medical databases and one in-house database, the proposed kCBAC-Net model achieved the best segmentation performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper is well organized and clear. Meanwhile, the proposed method has achieved good performance on three medical databases.
2. Each designed module is supported by corresponding experiment results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

[1] The major problem of this paper is that the technical novelty is limited. All designed modules are not new. (1.1) The complete-bipartite network is very similar to the existing MDU-Net (see: Zhang, J., et al. 2018. MDU-Net: Multi-scale densely connected u-net for biomedical image segmentation. arXiv preprint arXiv:1812.00352.) and HR-Net. (1.2) The asymmetric convolutions and deep supervisions are also common tricks. Please compare with the related inception v2 model and U-Net++ model. (1.3) Meanwhile, k- model also lacks novelty. The k is 2 in this paper. The authors should call it as a cascaded model rather than a k- model. (1.4) Generally speaking, I prefer to see some insights from this paper. However, There are no new ideas about medical image segmentation. Compared to similar existing models, the authors needed to clearly explain the differences and state the own motivations.

[2] Specifically, in Table 1, why is the performance of CB-Net w/o CBC (IoU: 0.757) much better than U-Net (IoU: 0.661) on the ultrasound lymph node dataset? I think the CB-Net model without CBC should be the U-Net. According to this, I am wondering that whether the SOTA performance of this model on three databases is based on a better baseline or not.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good Reproducibility
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Please clearly state the differences between the proposed model and similar existing models.
2. Please explain why the performance of CB-Net w/o CBC is better than U-Net?
3. Please clearly show what is the real challenge of medical image segmentation and give some insights rather than improve the segmentation performance.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although the performance of the proposed model is good, I do not see any new insights from this paper. Hence, the limited technical novelty of this paper led me to recommend probably reject (4) for this paper.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This work presents a deeply supervised k-complete-bipartite network with asymmetric convolutions by integrating multi-scale features and asymmetric convolutional blocks. Experimental results on three datasets show that the proposed network outperforms state-of-the-art methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This work presents a network to leverage multi-scale features and enhance the capability of standard convolutions for medical image segmentation.
2. K complete bipartite network with asymmetric convolutions is devised to progressively extract and fuse multi-scale information.
3. Selecting deep supervisions on different CBAC-Nets are designed to better assign deep supervision for medical image segmentation.
4. Three datasets are introduced for segmentation and experimental results on three datasets verify the effectiveness of the developed network.
5. Ablation study experiments are conducted to validate the effectiveness of major modules.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In the KCB-Net, how to select the K values? It is better to add an ablation study experiments to show the results with different K values.
2. Since many CB-Nets are integrated together for enhancing medical segmentation performance, it is better to present the training time and testing time of the developed network.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think the readers can implement the developed network. A released code and their results on three datasets will help a lot.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As shown in Fig. 2, k = 2. What are the segmentation results if increasing the value of k?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This work presents a k deeply supervised bipartite network with asymmetric convolutions for medical image segmentation. Experimental results on three data show that the developed network outperforms state-of-the-art methods. The authors are also suggested to resolve the raised weaknesses.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents a deeply supervised complete bipartite network with asymmetric convolutions for medical image segmentation. Experimental results on three datasets have demonstrated good performance. As reviewers mentioned, the proposed method seems to combine several existing tricks from the methodological view. Please address the following issues in the rebuttal: technical difference from existing methods (such as MDU-Net); small performance improvement magnitude while on relatively small test sets; ablation studies design on validating the effectiveness of major modules, etc.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Author Feedback

R4: Novelty?

Comparison with known methods (MDU-Net, HR-Net, inception-v2, U-Net++). 1) Regarding complete-bipartite connections (CBC), the dense cross connection block in MDU-Net can be viewed as a special case of CBC, since it fuses only feature maps from neighboring higher and lower layers with current layer. Though CBC and HR-Net both use a multi-resolution fusion scheme, our CBC is built upon multi-resolution feature pyramid like U-Net and aggregates all-scale feature maps at each stage while HR-Net gradually expands in deeper stages. 2) On asymmetric convolutions (Convs), inception-v2 replaces nxn standard Conv by a 1xn Conv and an nx1 Conv to save computation cost. But, our ACB combines standard Conv with asymmetric Convs in parallel to learn richer heterogeneous representations. 3) On deep supervision (DS), U-Net++ uses DS at multiple semantic levels in one model; our model uses DS in multiple stages (see detailed explanation in Sec. 2.3 for new insight of how to employ DS). Will add comparison to these methods in final version.

k-model (k=2 only) We aim to propose a new diagram of k-model. There is no constraint on using a larger k in our model. We tried different k in preliminary experiments to determine a balance of efficiency and efficacy. We chose 2 finally.

New insights of medical image segmentation? 1) kCB-Net can deal with the multi-scale object issue (kCB-Net (0.897) improves F1 score on base-model CB-Net w/o CBC (0.861); see Table 1). 2) ACB and DS address the missing/blurring border issue (in Table2, Jaccard improves by 1% w/ ACB and DS; visual qualitative results in Fig. 3 also illustrate the effect). 3) We add DS at different layers and show that adding DS at layers far away from the layer computing loss functions yields the best performance. Our method has no additional constraint and can be broadly applied to other tasks with multi-scale object and missing/blurring border issues. CB-Net w/o CBC setting. R2: What is it? It is a deeper U-Net with one more encoding and decoding block and normal shortcuts. Will clarify it in final version. R4: Outperforming U-Net. Is the performance gained from a better baseline only? 1) It is a deeper model than U-Net, thus outperforming the original version [19]. 2) As shown in our ablation study (Table 1), built upon this base-model (0.757), our kCBAC-Net + DS obtains 0.829 (+7.2%), a big improvement. On the other 2 datasets, we obtain clear improvement on kCB-Net (which is better than [19]). Thus the comparison is fair and the improvement is not small compared to a better baseline. R2: Small improvement while on small test sets. No significance analysis/standard deviation is reported. 1) The amounts of public training/test data are often small, especially in 3D scenarios. We follow the data splitting in [24-MICCAI’19, 26-MICCAI’19, 22-TMI’21] and outperform them all. Thus our comparison is fair and reasonable. 2) Though the improvement seems small (e.g., 1 point in Dice), the baseline is already quite high (e.g., 0.909 in [26]) and well-explored in previous work (e.g., [26] improved 0.9% Dice on MM-WHS and [22] improved 1.15% Dice on 2017 ISIC Skin Lesion). Thus it is not trivial to gain such improvement by our model. 3) We experimented on 3 diverse public datasets and the results are consistently good. We tried different random seeds; the performances were stable (e.g., on lymph node dataset, over 5 times, the mean and standard deviation (std) of F1 scores are 0.905, 0.002, resp.). Will add std to the tables. R5

How is k determined? Ablation studies of larger k? We experimented with different k on kCB-Net. On lymph node dataset, for k = 1, 2, 3, 4, F1 scores are 0.871, 0.897, 0.903, 0.902 and memory costs are 5.5GB, 11.8GB, 18GB, 24.4GB, resp.. The performance saturates with larger k and costs increase largely. Thus we use k=2 in final experiments for balancing efficiency and efficacy. Will discuss this in final version.

Show training/test time. Will do.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Although an in-depth discussion of k should be provided in the final version, the authors have addressed most of the concerns in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The interest of this work has been acknowledged in the reviews: novel architectural combination that led to good performance on various datasets. The authors rebuttal is reasonably responding to the concerns raised by the reviewers. Hence I recommend acceptance for this paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper propose a new deeply supervised k-complete-bipartite network with asymmetric convolutions (kCBAC-Net) to exploit multi-scale features and im- prove the capability of standard convolutions for segmentation. In this paper, the proposed method seems to combine several existing tricks from the methodological view. The innovative nature of this paper may not be enough to support its publication on MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

kCBAC-Net: Deeply Supervised Complete Bipartite Networks with Asymmetric Convolutions for Medical Image Segmentation