Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Ke Wang, Shujun Liang, Yu Zhang

Abstract

Accurate lesion segmentation in breast ultrasound (BUS) images is of great significance for the clinical diagnosis and treatment of breast cancer. However, precise segmentation on missing/ambiguous boundaries or confusing regions remains challenging. In this paper, we proposed a novel residual feedback network, which enhances the confidence of the inconclusive pixels to boost breast lesion segmentation performance. In the proposed residual feedback network, a residual representation module is introduced to learn the residual representation near missing/ambiguous boundaries and confusing regions, which promotes the network to make more efforts on those hardly-predicted pixels. Moreover, a residual feedback transmission strategy is designed to update the input of encoder by combining residual representation with original image features. This strategy could enhance the regions including hardly-predicted pixels, which makes the network can further correct the errors in segmentation. Experimental results on three datasets (3813 images in total) demonstrate that our proposed network outperforms the state-of-the-art segmentation methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87193-2_45

SharedIt: https://rdcu.be/cyhMn

Link to the code repository

https://github.com/mniwk/RF-Net

Link to the dataset(s)

https://www.ultrasoundcases.info/

https://radiopaedia.org/

Reviews

Review #1

Please describe the contribution of the paper

The authors present a segmentation strategy for breast lesions in ultrasound, which could be helpful in diagnosis and treatment planning. They employ a residual feedback network to learn pixel-level information and identify unclear boundaries and confusing regions that would otherwise be missed by previous approaches.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors describe a novel approach to the task of breast ultrasound lesion segmentation.
There is a detailed comparison with state-of-the-art approaches, like U-net, Deep-Lab, CE-Net and S2P-Net. The proposed model shows an improved DSC, JI, Recall and Precision metrics compared to previous approaches.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is no discussion of failed/ less accurate cases.
Various grammatical inconsistencies and typos exist in the paper, which sometimes make comprehension of the content difficult, for example, paragraph 1 of Section 2.2.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The dataset, network used, and the training details are well presented. However, the computational cost is not presented and there is no analysis of the results where the method has failed.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper addresses a strong clinical need of breast lesion segmentation to inform diagnosis and treatment. The technique produced has novelty—it improves the well-known encoder-decoder segmentation model. To remove ambiguity, it would be good for you to mention whether you are dealing with both benign and malignant lesions in your dataset. The breakdown of benign/malignant images should accompany your dataset description. It is well known that margins of benign lesions are much easier to figure out that the malignant ones and this could influence your results if the test set is skewed. Relatedly, although your results show marked improvement on state-of-the-art methods, there are clearly some outputs which are not perfect, but these are not well discussed. It is for example well known that some malignant lesions present with posterior acoustic shadowing, which would make delineation of the margin difficult. How does your model perform on these images? Although the writing style is generally good, it is important to address several typos and grammatical inconsistencies, for example: Introduction, second last line. Correct the sentence: “which makes the network can further correct the errors in seg-mentation,” and Paragraph 1 of 2.2 (Encoder-decoder baseline). Fig 1 should be improved to ensure that the text is easily discernible.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A novel method for breast lesion segmentation is presented, with strong relevance to the MIC community. The results are presented in good detail and show a step forward when compared to state-of-the-art methods.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

A residual representation module is introduced to learn the residual representation in an encoder-decoder architecture, near missing/ambiguous boundaries and confusing regions in lesion segmentation in breast ultrasound
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed architecure is a nivel proposal, and requieres no further parameter specifications to train the net.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Although results overcome exisiting methods. it is necessary to run experminets with different data sets to confirm superior performance. Furhtermore, the paper lacks of a deep and formal discussion on the residual representation and how it helps to the task. The paper has several language errors.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Not enough information for full reproducibility
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Further experiments should be done and a deeper discussion on the contribution should be added.The manuscrips has to be reviewed to correct language use.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A deeper discussion is missing, more experiments are necessary to suport claim and language should be revised.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The paper presents a novel method for the segmentation of ultrasound data using residual refinement.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is well examined and the results are convincing enough. The ablation experiment further strengthens the work.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The loss function especially residual representation loss needs to be clarified.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I didn’t find anything strange.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1- would the authors explain more about the loss function especially residual representation loss. 2- The residual part only works in training phase. Would authors explain more about why this strategy improves the results in test. 3- What is the equation for residual mask? Is it absolute difference? If yes, how the residual refinement module improve the results when the predicted mask is larger than ground truth mask. 4-The authors use a single refinement step? What happens if several steps of residual refinement happens?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Good ablation experiment Good experiments Suitable novelty
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Review #4

Please describe the contribution of the paper

This work presents a residual feedback network for breast lesion segmentation in ultrasound images. In the developed network, a residual representation module is introduced to learn features near boundaries and confusing regions, thereby better detecting hardly-predicted pixels. Experimental results on three datasets show that the proposed network outperforms state-of-the-art methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This work presents a residual feedback network for breast lesion segmentation in ultrasound images.
2. A residual representation module is formulated to learn feature representations near ambiguous boundaries and confusion regions.
3. A residual feedback transmission strategy combine the residual representation and original image features to update the encoder blocks.
4. Experimental results on three datasets verify the effectiveness of the developed segmentation network.
5. Ablation study experiments are conducted to validate the effectiveness of major modules.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In Eq. (1), the weights of three loss function are 1. Is there any ablation study experiment to discuss the results with different weights?
2. Writing issues: the mini-batch size of 16; momentum of 0.9; update the network parameters; The third one, datasetC, is.
3. According to Section 3.1, datasetA is employed for training and validating. Then, how to obtain the results of datasetA in Table 1? since there is no testing data in datasetA.
4. The authors only use a residual map of a coarse prediction and ground truth to predict hardly-predicted pixels, which is straightforward and lowers the novelty of this work.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think the readers can implement the developed network, and a released code and these datasets, as well as the segmentation results on three datasets will help a lot.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Eq.(1) should be ended with ‘,’.
2. There are many writing errors to be modified. Please see the weaknesses.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This work presents a breast lesion segmentation network in ultrasound images by learning residual presentations of hardly-predicted pixels. However, the idea of estimating hardly-predicted pixels is novel, but the authors only use a residual map of a coarse prediction and ground truth to predict hardly-predicted pixels, which is straightforward and lowers the novelty of this work. Experimental results on datasets show that the developed network outperforms state-of-the-arts. Moreover, the writing of this work should be improved.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

6
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes a novel method for the segmentation of breast lesions in ultrasound. As reviewers’ comments, the paper is written and organized well, but there are some important questions that need to be concerned. For example, there is no discussion of failed/ less accurate cases. Also, the paper lacks a deep and formal discussion on the residual representation and how it helps the task. The resolution of the used figures need to be enhanced in this paper. As a main contribution, the residual representation loss needs to be explained in detail. In addition, the authors should explain why the proposed module can improve performance, rather than just report good results. The authors are encouraged to revise the manuscript according to the suggestions of the reviewers point by point. Therefore, a rebuttal is appropriate.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We thank all the reviewers for their valuable comments. We appreciate that the key contributions of our work are affirmed by reviewers: 1) a novel approach for breast ultrasound lesion segmentation (all reviewers); 2) ‘the paper addresses a strong clinical need to inform diagnosis and treatment; the results are presented in good detail and show a step forward when compared to state-of-the-art methods’ (R1); 3) ‘the proposed requires no further parameters specifications to train the net’ (R2); 4) ‘good ablation experiment, good experiments, suitable novelty’(R3); 5) ‘the idea of estimating hardly-predicted pixels is novel’(R4).

R1 suggests to mention the breakdown of benign/malignant images, discuss the failed/less accurate cases, and the malignant lesion with posterior acoustic shadowing. We apologize for the vague description in the paper. Indeed, we have dealt with both benign and malignant lesions in our dataset, which contains total 1623 benign and 2190 malignant lesions. In fact, we have shown the results of malignant lesion with posterior acoustic shadowing in the second row of Fig.3. The results show that our proposed model achieves superior performance in this case. We will also add some failed/less accurate cases in experiment section, which contains the mucus-like lesion companied with microcalcificaiton, and the poorly visible mucinous carcinoma. The details will be discussed in the final version.

As suggested by R1, we will improve the quality of all figures.

R2 concerns about a deep and formal discussion on the residual representation and how it helps to the task. More metrics will be added in Table 4 to explain the reasons. For example, employing the residual representation yields significant recall improvements (0.70%,1.99%,0.55% on datasetA, datasetB, datasetC). This achievement can be explained by learning the representations of missing/ambiguous boundaries and confusing regions in residual representation module, and encouraging the network to explore more information about hardly-predicted pixels. Further, observed from Table 4, training our model using the proposed residual feedback transmission strategy achieves more accurate segmentation results in terms of all metrics, especially on the boundary-related metric HD. It proves that the residual feedback transmission strategy, which feedbacks the learned representations into the encoder, can enhance the weight of regions including hardly-predicted pixels, and make the network iteratively correct the errors in segmentation. These discussions will be added in the revision.

R3 suggests explaining the loss function especially residual representation loss. We used the weighted-balanced loss for segmentation and the weighted binary cross-entropy loss for residual representation module, which was proposed in our previous work[28] and adopted a weighted parameter to calculate the ratio of the number of foreground and background pixels (large differences in size between foreground and background areas, especially for residual representation). We will make this clear in the final version.

R3 is confused about the test phase in our model. For each BUS image in the test phase, the segmentation results can be generated recurrently for more precise performance owning to the proposed feedback transmission strategy. These will be added in the revision.

R4’s question is on how to obtain the results of dataset A in Table 1. Indeed, we perform the five-fold cross-validation on dataset A and report the validated results, then use dataset B and dataset C as two test datasets to illustrate the effectiveness of the proposed model based on five models trained on dataset A.

R2 and R4 require more information for reproducibility. We will release code and all segmentation results at GitHub and report the URL in final version.

All reviewers point out that the language using in our paper should be improved. We will carefully revise the paper in the revision.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This work presents a residual feedback network for breast lesion segmentation in ultrasound images. In the developed network, a residual representation module is introduced to learn features near boundaries and confusing regions, which can better detect hardly-predicted pixels. Experimental results on three datasets show that the proposed network outperforms state-of-the-art methods. In the author’s rebuttal, many issues have been addressed.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper explores the residual feedback for breast lesion segmentation from ultrasound images. Although the quality of the manuscript required to be improved significantly in the final version, the idea is interesting and experimental results demonstrated better performance over other methods. Therefore, I would like to recommend accept.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a novel residual feedback network to enhance the confidence of the inconclusive pixels and boost breast lesion segmentation performance. The performance of this proposed model seems quite good and the ablation study verifies the effectiveness of the proposed model. Three reviewers give positive feedback and the other reviewer raises concerns related to discussion of residual representation and the work strategy.

In the rebuttal, the authors addresses these issues and these information should be added in the final version. Therefore, I would accept this paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

back to top

Residual Feedback Network for Breast Lesion Segmentation in Ultrasound Image