Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Youbao Tang, Ke Yan, Jinzheng Cai, Lingyun Huang, Guotong Xie, Jing Xiao, Jingjing Lu, Gigin Lin, Le Lu

Abstract

Measuring lesion size is an important step to assess tumor growth and monitor disease progression and therapy response in oncology image analysis. Although it is tedious and highly time-consuming, radiologists have to work on this task by using RECIST criteria (Response Evaluation Criteria In Solid Tumors) routinely and manually. Even though lesion segmentation may be the more accurate and clinically more valuable means, physicians can not manually segment lesions as now since much more heavy laboring will be required. In this paper, we present a prior-guided dual-path network (PDNet) to segment common types of lesions throughout the whole body and predict their RECIST diameters accurately and automatically. Similar to [23], a click guidance from radiologists is the only requirement. There are two key characteristics in PDNet: 1) Learning lesion-specific attention matrices in parallel from the click prior information by the proposed prior encoder, named click-driven attention; 2) Aggregating the extracted multi-scale features comprehensively by introducing top-down and bottom-up connections in the proposed decoder, named dual-path connection. Experiments show the superiority of our proposed PDNet in lesion segmentation and RECIST diameter prediction using the DeepLesion dataset and an external test set. PDNet learns comprehensive and representative deep image features for our tasks and produces more accurate results on both lesion segmentation and RECIST diameter prediction.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87196-3_32

SharedIt: https://rdcu.be/cyl2C

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper presents a semi-supervised (a click is required by the user) lesion segmentation and RECIST diameter prediction method. It uses 2 encoders, first one with just the image as input, and the second with the image, click information (distance map and binary click image) and features from the first encoder as inputs. In decoder, it uses top-down and bottom-up connections between all levels and use deep supervision to train the network.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well written.
- The proposed method (each technique) is clearly motivated.
- The ablation experiments are nice and show the contribution of each proposed technique clearly.
- The results are promising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It seem nnUNet was only trained with segmentation loss. It would have been nice to use the diameter loss as well. At present, the difference in performance of nnUNet and PDNet might be partially due to different loss functions.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Most of the paper is quite clear. However, there are some details which are very briefly mentioned, e.g. morphological snake used for pseudo mask generation might be difficult to re-produce.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- It is unclear why exactly the stage 1 is needed. Since the user has to provide a click, which can be used to crop a large fixed sized region. Is it ONLY for dynamically selecting the cropped region size. A comment about this would be nice.
- It would be nice to increase the font of text in Fig. 1
- “The first stage extracts the lesion of interest (LOI) by segmentation rather than detection in [21], since sometimes the detection results do not cover the lesions that we click in.” Wouldn’t expanding the bounding box slightly resolve this issue as is commonly done when further processing is to be done.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method is well motivated, contribution of proposed compoenents are individually evaluated, and the proposed method improves the performance over the previous state-of-the-art.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Review #2

Please describe the contribution of the paper

This paper focuses on RECIST diameter prediction, guiding by the click-driven attention and dual-path connection. In this work, the detection baseline is replaced by a segmentation baseline, and the overall performance is not bad.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-This paper is well-written. -The performances is not bad. -Ablation study is helpful
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

-What I concern about is the innovation. The topic is not original (but not very common), and the technical improvement is not obvious. The click-driven attention is similar to the previous work[21], and the decoder part of framework is similar to some other known work (SSN, Feng et al. ISBI, 2020). The details of the proposed modules are similar to, but not identical to the known work. -Some fragments in Fig.1 is too small to see. Please make them larger.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It is okay.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As claimed above, I don’t see too much new things, but the paper is well-written and the performances is not bad. Besides, the topic is interesting. My attitude is just at “boardline” – it is really hard to decide.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is not innovative enough, but the topic and the writing is good.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper presents a lesions segmentation and characterization (lesion diameter) method which is based on a previously proposed solution and uses manual input to speed up and improve lesion localization. The method was evaluated on a public dataset of 3D CT images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is easy to follow
- The method is evaluated on a public dataset
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Motivation is not well explained
- Very little difference from prior/base work
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper appears to provide enough details to reproduce the results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The authors mention in the introduction that measuring the diameter of a lesion is a subjective task and more accurate and precise metrics could be derived from lesion segmentation masks. Yet the paper describes a method for diameter prediction using segmentation masks. It is unclear why this inferior metric was chosen as opposed to something more accurate and meaningful, such as volume.
- The method described in the paper appears to have very little difference compared to the previous/base method [21].
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Motivation is not well explained
- Very little difference from prior/base work
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper appears coherent with good results, solid experiments and general good clarity. However the motivation is not well presented and the novelty of the framework appears limited or should be better highlighted
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We thank all the constructive and helpful comments from both reviewers and AC.

[R4] Motivation: The most related previous work [21] has two major limitations: 1) the detection based LOI (Lesion of Interest) extraction method can produce erroneous LOIs when multiple lesions are spatially close to each other; 2) the click prior information is treated equally for all lesions with different scales. In this work, to precisely alleviate these two issues, we propose 1) a segmentation based LOI extraction strategy, and 2) a prior encoder with click-driven attention to learn lesion-specific attention matrices by effectively exploring the click prior information, and 3) a decoder with dual-path connection to aggregate the multi-scale features comprehensively. With these method improvements, our empirical quantitative performance has been boosted by a coherent and consistent margin compared to [21]. We will elaborate more on the motivation at a later edition. R1 commented that the proposed method (of each technique) was clearly motivated.

[R2&R4] “The novelty of the framework …”: 1) As described above, our proposed method is significantly different from [21]. 2) For the decoder part, the previous work SSN (mentioned by R2) used the dual-attention and SE modules on the extracted multi-scale features. Although we also used dual-attention to build our scale-attention module, the main difference is that we proposed the new dual-path connection to aggregate the multi-scale features comprehensively. The dual-path connection brings a larger performance gain than the scale attention module. Therefore, the novelty of this work is evident, comparing to [21] and SSN. To better summarize the novelty: 1) A prior encoder with click-driven attention is built to learn lesion-specific attention matrices to improve feature representations in parallel from the click prior information, enhancing the representation ability of the extracted features (for performance improvement). 2) A new decoder is built to aggregate the multi-scale features comprehensively for better handling our tasks by introducing dual-path connection and scale-aware attention. 3) Our proposed method achieves state-of-the-art performance of lesion segmentation and RECIST diameter prediction on both the DeepLesion test set and an external evaluation dataset.

[R1] nnUNet was also trained using both the segmentation loss and the diameter prediction loss, the same as done by our model. Thus, the superior performance of PDNet is mainly due to its own feature representation learning ability.

[R1] “why exactly the stage 1 is needed”: Stage 1 is used for LOI extraction. Based on the click information, it is able to adaptively extract a lesion-specific LOI where the non-lesion regions are well removed, and the lesion’s contextual information is preserved. With the extracted LOI, stage 2 can segment the lesion and predict its RECIST diameters precisely. If directly using the click to extract a LOI by cropping a large region with a fixed size, it is hard to decide how large the size we should select since the lesion sizes can vary significantly for different lesions. There are lots of non-lesion regions in the cropped region if the selected fixed size is large, which affects the performance of stage 2. Adaptively extracting a lesion-specific LOI (as done in stage 1) is better and more reliable than cropping a large region with a fixed size.

[R4] “using segmentation masks for diameter prediction”: Actually, we did not get the predicted diameters from segmentation results. This work treated the lesion segmentation and diameter prediction as two different tasks and learned them jointly in a unified framework. Both results are obtained from the input CT images directly. It is, with one model, we can get accurate RECIST diameters and lesion segmentations. If stacking the 2D lesion segmentation results slice-by-slice, we can even obtain the 3D lesion volumetric segmentations (out of the scope here).

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper presents good results and a cohesive structure. Issues regarding motivation and added novelty were convincingly addressed in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper presents an interesting approach to lesion segmentation and RECIST estimation. Main weaknesses identified by one of the reviewers was the lack of motivation and innovation. This was addressed in the rebuttal, highlighting the differences to previous work. Given the explanations in the rebuttal, I believe this is sufficient for acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper is clinically relevant and the proposed method has shown impressive results. Reviewers have concerns on its motivation, novelty and some experiments. The rebuttal clarified several points mentioned by the reviewers. The authors confirmed that nnU-Net was trained with the same loss as the proposed method. The motivation is also better explained, which seems to be reasonable. The rebuttal also helped to understand the difference from [21]. Thus, the paper is recommended to be accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

back to top

Lesion Segmentation and RECIST Diameter Prediction via Click-driven Attention and Dual-path Connection