Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Han Li, Long Chen, Hu Han, Ying Chi, S. Kevin Zhou
Abstract
Universal Lesion Detection (ULD) in computed tomography plays an essential role in computer-aided diagnosis. Promising ULD results have been reported by coarse-to-fine two-stage detection approaches, but such two-stage ULD methods still suffer from issues like imbalance of positive v.s. negative anchors during object proposal and insufficient supervision problem during localization regression and classification of the region of interest (RoI) proposals. While leveraging pseudo segmentation masks such as bounding map (BM) can reduce the above issues to some degree, it is still an open problem to effectively handle the diverse lesion shapes and sizes in ULD. In this paper we propose a BM-based conditional training for two-stage ULD, which can (i) reduce positive vs. negative anchor imbalance via a BM-based conditioning (BMC) mechanism for anchor sampling instead of traditional IoU-based rule; and (ii) adaptively compute size-adaptive BM (ABM) from lesion bounding-box, which is used for improving lesion localization accuracy via ABM-supervised segmentation. Experiments with four state-of-the-art methods show that the proposed approach can bring an almost free detection accuracy improvement without requiring expensive lesion mask annotations.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87240-3_14
SharedIt: https://rdcu.be/cyl5L
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper is an extension of the referenced paper [11], which aims to handle with the diversity in lesion shapes & size. The authors set a few conditions to deal with the positive vs. negative problems and make it adaptive to lesion size. The effectiveness is demonstrated with improvements on four SOTA approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper clearly analyzes existing problems in ULD and the related work BM[11].
- The authors provide some useful conditions to overcome the shortages of BM, and make it adaptive to the lesion size.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The experiments do not directly present improvements on small size lesions.
- Despite the improvements, this paper is prone to be considered as an increment of BM.
- The conditions proposed are actually specific to incorporate with the Bounding Map, therefore, it is hard to tell the significance of the proposed technique.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Implementation details are given; I think it’s easy to reproduce.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The proposed method can bring benefits as the experiments suggest. However, it is not fully evaluated through experiments to show its superiority on small-size lesions.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Accurate statement of motivation, and a reasonable solution based on BM.
- What is the ranking of this paper in your review stack?
3
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Review #2
- Please describe the contribution of the paper
The paper describes a novel method for universal lesion detection (ULD) by bounding map conditional training. Existing methods adapt anchored object detection frameworks. These methods have limitations of 1) few positive anchors to use as ROI proposals for anchor-based methods, 2) lack of supervision in stage 2. The proposed method introduces two mechanisms to deal with the mentioned limitations, 1) BM-based conditioning to reduce anchor imbalance, 2) size-adaptive BM (ABM) to provide more supervision in stage 2 and improve lesion localization accuracy. The proposed method has been evaluated on DeepLesion dataset and compared with SOTA ULD methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper analyzed the limitation of existing anchor-based methods and addressed the limitations with novel modules. The method is also extensively evaluated.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Major concerns: 1) could the authors further explain why BM can be a good GT indicator, and if the GT are soft labels, is it harder for the RPN to predict the objectness? 2) how does the positive anchor boundary value B affect the model performance and how to select a good value B? 3) the adaptive BM is interesting, is it possible or desired to adapt to the lesion size in a continuous manner instead of small, medium and large sizes?
Minor concerns: 1) some typos, e.g. in section 2.2, second paragraph, line 6, “objectness GT lable” -> “objectness GT label”; 2) some texts in Fig. 2 have slightly lower resolution, a bit hard to read, e.g. “anchor selection”
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Major details are provided, but it could well be some of the minor details are neglected in the main paper. It would be better if the code can be provided to ensure productivity.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The concerns are addressed in the weakness part.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is overall well-written and well-motivated. The issues that the authors would like to address with the proposed method are discussed and such issues can be addressed with the proposed method with technical novelty. The method is also properly and sufficiently evaluated.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
The paper aims to strengthen the bounding map mechanism’s application on Universal Lesion Detection. In particular, the bounding map is calculated in an adaptive way taking the object’s size and shape into consideration. Extensive experiments with SOTA 2D models demonstrate its consistent effectiveness.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
adaptive manner of BM generation theoretically drives the model treat more fairly to lesions of different sizes and shapes
-
per-pixel objectiveness map make the detection “denser” in this super sparse object detection scenario
-
experimental is well designed and thorough, making it convincing
-
consistent performance boost on SOTA models
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
do small objects really benefit from ABM? Theoretically yes, but I need the proof. One easy proof could be the performance boost wrt. lesion size. This doubt stems from the fact that most of the lesions DeepLesion are actually mid-sized, if they are the major beneficiary, the statement will not be true.
-
for reproducibility, what are the values for A_s and A_m? This should be provided as it is critical.
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
All experiments are conducted on public datasets and there are not too manny hyper-parameters. Should not be difficult to reproduce.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
This paper is already of high quality, but it can be better:
- do performance analysis wrt. the lesion size
- would it also work for 3D models? I understand 3D model is costly, but LUNA16 or Tianchi might be doable. If 3D models also benefit from this methods, it will make this paper even stronger.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Simple but effective approach; Consistent performance boost on SOTA approaches.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose methods to improve the training for two-stage detectors, commonly used in lesion detection. All reviewers agreed that experiments convincingly demonstrated the value of the contributions, particularly as they were shown to garner improvements when added to a variety of two-stage detectors. I agree with the reviewers that this would be a valuable addition to the MICCAI meeting.
Please try to address reviewer comments. In terms of shortcomings, two reviewers were doubtful as to the whether small lesions can be addressed with the authors’ approach. This is a very important question, given that small lesions are typically the most difficult to locate by clinicians.
I would also add that the authors’ reported CenterNet results (Objects as Points) are extremely low. Given that VULD [10] and other works (https://ieeexplore.ieee.org/abstract/document/9309244/, https://ieeexplore.ieee.org/abstract/document/9187431) have reported SOTA lesion accuracy with CenterNet-style anchor free methods , this is an important discrepancy that should somehow be explained (particularly as the authors’ contributions are only necessary when choosing to use a two-stage anchor-based detector).
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
1
Author Feedback
Q1: For the low performance of the anchor-free methods: A1: For the paper (https://ieeexplore.ieee.org/abstract/document/9187431): (i) they use new evaluation metrics but our results are based on the official evaluation metrics and test dataset ; (ii) Their best performance is given by two-stage anchor-based method faster R-CNN w/ ULDor instead of anchor-free methods; (iii)The faster R-CNN is not designed for ULD, and they didn’t test the performance of most SOTA methods (e.g., MVP-net, MULAN, and Alignshift), therefore the performance gap is small.
The performance of the CenterNet (anchor-free methods) in the listed paper( https://ieeexplore.ieee.org/abstract/document/9309244/) is inferior to the performance reported in our paper.
The VULD method is a brand new method that is different from anchor-free methods reported in our paper, they use a coarse-to-fine surface point training mechanism which has the same advantages as two-stage SOTA ULD methods. Unluckily, we failed to reproduce it hence we didn’t report their performance.
We believe the main reason for the low performance of the anchor-free methods is that the ULD task is a challenging task. Different lesions have very diverse shapes and sizes among different CT slices. Directly apply per-pixel training mechanism ( e.g., segmentation methods & CenterNet-like methods) are easily leading to false positive and false negative detections.
Two-stage anchor-based object detection methods are widely used in ULD and achieve good performance because: 1) The anchoring mechanism is a good reception field initialization for limited-data and limited-lesion-category datasets; 2) The two-stage mechanism is a coarse-to-fine mechanism for the CT lesion dataset that only contains two categories (‘lesion’ or not ), i.e., first finds lesion proposals and then removes the FP proposals.
Q2: For the performance of small lesions. A2: Compared with the BM branch, our ABM branch increase small-lesion performance by 0.07 (IoU 0.5), but the ABM is not just proposed for improving the performance of small or big lesions. The over-represent and under-represent problem is a common problem in many per-pixel training mechanisms (e.g., segmentation) and it will influence the entire network training, all lesions’ training should benefit by solving the over and under represent problem.