Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Reuben Dorent, Samuel Joutard, Jonathan Shapey, Aaron Kujawa, Marc Modat, Sébastien Ourselin, Tom Vercauteren

Abstract

We introduce InExtremIS, a weakly supervised 3D approach to train a deep image segmentation network using particularly weak train-time annotations: only 6 extreme clicks at the boundary of the objects of interest. Our fully-automatic method is trained end-to-end and does not require any test-time annotations. From the extreme points, 3D bounding boxes are extracted around objects of interest. Then, deep geodesics connecting extreme points are generated to increase the amount of “annotated” voxels within the bounding boxes. Finally, a weakly supervised regularised loss derived from a Conditional Random Field formulation is used to encourage prediction consistency over homogeneous regions. Extensive experiments are performed on a large open dataset for Vestibular Schwannoma segmentation. InExtremIS obtained competitive performance, approaching full supervision and outperforming significantly other weakly supervised techniques based on bounding boxes. Moreover, given a fixed annotation time budget, InExtremIS outperforms full supervision. Our code and data are available online.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87196-3_57

SharedIt: https://rdcu.be/cyl28

Link to the code repository

https://github.com/ReubenDo/InExtremIS

Link to the dataset(s)

https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70229053

Reviews

Review #1

Please describe the contribution of the paper

-A new weakly supervised method is proposed for training 3D deep image segmentation based on only 6 extreme points by increasing the number of foreground points by also including the points on a generated geodesic that connects point pairs on facing planes for x,y,z axes. A key contribution is the inclusion of an pixelwise background probabilities inferred from an initial network mostly trained on the background pixels into the length function for the geodesic, thereby avoiding the geodesic to fall outside of the foreground object pixels.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-Presents a novel weakly supervised method for training 3D deep image segmentation based on only 6 extreme points and thus reduce annotation time for constructing training images. -Combines older conventional techniques such as geodesics computed from Dijkstra’s algorithm from image gradients with new techniques based on deep learning in a clever way. -Includes means of regularization for further improvement. -Demonstrates improved performance compared to geodesics generated from only conventional length functions based on gradients and euclidian distances, as well as bounding box annotations. -Paper structure is fairly clear. -Experimental evaluation is clear and sufficient, including ablation study, comparisons on computation time, extreme point and full pixel segmentation annotations, manual and simulated points, etc.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

-Details regarding the method are a bit lacking, such as the structure of the whole framework, computation of CRF regularization loss, network structure, etc.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

As the authors have stated they are willing to provide their code, I believe that their work would be indeed reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

-Presentation of whole proposed iterative framework, perhaps visually as a figure, could be included to improve clarity. -More targeted comparative evaluation with previous methods, including [15] also based on extreme points, could be provided to clarify the contribution of this work.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

-The proposed method seems to be technically sound with sufficient technical novelty. -The results are clearly presented with comprehensive analysis in regards to various aspects of the proposed method. -The proposed method is likely to have a positive impact in the relevant field of research.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper proposed a new method to generate weak labels for the image segmentation problem based on extreme points. Compared to previous work [15] that uses gradient geodesic, the main novelty of the proposed method is the deep geodesic (gradient+Euclidean+CNN output). Significant improvements are observed when using the new geodesic labels and the CRF regularization loss.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea to introduce CNN output to restrict the geodesic generation is novel and seems to work well compared to the methods without the restriction. The generated geodesic better delineates the target object compared to bounding boxes and requires less annotation effort compared to scribbles (as well as full annotation).
- The paper is very well written, with clear introduction and method sections. This provides a good reading experience.
- The experiments are well designed and the results show clear improvements over the baselines. The comparison to simulated extreme points provides a good reference in an ideal weakly supervised case and the comparison under different annotation budgets shows the advantage of the weak annotation methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The performance of the proposed method may highly depend on the shape complexity of the target object. The authors assume the foreground object is path-connected, which could be true in the vestibular schwannoma segmentation in this paper, but may not be the case in many other medical image segmentation tasks, e.g., brain tumor segmentation (multiple large and small objects), vessel segmentation and airway segmentation (very complex structure), etc. Since the contribution of the paper is more methodological, there lacks a discussion about the generalization of the current method to other segmentation applications.
- When the shape complexity of objects increases, the CNN output may also get worse, which may hinder accurate geodesic generation. In this case, the CNN output may be even less accurate than the image gradient. I suggest the authors discuss when would this happen.
- Although the CRF loss provides additional performance improvements, it may share a conflict on the motivation of the paper. In the introduction, the authors claim that using only the image gradient information to generate geodesic may lead to paths outside of the object. In the CRF loss, the inaccurate intensity information may also lead the segmentation outside (or inside) the object and results in poor regularization if the CRF effect dominates the training. I suggest the authors discuss this issue.
- There lacks a comparison to the weakly supervised methods that use scribbles, which could be more interesting than the comparison to bounding box methods. The bounding box is more meaningful in detection problem and may not provide many insights as baselines in segmentation problem. The scribbles may share similar annotation budgets as the extreme points and makes more sense in segmentation problem.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors list sufficient details and would provide source codes and data after acceptance, which shows good reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- As mentioned before, the main weakness of the paper is the discussion. To better show the methodological contribution, I recommend the authors consider the generalization ability of the proposed method to different segmentation tasks with more complex target object shapes. In other words, to give a more concrete scope when the assumption (there exists connected paths in the object) can be applied, when CNN output gets poor and when CRF loss provides unreliable regularization on wrong intensity information.
- Comparison with scribble methods would be more interesting than bounding box methods.
- In Fig.2, the gradient magnitude may not show the information properly, since it is too complicated without zooming into the region of interest. I suggest the authors to zoom in the same ROI, just like that in the right subfigures (blue and green voxels).
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper proposes a new weak label generation method that generates more reliable geodesics as training data for image segmentation problem. The method is novel compared to previous works and shows promising results. The paper is well written and easy to follow.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

In this paper, the authors proposed an end-to-end trainable extreme-point-based weakly-supervised framework for vestibular schwannoma segmentation. There two novel components (1) a shortest-path-based method for locating more possible foreground pixels which allow more accurate segmentation for non-convex shapes (2) a CRF regularised loss to enable end-to-end training.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel idea of using shortest paths to connect extreme points to find more potential foreground pixels, this is very important for complex 3D shapes.
- Novel idea of using CRF regularised loss for end-to-end training, this is very appealing since previous methods will rely on multiple rounds of training and pseudo label generation
- Well written paper, easy to read.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Experiment only performed on a single dataset, and the foreground object looks simple (convex looking shape, homogenous intensities), need to test on more datasets to ensure the generalizability of the method. Otherwise, please update the title to reflect the method is only tested for vestibular schwannoma segmentation.
- No comparison experiment with previous state-of-the-art method [15].
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors will share code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Very interesting method. Please compare with previous state-of-the-art [15] and also test the method on more datasets.
- It seems only 3 pathes are used to supervised the model (i.e. G^x, G^y, G^z). Why not use all 15 pairs? Please discuss more.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novel paper and the proposed end-to-end method is very appealing. But lack of comparison experiment and only tested on a single dataset.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

There exists a consensus among reviewers that the proposed method is interesting and novel. Reviewers also acknowledge the clarity of the paper and the experimental evaluation, as well as the potential impact on current weakly supervised segmentation methods. I therefore recommend early acceptance of this work. Nevertheless, there are few concerns that I would like to see addressed in the camera-ready version. First, the three reviewers suggest to include results from scribble based methods (e.g., [15]) to better show the contributions of this work. And second, R2 and R3 bring an interesting point regarding the generalization of their method to other segmentation applications. In particular, a discussion about the dependence of this method to the shape complexity of the target object should be included. Authors should attend to the main points in the reviews, when preparing a final version.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

N/A

back to top

Inter Extreme Points Geodesics for End-to-End Weakly Supervised Image Segmentation