Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Wei Shao, Indrani Bhattacharya, Simon J. C. Soerensen, Christian A. Kunder, Jeffrey B. Wang, Richard E. Fan, Pejman Ghanouni, James D. Brooks, Geoffrey A. Sonn, Mirabela Rusu

Abstract

The interpretation of prostate MRI suffers from low agreement across radiologists due to the subtle differences between cancer and normal tissue. Image registration addresses this issue by accurately mapping the ground-truth cancer labels from surgical histopathology images onto MRI. Cancer labels achieved by image registration can be used to improve radiologists’ interpretation of MRI by training deep learning models for early detection of prostate cancer. A major limitation of current automated registration approaches is that they require manual prostate segmentations, which is a time-consuming task, prone to errors. This paper presents a weakly supervised approach for affine and deformable registration of MRI and histopathology images without requiring prostate segmentations. We used manual prostate segmentations and mono-modal synthetic image pairs to train our registration networks to align prostate boundaries and local prostate features. Although prostate segmentations were used during the training of the network, such segmentations were not needed when registering unseen images at inference time. We trained and validated our registration network with 135 and 10 patients from an internal cohort, respectively. We tested the performance of our method using 16 patients from the internal cohort and 22 patients from an external cohort. The results show that our weakly supervised method has achieved significantly higher registration accuracy than a state-of-the-art method run without prostate segmentations. Our deep learning framework will ease the registration of MRI and histopathology images by obviating the need for prostate segmentations.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_10

SharedIt: https://rdcu.be/cyhPR

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper tackles the problem of intermodal registration between histology sections and pre-operative MRI slices in the prostate. Their main contribution is to develop a registration algorithm that does not need MRI prostate segmentations at inference. They use a VGG network to extract features from each image and use a low dimensional deformation model (affine + non-linear) to align both domains. They use a novel loss function that combines (i) the cross-alignment between modalities using the dice score between prostate masks available only during training and (ii) the alignment between randomly deformed copies of each domain using the MSE.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The manuscript is organized nicely: the problem is properly motivated, the goal is presented clearly and the main contributions are well explained. To the best knowledge of the authors and the reviewer, this is the second attempt to using neural networks for histopathology-MRI registration in the prostate, improving the results upon the previous work (ProsRegNet, [9]).

They use a deformation model that can be found in the literatue (e.g., Shao et al. [9]) but many differences can be spotted regarding:
- The feature extractor used: VGG vs. ResNet.
- Model parameter computation: GAP + concatenation layer vs. correlation layer
- The loss function: a cross-domain term using the prostate segmentation (available only during training) from both MRI and histology images.
Another strong point is that they show the feasibility of the model on two different cohorts: one coming from the same distribution as training data and another external cohort. Significant improvements (p<0.05) are found compared to previous state-of-the-art using neural networks for registration.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

My main concern is that the authors did not explain clearly how their competitor method was trained: e.g., what parameters were used or whether prostate segmentations were used at training and/or inference or not. Hence, it is hard to properly assess the comparison results.

The methodology presented here is not entirely novel. They use weakly supervised training that doesn’t require labels at inference time, quite relevant when using pre-operative MRI scans.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have commited to make their code available. However, some of the design decisions were not justified (e.g., hyperparameter selection).

The cohort used for training is not publicly available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
While the text is well written, I believe that some minor corrections will improve readability:
- Theta is used to represent the affine parameters and the nonlinear parameters separately. It may be worth using a superscript to distinguish them.
- What is the magnitude of distance metrics (landmark error, urethra deviation).
Moreover, some decisions about the method used may be better justified in the text:
- The number of frozen layers from VGG. If the authors have tried different options it may be noted in the text why they decide to keep the presented configuration
- The weight losses used for each component: seg, int, reg.
Finally, it will be interesting to know how the landmarks were placed in the corresponding image pairs. For example: spatially random or mainly at the boundaries?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

While the formulation is not entirely novel, they use several ideas togehter and apply them for the first time to alignment of intra-operative MRI scancs and histopathology sections. They show the benefit of their models in two different cohorts for several metrics. Hence, I believe that this paper should be probably accepted.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

To compare two MRI to pathology registration methods. One based on grey level and the other on manual prostate segmentation. A dataset comprising 135 training and 22 test cases consisted of MRI and images of cut slices from pathology specimen pre-molded using 3D prostate segmentation models obtained from MRI. The grey value-based registration method used an ImageNet pre-trained VGG network that was partially retrained to produce a registration transformation. A downloaded but undescribed ProsRegNet was used to align based on segmentation. The resulting DICE overlap and other performance measures show a slight improvement of the gray value-based method over the segmentation-based method.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Well-written paper. Good dataset.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The claim should be limited to pathology images compiled using a 3D mold. This mold preforms the pathology specimen, which is crucial in the alignment process. Wihtout the mold the registration becomes much more difficult and the claim is definitely not justified in these cases.

The use of a set of patients that underwent prostatectomy introduced a bias in any method that will use this data. The authors claim that: “Cancer labels achieved by image registration can be used to improve radiologists’ interpretation of MRI by training deep learning models for early detection of prostate cancer.” Which is not true. The current most common case of MRI is low risk cancer patients that may or may not have clinically significant cancer requiring treatment such as prostatectomy. It is crucial for a radiologist to learn to discriminate cases with or without signs of clinically significant cancer. The selection to prostatectomy excludes cases without clinically significant cancer. This introduce a significant bias. I think this data set is very relevant to the problem of determining the size of a prostate cancer for treatment and various other related topics.

It is a bit strange that segmentation methods perform slightly worse. The segmentation presumably is used for the 3D mold model. The segmentation of the prostaet on the specimen should than be relatively simple. Aligning that to the 3D model should be fine right? The details of the mold segmentation are missing to verify this.

The segmentation based registration method selected is chosen because it was the only one that was available for download. I VGG was a dedicated algorithm fine tued to the problem at hand. I miss description of the downloaded method. There is always a disadvantage to ‘‘being” the algorithm compared to, as it will probably not receive the same amount of attention as the authors’ method. A sens a light bias.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Considering my review above I could imagine this may not be easily reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

See above
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Great dataset and intersting to discuss at a meeting.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper presents a novel regsitration framework that aligns the Histopathology Images with the cancer labels and the pre-operative MRI images without the cancer labels.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The registration algorithm is novel, the application has significant clinical value, the conclusion is supported real-world data and fair settings for all compared methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. It is not clear why the authors refer to their algorithms as un-supervised registration methods since the prostate segmentation is used in the training process, although the authors claim that the prostate segmentation is not used in the testing scenario. This point also challege the authors’ claim that one of the main advantage of their algorithm is that the prostate segmentation is not needed during the test stage, This is weird to some extent, in my opinion.
2. Some key(or state-of-the-art) references in the medical image registration are missing.
3. The novelty(what has been changed?) of the algorithm should be justifed over the existing registration methods.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Satisfactory.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. It is of great value if the author can elaborate more about the differences between their proposed registration algroithm and the compared state-of-the-art method ProsRegNet.
2. It is also very helpful if the authors can add experiments that compare ProsRegNet and the proposed methods, where the prostate segmentaons are also used in the training.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The writing is good, and the registration algorithm is well illustrated with moderate validations.

The major point that hinders the reviewer to a strong accept is that the reviewer doubt the authors’ claim of the advantage of not using the prostate segmentations, since prostate segmentation is still utilized in the training stage.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Although an interesting work with a long-standing challenge in prostate MR-to-histo registration, there are a few important comments from all three reviewers need to be addressed: 1) a better review and perhaps additional references in weakly supervised registration methods should be acknowledged and discussed (for future work, compared) would help to better position this work (as commented by Reviewer 1 and 3); 2) a strong rebuttal to comments raised by Reviewer 2, in regarding to the clinical relevance of the experimental design and results, is also needed; 3) address the comments on the compared segmentation methods, raised by all three Reviewers - is it a fair comparison or a comparison with existing method without any adaptation to the proposed one fine-tuned for the data/application?
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We thank the reviewers for their constructive comments. Abbreviation: Reviewer (R).

Clarification of the need for discarding prostate segmentations at inference and comparison with the ProsRegNet method. (R1, R2, R3) While prostate segmentations on MRI are required for ultrasound-MRI fusion targeted biopsy, such segmentations are not routinely performed in men undergoing prostatectomy after conventional prostate biopsy. Moreover, pathologists do not segment the prostate on digital histopathology. Since prostate segmentation on MRI and histopathology are not always immediately available and gland segmentation is a time-consuming task prone to errors, there is value in registration of MRI and histopathology without the need for gland segmentation. The major advantage of our method is that it avoids the above shortcomings of prostate segmentation by not relying on it at inference. Since the goal of this paper is to develop a registration approach that does not require prostate segmentation, both our method and the competing ProsRegNet method were tested without using prostate segmentations.

Describe the differences between the proposed method and the ProsRegNet method. (R1, R2, R3) (a) Network architecture. The ProsRegNet method used ResNet101 while our method used VGG16 for feature extraction. The ProsRegNet method used a correlation layer to compute the correlation between feature maps while our method used the global average pooling layer to reduce the size of each feature map. (b) Loss function. The ProsRegNet method used the mean squared error (MSE) as the training loss. Our proposed method used MSE and two additional terms: the Dice coefficient loss to align the prostate boundaries and the regularization loss to guarantee the smoothness of the transformations. (c) Training dataset. The ProsRegNet method used masked synthetic unimodal image pairs for the training. The proposed method used unmasked synthetic unimodal image pairs, unmasked real multimodal image pairs, and manual prostate masks for the training.

Discuss previous weakly supervised registration methods to justify the novelty of the proposed method. (R1, R3) Many weakly supervised registration methods have been proposed for image registration [A-C]. Our method used both label-driven and image-driven losses, unlike Hu et al. that only used label-based loss [A]. Moreover, others used multimodal losses for the training [B-C], however, these methods are not suitable for registration of MRI and histopathology due to their large differences in appearance. A. Hu, Y., et al. “Label-driven weakly-supervised learning for multimodal deformable image registration.” ISBI 2018. B. Balakrishnan, G., et al. “VoxelMorph: a learning framework for deformable medical image registration.” IEEE TMI 38.8 (2019): 1788-1800. C. de Vos, B., et al. “A deep learning framework for unsupervised affine and deformable image registration.” MEDIA 52 (2019): 128-143.

Biased caused by the use of patients that underwent radical prostatectomy. (R2) We acknowledge that the use of patients undergoing prostatectomy might introduce spectrum bias in our cohort that does not translate into other populations (e.g., candidates for active surveillance). However, we used grade as a surrogate for the aggressive disease since it has been demonstrated to be the most powerful predictor of outcome in localized prostate cancer. Moreover, grade is also used for clinical decisions (e.g., selection of patients for active surveillance). As such, radical prostatectomy cases are ideal for model building as they allow: (1) precise registration of MR and histopathology images, (2) provide ground truth cancer label at every voxel within the prostate and (3) diversity of low- and high-grade cancers as they often coexist within the same lesion. However, such granularity of labels cannot be achieved in subjects that only undergo biopsy as the prostate is only sampled at the biopsy sites.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The use of the technical differences to the existing registration frameworks to argue for their novelty claim, e.g. ResNet vs. VGG, a combination of two existing losses, MSE and label-driven dice. This does not justify a strong claim of novelty or avoiding reasonable comparing/discussing/acknowledging these prior work.
I also agree with three reviewers that the justification for not using segmentation needs to be a stronger, as this mr-tohisto registration task has a primary goal for retrospective validation and automatic segmenting prostate from either MR or WSI isn’t challenging. The rebuttal did not address the issues on the fair comparison. More importantly, the results may be misleading in current presentation with much inferior results from an un-optimised segmentation-based method that also is arguably easier to get better performance. However, given the nature of the preliminary results with a small data set, i consider the work has the potentials to be strengthened and could make an much greater impact.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

20

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This work tackles one of the MIC challenges that has been researched extensively in the past and yet not resolved with a clinically acceptable solution. The proposed solution is another interesting approach with promising results. Authors adequately addressed reviewers’ comments and concerns in their rebuttal response. Overall, I find this work of interest for MICCAI 2021 audience.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors rebuttal accounted for most of the critical issues raised in the first round. Their response to the bias caused by the patients’ cohort is convincing.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

back to top

Weakly Supervised Registration of Prostate MRI and Histopathology Images