Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Hongxiang Lin, Yukun Zhou, Paddy J. Slator, Daniel C. Alexander

Abstract

Multi-modal and multi-contrast imaging datasets have diverse voxel-wise intensities. For example, quantitative MRI acquisition protocols are designed specifically to yield multiple images with widely-varying contrast that inform models relating MR signals to tissue characteristics. The large variance across images in such data prevents the use of standard normalisation techniques, making super resolution highly challenging. We propose a novel self-supervised mixture-of-experts (SS-MoE) paradigm for deep neural networks, and hence present a method enabling improved super resolution of data where image intensities are diverse and have large variance. Unlike the conventional MoE that automatically aggregates expert results for each input, we explicitly assign an input to the corresponding expert based on the predictive pseudo error labels in a self-supervised fashion. A new gater module is trained to discriminate the error levels of inputs estimated by Multiscale Quantile Segmentation. We show that our new paradigm reduces the error and improves the robustness when super resolving combined diffusion-relaxometry MRI data from the Super MUDI dataset. Our approach is suitable for a wide range of quantitative MRI techniques, and multi-contrast or multi-modal imaging techniques in general. It could be applied to super resolve images with inadequate resolution, or reduce the scanning time needed to acquire images of the required resolution. The source code and the trained models are available at https://github.com/hongxiangharry/SS-MoE

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_5

SharedIt: https://rdcu.be/cyhUw

Link to the code repository

https://github.com/hongxiangharry/SS-MoE

Link to the dataset(s)

https://www.developingbrain.co.uk/data/

Reviews

Review #1

Please describe the contribution of the paper

In this paper, the authors suggest a Mixture of Experts method for MRI super resolution. The authors suggest a new module which is trained to discriminate the error levels of inputs estimated by multiscale quantile segmentation. Such a process of discriminating the inputs reduces the variance in each cluster of inputs. The authors show their result using a series of errors maps coronal slices of diffusion brain images and comparing their reconstruction with that of ground truth.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main contribution of this paper is that the proposed method discriminates the input data by predicting the error class in a self-supervised manner. This strategy clusters the input data into appropriate clusters reducing the variance in each cluster. This process makes the proposed method quite generalizable as the author claims. The method shows the results of through a series of coronal slices of brain images. Most of the super resolution methods apply the same reconstruction method across the whole image. However, the error characteristics for different tissue types across different modalities are different. The idea to cluster the inputs is a particularly novel one in my opinion and defnitely requires more research.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The errors are shown using color maps. However, an error histogram showing the extent of improvement between baseline and the proposed method will be nice. In addition, it will be great to see similar more slices on sagittal plane too. I do understand that showing all these results might be difficult due to space constraints. However, some comments would be greatly appreciated. I also wonder, if the method is limited to brain images or is quite general to other imaging modalities. I do not see a specific reason for the method to not be generalizable to other modalities. But authors should make some comments about the same. It wll also be great if the authors comment about the clusters they generated. If they reveal any patterns or any other similar insight. Doing so, would greatly improve the clarity of the paper.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have made the source code available for the future. However they are redacted at the moment. I will assume that the code will contain an appropriate readme file and example data for making the algorithm work.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper is well written and I have outlined some of the weakness of the paper. I acknowledge the space constraints of the conference article. However, adding few lines on them will make the paper well rounded. In the future, the authors can also test the limits of the algorithm. As I understand the authors have looked into a 2 fold isotropic upsampling and 2x upsampling in superior-inferior direction. I have encountered diffusion images with 1mm resolution in axial plane and 5.5 mm resolution in the coronal plane too. I am trying to hone at the point that the authors should try different upsampling scales and test the limits of the algorithm in the future.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

As mentioned earlier, it is a good direction of research in the super-resolution community. Through my personal experience in this field I found that super-resolution methods (when they work) do improve the visual quality of the image, but fail to add any value to any large scale statistical study using super resolved images. I am also not sure if a clinical decision can or should be made based on super-resolved images. I will be happy to hear any use cases of super-resolution images in a clinical workflow. Inspite of all this I do consider that the presented work is a good direction of research.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The paper presents a self-supervised mixture of experts model to predict high resolution contrast of quantitative MRI.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The use of SS-MoE is, to some extent, novel in this context. The topic is interesting as predicting high resolution quantitative MRI is challenging due to its wide range of intensity scale.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is a lack of test of significance in improvement. The visual results are also not impressive.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Method is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Please provide and discuss whether the improvement over base-line method is significant. From Table 1, while SS-MoE is better, it is not sure that the improvement is meaningful without numerical quantification.
2. From Fig. 3, the improvement over baseline method is not visually significant. Please provide more visual results or subsequent analyses (using the predicted data for other subsequent model) to discuss the value of the work. From what presented, it is hard to convince that the work would be beneficial for the community.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well-written but results are not convincing.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

This work present a method to explicitly assign an input to the corresponding expert based on pseudo error labels which is obtained by a gater module.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The motivation of this paper is clear.
2. The formulation is somehow novel.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. This paper does not show comparison between automatic aggregation MOE and the proposed explicit assignment MOE.
2. This paper only consider cases where different distributions are not overlapping. What the performance of this model is regarding the overlapping regions between different adjacent distributions is not clear.
3. Super MUDI is a famous challenge. It would be better if this paper can provide comparison results with papers coping with MUDI dataset.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The overall reproducibility is good. The dataset is publicly available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Corner cases such as the performance of the model in terms of different overlapping distributions are not demonstrated in this paper.
2. Since the overall novelty of the proposed model is limited. It would better to provide some evidence to prove that explicit MOE is better than automatic assignment MOE.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This work present a ensemble modeling way to solve super resolution task. It first generate the pseudo error label for each input according to their error value and then send it to the corresponding decoder to perform super resolution. The novelty of this model is limited while the motivation is clear.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers felt that the topic of self-supervised super resolution was important and that the method was reasonably novel. However some reviewers felt that the results did not clearly demonstrate significant improvement. The rebuttal should clarify how the benefits of the proposed algorithm are meaningful, and how it compares to other approaches, particularly those that entered the Super MUDI challenge and standard Mixture of Experts approaches.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

We thank all Reviewers who contributed their constructive comments. Here we respond to the major concerns from Meta-Reviewer:

The results did not improve significantly [R2] We do in fact test and report statistical significance of SS-MoE improvements over the other methods in terms of MSE score distribution over all volumes with p<0.001; see 1st paragraph, Section 3.2. We have added a pointer to Section 3.2 to highlight this.

How meaningful are the benefits of the proposed algorithm? [R1, R2, R3] We specify the benefits of SS-MoE originally in Section 4: 1) Our SS-MoE benefits the generalised SR reconstruction of multi-modal/multi-contrast images with large diverse voxel-wise intensities. 2) SS-MoE is a general framework that can build upon any encoder-decoder network architecture, which not only boosts the performance but also reduces the variance of MSE scores over all volume data. 3) Training SS-MoE by gradient descent works more robustly than the conventional MoE. 4) Our SS-MoE is economic and scalable. The memory cost of SS-MoE is not more than the baseline cost, whilst that of MoE highly depends on the number of baseline experts.

Compare to the methods that entered Super MUDI challenge [R3] We clarify that at the time of writing this paper, Image Quality Transfer using SR U-net with a 3D patch-based technique, one of the top four models on the leaderboard, was the only one which is documented sufficiently well to reproduce. Recently, V Nash et al, ISMRM’21 (16-20 May) indicates that our reimplemented baseline reasonably represents all the other top four models due to their similar MSE behaviours. We will pay attention to the source codes of other best models released in the future for comparison.

Compare to MoE [R3] Thank you for the great suggestion. We have adapted the MoE model with separate decoders as experts [8]. We ran MoE on 100k Iso. data and obtained Stats. of MSE score distribution over the 5-fold cross validation (CV): Average 0.30+/-0.16, Variance 7.46+/-10.72, Median 0.052+/-0.014, Max 67.90+/-70.27. MoE had smaller bias to ground truth but much larger deviation in between different fold CVs. It is known that the gradient descent algorithm for learning the conventional MoE may easily get stuck in local minima due to the non-convex loss. [A Makkuva et al, ICML’19]. SS-MoE can work as robust as the baseline model by retaining its convex loss. The full results will be appended to Table 1 with minor edits in Section 3.2.

Other concerns from Reviewers:

Generalisation of SS-MoE to other modalities [R1] It can be generalised to not only [26, 29] but also multi-parametric spine cord MRI [F Grussu et al, NeuroImage’20] and multi-centre multi-shell dMRI [Q Tong et al, Scientific Data’20].

General input resolution for SR [R1] This is a great future direction for the generalised SR task with non-integer scale factors. Using Meta Upscale Module [X Hu et al, CVPR’19] may solve it.

Show better visualisation of Fig. 3 and more sagittal-plane slices [R1, R2, R3] We have followed the suggestion from R1 to append an error histogram to appendix. We can clearly see SS-MoE behaves stably better than any others. Next, as the coronal views in Fig. 3, the concentration of the large-error voxels for the sagittal views was distributed at the top image boundary, cerebellum boundary and ventricles.

Consider the overlapping distribution of contrasts in SS-MoE [R1, R3] The second experiment already gives insight into this question. We observed the overlap in between some adjacent clusters, which may confuse to posit the hard decision boundaries. However, the combined result of Table 2 and Fig. S2 indicates that mis-classified labels were mostly concentrated around the hard boundary, which may not influence too much to the performance of SS-MoE. We will accordingly edit a few phrases in Section 3.2.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Although some aspects of the paper were found to be meritorious, the major issue was in the evaluation, which used data from the Super MUDI challenge. Although the data was used, the evaluation was not performed in the same manner that the challenge was conducted, making it difficult to compare the performance of the proposed method to the participants of the challenge. This issue was not sufficiently addressed in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Authors present a Mixture of Experts method for MRI super resolution. Reviewers acknowledge that the contribution in this application domain is somehow novel but that results did not clearly demonstrate significant improvement. I think though that the most sever reviewer indicated low confidence in his/her own review. In the rebuttal the authors clarify that indeed statistical significance was presented, at least for the MSE. They also justify why at the time of the submission other methods from Super MUDI challenge could not be chosen (not well documented in order to be reproduced). They also clarify and address many other points raised by R2 and R3, including a new comparison using separate decoders as experts (with smaller error but bigger variance). I think the paper is of interest and given the additional information given by the authors I would recommend it for acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I echo with the primary AC stating that the self-supervised image super-resolution is relatively novel. Although two reviewers were leaning to a boarderline reject, the authors have well clarified the major raised concerns and succinctly articulated their point of view. New results were also provided against the ablated version of the paper. I would lean for an accept —although I would not mind if it eventually gets rejected.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

back to top

Generalised Super Resolution for Quantitative MRI Using Self-Supervised Mixture of Experts