Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Mohammad Hamghalam, Alejandro F. Frangi, Baiying Lei, Amber L. Simpson

Abstract

In large studies involving multi protocol Magnetic Resonance Imaging (MRI), it can occur to miss one or more sub-modalities for a given patient owing to poor quality (e.g. imaging artifacts), failed acquisitions, or hallway interrupted imaging examinations. In some cases, certain protocols are unavailable due to limited scan time or to retrospectively harmonise the imaging protocols of two independent studies. Missing image modalities pose a challenge to segmentation frameworks as complementary information contributed by the missing scans is then lost. In this paper, we propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. Instead of designing one network for each possible subset of present sub-modalities or using frameworks to mix feature maps, missing data can be generated from a single model based on all the available samples. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing. Our experiments against competitive segmentation baselines with missing sub-modality on BraTS’19 dataset indicate the effectiveness of the MGP-VAE model for segmentation tasks.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_42

SharedIt: https://rdcu.be/cyl8C

Link to the code repository

https://github.com/hamghalam/MGP-VAE

Link to the dataset(s)

https://www.med.upenn.edu/cbica/brats2020/data.html

Reviews

Review #1

Please describe the contribution of the paper

The authors have proposed a method for the imputation of missing sub-modalities using Multi-modal Gaussian Process Prior Variational Autoencoder. They have used BRATS’19 datasets to validate the proposed method and have compared it with some of the methods in the literature.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The method is novel and interesting.
- The authors have clearly explained the data collection and processing pipelines.
- The dataset that has been used and the number of cases are representative of patient population. This is a very good paper with a strong technical innovation.
- The authors have performed a proper statistical significance analysis of results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The results in Table 1 is not sufficiency informative, as the authors have not provided any measures of spread.
- It is not clear how the authors have optimized the hyper-parameters.
- Number of subjects for the validation is not clear.
- There is no discussion about the limitations of the study.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Based on the text and the checklist, the paper provides sufficient details about the models datasets, and evaluation.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1- Please report median and IQR range for table 1. 2- Please provide more information about the validation strategy and hyper-parameter optimization.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a good paper with a very good technical innovation. Large number of subjects are used for the validation.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

In this paper, the authors propose a GP-VAE method for addressing the modality missing problem. The core of their method is the kernel design. A Kronecker production is used to link patient features and modality features together.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Generally, this paper well describes the problem and the solution. The logic behind the methodology makes sense. The usage of Kronecker product realized the coupling of two types of features. This paper provides an applicable application scenario.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The novelty of the proposed paper seems limited. I didn’t find explicit explanations about the differences between GPPVAE and the proposed method. In the last paragraph, the authors also mention that this work is the extension of GPPVAE. So Can I understand that the contribution is the perspective of application? (2) Since MGPVAE has close relationship with GPPVAE, I would say the authors should consider adding GPPVAE as a comparison method. (3) Some expressions are confusing. In Formulation section, what does the K mean in the definition of Y (R^N*K)? In Fig.1, what does the two columns of Y-tilda=Y mean? Does it mean the prediction of each z ? A clearer notation may be more helpful. What is the kernel basis used?
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Yes, the proposed method can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) In Fig. 2, I’m curious why the performance of CVAE is so much worse than linear interpolation. Any explanations? (2) The authors may be interested in cascaded residual autoencoder for this task, As far as I know, it may also works. It will an interesting and strong reinforcement to the experiment. (3) I notice in Table 1, all experiments have missing modalities. It is better to have a comparison with completed modalities if such data is available. It will be more interesting to see the difference between completed modality cases and cases that artificially losing some modalities.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I care more about the novelty of the methodology. I hope to see some clarifications about this point.
What is the ranking of this paper in your review stack?

6
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposes a new methodology for glioma segmentation from multi-modal MRI structural data under missing sub-modalities. The proposal employs the Gaussian process prior variational autoencoder and models correlations between sub-modalities and subjects. The authors designed an approach to share a common feature space from sub-sampled data instead of training combinations of modalities or synthesizing missing modalities.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper’s motivation is correctly defined, and the state-of-the-art presents recent advances in the field, pointing out (possible) limitations of the current methods. The method uses a publicly available multi-modal MRI data set from the BraTS’19 database used for previous MICCAI challenges. The technique follows a conservative mathematical formulation referring to the state-of-the-art where necessary. The method is based on the Gaussian process prior variational autoencoders that address a limitation of variational autoencoders presenting a latent variable as a i.i.d. The experiments in the paper are correctly handled, including state-of-the-art evaluation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The method benefits from the Gaussian process prior variational autoencoders presented in the paper by Casale et al, 2018 [1].

Although the improvement over the HVED method in terms of a Dice coefficient is small, I guess it is difficult to go beyond a barrier using variational autoencoders. On the other hand, significant improvement is observed in most cases.

In my opinion, section 2 is a bit too long. Consequently, experiments included in section 3 and discussion part (section 4) are minimized too much.

[1] Casale, Francesco Paolo, et al. “Gaussian process prior variational autoencoders.” arXiv preprint arXiv:1810.11738 (2018).
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The proposal uses a publicly available BraTS’19 database. The authors claim they will share the code and training parameters on GitHub soon. I trust the Authors to have reproducible research.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Fig. 2(a, third column) possibly shows incorrect bit encodings as the column includes two missing sub-modalities.
2. My suggestion is to extend Fig. 2 (c) towards including a separate analysis for one and two missing sub-modalities.
3. Fig. 2(b, top) shows sub-modalities covariances. I wondered if those covariances are only positive?
4. Another suggestion is to shorten section 2 and extend experiments (section 3) and discussion part (section 4).
5. I suggest the Authors revise the References as incomplete entries are provided.
Glimoa -> Glioma
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Generally, the paper (possibly) follows the requirement of the MICCAI conference in terms of current research trends, conservative mathematical formulation, and experimental set-up. The article is correctly written, and the organization of the paper is correct except a bit too long section 2 and too a short discussion part.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The methodological contribution of the paper was found sound and clearly illustrated. There are some remarks concerning the novelty of the contribution, in particular concerning the comparison with the original GPPVAE. Besides HVED and CVAE, there is also an important literature on multi-modal/multi-channel VAE that the authors should consider for this work. Finally, some aspects of the papers appear unclear, in particular respect to the experimental validation.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We thank the reviewers for their constructive comments and note that two reviewers and the meta reviewer ranked our paper in the 1st and 2nd positions in their stacks, suggesting that our paper is of high quality and of interest to the MICCAI community. The primary score driving consideration appears to be the comment by meta reviewer concerning the novelty of our work. We emphasize that the proposed method, MGP-VAE, is certainly not an application of GPPVAE. Although both approaches are based on Gaussian Process (GP) prior, the GP enables the specification of sample correlations through different kernel functions. To model covariance between MRI sub-modalities (Flair, T1, T1c, T2), we designed an appropriate kernel function to capture multiple levels of correlation between sub-modalities and subjects in 3D MRI scans. This kernel is discussed in Section 2.1 and designed to be used when one or even more MRI sub-modalities are missing. Our design is entirely different from the GPPVAE, in which its kernel is composed based on the rotation angle of faces in various poses inside 2D input images. Furthermore, as we mentioned in P2:L32, MGP-VAE has a 3D architecture to address input 3D MRI volumes and outperforms SOTA in terms of DSC from multi-modal brain tumour MRI scans with any configuration of the available sub-modalities.

Responses to other points in meta-review:

1) Regarding comparison with GPPVAE, as its kernel functions were optimized to predict unseen face poses based on view angle in 2D images, the comparison with this GPPVAE might not be possible. Indeed, we do not have any rotation in the input 3D MRI scans of patients and only model correlation between submodalities and subjects.

2) Regarding unclear items: In Y (R^N*K), K denotes k-dimensional representation for N samples.

In Fig. 1, Y-tilda and Y are the predictions of each z.

Hyperparameters were experimentally optimized, and important ones were described in P5-L14:19.

We divided the dataset into three separate subsets, including train, valid and test, with ratios of 70%, 15% and 15%, respectively. To bypass over-fitting, we applied early stopping on the validation set.

Responses to the individual reviewer:

R1: We will report the median and IQR range for table 1 in our final revision.

R2: In Fig.2c, R2 asked why the performance of CVAE is less than the linear interpolation of VAE. It is true that CVAE improves VAE in image generation by conditioning the encoder and decoder to the desired input. However, when we have missing input data (our case), CVAE has a confined ability to create the latent variable for unseen data compared to VAE. This might be because CVAE is more restricted to learn particular data features (latent representation) from observed input data. Therefore it has dedicated latent variables with limited features from missing data. In fact, the latent space of VAE contains more general features which can be used to predict missing data.

Regarding comparison with completed modalities, we have almost the same performance in Table 1 in case all sub-modalities are available (without imputation). Our method is designed and optimized to address problems where either one, two, or three of four sub-modalities may be missing. When all the sub-modalities are available, this is a different scenario.

R3:

I will correct the bit encoding of the first row in Fig. 2(a) the third column.

The magnitude of the covariances is shown in Fig. 2(b, top).

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The rebuttal mainly focuses on the discussion of the novelty of the proposed framework, especially with respect to the baseline GPPVAE model, and on the clarification of the experimental setup. Although the experimental section was originally found to lack clarity on certain aspects (e.g. statistical results of Table 2, optimization and validation scheme), the authors positively clarified a number of remarks. Overall, the proposed approach seems well motivated and interesting for the conference.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I am quite confused by the rebuttal of the paper – the paper is definitely a modification of the GPPVAE, which the authors rightfully state in the paper. There’s nothing wrong with this, they adapt a method to their problem – if done insightfully, with a clear explanation of why this is non-trivial and interesting. I believe the reviewers’ request for clarity on the difference of the methods is also appropriate, and easy to address while improving the paper.

I find the authors’ response quite weird, with statement like “We emphasize that the proposed method, MGP-VAE, is certainly not an application of GPPVAE”, “Our design is entirely different from the GPPVAE”, which overstate the contribution and in my opinion weaken the paper’s chance of getting accepted. The GPPVAE contribution was the idea of putting a sample correlation in a VAE prior, not the specific GP kernel used (which was clearly just for demonstration). Saying that the methods are completely different because a different kernel is used is inappropriate and in my opinion weakens the author’s grasp of literature.

I believe the extension and original work is borderline acceptable and will recommend acceptance. However, the authors must clarify the ‘unclear items’ and done in the rebuttal, and clarify that the proposed method modified the GP kernel of the GPPVAE and 3D aspect (thus addressing the reviewer/MR concern).
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a method using a multi-modal Gaussian process prior variational autoencoder to solve the problem of modality missing. The key point of the method is the kernel design to capture different levels of correlation between sub-modalities and subjects in 3D MRI scans. The work is interesting and has a certain novelty. The experiment shows some improvements compared to SOTA. My proposition is “accept”.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

back to top

Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation