Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Heran Yang, Jian Sun, Liwei Yang, Zongben Xu

Abstract

Cross-contrast image translation is an important task for completing missing contrasts in clinical diagnosis. However, most existing methods learn separate translator for each pair of contrasts, which is inefficient due to many possible contrast pairs in real scenarios. In this work, we propose a unified Hyper-GAN model for effectively and efficiently translating between different contrast pairs. Hyper-GAN consists of a pair of hyper-encoder and hyper-decoder to first map from the source contrast to a common feature space, and then further map to the target contrast image. To facilitate the translation between different contrast pairs, contrast-modulators are designed to tune the hyper-encoder and hyper-decoder adaptive to different contrasts. We also design a common space loss to enforce that multi-contrast images of a subject share a common feature space, implicitly modeling the shared underlying anatomical structures. Experiments on two datasets of IXI and BraTS 2019 show that our Hyper-GAN achieves state-of-the-art results in both accuracy and efficiency, e.g., improving more than 1.47 and 1.09 dB in PSNR on two datasets with less than half the amount of parameters.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_12

SharedIt: https://rdcu.be/cyl3O

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This work develops an unpaired method that enables efficient cross-contrast MR image translation. The authors present a hypernetwork-based approach to enable scaling of the networks filters (or normalization functions) according to the input/output contrast.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is very well written and the presentation is clear.
- As far as my knowledge goes, applying a hypernetwork-based approach for the synthesis of particular MRI contrasts is a novel and interesting idea, since only one encoder and decoder are necessary, while they learned filters are adapted to the particular contrast type.
- Good evaluation: comparison to many standard methods, ablation study concerning the losses and FS/CIN, different metrics and a significance test.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The main disadvantage of the method is that it is only applied on 2D slices of the brain. I assume this is due to computational restrains, however, the 3D nature of brain MRIs is crucial.
- In my experience, having such a large amount of different loss-functions is typically hard to optimize. I lack the training details concerning this issue.
- A general drawback of the method is, that the available amount of contrasts needs to be known prior to training, i.e. four different MRI contrasts. Thus, if new sequences (e.g. from another dataset) need to be translated to the known ones, it would be impossible with this method. The authors implicitly bump into this issue in their work, since the IXI dataset has less different contrasts than the BRATS dataset. This means that a reliable training (for a clinical application) requires a training on all possible contrasts, which is highly unlikely.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- In the form the authors answered most of the questions positively. While they use openly available data, they promise to release the code on GitHub. With the released code, the reproducibility will be very good, but at this stage it would be rather hard to reproduce (i.e. because of optimization issues).
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The general idea of the work is interesting, however not “ground-breaking”. The good evaluation shows the advantages of the presented method, however the lack of 3D application limits the practical use of the method. Also, the needed a-priori knowledge about all possible contrasts, limits the possible applications of the approach. I strongly advise to make the code available as promised by the authors.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Good novel idea and good validation of the method. However, the paper is more proof-of-concept that presenting an applicable approach.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

It intends to solve cross-contrast image translation task by proposing a unified hyper-GAN model. It has two dataset IXI and BraTS2019 for evaluation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea is interesting. It is well-written and has detailed explanation. It compares the proposed method with several baselines in the literature and also conducted ablation studies. It not only compares the performance but also compares the parameter efficiency of the model.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There are other state-of-the-art models in the literature with code such as Dual Generator Generative Adversarial Networks for Multi-domain Image-to-Image Translation. It would be great to compare with these methods.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The data are public datasets and the author promises to release the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It would be great to compare with the other sota methods in the literature. I wonder how helpful the generated data is for the segmentation tasks.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It is an interesting paper with evaluation on two public dataset. I think it is on the borderline of acceptance.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposed a unified multi-constrast MRI image translation framework based on unsupervised learning. A hyper-GAN model was built which included a hyper-encoder, a hyper-decoder, two contrast-modulators, a classifier, and several discriminators. A composite objective function was also proposed for improving the performance of the proposed network.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The authors built a unified multi-contrast MRI image translation network that be used to achieve any possible contrast conversion. Compared with existing multi-contrast MRI image translation method that can only achieve a certain type of contrast conversion, the proposed method worked in a multi-tasking way.
2. The proposed network were trained unsupervisedly, which mitigated the shortcoming of the requirement of paired data.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. There are two datasets used in the paper and the authors implemented their method on these two datasets separately. Since the proposed method is a unified multi-contrast method, it is unclear why the authors not choosing to implement their method by mixing the two datasets together.
2. The result comparison with other methods only focused on the overall level. Specific contrast translation result comparison should be added to show the effectiveness of the proposed method. The authors should also point out the percentage of every type of contrast conversion in the training data.
3. It seems that other methods for comparison like cyclegan can only implement a certain type of contrast conversion. Authors should discuss how they achieve other mentioned methods.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper can be easily reproduced since authors will make their code publicly available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors should consider adding experiments showing the result of specific types of contrast conversion. Also, the authors are suggested to train the proposed network on the dataset by mixing the two datasets and see if better results can be obtained.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Some factors like the novelty of the proposed method, experiment design and result analysis.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers agreed the idea of synthesizing different MR contrasts using a single encoder single decoder model is of interest. Most concerns are about the experimental validation. First, since some methods in comparison, e.g., cycleGAN, could not do arbitrary cross-contrast MR image translation, how their performance were obtained in Table 2 needs further clarification, since this is the major results of this work. Second, as Reviewer#3 pointed out, in addition to comparing the overall performance, comparing the performance of specific contrast translation is also important. Is there any discussion about this? Third, the proposed method is 2D in contrast to the 3D nature of MR images. This paper did not mention or discuss or compare with 3D unpaired medical image synthesis methods, such as 3D Cycle GAN. Could this be discussed during rebuttal since Review#1 concerned this might limit the practical use of the proposed method.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

Q1: Clarify the implementation of CycleGAN. A1: As discussed in “Comparison with CycleGAN” on page 8. CycleGAN is for one-to-one contrast translation. For a N-contrast translation task, we trained N*(N-1)/2 CycleGAN [1], each of which is for a specific pair of contrasts. We averaged the results over different contrast pairs as final accuracy of CycleGAN. [1] Zhu, J.Y., et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV 2017.

Q2: Compare specific contrast translation results. A2: Thanks. Due to space limit, we will try to put comparative results for specific contrast translation tasks in final version. For IXI dataset, our method achieved the highest accuracies for all 6 translation tasks in all metrics. For BraTS dataset, our method worked better than all compared methods for all 12 tasks in all metrics, except MAE score of StarGAN for FLAIR-to-T2 translation (0.0103 of StarGAN vs. 0.0105 of ours).

Q3: The method is only applied on 2D slices and might limit the practical use. A3: Thanks. First, in experiments, all the compared methods are trained on 2D slices, and evaluated on 3D volumes. The results in different metrics show that the learned 2D model works well on 3D volumes. In addition, our method is not restricted to 2D and can be naturally extended to 3D in essence. As reviewer pointed out, the major bottleneck is GPU memory instead of methodology. We will discuss this in “Conclusion”, and plan to extend our approach to multiple adjacent slices or 3D patches in future work.

Q4: Why not mix two datasets together? A4: In this paper, the two datasets are in different settings, i.e., one is with-skull and another is skull-stripped. As suggested, we additionally conduct experiments of training and testing on the combined dataset with 5 contrasts (T1w, T2w, PDw, T1CE, FLAIR) with both with-skull and skull-stripped brain images, which is more challenging than the separate training/testing setting. The results show that our method still performs best in all metrics (p<.001) and produces 0.0127/29.02/0.902 in MAE/PSNR/SSIM, compared with 0.0142/28.60/0.873 of StarGAN, 0.0168/27.00/0.831 of SUGAN, and 0.0145/28.00/0.879 of ComboGAN. We will try to add these results in paper.

Q5: The method needs to know the amount of contrasts before training, and is impossible to translate unseen sequences into the known ones. A5: We propose a unified GAN model that can flexibly translate between different contrast pairs using a unified network. It is much more efficient than the existing methods that use separate translators for different pairs of contrasts, or multiple encoders/decoders, with insufficient adaptiveness to different contrasts. All these methods including ours depend on the given contrast set. It is challenging to generalize to unseen sequences. One possible way to achieve this for our method is as follows. Different sequences can be represented by imaging parameters. We implicitly model the tissue parameters by common feature space, and imaging parameters by one-hot codes in hyper-encoder/hyper-decoder. If one-hot codes of imaging parameters can be generalized to continuous codes covering unseen sequences, our method might be able to generalize to unseen sequences. This will be left for future work.

Q6: Compare with SoTA methods, e.g., DGGAN. A6: We additionally compare with DGGAN. The results show that our method works better than DGGAN in all metrics (p<.001) and produces 0.0133/29.64/0.910 and 0.0070/31.90/0.930 in MAE/PSNR/SSIM on two datasets, compared with 0.0175/27.51/0.858 and 0.0081/30.81/0.908 of DGGAN. We will cite and compare DGGAN.

Q7: How to optimize the training loss? A7: As discussed in “Training loss” on page 5. We set the weights lambda_{id, rec, cla} to 0.5, 10, 0.2 to make each loss term within a similar range, and trained the model in 100 epochs using an Adam optimizer with a learning rate of 0.0002 and betas of (0.5, 0.999). We will add more details in paper.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

In the rebuttal, the authors have answered most questions from Reviewer #3 including the training details of the comparison methods like cycleGAN in Table 2, additional comparison results of specific contrast translation, and additional experiments on the mixed dataset. On the other hand, the rebuttal did not address the concerns with no comparison to 3D based synthesis methods, which although does not affect the major claims of this paper. My attitude to this work is neutral. The major concern is still that although arbitrary contrast translation is of certain interest, the methodological contribution of the proposed model seems to be a bit mild. It seems to be a borderline paper
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper presents an interesting take on image translation through an invariant latent space.

Overall the other reviewers, MR, and I found aspects/ideas in this paper interesting, and worthy of presentation at MICCAI. While several issues were brought up, I think they were quite well addressed in the rebuttal, such as why 2D vs 3D, how cycleGANs are tested, etc, and I believe the remaining issues brought up by the reviewers are minor. Overall, the idea bring some interesting insights that would be nice to appear at MICCAI. Of course, the authors should take all the feedback into consideration, make the necessary clarifications described in the rebuttal, as these are part of the reason for my acceptance.

Minor: HyperGAN might not be the best name, as there are already popular projects named this, which will make it harder for this paper to be found.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper included good new ideas and provides a good validation. Although implementation is done in 2D, authors discuss in their rebuttal that results on 3D images are promising and show feasibility, and that the algorithm itself would not be limited to 2D given sufficient computational resources. The paper has weaknesses also cited by reviewers, e.g. hard to optimize due to complexity, somewhat experimental validation, and not really be ground breaking, but it may have merit to be discussed with the MICCAI community.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

back to top

A Unified Hyper-GAN Model for Unpaired Multi-contrast MR Image Translation