Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ziqi Yu, Yuting Zhai, Xiaoyang Han, Tingying Peng, Xiao-Yong Zhang
Abstract
Automatic segmentation of mouse brain structures in magnetic resonance (MR) images plays a crucial role in understanding brain organization and function in both basic and translational research. Due to fundamental differences in contrast, image size, and anatomical structure between the human and mouse brains, existing neuroimaging analysis tools designed for the human brain are not readily applicable to the mouse brain. To address this problem, we propose a generative adversarial network (GAN) - based network, named MouseGAN, to synthesize multiple MRI modalities and to segment mouse brain structures using a single MRI modality. MouseGAN contains a modality translation module to project multi-modality image features into a shared latent content space that encodes modality-invariant brain structures and a modality-specific attribute. In addition, the content encoder learned from the modality translation module is reused for the segmentation module to improve the structural segmentation. Our results demonstrate that MouseGAN can segment up to 50 mouse brain structures with an averaged dice coefficient of 83%, which is a 7 ~ 10% increase compared to baseline U-Net segmentation. To the best of our knowledge, it is the first Atlas-free tool for segmenting mouse brain structures from MRI data. Another benefit is that with the help of the shared encoder, MouseGAN can handle missing MRI modalities without significant sacrifice of the performance. We will release our code and trained model to promote its free usage for neuroimaging applications.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87193-2_42
SharedIt: https://rdcu.be/cyhMk
Link to the code repository
https://github.com/yu56500/MouseGAN
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a novel generative adversarial network for simultaneous image synthesis and segmentation for mouse brain MRI. It contains a modality translation module to project multi-modality image features into a shared latent content space that encodes modality-invariant brain structures and a modality-specific attribute. In addition, the content encoder learned from the modality translation module is reused for the segmentation module to improve the structural segmentation. The method can segment up to 50 mouse brain structures with an averaged dice coefficient of 83%, increased by 7~10% compared to U-Net segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors claim that it is the first atlas-free tool for segmenting mouse brain structures from MRI data.
- This paper proposed a translation module, where different domains can be encoded into a shared space.
- The experimental results show superior segmentation performances.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some modules in Fig 1. lack description, i.e. what is domain code?
- It would be interesting if compared to atlas-based methods.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The auhors claim to open source their codes with trained model.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
This paper presents a novel generative adversarial network for simultaneous image synthesis and segmentation for mouse brain MRI. In particular, a translation module is proposed, where different domains can be encoded into a shared space. The experimental results show superior segmentation performances. My comments are listed below:
- Some modules in Fig 1. lack description, i.e. what is domain code?
- It would be interesting if compared to atlas-based methods.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper presents a cycle-GAN style network to model multi-parametric MRI mouse brain images. Overall, the idea is interesting and the exprimental results looks promising. I, therefore, recomand to accept this paper.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
1) In the paper, the authors proposed an atlas-free method for the segmentation of mouse brain which is quite different from human brain. 2) The proposed method is very flexible as it can generate segmentation with any combination of the five common modalities, i.e. T1, T2, T2-star, QSM, magnitude MR images as input. 3) The content encoder learned from the modality translation module is reused for the segmentation module, which allows for an effective training with a small number of segmentation annotations.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper present an important segmentation method for mouse brain which is quite different from human brain in contrast, image size, and anatomical structure (a novel application).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The idea that uses GAN-based multiple MRI modalities synthesis to medical image segmentation is not new [1,2,3]. [1] Chen, C., Dou, Q., Jin, Y., Chen, H., Qin, J., Heng, P.-A.: Robust Multimodal Brain Tu-mor Segmentation via Feature Disentanglement and Gated Fusion. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., and Khan, A. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. pp. 447–456. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_50. [2] Yang, J., Dvornek, N.C., Zhang, F., Chapiro, J., Lin, M., Duncan, J.S.: Unsupervised Domain Adaptation via Disentangled Representations: Application to Cross-Modality Liver Segmentation. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., and Khan, A. (eds.) Medical Image Computing and Computer Assisted Intervention –MICCAI 2019. pp. 255–263. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_29. [3] Jiang, J., Veeraraghavan, H.: Unified Cross-Modality Feature Disentangler for Unsuper-vised Multi-domain MRI Abdomen Organs Segmentation. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., and Joskowicz, L. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 347–358. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_34.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I think the reproducibility of the paper is acceptable. But still hope that the mouse brain dataset the authors used in the paper can be publicly released.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Can the proposed method be used for other challenging segmentation tasks, such as infant human brain segmentation?
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The study of mouse brain segmentation is very little compared to human brain segmentation, and the segmentation results are good. It can segment up to 50 mouse brain structures with an averaged dice coefficient of 83%, which is a 7 ~ 10% increase compared to baseline U-Net segmentation.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
4
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
The authors present a GAN based approach to segmenting mouse brain structures from MRI. Existing brain segmentation tools developed on humans are ineffective in mouse images, and there are additional difficulties in modeling mouse images (e.g., scarcity of data, variance in imaging modality). The authors propose a unified model across imaging modalities that can incorporate multimodal input as well as translating across modalities.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors approach a problem that has relatively little prior work, but could have a research impact. The proposed model is able to provide brain segmentations across several MR protocols and is also able to generate images of different protocols for a given subject.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The Dice scores achieved are moderate and it is hard to contextualize without prior state-of-the-art methods for comparison. The evaluation of the image synthesis is limited with only a single example shown and no quantitative evaluation.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
All criteria are met, so it appears to be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The authors mention in the introduction that an atlas would be impractical to produce for mouse brains, which is part of the motivation for creating an atlas-free approach. However, in the data description, the authors state that their labels were generated using an unspecified atlas-based method. This would suggest that an atlas already exists and that methods are already available for applying it to this task. The existence of an atlas would seem to contradict the argument in the introduction. Is there a reason why this atlas might not be available for others?
The authors qualify the novelty of their method by saying it is the first “atlas-free tool” suggesting that there are atlas-based methods (possibly including the one used to help generate labels for the current study). A comparison of the results from the proposed model to some of these methods would help contextualize the results. If the atlas-based methods outperform the proposed method, it could still be useful if the atlas is not widely available or if it is not applicable in some situations.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The problem of segmenting mouse brain MRIs is necessary as a first step in some research tasks, and there is limited existing work in this space. The proposed model is interesting in that it not only can segment mouse brains using several MRI protocols, but it is also able to translate images across protocols.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
4
- Reviewer confidence
Somewhat confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Reviewers came to consensus on the excellence of this work. Please check the detailed comments by the reviewers and update the paper accordingly.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
2
Author Feedback
We thank the reviewers for their valuable feedback. In the following, we address their major comments on a point-by-point basis.
R1: Some modules in Fig 1. lack description, i.e. what is domain code?
We used one-hot codes to represent different MRI modalities. We have updated Fig. 1 and corresponding caption descriptions.
R1: It would be interesting if compared to atlas-based methods.
This is a good suggestion. We will work on experiments of atlas-based methods and will present results in the extended version of the conference paper in the near future.
R2: Can the proposed method be used for other challenging segmentation tasks, such as infant human brain segmentation?
As the reviewer pointed out, infant human brain segmentation is an important and challenging task. We do not have infant MRIs ourselves at present but we are happy to collaborate to see how the method works on infant brain segmentation when the data are available.
R3: The Dice scores achieved are moderate and it is hard to contextualize without prior state-of-the-art methods for comparison.
The mouse brain structural segmentation is a very challenging task and we do not agree that our average dice of 0.82 is only “moderate” for 50 brain structures, e.g. mammillothalamic tract structures are tiny and extremely difficult to segment. In fact, there is a lack of ‘state-of-the-art’ methods that focus on mouse brain segmentation, particularly in such a fine structural segmentation. Thus, we compare to the U-net segmentation, which is a standard baseline for most medical image segmentation problems.
R3: The evaluation of the image synthesis is limited with only a single example shown and no quantitative evaluation.
In our paper, image synthesis is only byproduct and it was mainly used for improving the segmentation task. So we did not make a thorough quantitative evaluation in the current manuscript but plan to include it in our extended version in the future.
R3: “Atlas-availability” and “atlas-free approach”
As already stated in the manuscript, atlas-based segmentation often requires non-rigid registration which is challenging by itself and often manual curation afterwards. This is possible by building up ground-truth labels of training data (as we only need to do it once), but is not feasible for routine usage. Moreover, the available atlas is limited for single modality, e.g. T2 and there are no atlas for every modality. Using our image synthesis module, we can segment brain structures on multiple modalities, beyond the capacity of atlas-based segmentation. In fact, our MouseGAN can potentially be used to help the generation of altas, which will be our future target.