Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Chia-Hsiang Kao, Yong-Sheng Chen, Li-Fen Chen, Wei-Chen Chiu

Abstract

Recent development of image-to-image translation techniques has enabled the generation of rare medical images (e.g., PET) from common ones (e.g., MRI). Beyond the potential benefits of the reduction in scanning time, acquisition cost, and radiation exposure risks, the translation models in themselves are inscrutable black boxes. In this work, we propose two approaches to demystify the image translation process, where we particularly focus on the T1-MRI to PET translation. First, we adopt the representational similarity analysis and discover that the process of T1-MR to PET image translation includes the stages of brain tissue segmentation and brain region recognition, which unravels the relationship between the structural and functional neuroimaging data. Second, based on our findings, an Explainable and Simplified Image Translation (ESIT) model is proposed to demonstrate the capability of deep learning models for extracting gray matter volume information and identifying brain regions related to normal aging and Alzheimer’s disease, which untangles the biological plausibility hidden in deep learning models.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_38

SharedIt: https://rdcu.be/cyl4l

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper proposed two efforts for interpretability understanding in MRI to PET image translation model. The first is an analysis of the similarity between network internal representation and tissue/region maps. The second is a proposed network that take spatial encoding and tissue map as inputs and generate PET map.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is novel in its problem formulation and technical solution trying to tackle the explainability in image translation in brain anatomy context, and have some interesting findings.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

While I think the results are interesting, I don’t find much surprise in the findings.

For example, for 4.2 it’s well recognized one primary information T1 MRI can provide is tissue map (e.g. tissue segmentation using T1), and regional segmentation is another. And both tissues and regions are spatially clustered, so should still be homogeneous in CNN learned features.

Well I appreciate the motivation for the first analysis, I don’t find clear utility of the second proposed network. E.g. why do we need to understand regional gray matter volume information from a MRI-PET translation model.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Some more details are needed.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Why were the operations substituted from typical UNet? Can add a dashed line in Fig-3a to indicate early encoding, similarly for 3b
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think there are some interesting findings and the way the authors tackle interpretability in this paper is novel. But I am conservative on how much insight this paper could provide to readers and not sure of the utility.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper details the creation of an explainable model that is able to understand key features for doing FDG-PET to T1 translation. The model helps explain features “decisions” made by the network and verifies them through understanding the biological basis of certain neurodegenerative diseases such as Alzheimer’s disease.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This is a very cohesive paper where every section is well explained and easy to follow.
2. Biologically, the method is sound, for example, when writing about the activation maps, it was shown that the CCA between activation maps and grey matter maps were similar to the ones between activation maps and PET, which help t emphasize the higher metabolic rate in the grey matter.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

N/A
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. All equations are well explained and each variable is defined.
2. A U-Net was used, which is a widely implemented architecture
3. Data was taken from a widely available open source neuroimaging database ADNI.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Introduction
1. MRI, FDG, PET, MR all need to be defined before the acronym is used… a. Write out [18F]-Fluorodeoxyglucose (18FDG)-proton emission tomography (PET)
2. Maybe replace the 3rd sentence of the introduction with a sentence similar to this one…”Typically, the T1, a type of structural magnetic resonance imaging (MRI) scan, is an anatomical image used to observe brain structures…”. The reason is that most MRI modalities are structural images (T1, T2, DWI, etc).
3. In the last line of the introduction, it is written “informative ness on clinical status”, however, no information is given about a clinical
Image Translation Model, Dataset, and Analysis Tool
1. ADNI stands for Alzheimer’s Disease Neuroimaging Initiative. In the paper, the word ‘Disease’ was left out. Make sure to include the full name.
Preprocessing
1. When mentioning the transformation of T1-MR to MNI space, it would be helpful if you could include if this was an affine transformation or a non-linear transformation.
Please state your overall opinion of the paper

ground-breaking (10)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper is well written, introduces all hypothesis with background information, is well organized and easy to follow, and reports novel science. Also one of the main problems with clinical adoption of deep learning applications is that it produces a black box of decisions. In medicine, it is very important to understand why certain patterns exist and how they inform decisions. This worked aimed to do this, using T1 to PET models, creating ESIT.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper attempts to shed light on the image-to-image translation process, specifically between T1-MRI (structural) and PET (functional) images which are implemented with a U-Net model.

They proposed the use of Canonical Correlation Analysis to perform ‘representational similarity analysis’. Similarity between hidden layer activations and brain tissue maps / brain region templates were computed and they found that brain tissues are segmented in the early encoding stage (and such information plays a key role in PET image synthesis) while brain regions are recognised later in the translation process.

These insights were then used as the basis of their proposed ESIT model, which also performs image translations but is touted to be more explainable than the U-Net model used initially for image translation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- ESIT is able to get very similar image quality for image translation tasks, as compared to the traditional U-Net model, with 20 times fewer parameters
- Experiments performed are well-motivated (from prior knowledge on PET and insights from data visualisation) and the results (MRI vs Tissue input, U-Net vs ESIT) seems to verify their hypothesis pretty well. If this breakdown of the image translation process is indeed correct and robustly shown across datasets, it could motivate a new wave of interpretability-driven deep learning models that are leaner and work as well as present established methods (e.g. U-Net) which have the notoriety of being a black box.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Due to ‘expensive computational cost’, only 20 subjects were sampled to perform the analysis. It is likely that more robust experiments are needed to verify the findings of this work over more and larger datasets. This is especially so in view of recent works on CCA (https://www.biorxiv.org/content/10.1101/2020.08.25.265546v1.full.pdf) which found that ‘resulting CCA/PLS associations could be highly inaccurate when the number of samples per feature is relatively small’
- Error bars are not provided for the results in Table S2, preventing a more robust comparison between the experiments.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

No major issues. It will be great if the code is released after acceptance so that the community can go deeper and extend the findings to other domains and applications.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The gap between drawing insights from the joint histogram and arriving at the second hypothesis that ‘there would be an in-between stage of brain regions identification followed by performing region-dependent transformation’ feels quite large (though well-motivated when considering that PET captures information about glucose metabolism, which varies across brain regions). It would have been interesting if a similar visualisation was done but with different colours assigned to pixels belonging to different brain regions instead.
- Fig 3(b) showed the CCA similarity for output. For completeness, the same should be shown for Fig 3(a) too.
- There wasn’t any discussion about the red line in Fig 3(a) - why would the CCA similarity for WM start of high and remained so throughout all the layers?
- While it is very interesting how MRI and tissue inputs to the U-Net can give similar quality metrics, Section 4.2 doesn’t really clearly explain how the experiment was conducted - for the row in Table S1 with the U-Net model and Tissue as input, how was a single score computed when there are 3 tissue maps (WM, GM, CSF)?
- Only the final ESIT architecture was given and it seems like no details about how the architecture was arrived at (e.g. number of convolution layers used in various parts of the architecture)
- ESIT was demonstrated on T1-MRI to PET translation using ADNI data. The results would have been even stronger if it was replicated on another dataset as well, or even for other modality pairs (perhaps as an extension of this work)
Minor comments
- Is CCA the only tool that can perform such an analysis? There are many variants of CCA - could non-linear forms of CCA shed even more insight?
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper presents a very neat flow from data visualisation and hypothesis development to the generation of insights that led to the development of the proposed ESIT model. This made ESIT look very well motivated and the architecture choice seems well justified. Main caveat is the very small sample size used (20), but I think the paper will be of significant interest to the community and could spark some debate during the conference, or even motivate further similar research for other deep learning models / tasks.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers came to consensus on the great merits of this work. Please check the detailed comments by the reviewers and update the paper accordingly.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We thank the reviewers for the insightful and constructive feedback and will revise the paper according to your precious suggestion. We address one major concern in the following. Q1: Small sample size for representational similarity analysis. A1: We use five-fold cross validation for data splitting and model training. For each trained model, we sample 20 subjects from corresponding training data to perform the CCA analysis. The standard deviation of the CCA similarities is plotted in Fig. 3 (a) and (b) as the shaded region. We will add details in CCA computation in the camera-ready.

back to top

Demystifying T1-MRI to FDG18-PET Image Translation via Representational Similarity