Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Bo Zhou, Rui Wang, Ming-Kai Chen, Adam P. Mecca, Ryan S. O’Dell, Christopher H. Van Dyck, Richard E. Carson, James S. Duncan, Chi Liu

Abstract

Positron Emission Tomography (PET) is an important tool for studying Alzheimer’s disease (AD). PET scans can be used as diagnostics tools, and to provide molecular characterization of patients with cognitive disorders. However, multiple tracers are needed to measure glucose metabolism (18F-FDG), synaptic vesicle protein (11C-UCB-J), and beta-amyloid (11C-PiB). Administering multiple tracers to patients will lead to high radiation dose and cost. In addition, access to PET scans using new or less-available tracers with sophisticated production methods and short half-life isotopes may be very limited. Thus, it is desirable to develop an efficient multi-tracer PET synthesis model that can generate multi-tracer PET from single-tracer PET. Previous works on medical image synthesis focus on one-to-one fixed domain translations, and cannot simultaneously learn the feature from multi-tracer domains. Given 3 or more tracers, relying on previous methods will also create a heavy burden on the number of models to be trained. To tackle these issues, we propose a 3D unified anatomy-aware cyclic adversarial network (UCAN) for translating multi-tracer PET volumes with one unified generative model, where MR with anatomical information is incorporated. Evaluations on a multi-tracer PET dataset demonstrate the feasibility that our UCAN can generate high-quality multi-tracer PET volumes, with NMSE less than 15% for all PET tracers.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_4

SharedIt: https://rdcu.be/cyhUv

Link to the code repository

https://github.com/bbbbbbzhou/UCAN

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

In this work, the authors developed a 3D unified cyclical adversarial network (UCAN) to translate multi-tracer PET volumes. The study was motivated by limitations in the use of multiple tracers in human studies. The authors trained and tested the model on a dataset of 35 subjects with 3 tracers. Quantitatively, the authors found that the UCAN generated better results when compared to other generative adversarial models (GANs), namely cGAN and StarGAN. Ablation studies revealed that the use of MR information led to better synthesis results.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The motivation and clinical need for the work is high.
- The model utilized to generate the images is quite interesting.
- The dataset used included 3 different tracers.
- The authors compared their results to two other GANs, demonstrating improved reconstructions/synthetic maps.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- While the prospect of using a UCAN to translate multi-tracer PET volumes is interesting, the number of human subjects used to train and test the model seems rather small for this difficult task.
- The dataset seems limited and it is unclear which populations are included (whether from normal controls, MCI or AD), which limits the generalizability of the model.
- While the translated images are perceptually similar to the ground truth, it is difficult to determine if the translated images can be applied to diseased cohorts or to what extent it can represent amyloid positivity or hypometabolism for example.
- The authors mentioned the potential of using the UCAN for AD studies, but did not provide any validation in relation to clinical decisions based on these PET scans (for example, deciding amyloid positivity in AD based on UCAN results)
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sufficient details were provided for the architecture of both the generator and the discriminator. The methods for data collection, model development, and testing are mostly explained clearly. Implementation details were also provided in the Supplementary document.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- I think it will be beneficial to include a range of participants across the AD spectrum (and AB/tau positivity) to train and test this model.
- While the main motivation of this work is clinical, the results and comparisons did not elude to the utility of the model in any cohorts. For example, it would have been beneficial to show differences in regional or global SUVR values in AD subjects and how accurate the synthetic maps are in classifying a patient as AB positive or negative.
- Section 2.1 is lacking in detail, namely concerning the cohort/subject population and their diagnosis/demographics.
- In addition, it is difficult to judge the bias in AD-related ROIs results (pg 8) and there was no regional comparison with other GANs. A percentage difference in SUVR for a specific translation across methods maybe be easier to interpret.
- It is unclear why the other three loss functions were excluded from the ablation studies.
- Page 2: “Our UCAN consists of one generator G (red block) and one discriminator D (blue block) with detailed structures demonstrated in Figure ??.” The authors may have intended to state Supplementary Figure 1
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

While the study is interesting, it is difficult to approve based on the lack of subjects utilized to generate the model. It would benefit the authors to account for this. In addition, it would benefit the authors to test the model on different disease groups.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors propose a novel framework for synthesizing PET images for multiple scenarios using a unified framework. The method also leverages MRI anatomical information. The image translation framework is based on cycle consistency and adversarial training aided by anatomical information.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The application is of interest to the community especially in problems related to disease classification and identification of biomarkers. The evaluations appears to be strong but the manuscript is unable to contain all the information within the page limit. For example, qualitative results from baseline methods are not included. The problem is well motivated and the Introduction section is well written.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Overall, it seems the proposed method works better than other baselines that are considered. But i am worried about the false activations in several regions of the brain. For the synthesized tracer C, across the orthogonal views, the activity distribution is substantially different from the one used as input (row 3). In rows 1 and 2, significant activation is estimated at caudate nuclei, putamen. This is evident even if MRI information is not used! Also, the anatomical MRI image is not shown!

Tracer A (FDG) seems to be estimated well from B but suffers from over estimation when using C. Nevertheless, structures are in tact as MRI information is used. This is clear from Supplementary Figure 2 as well.

Images from other baseline methods are missing! The performance improvement is shown only via Table 1 . I believe qualitative assessment of the predicted images across all the methods is extremely important.

Did you tune the regularization parameters in SSIM? In either case, report the values used. Figure1: How were the images normalized? Are all images divided by max SUV across tracers?

Figure 3: There seems to be positive bias in all the regions for all the methods. Comments?

Certain symbols and cross-references are missing! Please fix these.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors claim that code will be released, which is good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please see comments above.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The problem is well motivated and of interest to the community. Extensive evaluation has been carried out but the manuscript should contain both qualitative and quantitative results. I am worried about false activations for tracer C. The argument that activations in certain regions are in agreement with the ground truth does not work.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The article entitled “Synthesizing Multi-Tracer PET Images for Alzheimer’s Disease Patients using a 3D Unified Anatomy-aware Cyclic Adversarial Network” proposes a new approach for translating multi-tracer PET volumes using Cycle-GAN. The proposed unified generator, accepts as input tracer volume, MR prior and target tracer label and outputs a target tracer volume. The authors use DuSE-Net as a generator in the cycle GAN framework for fusing multiple inputs. The quantitative analysis on AD related ROIs establishes that the preliminary results are promising.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The article presents a novel formulation of multi-tracer volume synthesis using the Cycle GAN based framework. It is still preliminary work so there is no thorough evaluation/comparison but the results look promising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Being novel it doesn’t have many approaches to compare with or any benchmark dataset to assess the performance.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The results a reproducible thanks to the additional material provided
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
A couple of comments to the authors:
- Does it benefit to have a unified network for the multi-tracer translation ? Or having multiple networks for the various translation paths help in better results ?
- Does the architecture still work on a different dataset acquired on a different machine (after required pre-processing) ?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It is a good paper.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a new 3D unified cyclical adversarial network (UCAN) based on cycle consistency and adversarial training aided by anatomical information to synthesize multi-tracer PET images. The results show that it is more performant than other generative adversarial models (GANs). The work is very interesting. However, the authors need to address critical points of the reviews:
- A dataset of 35 subjects with 3 tracers is a small data. How to guarantee there is no overfitting problem?
- More description about choosing regularization parameters is needed.
- How to be sure if the synthesized images can be applied to diseased cohorts?
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

R1 pointed out that we have a relatively small dataset and we should include a range of participants across the AD spectrum for training/test. R1 also suggested that we should include more patient information. We agree that we have a relatively small dataset for training/test, and it is due to the fact that a clinical exam with patients undergoing 3 tracers is not a clinical routine. To avoid overfitting, we performed data augmentation by randomly cropping 100 64x64x64 patches from each volume. This information is provided in our supplementary. We actually included a range of participants across the AD spectrum in our dataset. We included 15 patients diagnosed with AD, 9 patients diagnosed with MCI, and 11 patients as healthy control. We will include this detail in our dataset section.

R1 suggested that we should validate how our UCAN can aid the clinical decision on AD studies. We agree that it is important to validate how UCAN can help the clinical decisions on AD studies by synthesizing different tracer images. In this work, we focused on demonstrating the feasibility of generating a multi-tracer from a single tracer using our UCAN, as mentioned in our conclusion. Comprehensive clinical evaluations with more patient data will be investigated in our future works.

R1 asked why ablation studies are not performed for the other three losses. Pair loss, anatomy prior, and DuSE are the three key components we proposed in UCAN, while the adversarial loss, classification loss, and cyclic loss are the losses already used in StarGAN [16]. Therefore, we focused on ablation studies on our new components, as shown in Table 2 & Fig 2 (supplementary).

R2 pointed out that there are inaccurate estimations in our synthesis image. We agree that our synthesis results are still not perfect. However, as compared to previous methods, our UCAN can achieve better synthesis performance. In our future work, we will continue to investigate how we can further improve the performance by collecting more data, followed by comprehensive clinical evaluations. In addition, due to limited space, we will include visual examples of the previous methods in our supplementary.

R2 asked how we tune the parameter in SSIM. We want to clarify that we don’t have SSIM in our loss, and we used SSIM for evaluation only. In Eq 6 & Eq 7, we empirically chose our weights and found the stated parameters can achieve a balanced training with the best results produced. We set the weights of pair loss and cyclic recon loss to bigger values than adversarial and classification loss weights as the ground-truth tracer volume can provide direct supervision.

R2 asked how the images are being normalized. In our dataset, we preprocessed the images by first dividing it by its maximal SUV value, and then multiplying it by 2, and then minus 1 to ensure the intensity lies in [-1,1]. We will include this detail in our dataset section.

R2 asked why positive bias is observed in Fig. 3. We believe it is caused by the network behavior that tends to overestimate the SUV. However, we can see the mean bias is no more than 15% for all the ROI, including the hippocampus & entorhinal cortex which are two of the first areas impaired by AD. In the future, we will continue to investigate how to further reduce the bias.

R3 asked what is the advantage of the unified model. Compared to previous methods focusing on 1-to-1 fixed domain translation, our unified model can not only reduce the number of models to one and allows multi-tracer domain feature learning which leads to better synthesis results. These details are already provided in our abstract, introduction, and result sections.

R3 asked if our model can be applied to different datasets acquired from a different machine. We believe our model can potentially be applied to datasets acquired from a different machine if the domain differences were properly addressed.

In our final version, we will fix the notation errors and typos.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a new 3D unified cyclical adversarial network (UCAN) based on cycle consistency and adversarial training aided by anatomical information to synthesize multi-tracer PET images. The results show that it performs better than other generative antagonist models. The work is very interesting. Despite the small size of the data, the method has potential for future work. My proposition is “Accept”.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper aims to synthesize multi-tracer PET images. The solution is a new 3D unified cyclical adversarial network (UCAN) based on cycle consistency and adversarial training aided by anatomical information. The work in general has good potentials in clinical scenario, and can be attractive to the community. The size of the dataset is not large (#35), yet it is acceptable considering the specific PET task in this paper. Reviewers also raised concerns upon validation, while the authors made proper responses in their rebuttal
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper addresses a relatively novel research problem of multi-tracer PET volumes translation. The proposed technique is sound and did not receive much questions from the reviewers. The responses to the major concerns from Reviewer#1 about the dataset (patch-based training and including AD patients) and the ablation study are OK to me. Please include the discussions about the visual results in the rebuttal into the final paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

Synthesizing Multi-Tracer PET Images for Alzheimer's Disease Patients using a 3D Unified Anatomy-aware Cyclic Adversarial Network