Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xinxin Shan, Ying Wen, Qingli Li, Yue Lu, Haibin Cai
Abstract
In the practical application of medical image analysis, due to the different data distributions of source domain and target domain and the lack of the labels of target domain, domain adaptation for unsupervised cross-domain classification attracts widespread attention. However, current methods take knowledge transfer model and classification model as two separate training stages, which inadequately considers and utilizes the intrinsic information interaction between modules. In this paper, we propose a coherent cooperative learning framework based on transfer learning for unsupervised cross-domain classification. The proposed framework is constructed by two classifiers trained by transfer learning, which can respectively classify images of source domain and target domain, and a Wasserstein CycleGAN for image translation and data augmentation. In the coherent process, all modules are updated in turn, and the data is transferred between different modules to realize the knowledge transfer and collaborative training. The final prediction is obtained by a voting result of two classifiers. Experimental results on three pneumonia databases demonstrate the effectiveness of our framework with diverse backbones.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87240-3_10
SharedIt: https://rdcu.be/cyl5F
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes an end-to-end unsupervised cross domain classification model. Specifically it first combines Wasserstein GAN with Cycle GAN to form the generator network. In addition to this, two classifiers (one in source domain and another in the target domain) are learnt with the sampled labels (this is to overcome the class imbalance problem).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The problem is well motivated and most of the individual aspects are sound and intuitive.
- The experiments are extensive with detail ablation study. In addition to this, the results of the model seem impressive.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I think the claim of having Wasserstein CycleGAN as a novel contribution is incremental. Although the results of CycleGAN are not great visually as shown in Fig 2 but there is not theoretical grounding to this claim.
- How are the values for λ choosen? There are numerous losses in the objective and since it is an end-to-end model, it becomes cumbersome to do ablations on each loss term or even perform grid search. An insight into this would be useful. 3.In the cooperative mechanism, if only MMD is used throughout to train C_t classifier, then there is a significant drop in accuracy (Fig 3 #2). Why is this the case? The motivation behind sharing parameters for C_t is unclear from the text.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Details of architecture, losses, loss weights, optimizer etc. are all mentioned in the paper. I hope the authors release the code in future, if accepted.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
See weaknesses section.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper does include several new insights and the experiments are through. Although I am doubtful about the novelty of Wasserstein CycleGAN claim but I would still vote for a weak accept of the paper on the grounds of results and experiments. Also, I’d request authors to make changes to accommodate the comments in the Weaknesses section.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
The authors propose a coherent cooperative learning method exploiting transfer learning for unsupervised domain adaptation. The framework consists of a Wasserstein CycleGAN and two classifiers. All the elements are trained in an iterative process.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Comparison to 3 transfer learning and 1 domain adaptation methods.
- Experiments with different backbones and discussion about the suitability of the different models.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Numerical results reported are the result of a single run. No statistical significance in the difference with respect to compared methods.
- No cross-validation of the results.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Reference to the datasets used are included.
- Link to the code is missing.
If the authors make their code publicly available, as stated in the reproducibility checklist, results could be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- I would suggest clarifying contribution (ii) “The proposed collaborative training makes different modules complement each other.”
- Please give more details about how you obtain the “intermediate images” that are supplemented from \bar{X} in Section 2.1.
- I would suggest including an introduction before jumping into the equations of the losses in Section 2.1.
- In Equation (8) you could include the three loss components in order to have a final equation with all the losses.
- The last sentence of Section 2.1 sounds vague, what do you mean by “The WCycleGAN G is an important part of CCL”?
- Why is the objective function for the supervised classifier Cs a combination of the cross-entropy loss and the center loss? Did you find that the combination was better than just the cross-entropy loss?
- Comparative methods should be further explained: why are they relevant and what are the differences with respect to the proposed approach.
Minor comments: Some sentences need rewriting, for example:
- “is independent identically distributed” -> is independent and identically distributed.
- “Fortunately, transfer learning [15,7] and unsupervised learning [26,2,18] can be exploited to solve it” -> Fortunately, transfer learning [15,7] and unsupervised learning [26,2,18] are employed to deal with such challenges.
- “Supervised Classifier Cs is obtained by fine-tuning a CNN that pre-trained on ImageNet, which can be used…”
- “For more reliable prediction, we utilize a voting strategy”.
- “Cs with diverse backbones is always outstanding compared with other methods, which indicates that Cs is able to classify images in the same domain very well”.
- “so as to complete unsupervised domain adaptation well”.
- “During the training, the initial learning rate…” -> During training, …
- “From Fig. 3, it is obvious that …” and there is a missing dot at the end of this paragraph. Is the column “Class-imbalanced” in Table 1 necessary? Why do you only show training/test number of samples for Chest X-Ray dataset?
- Please state your overall opinion of the paper
borderline reject (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Results of the paper seems promising however they are based on a single run, without statistical significance.
- The writing of the paper could be improved for a better readability.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
2
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
The paper proposes a framework for unsupervised domain adaptation. It uses a Wasserstein CycleGAN for image translation and data augmentation, a cooperative training strategy to transfer knowledge between classifiers of two domains, and an MMD loss to minimize the distribution distance. It also tackles the class imbalance problem. Experiments on three Chest X-ray datasets showed that the proposed framework outperformed multiple existing methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Transfer learning is an important topic for medical images.
- This paper tackles the problem from multiple aspects including data translation, feature alignment, and parameter transfer. The cooperative training strategy seems novel (although I am not an expert in this topic), and the way of using generated intermediate images for data augmentation is also interesting.
- Extensive experiments and comparison were performed.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
In the method part, there are multiple confusing places requiring clarification. 1.1. In Eq 1, I think Ds should be used to discriminate samples x and G_{t2s}(y), and vice versa. So Ds(G_{s2t}(x)) should be Dt(G_{s2t}(x)), like the one in Eq 5. 1.2. In Eq 9, if f is a feature map, how can you use it in the softmax experssion, which should use prediction scores? 1.3. On the 3rd line of page 5, why would you want to transfer parameters of Cs to Ct when the accuracy of Cs does NOT reach the threshold? I think when the accuracy is bad, parameters should not be transferred. 1.4. In Eq 12, what is the meaning of argmax_k{*, *}? The first line of this equation is confusing.
-
In Fig 3, why is F1 score 0 for all methods? Should the authors multiply it by 100 before plotting?
-
Fig 2 is not very informative. For example, I cannot tell the domain difference between x and y by only looking the two examples, thus not clear why G_{s2t}(x) can be more similar to y.
-
The results without transfer learning should also be reported for comparison as a baseline (train on source and test on target).
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Questionable, because the framework has many hyper-parameters.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
See weaknesses.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is interesting, however, the method description and result presentation seems to contain unclear parts.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Somewhat confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper receives diverged review ratings. Please the authors address the issues and concerns as listed by the reviewers in their comments in box 4 for weakness and the questions raised in box 7 for detailed inquiries. In particular, please answer/clarify the following inquiries:
- How the parameters are selected in the experiments?
- Please provide more experimental details as inquired by the reviewers, such as “how the intermediate images are obtained” and many others as listed in the review comments?
- Provide some explanation on the mathematical derivations as inquired by R#2 and R#3.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7
Author Feedback
R#1 Q1:What’s the theoretical grounding of Wasserstein CycleGAN? A1:Since Wasserstein distance provides more stable gradient, Wasserstein CycleGAN improves the performance of CycleGAN, which was verified in [14].
Q2:How to select parameters? A2:There are 5 parameters in our model. The value of a defined parameter τ is obtained from the experiments shown in the supplement, while the optimal values of the rest λ1-λ4 are from the cited papers in Section 3.1.
Q3:If only using MMD throughout to train Ct, why there’s a significant drop in accuracy (Fig3#2)? What’s the motivation of sharing parameters for Ct? A3:In the early training, the unsupervised classifier Ct without labels is too weak to avoid overfitting, thus using MMD without passing parameters to train Ct causes performance drop (Fig3#2). Hence, Ct receiving parameters from labeled source domain can better complete unsupervised domain adaptation.
R#2 Q1:Numerical results reported are of a single run. No cross-validation of the results. A1:We follow the common leave-one-domain-out strategy as [Q.Dou et al., MICCAI2020][Q. Meng et al., MICCAI2020] and get the average result of 3 runs. To ensure the comparison and repeatability of experimental results, we use the existed divided datasets for training/testing/validation, instead of using cross-validation that will change the composition of datasets. All samples of the source domain are used for training, and Chest X-Ray (for training) and Chest X-Ray (for testing) in the target domain are respectively used for validation and testing to ascertain the iterations when convergence. Then we can apply the iterations as stop criteria to test other datasets in the target domain. We will add the explanation in the revision.
Q2:How to obtain intermediate images? A2:G_t2s and G_s2t are generators shown in Fig2, from which we get G_t2s(G_s2t(x)) of Eq2 and G_t2s(x) of Eq3 as intermediate images.
Q3:In Eq8 you could include the 3 loss components to have a final equation with all the losses. A3:We do as what you said indeed.
Q4:Why is the objective function for Cs a combination of the cross-entropy loss and center loss? A4:It’s verified by [X. Shan et al., ICASSP2021] that the combination is better in unsupervised domain adaptation.
Q5:Comparative methods should be further explained. A5:We compare to 3 transfer learning methods[3][12][7] using diverse loss functions and backbones; 2 domain adaptation methods[5][7]-GAN: [5] is classical, and [7]-GAN also uses GAN.
Q6:Is “Class-imbalanced” in Table1 necessary? Why do you only show training/test number of Chest X-Ray dataset? A6:Yes. One of the goals of our method is to avoid overfitting caused by class imbalance in medical images. The training/test partition of Chest X-Ray dataset is the same as[12], but other datasets aren’t pre-divided.
R#3 A1:There’re some confusions in Section2. Q1:We will modify G_t2s(y) of Eq1 and replace ‘softmax’ with ‘classification’. But we only take Ds for example to explain Eq5, and Dt in Eq8 is similar.
A2:What’s the meaning of argmax_k{, *} and the 1st line of Eq12? Q2:In argmax_k{, }, k is the label of max(, *). The 1st line of Eq12 is to get the label with the highest prediction probability as the final prediction.
A3:Why is F1 score 0 for all methods in Fig3? Should you multiply it by 100? Q3:F1 score≠0, and its range is [0,1]. We mark the value of F1 score near per red dot in Fig3.
A4:Fig2 isn’t very informative. Q4:Fig2(c) G_t2s(y) is clearly different from others. We can achieve improvement by using it, indicating that it provides valid information.
A5:Results without transfer learning should be compared as a baseline. Q5:[5][12] had verified that transfer learning on domain with limited data has better effect than without transfer learning, so we don’t give a comparison. We can add it if necessary.
Thanks for the reviewers’ comments. We’ll fix other linguistic and typo mistakes in the revision. We’ll open the code in future if accepted.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper studies the problem of transfer learning for unsupervised cross-domain classification, which is of practical interests for medical image classification. The presented neural networks structure in Fig.1 makes sense, which are supported by the experimental results. The authors’ rebuttal have largely addressed the questions and concerns raised by the reviewers. So I recommend an acceptance to this paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The novelty of this method is incremental over existing techniques. The reviewers had mixed opinions about the experimental sections. In response to one of the reviewer’s comments, the rebuttal; states that the average over 3 runs is reported. However, the reviewers point was about statistical significance which would also require reporting standard deviations. Overall, my opinion of this paper is borderline reject.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
12
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The work proposed to an unsupervised cross domain classification model. It adopted the Wasserstein CycleGAN for image translation and data augmentation. Additionally, two classifiers are learnt with the sample labels to overcome the class imbalance problem.
The work provided new insights by adapting a theoretically solid method and performed thorough and well designed experiments. The weakness is that some algorithm and experimental details are missing in the original submission.
The authors made a good rebuttal to clarify the details. The work may inspire new ideas to study sophisticated adversarial networks in medical imaging field. Therefore, an “Accept” recommendation is made.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7