Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Cheng Chen, Quande Liu, Yueming Jin, Qi Dou, Pheng-Ann Heng

Abstract

Domain adaptation typically requires to access source domain data to utilize their distribution information for domain alignment with the target data. However, in many real-world scenarios, the source data may not be accessible during the model adaptation in the target domain due to privacy issue. This paper studies the practical yet challenging source-free unsupervised domain adaptation problem, in which only an existing source model and the unlabeled target data are available for model adaptation. We present a novel denoised pseudo-labeling method for this problem, which effectively makes use of the source model and unlabeled target data to promote model self-adaptation from pseudo labels. Importantly, considering that the pseudo labels generated from source model are inevitably noisy due to domain shift, we further introduce two complementary pixel-level and class-level denoising schemes with uncertainty estimation and prototype estimation to reduce noisy pseudo labels and select reliable ones to enhance the pseudo-labeling efficacy. Experimental results on cross-domain fundus image segmentation show that without using any source images or altering source training, our approach achieves comparable or even higher performance than state-of-the-art source-dependent unsupervised domain adaptation methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_22

SharedIt: https://rdcu.be/cyl5W

Link to the code repository

https://github.com/cchen-cc/SFDA-DPL

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors propose a new setting for the unsupervised domain adaptation on fundus image segmentation. In this setting, we only have access to the pertained model on the source domain and the unlabeled data on the target domain. Secondly, this paper proposes a method to create the pseudo label from the target domain for self-training.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The new proposed setting is quite interesting and more realistic for real-life applications.
2. The author applies the uncertainty method to de-noise pixel-level pseudo label and prototype feature for class level de-noise.
3. The experiments are sufficient and achieve superior performance than the baseline methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The Monto-Carlo dropout uncertainty and the prototype de-noise are already proposed by the previous method.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sufficient details for reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The idea of this paper is quite interesting; more experiments on different datasets would be great.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Interesting idea and good technical quality.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors introduce a new unsupervised domain adaptation (UDA) task, where source data can not be accessed nor can alter source training. They also present a self-training method for this task, and they further propose a denoising method, with a class-level and pixel-level denoising scheme, to denoise the pseudo labels in self-training and gain a better performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The authors introduce a new UDA task, which has high practical value under the data privacy and security issues in medical field.
2. The authors propose a self-training method, which seems achieve a good performance in this task.
3. The authors propose a denoising method to denoise the pseudo labels in the self-training, which helps gain better performance.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The effectiveness of the self-training scheme is not clear, an ablation study compare the model with/ without self-training is needed.
2. The reasons to exploit the denoising method is not well discussed. In fact, the proposed denoising method can also serviced as post-processing. What is the advantage of exploiting it to denoise the pseudo labels rather than the final segmented results.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Can be reproduced
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1.The effectiveness of the self-training scheme is not clear, an ablation study compare the model with/ without self-training is needed. 2.The proposed denoising method can be better justified by exploiting it as post-processing 3.In the first sentence of Ablation study, ‘complonent’ should be ‘component’ 4.In Fig. 3, it would be better to present the correct recognized noise and wrongly recognized noise.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The authors introduce a new and meaningful task, and they also proposed a method for this task with well performance. However, the proposed method is not well justified.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper tackles Source-Free Unsupervised Domain Adaptation for image segmentation, i.e. adaptation using only a model trained on unavailable source data and unlabeled target images. The adaptation is done through self-training in the target domain, generating pseudo-labels. This classical method improved by filtering out “unreliable” pseudo-labels, as measured by two distinct uncertainty measures. The first is calculated at the pixel-level, using the variance of prediction maps generated by stochastic dropout. The second is calculated at the class-level, using the distance of features to the estimated class centroid. Experiments are done on one application: retinal fundus images, using 3 datasets collected from different clinical centers with distribution shifts.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The idea of source free domain adaptation (SFDA) is still novel in the medical imaging community and is valuable in practical settings, where source data is often unavailable.
2. The paper is well written and easy to follow. The motivation and the method are well explained. The method draws from a combination of losses and concepts which have been proven effective in other settings, and their application and combination to SFDA is novel. 3.Sufficient experimental section: Although one application only is tackled, retinal fundus images, experiments are done on 2 different target datasets; Comparison to various SOTA methods; Ablation study is done to investigate the effect of each component in the proposed method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is an assumption of closeness in this paper between the source and target domain, that the domain shift is not too wide. Indeed, the experimental setting is one where both source and target datasets are from the same image modality, the shift comes from scanner differences, which, as authors not, is generally minor compared with cross-modality discrepancy (ex. CT to MRI). As a result, the lower baseline “no adaptation” is already very high, 83.2 Dice (for RIM-ONE-r3) and 93.84 (for Drishti-GS) respectively. In more general settings with bigger domain shifts, the method’s efficiency may not hold. This is not discussed in the paper.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper has sufficient details about the framework, hyperparameters choosing, and evaluation. The datasets used are public fundus image datasets. However, the paper doesn’t mention any code available, which limits reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

In settings with bigger domain shifts, it is common to that a network’s predictions are skewed towards the dominant class, ex. the background, resulting in “undersegmentations” (see, for ex, [1], Figure 3, Seg-CT-noDA). Could the proposed method “get out” of such a situation, if it only keeps to “non-noisy” pseudo labels ? Did authors see a general tendency to undersegmentation in the output segmentation masks, and if not so, how come ?

[1] Dou et al, PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network with a Benchmark at Cross-modality Cardiac Segmentation, IEEE TMI.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper tackles an interesting and original problem,source free domain adaptation, and explores the potential quite simple and well motivated measures of uncertainty to improve classical self-training schemes. The paper is well written, easy to follow. The method is simple, easy to implement and well-motivated, with only one network to optimize, making it easier than SOTA adversarial methods. I find it particularly interesting to show that simple and sound uncertainty measures help select the best pixels to improve self-training. I believe this is an intersesting contribution to the MICCAI community.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

All reviewers appreciated the novel experimental setting, i.e., source-free unsupervised domain adaptation. While reviewers provided positive comments (probably/borderline/probably accept), they gave some suggestions to improve the manuscript, e.g., clarify the necessity of using the denoising method for pseudo labels, discuss if the proposed method can handle big domain shifts, and provide a better illustration of Fig. 3. Please consider these suggestions when preparing the final version.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We thank all the reviewers and meta-reviewer for the affirmative comments on recognizing the novel source free domain adaptation setting and sufficient experimental evaluation of our method.

We will include the reviewers’ suggestions in our final version and carefully proofread our paper.

back to top

Source-Free Domain Adaptive Fundus Image Segmentation with Denoised Pseudo-Labeling