Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Tariq Bdair, Nassir Navab, Shadi Albarqouni

Abstract

Skin cancer is one of the most deadly cancers worldwide. Yet, it can be reduced by early detection. Recent deep-learning methods have shown a dermatologist-level performance in skin cancer classification. Yet, this success demands a large amount of centralized data, which is oftentimes not available. Federated learning has been recently introduced to train machine learning models in a privacy-preserved distributed fashion demanding annotated data at the clients, which is usually expensive and not available, especially in the medical field. To this end, we propose FedPerl, a semi-supervised federated learning method that utilizes peer learning from social sciences and ensemble averaging from committee machines to build communities and encourage its members to learn from each other such that they produce more accurate pseudo labels. We also propose the peer anonymization (PA) technique as a core component of FedPerl. PA preserves privacy and reduces the communication cost while maintaining the performance without additional complexity. We validated our method on 38,000 skin lesion images collected from 4 publicly available datasets. FedPerl achieves superior performance over the baselines and state-of-the-art SSFL by 15.8%, and 1.8% respectively. Further, FedPerl shows less sensitivity to noisy clients.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_32

SharedIt: https://rdcu.be/cyl4f

Link to the code repository

https://github.com/tbdair/FedPerlV1.0

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a novel federated learning approach for semi-supervised skin lesion classification. This paper introduces the concept of building communities and integrating peer learning in a federated learning scenario to compensate for the missing classes in the similar clients of a federated learning model in a privacy preserving format and shows the sate of the art performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of this paper is novel, very interesting and super promising. I like the idea of building communities and sharing knowledge of clients based on similarity although it’s not fully novel in its essence.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

None
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper is reproducible in my opinion
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The overall idea of creating communities in federated learning is not novel and is normally called device to device communication in the wireless communication field. Here is an examples of such works that authors can add recognition to if they see fit. Hosseinalipour, Seyyedali, et al. “From federated to fog learning: Distributed machine learning over heterogeneous wireless networks.” IEEE Communications Magazine 58.12 (2020): 41-47.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novel idea, great results and justifications and very nice motivation.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper introduces a deep learning mode FedPerl to produce accurate pseudo labels in skin cancer classification. FedPerl, a semi-supervised federated learning (SSFL) method uses a new technique peers anonymization(PA) for the first time in SSFL to avoid privacy breaching and communication cost. FedPerl have been validated on 38,000 skin lesion images (out of which only 12% were annotated) from 4 publicly available datasets distributed over 10 clients.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1) The proposed technique peer anonymization is noble because it improves the privacy of data and communication cost. 2) The performance accuracy is better compared to baselines and SSFL by 15.8% and 1.8% respectively. 3) Related work is well explained in the introduction section
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1) It says that communication cost will be reduced because of peer anonymization but it didn’t provide any result on communication cost. 2) There is no discussion on performance improvement of 1.8% compared the state-of-art SSFL. 3) It is hard to understand Fig2 which shows the clients and classes distribution from four datasets 4) There is no proper classification/reason how the clients and datasets are divided 5) In client level results(Table 3) of F1- score, client 8 had less performance when compared with others clients due to class distribution mismatch
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

There is no code/data provided
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It is recommended that results of communication cost to be added. It is better to explain the reason about performance accuracy improvement compared with baselines and state-of-the-art SSFL.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Boderline accept because the technique implemented in this paper is new even though it had slight performance accuracy compared with SSFL
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper has only two reviews. There is agreement about the interest of the proposed methodology, and on the encouraging experimental results. Concerns are about the novelty of the contribution with respect to the baseline approach SSFL. A further consideration of the AC is about the privacy guarantees of FedPerl. The identification of peers exposes the system to information leakage between centers. This is particularly relevant when the number of clients and comunities is low, which is a typical case for this work ( T<5). The peers anonymization strategy attempts at addressing this issue, but it is not clear which privacy guarantees are provided by the simple averaging operation. Some sort of formal analysis would be much needed. The notation appears also not clear, e.g. concerning the relationship between \phi_a and \phi_t. Moreover ,the robustness and convergence of the framework with respect to the different hyperparameters controlling the optimization doesn’t seem to be investigated. It is therefore difficult to appreciate the contribution of each component to the resulting performance.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

Dear Area Chair,

We would like to thank you and the reviewers for the constructive feedback on our manuscript entitled “FedPerl: Semi-Supervised Peer Learning for Skin Lesion Classification” and for allowing us to clarify a few points.

The novelty of our method was acknowledged by the reviewers as reported by R2, “The idea of this paper is novel, very interesting and super promising”, and “Novel idea, great results, and justifications and very nice motivation”, and by R4 as “the technique implemented in this paper is new”, and “The proposed technique peer anonymization is noble”. Moreover, our manuscript is acknowledged by R2 as reproducible, very clear, and organized, with no weaknesses. Besides, our novel Peer Anonymization (PA) technique is acknowledged by R4 as “noble because it improves the privacy of data and communication cost”. This has also been reflected in their decisions “Strongly Accept” (R2) and “Borderline Accept” (R4). Nevertheless, we would like to highlight a few points raised by the AC and reviewers, which will better shape our manuscript for presentation at MICCAI

(AC) Contribution w.r.t the baseline approach SSFL: We have clarified and explained our three main contributions w.r.t. baseline SSFL [9,26]. Whereas our method has employed peer learning such that the clients gain extra knowledge by helping each other in a privacy-preserved way to leverage the unlabeled data, [26] was limited to the local knowledge by the client itself. Such extra knowledge has been shown to be useful in [9], however, at the cost of communication and privacy, and this is what we tried to mitigate in our work by first, the way we profile the clients to build communities and clusters; second, the novel peer anonymization (PA), first of its kind in SSFL, which is communication efficient and avoids privacy breaching. We showed that FedPerl significantly outperformed the baseline SSFL methods in proof-of-concept (CIFAR10, FEMNIST) and skin lesion classification experiments.

(AC) Privacy: Our anonymized peer is created by aggregating/averaging the model parameters of the top T similar peers. This process creates a virtual model that is not related to a specific client and offers a harder target for attackers seeking information about individual training instances [13,14]. Nevertheless, a privacy guarantee for aggregated models (not individuals) is an open issue and hasn’t been thoroughly investigated in the community and mathematical analysis is yet to be proven.

Apart from this extended explanation of methodological detail, we improved the following minor issues raised by the reviewers and AC:

(R4) Additional cost: The additional cost is calculated relative to the baseline (SSFL). For simplicity, we assume the initial cost for the SSFL is 0%. The additional cost increases proportionally with the number of peers T. Yet, with Peer Anonymization (PA), the cost is reduced to a fixed value O(1) regardless of T.

(R4) Data split: is intended to resemble a realistic and challenging scenario, where the medical data is heterogeneous, distributed with little annotations, and has a severe class imbalance. We have added a few lines to better describe the data split.

(AC) notation: \phi_a and \phi_t notations appeared in the PA formula, where \phi_a and \phi_t represent the anonymized and the similar peers weights respectively. The relationship can be simply realized as \phi_a is the average of \phi_t’s.

(AC) Hyperparamters: Information on the hyperparameter selection was left out and only reported in the supp. material, and will be included in the final version.

We hope that the above clear explanations would improve the clarity and allow for our contribution to be presented to the MICCAI community.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The major points of the rebuttal concern the positioning with respect to baseline SSFL approaches, and the discussion of the privacy guarantees. Overall, the rebuttal provides a fair discussion of the approach, and is satisfactory in addressing the reviewers’ concerns.

While the privacy aspect should deserve further discussion and clarification in the manuscript, the method was found novel, interesting, and well formulated. For this reason this work will represent a positive contribution to the conference.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I agree that the paper introduces an interesting approach but there still remains a problem of guaranting privacy for aggregated models. Yet, I think that this paper could be still interesting for MICCAI community.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The main reason for using federated learning is to preserve privacy. Nevertheless, according to authors’ rebuttal response, the privacy guarantee of the proposed approach is yet to be proven. This, in my opinion, is a major concern of the proposed work. Until this is proven, I am not sure one can trust such method in real-world application.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

14

back to top