Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xian Wu, Yangtian Yan, Shuang Zhao, Yehong Kuang, Shen Ge, Kai Wang, Xiang Chen
Abstract
Psoriasis is a chronic skin disease which occurs to 2%~3% of the world’s entire population. If treated properly, patients can still maintain a relatively high quality of life. Otherwise, Psoriasis could cause severe complications or even threat to life. Therefore, continuous tracking of severity degree is critical in Psoriasis treatment. However, due to the shortage of dermatologists, it’s hard for patients to receive regular severity evaluation. Furthermore, evaluating the severity degree of Psoriasis is both time-consuming and error-prone which poses a heavy burden for dermatologists. To address this problem, we propose an automatic rating model which measures the severity degree quantitatively based on skin lesion pictures. The proposed rating model applies coarse to fine grained neural net to evaluate skin lesion from multiple perspectives. According to experimental results, the proposed model outperforms experienced dermatologists.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87234-2_18
SharedIt: https://rdcu.be/cyl8d
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposed a siamese-like approach with attention modules to estimate PASI score and related subscores to quantify the severity level for Psoriasis patients. They compared the results against dermatologists with different background, beating average dermatologists.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper attempts an important clinical application that has been somewhat under explored before - assessing the severity level of Psoriasis disease using the PASI standard. The proposed method not only outputs an overall PASI score, but also subscores that describe different aspect of the severity. The dataset used in the paper is not huge, but the ground truth construction is very solid, based on consensus of three experienced dermatologists. The evaluation against humans is also well thought out, comparing against a large number of dermatologists with varying level of experience.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This paper is more of an interesting ‘clinical’ paper, instead of a ‘methodology’ paper. The methodology is somewhat novel (i.e., introducing lesion attention module), but similar idea has been seen in other papers on skin lesion classification topic. It also lacks comparison against other automatic approaches, despite the large comparison against dermatologists. From the clinical perspective, it also lacks in-depth discussion to answer key clinical questions - e..g, why PASI is important in terms of severity monitoring, especially since many dermatologists don’t even use it, as reported in the paper? How big of the error will it matter? How reliable the ground truth is since inter-dermatologist agreement is high?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The general architecture is reproducible since training/evaluation code is provided, but since no dataset nor pretrained models are provided, reproducibility is limited.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
See weakness section above.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses an important yet under explored clinical application to assess severity level for Psoriasis. The method they proposed is somewhat novel and the evaluation against dermatologists is well done. However, it lacks comparison against other approaches and key discussions to emphasize its clinical impact.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
3
- Reviewer confidence
Very confident
Review #2
- Please describe the contribution of the paper
This work has proposed an automatic rating model for psoriasis. To conduct this study, this work has collected a new dataset containing 5,205 images and compare the proposed method to the PASI score for analysis. The method simulates the PASI scoring system and achieved improved performance compared to the baseline.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- New loss function to address the feature learning between two sub-networks
- A novel topic in dermatology
- Article structure and motivation is sound and clear
- Comparision to human ratings
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of other technical baselines
- Some important references for dermatology AI papers are missing
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The technical details are clear to reproduce the method.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
This paper can include more qualitative studies for further analysis. The potential clinical values can be discussed.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper has addressed a new problem in dermatology AI and technical novelties are good. The experiments have been strong in comparison to clinicians. Therefore, I think this paper makes a good contribution to the community.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
This paper introduces to automatically rank the severity of psoriasis disease. A Siamese network with attention module was used for comparison and ranking across different body regions. The experimental results with a private dataset appear to be reasonable.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well organized, which made audience easy to follow. There are detailed comparisons of the results derived from machines against clinicians.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It seems that the proposed method is quite similar to the method mentioned in Reference 10, where both methods were applied to the same problem and used similar network architecture e.g., Siamese network.
There are quite limited experiments on comparing to the existing deep learning based methods. It’s understandable that the proposed method is one of the first few methods applied for psoriasis disease. However, without comparison to the existing methods, it’s relatively difficult to understand that the Siamese network with attention module is the optimal solution to this problem.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed method was developed with a self-organized dataset. The source codes were released, which made easier for reproducing the results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Authors should consider to explain the key differences to Reference 10. It’s relatively difficult to understand that there are sufficient differences to Reference 10.
There is no comparison to the existing methods e.g., deep learning based methods for medical image classification. Therefore, it’s challenging to understand that the proposed method has improved the state-of-the-art.
- Please state your overall opinion of the paper
probably reject (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
There exists limited new technical contribution to the published work [10]. [10] Li, Y., Wu, Z., Zhao, S., Wu, X., Kuang, Y., Yan, Y., Ge, S., Wang, K., Fan, W., Chen, X., et al.: Psenet: Psoriasis severity evaluation network. In: Proceedings of the AAAI Conference on Arti cial Intelligence. vol. 34, pp. 800{807 (2020)
There are also quite limited number of comparison to the existing methods.
- What is the ranking of this paper in your review stack?
5
- Number of papers in your stack
7
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper tackles a new and interesting problem. Reviewers 3 and 4 suggest that the paper has merits but expressed concerns on the lack of comparative evaluations against the other baseline methods. Moreover, the technical contributions of the proposed method is limited as similar approaches have been introduced in the literature (e.g., ref 10). Please carefully address issues raised by all reviewers in your rebuttal.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7
Author Feedback
We thank all reviewers for their helpful comments.
Response to Reviewer 3:
Q1: Comparison against other automatic approaches
A1: To the best of our knowledge, this is the first piece of work that automatically predicts the PASI score as well as all the 16 sub-scores. There are few automatic baselines for this particular task. Therefore we mainly compare with experienced dermatologists. Nevertheless, we conduct ablation studies in Table 2 to prove the effectiveness of two proposed modules.
Q2: Emphasize its clinical impact.
A2: PASI is the most frequently adopted indicator which helps in Psoriasis diagnosis and proper therapy selection. As shown in the first column of Table 1, with the increase of the hospital level, the adoption rate of PASI is increasing, which shows from one perspective that the PASI is helpful in clinics. In the questionnaire to 223 dermatologists, we also ask what’s the major drawbacks of PASI in practise. 87% choose complicated to calculate, subjective and inconsistent. In this paper, we learn from the consensus of three dermatologists who have 10 years’ PASI scoring experiences, and target to produce objective and consistent PASI scores in milli-seconds. The proposed model is proven to be comparable to experienced dematologists. This might help to increase the PASI adoption rate especially in community and county level hospitals and improve the Psoriasis treatment quality.
Response to Reviewer 4:
Q1: Other technical baselines
A1: To the best of our knowledge, this is the first piece of work that automatically predicts the PASI score as well as all the 16 sub-scores. There are few automatic baselines for this particular task. Therefore we mainly compare with experienced dermatologists. Nevertheless, we conduct ablation studies in Table 2 to prove the effectiveness of two proposed modules.
Q2: More dermatology AI papers
A2: We will cite more dermatology AI papers including both Psoriasis and other skin diseases.
Response to Reviewer 5:
Q1: Relation to the published work [10].
A1: [10] targets to give a local severity score of a single skin lesion image which differs from our work in the following perspectives: (1) [10] only outputs a single score to denote the severity which is hard to interpret. While ours outputs all the 16 sub-scores of PASI including of erythema, induration, desquamation and area ratio for four body parts; (2) [10] only accepts single image input, but the PASI score needs to consider all skin lesions of four body parts. Our model follows the excat PASI calculating procedure and accepts multiple input images for all four body parts; 3) [10] didn’t conduct a comparison with dermatologists to prove its effectiveness in clinics.
Q2: Compare to deep learning based methods for medical image classification.
A2: We compare with a state-of-the-art model [1] recently proposed in MICCAI 2020 for pulmonary edema severity assessment of chest radiography images. We adopt their major model, but remove the textual part since we do not have any reports. Under comparable settings, the MAE results on four aspects: Erythema, Induration, Desquamation and Area Ratio are 0.495, 0.545, 0.555 and 1.11, which is larger (worse) than 0.463, 0.481, 0.487 and 0.634 of our proposed model equipped with the LAM module and the siamese network. These results confirms the effectiveness of our introduced model.
[1] Chauhan, Geeticka, et al. Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment. MICCAI 2020.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper tackles a new and unexplored problem in automated skin disease analysis. The further comparison with a recent MICCAI paper was good and therefore overall contributions of the paper have been improved. This paper, therefore, could be interesting to other researchers in MICCAI community.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
9
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposed a siamese-like approach with attention modules to estimate PASI score and related subscores to quantify the severity level for Psoriasis patients. The proposed method is not novel enough since it is similar to the method mentioned in Reference 10. Also, it lacks comparison against other automatic approaches. In view the high quality requirements of MICCAI, we suggest to reject the paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
16
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Despite relative limited methodological improvements, the paper presents an interesting clinical application with promising results and an ablation study that demonstrates the relevance of the proposed solution. The rebuttal helps greatly clarifying the key points highlighted by the reviewers a possible weaknesses.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
8