Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Shanmukh Alle, U. Deva Priyakumar

Abstract

Parkinson’s Disease (PD) is a chronic and progressive neurological disorder that results in rigidity, tremors and postural instability. There is no definite medical test to diagnose PD and diagnosis is mostly a clinical exercise. Although guidelines exist, about 10-30% of the patients are wrongly diagnosed with PD. Hence, there is a need for an accurate, unbiased and fast method for diagnosis. In this study, we propose LPGNet, a fast and accurate method to diagnose PD from gait. LPGNet uses Linear Prediction Residuals (LPR) to extract discriminating patterns from gait recordings and then uses a 1D convolution neural network with depth-wise separable convolutions to perform diagnosis. LPGNet achieves an AUC of 0.91 with a 21 times speedup and about 99% lesser parameters in the model compared to the state of the art. We also undertake an analysis of various cross-validation strategies used in literature in PD diagnosis from gait and find that most methods are affected by some form of data leakage between various folds which leads to unnecessarily large models and inflated performance due to overfitting. The analysis clears the path for future works in correctly evaluating their methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_59

SharedIt: https://rdcu.be/cyl6z

Link to the code repository

https://github.com/devalab/parkinsonsfromgait

Link to the dataset(s)

https://physionet.org/content/gaitpdb/1.0.0/

Reviews

Review #1

Please describe the contribution of the paper

The paper presents an approach to Parkinson’s disease detection using gait analysis. The method proposed in the paper is an efficient model that can be run on a device with low memory and power footprint.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The work is presented on a public dataset, hence comparison with state of the art is clear. The discussion about data leakage is also compelling. The presentation on results is clear.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

A detailed comparison with the other methods is not done. The argument of the authors is the way in which the other methods treat the data leads to data leakage. However the main aspect of this paper in my opinion is to present something that is efficient in its memory and power footprint.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Nothing about the sharing the code is presented. However the fact that the data is publicly available and the training approach has been presented clearly, the work can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Comparison with other approaches in literature would be useful using the same data split.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The methods is novel and the results and analysis
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Somewhat confident

Review #2

Please describe the contribution of the paper

The authors present a novel method LPGNet that uses linear prediction residual with a 1D CNN to efficiently diagnose Parkinson’s from VGRF recordings. In this work, the authors also evaluate different validation strategies used in literature and identify the presence of data leakage and show that a subject level separation is necessary for correct evaluation of a method.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed method uses LPGNet (Linear Prediction residual based Gait classication Network) which is a deep learning based model that diagnoses Parkinson Disease (PD) from gait with good accuracy while being fast and small enough to be used in embedded systems to enable the method to be cheap and widely accessible. The method proposed achieves good discriminative performance with an accuracy of 90.3% and an F1 score of 93.2%. The model proposed is also orders of magnitude smaller and faster than methods described in literature.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The major weakness of the paper is the comparison with the state of the art methods. Even though one is more recent (2020), the other 2 methods are not so recent .
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

In my view, the methodology is not easily reproducible with just the information contained in the paper. Despite some details are given, some important aspects of the network architecture are not provided in the paper. These details about the network are provided in the supplementary work however this may hinder the reproducibility based solely on the paper. Also, the data split is not provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

In my view, it would be profitable for the reader if more details on the network and training and testing process would be provided in the paper.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

My reasons to recommend an acceptance of the paper are: the evaluation of different validation strategies used in literature and identification of the presence of data leakage; the results that show a good accuracy of the method while being fast and small enough to be used in embedded systems enabling the method to be cheap and widely accessible; the fact that the model proposed is smaller and faster than methods described in literature.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

The authors propose a PD diagnosis method using the CNN method based on linear prediction residual from gait (LPGNet). Specifically, LPR was used to extract discriminative patterns from gait recording then 1D CNN was adopted to perform the diagnosis. The proposed method had a fast and accurate PD diagnosis. In addition, different validation strategies were used to identify the presence of data leakage and evaluate the subject level separation for precise evaluation of the method.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The main objectives and motivations of the work are clear.
2. Different validation strategies were used and evaluated.
3. Comparisons and discussions were present and sufficient.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Proposed CNN is very simple.
2. The dataset used contains limited subjects.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. The dataset used is a publicly available one.
2. Code is not disclosed for public.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Why the predicted value in Eq. (1) is negated? Is it typo?
- Details of the different components of the CNN are missing in both text and Fig. 2.
- I do not think there is need for supplementary material for the baseline CNN since it is same architecure as in Fig. 2.
- Importantly, authors did not justify their choice of CNN, it is expected that LSTM can yield better berformance in the underlying task.
- Writing has few typos such as “the state of the art”, “ELU”, the word “use” in Introduction.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Manuscript is clear and easy to follow.
2. Methodology is well-explained and the results are plausible.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Somewhat confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper presents an approach to Parkinson’s disease detection using gait analysis. The idea and design of this paper are good, which can provide a design perspective for the diagnosis of Parkinson’s Disease. According to the reviewers’ comments, it’s better to add some comparison results with other methods. The preprocessing of the raw signals seems to be important, the authors should describe this part in detail. For example, the authors should explain the reason that the raw signal is filtered with a moving averagelter of order 2. What is the principle of stratified splits and how to conduct it? As reviewers’ comments, the paper is highly scored, which is appropriate to accept. But the authors should modify some issues according to the suggestion of reviewers point by point.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

The reviewers have pointed out that the codebase is not publicly shared (we avoided sharing the code repository in the initial submission in view of the double-blind review process). All the implementations of the experiments discussed in the manuscript along with the data splits used will be publicly shared with a link to the code repository in camera-ready version of the paper.

The lack of enough comparisons with methods in literature has also been mentioned. As discussed in the manuscript, most methods in literature were susceptible to data leakage and hence the tuned models ended up being extremely large and large models overfit easily and perform sub optimally. Hence, to avoid redundancy we compared with one recent, well cited work for which the code is publicly available.

Response to the points raised by Reviewer #3

The –ve sign in eq (1) is to compensate for the negative coefficients obtained by optimizing eq (3)/(4).

We experimented with both CNNs and LSTM networks and it was observed that CNNs generally performed better.

back to top

Linear Prediction Residual for Efficient Diagnosis of Parkinson's Disease from Gait