Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Michael Vasilakakis, Georgia Sovatzidi, Dimitris K. Iakovidis
Abstract
Wireless capsule endoscopy (WCE) constitutes a medical imaging technology developed for the endoscopic exploration of the gastrointestinal (GI) tract, where-as it provides a more comfortable examination method, in comparison to the con-ventional endoscopy technologies. In this paper, we propose a novel Explainable Fuzzy Bag-of-Words (XFBoW) feature extraction model, for the classification of weakly annotated WCE images. A comparative advantage of the proposed model over state-of-the-art feature extractors is that it can provide an explainable classi-fication outcome, even with conventional classification schemes, such as Support Vector Machines. The explanations that can be derived are based on the similarity of the image content with the content of the training images, used for the con-struction of the model. The feature extraction process relies on data clustering and fuzzy sets. Clustering is used to encode the image content into visual words. These words are subsequently used for the formation of fuzzy sets to enable a linguistic characterization of similarities with the training images. A state-of-the-art Brain Storm Optimization algorithm is used as an optimizer to define the most appropriate number of visual words and fuzzy sets and also the fittest parameters of the classifier, in order to optimally classify the WCE images. The training of XFBoW is performed using only image-level, semantic labels instead of detailed, pixel-level annotations. The proposed method is investigated on real datasets that include a variety of GI abnormalities. The results show that XFBoW outperforms several state-of-the-art methods, while providing the advantage of explainability.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87199-4_46
SharedIt: https://rdcu.be/cyl4G
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
In this paper, author has proposed a novel Explainable Fuzzy Bag-of-Words (XFBoW) feature extraction model, for the classification of weakly annotated WCE images. A state-of-the-art Brain Storm Optimization algorithm is used as an optimizer to define the most appropriate number of visual words and fuzzy sets and al-so the fittest parameters of the classifier, in order to optimally classify the WCE images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Some strength of the papers are mentioned below. 1) The paper is well written and categorized including feature extraction, bag-of-words and swarm intelligence. 2) In the XFBoW the FFE method is extended using visual class-specific visual vocabularies from weakly annotated images to describe the visual content of a WCE image. 3) According to author, the proposed XFBoW is the first BoW extension able to provide explainable histogram features, where the similarity of an image to every visual word is explained with a linguistic value.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The clarity in Model optimization is not understandable and supportive to the work needs to be performed. 2) To show the robustness of the algorithm, where author is comparing XFBoW to all BoW based method it is only focussed on the AUC.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
NA
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The author has contributed for paper entitled “Explainable Classification of Weakly Annotated Wireless Capsule Endoscopy Images based on a Fuzzy Bag-of-Colour Features Model and Brain Storm Optimization”. The paper is extending the previous work done with the BoW methods for WCE image dataset. Here I would like to provide some comments mentioned below: 1) What made you chose to extend the BoW methods over the previous work done and compared to the CNN methods? 2) Explainability analysis is not very clear to chose the linguistic variables such as none, low, medium and high. 3) The dataset chosen seems very small for the validation and comparison with the earlier method which might use the other and/or bigger dataset. 4) How this extended BoW work might perform for the disease severity on mild, moderate and severe cases. 5) Any need of creation the training and testing dataset for validation for the proposed work?
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The author have presented the work to modify the BoW to be more effective and by providing the advantage of explainability. The work flow need to be organized in better way to support the outcome.
- What is the ranking of this paper in your review stack?
3
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
The paper proposed a novel variant of bag of words called XFBoW feature extraction model for classification in weakly annotated settings. Authors’ have claimed that their models are explainable histogram features and perform better than other approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well structured, experiments are satisfactory.
- The proposed method is a classic Machine learning approach for a weakly supervised setting, it should work even with less amount of data.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some grammatical mistakes and spelling errors are there.
- For such work which is well explored I will suggest authors to cite more papers.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
NA
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Please correct the spelling erros and grammatical mistakes of the work.
- Add more relevant citations such as with metrics, consider citing the following papers:
- https://arxiv.org/abs/2006.01263
- https://arxiv.org/abs/2006.14822
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the work is a good contribution in classic Machine learning setting, but I believe there are more advanced approaches exists which are both explaianable and have more efficient feature extractor.
That being said, this paper is a contribution towards a setting when there is less data, and application doesn’t want to handle the deep learning model maintenance. So, Overall I will say this paper belongs to accept cateogry.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
Authors propose an Explainable Fuzzy Bag-of-Words feature extraction model, which is an extension of the BoW method. This provides explainable historgram features for a linguistic value.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Well written paper. The method is clear and it is easy to understand. The reasons for the different choices are well explained. The results are well explained and it is easy to follow the ideas.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The tests are performed in KID database, which has a small number of images, specially regarding some of the lesions.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Use of a public database and all the parameters are given or cited. Nothing to declare.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The only thing I see where you could improve is the choice of the database. Try to look for Kvasir Capsule database which has a larger number of images.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A good paper, with the major concern regarding the choice of database. I propose to accept it.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a novel Explainable Fuzzy Bag-of-Words (XFBoW) feature extraction model, for the classification of weakly annotated WCE images. Although three reviewers give relatively high comments, I have several concerns related to this paper. 1) Actually some bag of words-related papers have been proposed for endoscopy images in recent years (https://scholar.google.com/scholar?as_ylo=2017&q=bag+of+words+polyp). The author should make a comprehensive introduction of related papers (instead of old ones in the paper [2,4,5,7]) and clarify the corresponding contribution of this paper. 2) A lot of grammar errors or wrong words in this paper, for example, “2.1 Explainable Fuzzy Bug-of-Words Model”; the line after the equation 2 “𝑤𝑖,𝑚 represent the 𝑖th visual word”. 3) Since there are a lot of existing similar papers, the author should choose the recent papers to conduct a comprehensive and fair comparison. The reason for choosing the current comparison methods of this paper should be added. 4) A lot of deep learning-based models have been proposed for WCE image classification, the author should comment on this and emphasize the contribution/novelty of this paper. Therefore, I suggest “rebuttal” for this paper and give the opportunity for the authors to clarify these concerns.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
1
Author Feedback
We thank reviewers R1, R2, R3 and meta-reviewer MR for their valuable comments and for suggesting possible acceptance of our paper, recognizing its novel contribution and the fact that it is well -written, -explored and -structured, and easily understandable (R3) with satisfactory experiments (R2).
MR-4 A lot of deep learning-based…paper & MR-3 Since there are a lot of existing…comparison & R1-7.1 What made you choose…methods?: A comparative advantage of the Bag of Words (BoW) approach is that it resembles the way humans use specific vocabularies of words for the description of real-world concepts. Deep learning (DL)-based models are black-box classifiers. Unlike current DL or BoW models that have been applied to date for WCE image classification, XFBoW is explainable. As indicated in Section 2, XFBoW “can be applied on different features and can be combined with any classifier” including DL-based features and classifiers. To evaluate the performance of XFBoW, due to the limited availability of source codes implementing other state-of-the-art methods and considering its advantage of explainability, we performed comparisons both with the baseline and state-of-the-art BoW-based methods including models that inspired us to create XFBoW. This is pinpointed in the introduction. Also, we included comparisons with DL-based models from which the best experimental results have been reported on the same dataset.
MR-1 Actually some bag of words-related…contribution of this paper: We would like to thank MR for bringing these papers to our attention; we will replace older references with recent ones and make a comprehensive introduction as suggested. We will further clarify the main contribution of our paper, which as indicated in our previous answer is that XFBoW is explainable, and we will also highlight that XFBoW is applicable to other BoW variations, such as the generalized BoW model used in [García-Rodríguez, et al(2020)Polyp fingerprint: automatic recognition of colorectal polyps’ unique features. Surg. endoscopy 34(4), 1887-89]
@R1-4.2 To show the robustness…on the AUC: As indicated in Section 3.1, our comparisons are focused on the AUC, because unlike the rest of the evaluation metrics used i.e. accuracy, sensitivity, specificity, the AUC metric is more robust for datasets with imbalanced class distributions [17].
R1-7.2 Explainability analysis is not very clear to choose the linguistic variables…: The linguistic variables (LVs) express the similarity degree, between the image features and the visual words. They correspond to fuzzy sets (Fig.1). Thus, for a fuzzy set corresponding to a centroid that is closer to 0, we chose LV “Low”. For the fuzzy set corresponding to a centroid which is closer to 1, we chose LV “High”. Similarly, for the fuzzy sets corresponding to centroids in between, LVs expressing intermediate similarity degrees, e.g. “Medium” are chosen.
R1-7.3 The dataset…method & R3-4,7 The tests are performed in KID…lesions: For our experiments we used the KID Dataset as it has been proved sufficient and challenging for similar comparisons in well-recognized journals [7,10]. Our intention to perform experiments on larger datasets is also indicated in our paper.
R1-7.4 How this extended BoW…severe cases: In this paper we do not propose a methodology to assess the severity of a disease. This would require changes to our methodology and data that would include annotations about the severity of the cases. This is an interesting perspective for future research.
R1-7.5 Any need of creation the training…work?: Different training and testing sets were used. As indicated in Section 3.1, a 10-cross validation scheme was used to evaluate the model, i.e. the dataset was randomly partitioned into 10 equally sized disjoint subsets, a single subset was retained as the validation data for testing the model and the remaining 9 subsets were used as training data. This was repeated until all subsets are used for testing.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposed a novel Explainable Fuzzy Bag-of-Words (XFBoW) feature extraction model, for the classification of weakly annotated WCE images. Although three reviewers give relatively high comments, several concerns are related to this paper. 1) Actually some bag of words-related papers have been proposed for endoscopy images in recent years (https://scholar.google.com/scholar?as_ylo=2017&q=bag+of+words+polyp). The author should make a comprehensive introduction of related papers (instead of old ones in the paper [2,4,5,7]) and clarify the corresponding contribution of this paper. 2) A lot of grammar errors or wrong words in this paper, for example, “2.1 Explainable Fuzzy Bug-of-Words Model”; the line after the equation 2 “𝑤𝑖,𝑚 represent the 𝑖th visual word”. 3) Since there are a lot of existing similar papers, the author should choose the recent papers to conduct a comprehensive and fair comparison. The reason for choosing the current comparison methods of this paper should be added. 4) A lot of deep learning-based models have been proposed for WCE image classification, the author should comment on this and emphasize the contribution/novelty of this paper. Therefore, I suggest “rebuttal” for this paper and give the opportunity for the authors to clarify these concerns.
In the rebuttal session, the author addressed most of these concerns and this paper does have some contribution in the endoscopy image analysis. Acceptance is recommended.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors rebut that the main contribution of the paper is “explainability” of the model. However, the technical ideas of the paper are all ideas taken from the literature: BOW, Swarm and just put together. At the same time, CNN methods are currently performing better than the proposed method and there isn’t a contribution in terms of performance. The authors have also not discussed how much less data the propsed model would need or done any ablation experiments for that. Therefore, I do not think the paper is ready to be published yet.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
18
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The work is novel within the machine learning framework.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
2
Meta-review #4
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a new method combining bag of visual words (BoW) and fuzzy set for wireless capsule endoscopy image classification. The main contribution to existing BoW methods is the additional fuzzy set applied on the visual similarity histogram. Besides improving the classification accuracy, it provides better explainability than the BoW methods and the convolutional neural network (CNN). It is one of handful submissions to recent MICCAI conferences without using deep learning.
This paper received relative high ratings from reviewers, but it has several limitations. 1) Though outperforming other BoW methods and one CNN approach, its AUC is significantly lower than LB-FCN, another CNN-based approach (0.841 vs. 0.935). 2) The added explainability is still weak. Basically, the proposed method can provide a few similar cases from the training set for the query image and claim, for example, “because the query image is more similar to an abnormal image, it should be classified as abnormal.” However, such trick can be easily extended to other approaches. For example, we can take the extracted features from a CNN, use them to retrieve similar training images, and present these images (together with similarity scores) to a user as explanation. 3) There are quite a few grammar errors. 4) In a standard setting for a classification problem, a label is provided for the whole image. So, I do not think the proposed method can be claimed to be a weakly supervised method.
Of all, this is really a borderline paper. Considering the pros and cons, I bias a little bit towards accepting.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
-