Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Peng Huang, Xiuzhuang Zhou, Zeqiang Wei, Guodong Guo

Abstract

Content-based image retrieval (CBIR) has attracted increasing attention in the field of computer aided diagnosis, for which learning-based hashing approaches represent the most prominent techniques for large-scale image retrieval. In this work, we propose a Supervised Hashing method with Energy-Based Modeling (SH-EBM) for scalable multi-label image retrieval, where concurrence of multiple symptoms with subtle differences in visual feature makes the search problem quite challenging, even for sophisticated hashing models built upon modern deep architectures. In addition to similarity-preserving ranking loss, multi-label classification loss is often employed in existing supervised hashing to further improve the expressiveness of hash codes, by optimizing a normalized probabilistic objective with tractable likelihood (e.g., multi-label cross-entropy). On the contrary, we present a multi-label EBM loss without restriction on the tractability of the log-likelihood, which is more flexible to parameterize and can model a more expressive probability distribution over multimorbidity image data. We further develop a multi-label Noise Contrastive Estimation (ml-NCE) algorithm for discriminative training of the proposed hashing network. On a multimorbidity dataset constructed by the NIH Chest X-ray, our SH-EBM outperforms most supervised hashing methods by a significant margin, implying that the energy-based supervised hashing possesses better expressiveness for representation of multi-label medical images, facilitating multilevel similarity preservation in multimorbidity image retrieval.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_20

SharedIt: https://rdcu.be/cyl5T

Link to the code repository

N/A

Link to the dataset(s)

https://academictorrents.com/details/557481faacd824c83fbf57dcf7b6da9383b3235a

Reviews

Review #1

Please describe the contribution of the paper

The paper covers the development of a CBIR technique, using Supervised Hashing with Energy-Based Modeling, for computer aided diagnosis. A multi-label energy-based loss is itroduced, which has no restriction on the tractability of the log-likelihood, which is more flexible to parameterize resulting in a more expressive probability distribution for the multimorbidity image data used.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The developed method has novel aspects, but also learly building on existing work.

Evaluation is based on a large multimorbidity dataset using NIH Chest X-ray data.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is a claim of significance, but no appropriate stats test is used to support this claim.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This is using public data and the descriptions are clear enough to repeat.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Start section 2.1 with “We” not “we”.

Would it make sense to use an alternative to AlexNet?

In section 2.3, use “is trained by optimizing” instead of “is trained by optimize”.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is incremental work. The evaluation is good, but the claim snot fully supported by the presented results.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

Focusing on content-based medical image retrieval, the authors proposed a supervised hashing model with energy-based modeling to tackle the multi-label challenge. According to the comparison results on a large-scale dataset, the proposed method could achieve superior performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) The proposed energy-based modeling strategy is reasonable to tackle the clinical challenges of the multi-label problem on imaging data study; (2) The paper is well organized and the proposed method is well illustrated both visually and objectively;
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Even though the proposed method is a more competitive choice on the NIH chest x-ray dataset, it’s better to include more imaging and disease types to further demonstrate the efficiency of this method.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors provide enough information to reproduce the reported results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) Since the title of this paper is “xxx for multimorbidity image retrieval”, it’s better to include more datasets to avoid data bias caused by experiments on a single dataset. (2) Since the hashing method often emphasizes computing/arithmetic speed, it’s better to include the comparison of computing time.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

(1) The proposed method is well illustrated and the oraginzation of this paper is good. (2) Some issues as I point in 7 should be further improved.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposed a Supervised Hashing method with Energy-Based Modeling (SH-EBM) for scalable multi-label image retrieval. And a multi-label EBM loss without restriction on the tractability of the log-likelihood, which is more flexible to parameterize and can model a more expressive probability distribution over multimorbidity image data.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This work has a certain originality and technical depth. And the motivation setting is reasonable.

It is interesting to develop a supervised hashing method with energy-based modeling for scalable multi-label image retrieval. Furthermore, this paper proposes an effective training algorithm based on noise contrast estimation called the multi-label EBM loss, which overcomes the limitation of logarithmic likelihood traceability and makes parameterization more flexible. The results of experiments are also highly competitive.

Overall, the writing and organization of this paper are good. In this paper, the description of the method is clear, and the analysis of the experimental results is detailed.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The experiment is not comprehensive.

The baseline selection is not appropriate. The selected baselines are very limited. The comparison with the best current methods, as stated in the article, is not comprehensive. This article only compares some supervised hashing methods, and there are many other supervised hashing methods, such as DLBHC(https://homepage.iis.sinica.edu.tw/~kevinlin311.tw/cvprw15.pdf).

Although the results presented in this article are relatively good, the effectiveness of the method can only be reflected through comprehensive comparison. What’s more, we suggest conducting experiments on multiple datasets. It is not enough to compare with other methods on only one dataset.

In addition, In Fig.1, the interactive part of the upper subgraph and the lower subgraph of the model graph is not clearly described; Fig.3 only shows the visualization of the results of the method proposed in this paper on one dataset, which is not sufficient.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Some reproducibility, but not particularly high.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

In section 3, the method proposed should be compared with all current state-of-the-art supervised hashing methods and the experimental results should be presented on multiple datasets to show the effectiveness of the method.

In addition, In Fig.1, we suggest that the interactive parts of the upper subgraph and the lower subgraph of the model diagram should be more clearly described and represented; In Fig.3, we recommend comparing and presenting visualizations across multiple datasets using multiple approaches.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It is interesting to develop a supervised hashing method with energy-based modeling for scalable multi-label image retrieval. What’s more, this paper proposed a multi-label EBM loss without restriction on the tractability of the log-likelihood, which is somewhat novel.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper developed a supervised hashing method with energy-based modeling for medical image retrieval. This paper is well-organized and easy to follow. The major concern is that the experiment is not comprehensive. Results on multiple datasets could be helpful to further validate the proposed method. Also, more baseline and SOTA methods on supervised hashing could be used for solid comparison.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We thank the Area Chair and all reviewers for their time, efforts and insightful comments that are helpful to improve the quality of our manuscript. We have clarified most concerns raised by the reviewers, and the detailed responses are summarized as follows.

Q1: It’s better to include more datasets to further demonstrate the efficiency of this method.

Due to the page limit, more multi-label medical datasets haven’t been included for experimental evaluation in the paper. In future work, we would like to further validate the efficacy of our method on additional Multimorbidity datasets, such as ODIR-5K (a structured ophthalmic database of 5,000 patients, introduced by Peking University International Competition on Ocular Disease Intelligent Recognition in 2019).

Q2: More comprehensive baselines and SOTA methods on supervised hashing could be used for solid comparison.

Due to the page limit, we reproduced some representative supervised hashing algorithms for experimental comparison on the Multimorbidity dataset. Under the page limit, in final version we will try our best to present solid comparison with additional supervised hashing methods (such as DLBHC suggested by the reviewer).

Q3: Would it make sense to use an alternative to AlexNet?

The backbone can be any off-the-shelf networks like VGG and ResNet. For fair comparison, all the hashing methods chosen for comparison are built upon the same backbone (i.e., AlexNet).

Q4: There is a claim of significance, but no appropriate stats test is used to support this claim.

Experimental evaluation of multi-label image retrieval algorithms often adopts widely used metrics like NDCG, ACG and mAP. Such evaluation metrics were also adopted in our experiments, and the results demonstrated that our method outperformed most SOTA methods by a significant margin. As far as we know, few works (in image retrieval literature) adopt the stats test for such significance evaluation. In future work, however, we would like to investigate on such significance test for our method.

Q5: It’s better to include the comparison of computing time for the hashing method.

Note that the EBM branch of our proposed network is introduced for training more powerful hash features. The branch is no longer included in the inference stage. In this setting, most hashing methods being compared in the experiments have almost the same inference cost. Therefore, we haven’t include the comparison of computing time in the experiments.

Q6: In Fig.1, the interactive part of the upper subgraph and the lower subgraph of the model graph is not clearly described.

We will improve the visualization of the figure in final version to better illustrate the interaction of two sub-networks in model training and inference.

back to top

Energy-Based Supervised Hashing for Multimorbidity Image Retrieval