Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Tuong Do, Binh X. Nguyen, Erman Tjiputra, Minh Tran, Quang D. Tran, Anh Nguyen

Abstract

Transfer learning is an important step to extract meaningful features and overcome the data limitation in the medical Visual Question Answering (VQA) task. However, most of the existing medical VQA methods rely on external data for transfer learning, while the meta-data within the dataset is not fully utilized. In this paper, we present a new multiple meta-model quantifying method that effectively learns meta-annotation and leverages meaningful features to the medical VQA task. Our proposed method is designed to increase meta-data by auto-annotation, deal with noisy labels, and output meta-models which provide robust features for medical VQA tasks. Extensively experimental results on two public medical VQA datasets show that our approach achieves superior accuracy in comparison with other state-of-the-art methods, while does not require external data to train meta-models.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_7

SharedIt: https://rdcu.be/cyl5z

Link to the code repository

https://github.com/aioz-ai/MICCAI21_MMQ

Link to the dataset(s)

VQA-RAD dataset: https://www.nature.com/articles/sdata2018251

Pathvqa dataset: https://arxiv.org/pdf/2003.10286.pdf

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a multiple meta-model quantifying method to learn meta-annotation for pre-training and leverage the pre-trained knowledge for the medical VQA task. The proposed method aims to address some problems of meta-learning pre-training for medical VQA tasks, e.g., previous methods are heavily impacted by the meta-annotation phase for all images in the medical dataset so that noisy labels may occur when labeling images in an unsupervised manner, high-level semantic labels cause uncertainty during learning, and so on. The proposed method does not make use of additional out-of-dataset images, while achieving high accuracy in two challenging medical VQA datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A meta-learning pre-training method was proposed for medical VQA tasks to address some problems that exist in previous works.  
- The model does not make use of additional out-of-dataset images, while achieving high accuracy in two challenging medical VQA datasets.
The motivation and problem description are clear and significant.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Although the model outperforms all the baseline models, the baseline models are weak.  
- There are some grammar issues.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors do not provide code and other implementation details of their model. I am not sure if the results reported in this paper are reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The motivation and problem description are clear and significant.

You should choose more recently proposed strong VQA methods as your baseline.

You can provide more implementation details in your paper for reproducibility.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The strengths and weaknesses listed in parts 3 and 4.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

6
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The author proposed a framework using meta-learning to refine the training dataset, and showed that the pre-trained meta-learning models are useful in the downstream medical visual question answering task. The proposed method is validated on two public datasets and can outperform the MEVF method published in MICCAI 2019.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper proposed a novel algorithm for the data refinement based on the predicted score from the pre-trained meta-learning model.
- The pre-trained meta-models are successfully applied in the visual question answering task which means the feature extractor probably learned the semantic information from the meta-training tasks.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The main contribution of this paper is using meta-learning for the data refinement, so it will be very interesting to add another baseline where the MEVF method uses meta-model pre-trained on the refined dataset to show the effectiveness of using multiple meta-learning models.
- The author mentioned that they used uncertainty in the data refinement process, but from what they described in the algorithm part, they were using predicted scores rather than uncertainty.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It should be reproducible as long as the code is published.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

In generall, it is a good paper using meta-learning. Besides what mentioned in the weakness section, One thing is that the data pool used in the paper does not have unlabeled data. I believe all of them are labeled as required for the meta-training process. I guess using noisy-labeled data is more appropriate.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The data refinement part is novel. It is a good paper using meta-learning to obtain a more powerful feature extractor. It improves over the previous SOTA method. The writing can be improved and one baseline is expected as the ablation study for using multiple meta-learning models.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposes an approach for VQA on two public medical datasets. In this paper MMQ, and adaption from MAML, is introced. This adapation is mostly directed to overcome the extra complications with transfer learning in medical imaging. Authors show how this meta-learning method outperforms current VQA methods, as well as how data refinement on the meta-model can increase performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-creative approach in medical vqa, described accurately in the methods section -Shown good performance against earlier methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

-grammatical errors and double paragraph headers -Structure of paper is sometimes unclear -algorithms on page 5 are not clear. It might be possible to integrate these more with the rest of the paper -best performance in tables should be in bold for optimal clarity -No example of output is shown
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors indicated to make code publicly available. Method is clearly described, but the paper is missing relevant information on training settings and hyperparameters
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Interesting contribution, with interesting results. Integration of the algorithms on page 5 with the paper and showing an example of the model output are major area’s for improvement.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Quality of writing, strucuture of paper
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The reviewers and myself agree that the paper is of high enough quality for acceptance at MICCAI. All reviewers have favorably reviewed the work, and have also provided constrcutive feedback to the paper to improve its quality. I would ask the authors to take these into account before submitted their final version. In particular, there are number of references on the topic of med-VQA that could be added.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We sincerely thank the Reviewers and the AC for taking the time to consider our paper and give constructive feedback. We address the main concerns below.

Writing issues: grammatical errors, algorithms, and the structure of paper (R1, R2, R3).

We thank reviewers to point out our mistakes and unclear parts of our paper. All of them will be fixed and re-organized when we publish our camera ready version.

Source code: training settings, hyper parameters, and implementation details (R1, R2, R3).

The training setup, hyper parameters, and implementation details were presented in our Supplementary Material due to the paper’s length limitation. We will publish the source code with all these details for further research.

The baseline (R1,R2).

We agreed that there maybe other recent VQA baselines in computer vision that outperforms the baseline we used. However, they are not directly applied in medical images. Furthermore, as suggested by R2, we can also integrate our multiple meta-models learning process into MEVF. This will be done in our future work.

The uncertainty in the the data refinement process (R2).

From our understanding, the uncertainty occurs when the model cannot produce a high enough prediction score for a specific sample during training. Regardless of the reason behind is the incorrect annotation or the highly semantic label, the most recognizable sign of this problem is the “uncertainty” probability score of training samples, i.e., the predicted score (as mentioned by R2). We agreed that there are other methods that can output uncertainty scores (e.g. using label distribution). This could be an interesting direction to improve the accuracy of the medical VQA task in the future.

Example output (R3).

The example outputs of our proposed MMQ will be added.

References (AC).

We have intensively searched for more related papers and added them to our camera ready version. Thanks for your suggestion.

back to top

Multiple Meta-model Quantifying for Medical Visual Question Answering