Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jihun Yoon, Jiwon Lee, Sunghwan Heo, Hayeong Yu, Jayeon Lim, Chi Hyun Song, SeulGi Hong, Seungbum Hong, Bokyung Park, SungHyun Park, Woo Jin Hyung, Min-Kook Choi

Abstract

Automated surgical instrument localization is an important technology to understand the surgical process and to analyze them to provide meaningful guidance during surgery or surgical index after surgery to the surgeon. We introduce a new dataset that reflects the kinematic characteristics of surgical instruments for automated surgical instrument localization of surgical videos. The hSDB (hutom Surgery DataBase)-instrument dataset consists of instrument information of cholecystectomy videos obtained from 24 cases of laparoscopic surgery and gastrectomy videos obtained from 24 cases of robotic surgery for gastric cancer. Localization information for all instruments is provided in the form of a bounding box for training using the object detection framework. To handle the class imbalance problem between instruments, synthesized instruments modeled in Unity for 3D models are included as training data. Besides, for 3D instrument data, a polygon annotation is provided to enable instance segmentation of the tool. To reflect the kinematic characteristics of all instruments, they are annotated with head and body parts for laparoscopic instruments, and with head, wrist, and body parts for robotic instruments. Annotation data on assistive tools (specimen bag, needle, etc.) that are frequently used for surgery are also included. We provide statistical information on the hSDB-instrument dataset and the baseline localization performance of the object detection networks trained through the MMDetection library and the resulting analyses.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_38

SharedIt: https://rdcu.be/cyhQA

Link to the code repository

https://hsdb-instrument.github.io/

Link to the dataset(s)

https://hsdb-instrument.github.io/

Reviews

Review #1

Please describe the contribution of the paper

The paper describes a database of laparoscopic video images annotated with bounding box information for various instruments. The database covers 24 laparoscopic cholecystectomies and 24 robot assisted gastrectomies. The paper itself does not propose any novel algorithms to process the data, but does present the results of evaluations using multiple existing classifiers. The database and associated analyses seem a useful contribution to the field.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Automated surgical tool detection is an important and very popular field of research. The addition of a well curated and documented data set should be of benefit to the community. Curating and publishing such a data set is a substantial undertaking and I applaud the authors. There are extensive instructions on how to use the data set on the associated website (https://hsdb-instrument.github.io/) and the results presented in the paper give some confidence that the data is of use.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is no real novelty here. I don’t know whether something like this would be better submitted as a MICCAI challenge. Without actually trying to use the data set I can’t say for certain whether or not it is useful yet.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This should be good, reproducibility is the whole point of data sets like this.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This seems a very useful contribution to the field. Thank you. There are a few bits that I did not understand and would like you to clarify.

1: The real data is annotated by 2 trained annotators and a supervisor? I’m assuming this is manual annotation by humans? What do you mean trained? Are they medical experts? What training does the supervisor have? More details of this process would be helpful.

2: Synthetic data from CT. Please can you supply more information on how this was done. The CT data must have been segmented and texture mapped. But also presumably the patients were not insufflated for the scans, so how did you create the air space necessary for simulated surgery?

3: I had to do a bit of searching to work out how you calculated mean average precision (mAP). My understanding now is that you calculate mAP at 10 values of IoU between 0.5 and 0.95 and the reported figure is the mean of all these. This may be because I am not expert in the field, but in any case it would make the paper easier to read if you explained it a bit more.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The dataset and baseline analysis is a useful contribution to the field. It’s only a 7 because there’s no actual novelty present. If I was more confident in my understanding of how they generated the synthetic data and did the annotation I might move more to an 8.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Review #2

Please describe the contribution of the paper

This paper presents a dataset for MIS surgical tools localization. Additionally, a baseline performance was provided to test the dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A new dataset of surgical tools in two different surgical procedures
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- No comparison of contributions or improvements of this dataset over the rest publicly available
- No reading of baseline results
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Everything ok
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The paper is well-written and expose clearly the generation and annotation of the dataset. However, there are other public dataset (I just include here some of them, but there are more). I strongly recommend to briefly introduce them and argue their limitations. So, it will be welcomed a comparative discussion about the main contributions of your dataset regarding previous ones. https://endovissub-instrument.grand-challenge.org/Data/ https://datasets.bifrost.ai/info/848 http://camma.u-strasbg.fr/datasets

Currently it is not clear the need for a new dataset and what specific features or contributions provide your dataset that improves all the previous ones. Regarding to the baseline, I really appreciate this point. It is interesting to know how well-known algorithms performs with this dataset. However, the authors only present a huge amount of values without any kind of reading or discussion about its meaning. I think it will be really interesting expose the main contributions of this dataset to get better results (for this it will be useful to compare with any other previous work that used the same network with other dataset). Do your dataset get any performance improvement for a specific network model? I mean, could you compare your results with other studies? e.i. is a value of 25.7 a good or bad results for the mPA metric? As far as I can observe mPA values are very similar with and without synthetic and DR. Could you justify the importance of the synthetic dataset? What is the meaning of DR? Maybe it is my fault but I cannot find it in the manuscript.

Minor changes:
- Figures 3 and 4 are almost impossible to read. Text size is too small, I would suggest to make it bigger and maybe expose it in a landscape orientation.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well-written and provide a new dataset that could be useful but in the current state it is not clear what are the better features of this dataset regarding to other ones.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper descrives an open dataset including annotated images (position of the different parts of the instruments) in laparoscopic surgery and robotic laparoscopic surgery.The objective is to provide data for machine learning in the context of modelling and recognizing surgical interventions. Synthetic data are added to deal with non uniform data distribution w.r.t the type of instruments. A baseline performance of CNN for localization is also provided.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- such data may be very useful to people wanting to test their instrument localization method.
- it represents a big amount of work
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- the scientific contribution of the paper is not clear.
- the augmentation of data with synthetic images (the randomized part) may introduce some bias since instruments may not be represented in their “conventional” position with the conventional contextual image information. It seems to me that adding images with instruments which are rarely present in real situations could also result in an increase of false positive.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

it is an open dataset produced for reproducibillity concerns.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper presents a dataset of annotated images of laparoscopie images (robotic or not) for the localiztion of surgical tools (and their subparts). I wonder about the information given about the instruments : provided that they are often very angulated with respect to the image axes, a rectangular bouding box aligned to these axes may be a quite inaccurate information about the real pixels including the instrument. In section 2.1, the third sentence (“All the tools were divided etc.”) is really unclear. You mention that two experts judge the annotations and one annotation is kept when both agree on it. But who annotates the images? If this results from your baseline networks who provided the labels for training? It would be worth giving the number of annotated images of the different kinds (real surgery, simulated surgery and randomized synthetic images). The first time an acronym is used, you have to expand it.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The main concern is about scientific contribution - I am not sure that MICCAI is a place to publish information about a database even if the database is useful to ithers and buidlding it represent a huge amounf of work. Moreover I am not convinced that augmentation (the random part) is acceptable (since images may not correspond to real situations).
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper presents a new dataset for instrument localisation prepared using 24 videos from Cholec8- dataset. The dataset is interesting for the community. However, there is no technical contributions. And the motivation of having this dataset is not clear. What is the need of this new dataset compared to already existing datasets which provides instrument detection labels?
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

Thank you for the efforts of all reviewers. We notate ‘C#-#’ for each reviewers and comments and ‘A#’ for the answer.

[Common comments] (C-1) Technical novelty. (A-1) hSDB-instrument is the first and most large-scale dataset for localization of surgical instruments utilizing robotic gastrectomy surgery, to our best knowledge. It is also the first study to apply domain randomization using a virtual patient model in surgical vision. Based on table 2. in main manuscript, all models in all types of surgeries were improved by using this dataset (real+synthetic+domain randomization(DR)). There were also cases in which the existing datasets for surgical instrument localization didn’t follow the standard metric of object detection, especially in the evaluation criteria. We applied the well-known MS-COCO dataset interface to our dataset and configured the database API so that anyone can easily obtain objective indicators. (C-2) Dataset novelty. (A-2) According to studies, [1] is the most representative laparoscopic surgery dataset in MIS instrument localization, including 30 surgery cases with 7 instruments. For robotic surgical instrument dataset, [2] is the most representative dataset including 10 clips of In-vivo test surgery (pig) with 7 instruments. Compared to these latest surgical instrument recognition datasets, hSDB-instrument dataset includes 24 cases of cholecystectomy with 10 instruments and 24 cases of gastrectomy with 17 instruments and also synthetic data. [3] which reviewer 2 referred, doesn’t provide annotations of surgical instrument localization but the instrument presence for frames and [4] doesn’t provide videos in real surgical environments but in simulation. We will update a comparison table of these datasets in the updated version of main manuscript. (C3) Advantage of domain randomization (A3) Domain randomization method introduced in [5] is known for not just simple augmentation but improving performance in exceptional situations by generalizing pixel distribution. We found that using it together with our synthetic data caused significant performance improvement. (C-4) Annotation process. (A-4) A supervisor is a medical expert with a lot of medical field experiences and trained two non-medical annotators on overall surgical procedures and surgical instruments. The supervisor was also trained on Computer Vision Annotation Tool to inspect all annotations from the annotators. All the annotations were marked by two trained annotators, cross-validated with each other, and only the annotations finally approved by the supervisor were stored in the database.

[Reviewer comments] (C1-1) Supply more information on CT data. (A1-1) To get pneumoperitoneum models, we 3D scanned patients before and after pneumoperitoneum and manually annotated CT Dicom files for each organ. For texture mapping, we used the Blender and at the end, bound the models with the Unity. (C3-1) Bounding box is an inaccurate information for the instruments. (A3-1) Object detection with bounding box is a general method to localize object in computer vision although it doesn’t give us a tight boundary of an object. For further studies. segmentation can be utilized to get more exact pixels of instruments. (C3-2) Dataset in MICCAI. (A3-2) Cell detection and counting dataset [6] was already accepted in MICCAI 2020 and our research not only provides new dataset for the surgeries but also suggests a novel method to generate data for performance improvement.

[References] [1] Heidelberg colorectal data set for surgical data science in the sensor operating room, [2] 2017 Robotic Instrument Segmentation Challenge, [3] EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos, [4] Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos, [5] Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization, [6] BCData: A Large-Scale Dataset and Benchmark for Cell Detection and Counting

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have successfully justified the major concerns. The contributed data is the first large-scale data of its kind. The camera ready should be updated to reflect the justifications provided in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper presents a new open database to be used for automated surgical instrument localization of surgical videos in laparoscopic and robotic surgery, that includes the kinematic characteristics of the instruments, and shows the baseline performance. The paper is well-written and shows a substantial amount of work, and the addition of a well-curated and documented dataset for surgical tool localization is a welcome contribution to the CAI community. The main concerns are regarding lack of scientific contribution/novelty, (and contribution over prior publicly available datasets), which are addressed by the rebuttal to an extent.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper introduces a novel dataset of laparoscopic videos with bounding box annotations for various surgical instruments. One of the key strengths is that synthetic data was included to address unequal data distribution. The reviewers appreciate the effort that was put in creating this novel dataset and the extensive benchmark of various classifiers. One of the major concerns, however, is that this paper does not present significant novelty and it is not clear how it relates to existing datasets in this field. In my opinion, a novel dataset can be valuable MICCAI submission, for example if it poses a novel problem for the community or addresses an currently underrepresented task. However, also considering the arguments in the rebuttal, I am unfortunately not convinced of a significant added value or dataset novelty for this submission. I do believe that this work is valuable and I would love to see it in form of a open challenge, for example. But unfortunately, I tend towards “reject” for this paper as MICCAI main conference paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

14

back to top

hSDB-instrument: Instrument Localization Database for Laparoscopic and Robotic Surgeries