Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Viswesh Krishna, Anirudh Joshi, Damir Vrabac, Philip Bulterys, Eric Yang, Sebastian Fernandez-Pol, Andrew Y Ng, Pranav Rajpurkar

Abstract

The application of deep learning to pathology assumes the existence of digital whole slide images of pathology slides. However, slide digitization is bottlenecked by the high cost of precise motor stages in slide scanners that are needed for position information used for slide stitching. We propose GloFlow, a two-stage method for creating a whole slide image using optical flow-based image registration with global alignment using a computationally tractable graph-pruning approach. In the first stage, we train an optical flow predictor to predict pairwise translations between successive video frames to approximate a stitch. In the second stage, this approximate stitch is used to create a neighborhood graph to produce a corrected stitch. We extend GloFlow to leverage visual information at multiple resolutions to align separated tissue sections on a slide. On datasets of simulated video scans of pathology slides, we find that our method outperforms known approaches to slide-stitching, and stitches images resembling those produced by slide scanners. Our method allows for creation of whole slide images using widely-available low cost microscopes.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_50

SharedIt: https://rdcu.be/cymbc

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper introduces a computationally tractable approach for image stitching/alignment.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

I do not see any novelty in this paper. The main strength of this paper is the clear description of the method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This paper has the following weaknesses:
1. There is no technical contribution. The proposed methods consists of two main stages. In the first stage (Creating Approximate Stitch Using Sequential Pairwise Registration) it is just a minor modification of the existing FlowNet, and the authors name it FlowNetMod (This name is misleading). I would say it is a reduced version of the FlowNet since the original FLowNet output a dense pixel-wise translation map between two input images while the modification just output a single offset. For the 2nd stage there are well studied methods in literature (especially in panorama stitching), and I did not see any superiority over those methods.
2. The comparison is not enough. The authors should compare their method with state-of-the-art methods, such as Adobe Image Stitching Tool, “As-Projective-As-Possible Image Stitching with Moving DLT”, “Seagull: Seam-guided local alignment for parallax-tolerant image stitching”.
3. Figures are not of high quality. For example, text in Fig. 1 is too small to see. In Fig.2 and Fig.3 I do not see “original” images before stitching.
4. Reference is not enough. Much SOTA work in image stitching have been ignored.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think this paper can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
I have the following suggestions:
1. When studying a problem, it is not enough just reviewing most well known methods (such as Lucas-Kanade (LK) optical flow). Image stitching has been well studied recently. You can check this paper: “As-Projective-As-Possible Image Stitching with Moving DLT”. See which paper cites this one and what papers it cites.
2. The formatting of the paper can be improved. Please check previous MICCAI papers.
Please state your overall opinion of the paper

strong reject (2)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think this paper is definitely below MICCAI quality. Two major factors:
1. Lack of technical contribution.
2. Not enough comparison with SOTA methods.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

2
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The paper introduces a 2-stage method to create a whole slide image using optical flow-based image registration with global alignment.The method uses a graph-pruning approach. The first stage consists on an optical flow predictor trained to predict pairwise transformations. The second stage uses creates a neighborhood graph used to improve the overall registration.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Method claims to be 36x more accurate than pairwise sequential registration methods and 10x as computationally efficient as global alignment method

Evaluation is the best section of the paper, comparing against multiple relevant competing methods, utilizing good metrics and with good comparison figures
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Introduction and related work sections are not detailed enough and don’t provide enough information about the state of the art.

More specific mathematical details about the proposed method would be welcome.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Nor code nor data sets seem to be publicly available. Code is described but more specific mathematical details would be needed for replication
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Paper is robust, and the method seems to be an interesting advance in the field. Introduction and related work sections are short and weak Method is well described with words, but lacking mathematical detail. Evaluation is the best part of the paper.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Paper is robust, and the method seems to be an interesting advance in the field. Introduction and related work sections are short and weak Method is well described with words, but lacking mathematical detail. Evaluation is the best part of the paper.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposed a two-stage method called GloFlow to stitch a Whole Slide Image(WSI) from a video. A new metric called Re-EPE was also proposed to evaluated the stitching performance. The author claimed that GloFlow mehtod outperformed pairwise registration and global alignment on accuracy and computational time, respectively.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed GloFlow method offers a new way to digitize the histological slides. From methodology point of view, the GloFlow just combined two exiting methods (Flownet and template matching), however, the proposed method was quite useful for their specific clinical application. The method may be extend for creating WSIs from a low cost microscope.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The data (histological video) they used in the paper was only created from existed WSI. It is more convincing if they collected a real video from the microscope or other digital scanners.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The supplement files gave the detailed description of their method.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The training of the Flownet was not clearly described, e.g. the data distribution, parameter selections etc.
2. The comparison of different methods was not clear. For instance, in fig.2, I can’t understand how the GloFlow method produce the final result (4th colunm) from the FNM(2nd column). A lot of image information was missing in 2nd column, how is possible to globally align these images?
3. The author used the term of “significantly”. That means statistic evaluation is need, such as p-value.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method may be used to help the pathologists reading WSI from a low cost microscope.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #4

Please describe the contribution of the paper

GloFlow, a two-stage method for creating a whole slide image using optical flow-based image registration with global alignment using a computationally tractable graph- pruning approach was proposed. The method allows for creation of whole slide images using widely-available low cost microscopes.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. An efficient method for slide stitching was proposed;
2. The approach validated on 40 videos (200k frames) from 5 H&E slides;
3. The paper has a thorough description of the method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. More evaluation on data from another hardware would be beneficial as well as more description on how the data for evaluation was captured (hardware, camera, etc);
2. Maybe some elaboration on other WSI modalities would be interesting to include, such as IHC and mIF images?
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sharing the code and a test dataset will be highly desired for reproducibility of the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. More evaluation on data from another hardware would be beneficial as well as more description on how the data for evaluation was captured (hardware, camera, etc);
2. Maybe some elaboration on other WSI modalities would be interesting to include, such as IHC and mIF images?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

GloFlow, a two-stage method for creating a whole slide image using optical flow-based image registration with global alignment using a computationally tractable graph- pruning approach was proposed. The method allows for creation of whole slide images using widely-available low cost microscopes.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

7
Reviewer confidence

Not Confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes GloFlow - a 2 stage method for computationally tractable image stitching/alignment. The paper seems well written. However, the reviewers have brought up questions about novelty and comparisons to state of the art. Please respond regarding these issues in your rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

Reviewer 2 states that “Introduction and related work sections are not detailed enough and don’t provide enough information about the state of the art.” and Reviewer 1 states “Reference is not enough. Much SOTA work in image stitching have been ignored.” and “When studying a problem, it is not enough just reviewing most well known methods (such as Lucas-Kanade (LK) … You can check this paper: “As-Projective-As-Possible …”. While we could have provided a review of current state of the art approaches to general image stitching, we consider a more restrictive problem formulation where only planar transformations are possible and thus planar transformation approaches such as Lucas Kanade optical flow work effectively. We also found it more appropriate to discuss related work specific to image stitching in microscopy and pairwise image registration as it helps us demonstrate current approaches in stitching multi-row and column images and their drawbacks.

Reviewer 1 states ”The comparison is not enough. The authors should compare their method with state-of-the-art methods, such as Adobe Image Stitching Tool, “As-Projective-As-Possible …”, “Seagull: Seam-guided local …”.” The current state of the art stitching methods such as APAP (Zaragoza et. al., 2013), Seagull (Lin et. al., 2016), Adaptive As-Natural-As-Possible Image Stitching (Lin et. al., 2015), and Parallax-tolerant Image Stitching (Zhang et. al., 2014) develop approaches that extend traditional homographies or seam methods to account for general projective warps and remove ghosting and other artifacts. However, our problem formulation of stitching successive patches from a whole slide image ensures that we only need to consider planar transformations with no possible projective warping (so current approaches such as APAP and Seagull are simply not effective comparisons). Further, we find that in whole slide image stitching white empty image patches between tissue sections pose a significant problem to all general image stitching algorithms since they cannot extract features from these empty regions and thus always break continuity of the stitch. In such situations homography methods (RANSAC) and state-of-the-art panoramic stitching methods (e.g. A novel panoramic image stitching algorithm based on ORB (Wang et. al., 2017)) predict large translations which prevents them from generalizing to a multicolumn scenario which is required in whole slide image stitching. While it may still be hypothetically feasible to utilize stitching approaches (e.g. Seagull) which compute iterative correspondences between all pairs of images, such approaches are not practically feasible as there are hundreds to thousands of patches to be stitched together which causes runtimes to shift from several minutes to days. The only comparative baseline which we identified that could account for four criteria - planar transformations, white patches without features in between, multiple rows and columns, and practical speed - was the multigraph optimization approach from “A robust method for image stitching.” Pellika et. al, 2020 which was our primary baseline.

Novelty: Optical flow predictors (FlowNet, Lucas-Kanade) and pairwise homography approaches have shown strong predictive power for local alignment but accumulate error globally. Multi Graph optimization (Pellika et. al, 2020) is effective only when relative positions between images are known. To the best of our knowledge GloFlow is the first work that aims to solve the limitations of pairwise image registration approaches on whole slide image stitching by using global graph based optimization as well as the first work that aims to solve the need for relative position information by using optical flow as an initialization.

Reviewer 3, 4: We can include a stitch from video captured from an iPhone mounted on the microscope (this is a direction for future work). The data was captured on a Phillips scanner at a 20x magnification.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors seem to have explained the concerns of R1. I would recommend acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

R1 underlines that the paper lacked a deep review of the state of the art in image stitching. The AC concurs. The stitching methods are numerous and it forms an old computer vision problem, especially in the planar case. The rebuttal arguing that the state of the art has focused on the planar case is thus not convincing. The paper’s technical novelty is overall very limited and the state of the art review too limited.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

13

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a two-stage method for WSI stitching from video. While the 1st stage utilizes optical flow to predict pairwise transformations, the 2nd stage employs a neighborhood graph to produce a corrected stitch. Although more details regarding the method and related work should be introduced, most of the reviewers agree that the proposed method is effective and promising for the field. For the rebuttal, the authors have clarified the concerns regarding the contribution of the paper, but not mention too much about the questions of method and experiments, e.g., lacking mathematical detail and the training is not clear.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

back to top

GloFlow: Whole Slide Image Stitching from Video using Optical Flow and Global Image Alignment