Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Bovey Y Rao, Alexis M Peterson, Elena K Kandror, Stephanie Herrlinger, Attila Losonczy, Liam Paninski, Abbas H Rizvi, Erdem Varol

Abstract

Spatial transcriptomics techniques such as STARmap enable the subcellular detection of RNA transcripts within complex tissue sections. The data from these techniques are impacted by optical microscopy limitations, such as shading or vignetting effects from uneven illumination during image capture. Downstream analysis of these sparse spatially resolved transcripts is dependent upon the correction of these artefacts. This paper introduces a novel non-parametric vignetting correction tool for spatial transcriptomic images, which estimates the illumination field and background using an efficient iterative sliced histogram normalization routine. We show that our method outperforms the state-of-the-art shading correction techniques both in terms of illumination and background field estimation and requires fewer input images to perform the estimation adequately. We further demonstrate an important downstream application of our technique, showing that spatial transcriptomic volumes corrected by our method yield a higher and more uniform gene expression spot-calling in the rodent hippocampus. Python code and a demo file to reproduce our results are provided in the supplementary material and at this github page: https://github.com/BoveyRao/Non-parametric-vc-for-sparse-st

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_45

SharedIt: https://rdcu.be/cymaX

Link to the code repository

https://github.com/BoveyRao/Non-parametric-vc-for-sparse-st

Link to the dataset(s)

https://drive.google.com/file/d/1ea08Vtb9B-iJ0yybgNu6FZy77gtgVohm/view?usp=sharing

Reviews

Review #1

Please describe the contribution of the paper

To remedy the transcriptomics microscopy issues, such as shading or vignetting effects from uneven illuminations, this paper presents a non-parametric approach to correct such vignetting effects. Empirically it outperforms existing methods. Python code is available.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Pros: *Overall I feel the paper is well presented, with good empirical results and performance. *The figures are well prepared, and easy to understand *implementation is made publicly available
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Cons: *relatively lack of sufficient empirical evaluations, and lack of comparison methods
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

implementation is available
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

To remedy the transcriptomics microscopy issues, such as shading or vignetting effects from uneven illuminations, this paper presents a non-parametric approach to correct such vignetting effects. Empirically it outperforms existing methods. Python code is available. Pros: *Overall I feel the paper is well presented, with good empirical results and performance. *The figures are well prepared, and easy to understand *implementation is made publicly available

Cons: *relatively lack of sufficient empirical evaluations, and lack of comparison methods.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall the paper is well-presentation
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors propose a non-parametric method to remove artifacts from stitched mosaics of images, to mitigate effects of uneven illumination and allow mosaic-wide operations that depend on homogeneous image statistics.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The problem addressed is important.
2. The solution is creative, practical, and appears to be effective.
3. The exposition is clear.
4. A clear codebase is provided.
5. The comparisons are well-done.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Some of the figure subsections are uninformative (eg fig 2 A, C, E; fig 2 G time-series of Basic and Cidre).
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors provided a codebase.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The opening sentence did not connect for me.
2. Confusing terminology: “FoV” seems to be used for both the stitched mosaic and for individual images. Also, on page 3: “all FoVs” should be “each FoV”?
3. Fig 1 is good (except: why “volumes” for 2D image).
4. I wish you had said “hat matrix” relative to the solution to equation 2.
5. Inconsistent concatenated vector notation: [A B] is used in eqn 2, but 11 is used in eqns 3 - 5.
6. The second validation (effect on gene spotting) is a good addition.
Note: The ranked ordering given below is a dummy value, and meaningless, as I did not compare the various papers.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The many strengths listed above.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The authors propose a new method for vignetting correction in images, which is essentially an iterative histogram normalization algorithm with linear regression setting that converges to a reference histogram. In contrast to the other baseline methods they compared with (BASIC and CIDRE), the authors used a prospective method that requires reference images. Authors demonstrated their better performance over the baseline methods on a STARmap dataset using their own metric, which is, in my opinion, different from the others.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Clear Mathematical Formulation of the Problem: The authors formulated vignette correction as a marginal histogram matching problem which was further converted to a sliced Wasserstein distance minimization problem. Such flow of formulation was an interesting readout of the paper.

Evaluation in STARmap: It was interesting to read about the effect of vignetting correction on Spatial Transcriptomics images, which is an exciting new field.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Reference Image Selection: The algorithm proposed by the author is solely dependent on a reference strip. The method tries to converge the histograms of the input strips to its reference strip using a linear regression setting and thus achieve uniformity across all the strips. The uniformity is expected to be achieved, but how to select the reference strip is a concern. The authors did not discuss how they selected the reference strip and how the selection is finally affecting the downstream analysis results. If the gene spot detection results get solely dependent on the nature of reference strip, then the method may not be useful at all.

Inconsistency Among Evaluation Criteria: The authors stated that they used similar approach to BASIC for evaluation. But the evaluation criteria of them seems different from BASIC and other methods. In previous methods, the correction score is used when the ground-truth, shading-free images are unavailable, This score is based on the absolute difference between pairs of overlapping, corrected images that are precisely aligned, normalized by the benchmark error of the uncorrected image pairs. The authors of this paper instead used standard deviation of pixel intensities in aggregated mean images or maximum images as evaluation metric. The authors need to justify how their metric is equivalent to others. Moreover, they gaussian filtered the images before evaluating which was not the case in any of the previous methods. If the way of evaluation or the metric does not match with the previous methods, then such evaluation is not fruitful. The authors need to justify how their evaluation is fair considering the above cases. Limited Clarity in Method Description: The authors mentioned that their method only requires the setting of number of bins for histogram discretization. But it is not clear from the method description how number of bins are used and how they play their role.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors provided all the materials they selected in the reproducibility checklist. The provided codes and dataset commensurate with the results provided in the paper. The results are reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I would recommend the authors to explain how they selecting the reference strip and also show whether their method is robust to the selection of reference strip. The authors need to clarify the concerns regarding evaluation metrics and comparison to BASIC and CIDRE I have raised as a weakness of the paper. I would also suggest clarifying the use of a number of bins in their algorithm and how it emphasizes middle or extreme quantiles for vignette correction. It would be also interesting to see how methods like BASIC or CIDRE perform in the downstream gene spot-calling analysis to ensure a fair comparison. It would be great if the authors can show results for more STARmap spatial transcriptomic images instead of only one.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper was overall a good read, with a new way of formulating the vignette correction problem. The results obtained in downstream analysis- Higher gene expression spot calling by vignette correction in spatial transcriptomics images was also very interesting. However, the concerns regarding to robustness of the method and fair evaluation lead me to a borderline reject.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposed a novel method for vignette correction for spatial transcriptomic images based on an iterative sliced histogram normalisation. Such a vignette correction will greatly improve the downstream gene spot calling analysis and boost the sensitivity of STARmap for gene detection. It also outperforms two other state-of-the-art illumination correction methods, CIDRE and BaSiC, according to the authors. All reviewers agree that the paper addressed a critical biological problem and generate nice results, yet R3 has some concerns about the consistency of evaluation criteria for the proposed method and comparing methods (CIDRE and BaSiC). Why authors choose to use the standard deviation of pixel intensities in aggregated mean images or maximum images as an evaluation metric, why not use the common region of two partially overlapped FOV as CIDRE and BaSiC (R3). Using mean/maximum as metrics will bring an advantage to the proposed method over CIDRE and BaSiC as the method itself is based on histogram matching. In my opinion, the key trick of the proposed method is to break a normal histogram matching of FOVs at each pixel position into marginal histogram matching along the horizontal and vertical direction, which makes it robust with a low number of available FOVs. Yet this also implies that the estimated flatfield (S) and darkfield (D) are both rank-one matrix. Authors should discuss how valid this assumption is and indicate when it may fail if this underlying assumption is not satisfied. Overall the paper is well written, the method is simple and effective, and the demo code is easy to run and substantially improves paper reproducibility (which also help me to better understand the method). I think the paper can be accepted if the authors can address the two issues (evaluation metric and underlying assumption) in the rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We extend our thanks to the reviewers and the meta-reviewer for the thoughtful comments. Here we address the two primary concerns: 1) the performance evaluation metric and 2) the assumption of a rank-one matrix of illumination correction.

We updated the performance evaluation to the correction score (Icorr) metric that is used in BaSiC (Peng et al. 2017) and showed that our method outperforms BaSiC and CIDRE (Smith et al 2015). Accordingly, we revised figure 2 in the updated manuscript and tabulated the quantitative results below for the inspection of the reviewers since we are unable to attach figures.

The second concern raised by the reviewers was the validity of rank-one assumption of the correction field. Rank-one modeling captures the dome shaped vignetting field observed in imaging setups and is a consequence of how we slice the histograms (by rows/columns). Since we can normalize arbitrary shapes of histograms, it is possible to sample “checkerboard” patterns of histograms and model any arbitrary rank/shape of vignetting fields. We have updated our implementation to model more complex vignetting using this approach and have updated our discussion in the revised manuscript to detail the extension of the method.

Below are technical responses and the tabulation of new performance evaluations for figure 2 and figure 3. We updated figure 2 to reflect the evaluation metric that is used in BaSiC and updated figure 3 to reflect gene spot calling performance in mouse hippocampus comparing all the three methods.

For the Icorr, the absolute difference of the overlap from corrected slices is divided by the absolute difference from raw slices. A lower Icorr suggests that the vignetting effect is neutralized more effectively. We assessed 4 pairs of overlapping sections (2 overlapping on the top/bottom and 2 on the left/right), and examined their Icorr with different numbers of input images for correction.

O=Our method,B=BaSiC,C=CIDRE top/bot1

images|10|25|50|100

O|0.59|0.65|0.61|0.42 B|2.53|1.21|1.30|0.94 C|1.48|2.25|0.29|0.18 top/bot2 O|1.27|1.15|0.74|0.65 B|1.12|1.60|1.17|1.53 C|1.83|1.71|0.79|1.49 left/right1 O|1.18|3.35|1.28|0.93 B|0.86|1.06|0.94|0.91 C|1.99|1.64|9.07|2.81 left/right2 O|1.40|1.42|1.12|1.50 B|1.59|0.95|2.42|1.01 C|2.98|2.38|2.46|12.6 Mean O|1.11|1.64|0.94|0.88 B|1.53|1.21|1.46|1.09 C|2.07|2.00|3.15|4.26

Our method had a mean Icorr<1 with 100 images, which suggests better removal of the vignetting effect than the other approaches, both >1. CIDRE’s high Icorr is likely due to noted difficulties with hot-pixels. We will continue to add samples to this validation for the updated figure 2.

R3Q4c: Vignetting effects are regional, so we utilized bins to reduce an image to its approximate optical vignetting. Fewer bins will favor the maximum projection, and vice versa.

R3Q7a: We used the middle slice from the input images as the reference. To address the robustness of the middle reference slice, we selected a random slice as referenced and assessed the Icorr of 2 pairs of sections with 100 input images. The columns are: both mid, 1 mid and 1 rand, 1 rand and 1 mid, and both rand. top/bot 0.43|0.46|0.42|0.47 0.43|0.44|0.42|0.42 0.43|0.43|0.43|0.42 left/right 0.96|0.91|1.01|1.10 0.96|0.91|1.01|1.05 0.96|0.92|0.98|1.02

R3Q7b: Gene spot-calling using BarDensr was performed on the raw(R) and vignetting corrected images from our method(O), BaSiC(B), and CIDRE(C). Gene | R | O | B | C CALB2 |259|3254|2985|2529 CLD5 |130|2296|1837|1320 CPNE5 | 44 | 939 | 551 | 482 CRH |138|1879|1639| 1387 CXCL14| 3 |239 | 149 | 138 EMC10 |74 |1436|1079|219 NPY |135|1680|1144|1079 RGS10 |635|4738|4184|3563 SST |160|1191|571|620 WFS1 | 46 |1024|708|840 Total |1624|18676|14847|12776 Spot detection results for ten genes highly expressed in the hippocampus are shown above. Vignetting correction by our method increases the number of spots per transcript detected as compared to the raw, BaSiC, and CIDRE.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have excellently made their rebuttal. Added new comparison results and explained the assumption of the correction method. I happily recommend the paper acceptance and suggest oral presentation/award.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think the authors do a good job of addressing the issues raised by the reviewers. I would recommend acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This is a well written paper proposing a novel method for vignetting correction for sparse spatial transcriptomics images. The experimental results show the method outperforms two other state-of-the-art methods. Overall the reviewer support for this paper is strong. The main concerns, regarding the used evaluation metric and underlying assumption, are sufficiently addressed by the authors in their rebuttal, leading to an improved paper that in my opinion can be accepted for MICCAI 2021.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

back to top

Non-parametric vignetting correction for sparse spatial transcriptomics images