Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Seungjae Han, Eun-Seo Cho, Inkyu Park, Kijung Shin, Young-Gyu Yoon

Abstract

Calcium imaging is an essential tool to study the activity of neuronal populations. However, the high level of background fluorescence in images hinders the accurate identification of neurons and the extraction of neuronal activities. While robust principal component analysis (RPCA) is a promising method that can decompose the foreground and background in such images, its computational complexity and memory requirement are prohibitively high to process large-scale calcium imaging data. Here, we propose BEAR, a simple bilinear neural network for the efficient approximation of RPCA which achieves an order of magnitude speed improvement with GPU acceleration compared to the conventional RPCA algorithms. In addition, we show that BEAR can perform foreground-background separation of calcium imaging data as large as tens of gigabytes. We also demonstrate that two BEARs can be cascaded to perform simultaneous RPCA and non-negative matrix factorization for the automated extraction of spatial and temporal footprints from calcium imaging data. The source code used in the paper is available at https://github.com/NICALab/BEAR.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_56

SharedIt: https://rdcu.be/cyl8S

Link to the code repository

https://github.com/NICALab/BEAR

Link to the dataset(s)

https://github.com/NICALab/BEAR

https://drive.google.com/file/d/115lCnwIVU0TtKedQ_31FDaOG8wksGmG9/view

Reviews

Review #1

Please describe the contribution of the paper

A variant formulation of Robust PCA -implementation agnostic- called BEAR, and 3 different implementations (basic, greedy and cascaded) all based on neural networks. The construct validity of the greedy version, and its application exemplification on calcium imaging.

The rationale of the new RPCA variant BEAR is based on replacing the minimization of the trace norm (here referred to as nuclear norm I suppose in reference to Schatten norms family) with a maximum rank constraint. Then some other additional constraints follow but they are dependent on the implementation.

While the basic implementation requires a prior guess on the rank of L, the greedy version eliminate such demand but still keep a low complexity.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Many! A good rationale on the proposed solution, the separation of the formulation of the solution from the implementation, the 3 variant implementations, …
- No need to show off, or overtone… Just sober good experimentation with convincing results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

If original motivation was computational complexity of RPCA there are stronger competitors to compare to.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sufficient maths and pseudo code -supplementary material- to guarantee replicability (even if code wouldn’t have been provided…which it has anyway!)
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
OVERALL COMMENTS It took me two rounds of reading to fall in love with this research. During my first round, I was so driven by the computational complexity that I admit I slightly loss focus on the big picture; the authors were claiming an improvement in computational complexity but they were actually referring to unenhanced versions of PCA, when nowadays optimizations exist that under certain constraints can reach sublinear complexity, see [Ma and Aybat (2018) Proc of IEEE 106(8):1411-1426], and BEAR also has constraints, so what was the big deal?. But the second reading put things into focus; even without sophisticated optimization BEAR was actually really good in complexity, it is remarkably stable in its error, and the rest of the paper is actually very good in almost every aspect! Not much to say other than congratulations to the authors on a beautiful work! Some minor comments below follow but feel free to ignore them.

SUGGESTIONS TO IMPROVE THE DRAFT
- Fig 3a is excellent to quantitatively report errors. Can it be further accompanied by another figure qualitatively showing good/success and bad/failure examples on the synthetic data?
- Sect 3.2 is perhaps unnecessarily long given that you are already giving the computational complexity analysis… Let me push my luck here; expand Sect 3.3 at the cost of Sect 3.2
- Evaluation is made over the greedy variant but application results are reported with the cascaded variant. Why not complete both ways? Evaluate and validate all 3 variants, and comparatively show the results of the 3 variants on real images.
- Show also results of the another RPCA variant in Sect 3.3. for visual comparison.
- Report exact sizes (number of neurons per layer) and number of parameters of the neural network. If I got this right, since one of the constraints is L=WW^{T}Y then it is likely the size of the network does depend on Y. Correct?
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Robust PCA is well studied. There are now tens of variants each one trying to optimize here or there. However, to the best of my knowledge the variant in Eq 3 is novel and even if perhaps not as optimal as some other variants for very specific cases, fig 3a justifies compromising a bit of efficiency for a remarkably stable error which means that for practical purposes, the end-user can employ BEAR without worrying much of particularities.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors proposed BEAR, a simple bilinear neural network for the efficient approximation of RPCA which achieves an order of magnitude speed improvement with GPU acceleration compared to the conventional RPCA algorithms. In addition, BEAR can perform foreground-background separation of calcium imaging data as large as tens of gigabytes.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors introduce a Bilinear neural network for Efficient Approximation of RPCA which is a computationally efficient implementation of RPCA as a neural network. The work solves the key problem of efficiency in RPCA and it is general for similar tasks.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

But it lacks some reference of similar tasks.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The parameters and environment of the experiment is given, it can be easily reproduced as the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Add the reference: [1]Van Luong H, Joukovsky B, Eldar Y C, et al. A Deep-Unfolded Reference-Based RPCA Network For Video Foreground-Background Separation[C]//2020 28th European Signal Processing Conference (EUSIPCO). IEEE, 2021: 1432-1436. [2]Hovhannisyan V, Panagakis Y, Parpas P, et al. Multilevel approximate robust principal component analysis[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops. 2017: 536-544. [3]Solomon O, Cohen R, Zhang Y, et al. Deep unfolded robust PCA with application to clutter suppression in ultrasound[J]. IEEE transactions on medical imaging, 2019, 39(4): 1051-1063.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This work give a neural network to solve the problem which is hard to be optimized. The novelty and contribution of this work is clear. But it lacks some reference of similar tasks.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper casts the problem of background vs. foreground separation of the calcium recordings as a constrained optimization problem that can be efficiently optimized using a neural network.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The problem formulation is interesting, and of potential use for the neuroscience community. The application to multiple datasets is shown (mouse, drosophila, zebrafish).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

See the detailed comments below
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Codes are provided and some hyperparameters are given. Mostly clear but can still be improved (e.g. which 1/3 of the data is used for training).
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The idea of using neural nets to solve PCA/RPCA/matrix factorization existed way before this paper, please make sure that the relevant papers are cited.
- How does this compare to CNMF-E/OASIS? https://elifesciences.org/articles/28728 https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005423
- Eq. 1 is called low-rank+sparse recovery, please cite the relevant papers.
- Typo below Eq. 1 -> Y, L, S, S
- Reformulated -> better wording: Eq. 2 is a surrogate for Eq. 1 and does not necessarily give the same solution
- We do have faster SVD methods instead of doing the full decomposition on the whole data matrix https://arxiv.org/pdf/1502.00182.pdf
- O(nmr) -> how about the number of iterations? How does it interact with m, n, r? This gives a more realistic view of “time complexity”
- Eq. 3 again is not a “reformulation”, the solution is different
- Shouldn’t L be WM instead of WMY? Why are we multiplying by Y?
- M = W^T, what is the interpretation? It finds a low-rank AND symmetric matrix which we might not want.
- What kind of batching is used? Over time, over space?
- Inference only mode can be applied to SVD too, when the singular vectors are known
- The time comparison should include the training of the neural net (training is the same as finding singular vectors & values).
- In Eq. 5, what does “localized” in localized sparse footprints correspond to? How is this condition enforced in the cost?
- How is the non-negativity enforced? We have specific optimization algorithms for NMF, do the authors simply use back-prop for NMF?
- Please include a short description of the compared methods. If some methods are based on full SVD on the data matrix how can BEAR outperform them?
- Results in Fig. 4 are nice, but videos would be much more helpful, also can the authors show “residual” too (Y-L-S).
- The figures on real data as illustrated are not quite informative, please include more informative visualizations (use the papers provided above as examples).
- The experiments are limited, since the methods existed before, it’s important to show that the proposed approach existing methods in multiple common datasets
- Why training only on 1/3 of data? How is that 1/3 selected? What if there’s a shift in the dynamics of the data in the initial stages of recording vs. later later?
- In many cases the low rank component is itself variable in time (there’s overall intensity change in the tissue), how can that be added to the model?
- Simulation experiments (simulated calcium videos) are crucial to see how well the method can recover ground truth signals.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Experimental evaluation is limited (on simulations and real data). A clearer empirical understanding of the method is required before sharing it with experimentalists. Literature review seems incomplete, there already exists some literature on sparse+low-rank recovery.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

1
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed BEAR, a simple bilinear neural network for the problem of background vs. foreground separation of the calcium recordings.

The key strengths include 1) The problem formulation is of potential use for the neuroscience community. 2) The Bilinear neural network for Efficient Approximation (BEAR) is a computationally efficient implementation of robust PCA (RPCA). 3) Multiple species data including mouse and zebrafish are used in the validation of the new method.

The key weaknesses include: 1) lacks some reference 2) some minor issues mentioned in reviewers’ comments.

In sum, this paper has some theory contribution and I would like to recommend “accept” once the authors can address/clarify the concerns from the reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We thank the reviewers for their constructive comments. Below is our response to the major comments.

The manuscript lacks some reference. A) We will add references including the papers suggested by the reviewers.

Include short descriptions of the compared methods. A) We will include a short description in the revised manuscript.

Evaluation is made over the greedy variant but application results are reported with the cascaded variant. Why not complete both ways? A) Cascaded BEAR performs RPCA and NMF simultaneously, whereas the synthetic data is suited only for validation of RPCA. In line with the reviewer’s comment, we will add supplementary videos that show the performance of different algorithms on the real data.

What is the number of parameters of BEAR? A) The total number of parameters in BEAR is n × r, where n is the number of rows in the data matrix and r is the target rank.

How does BEAR compare to CNMF-E/OASIS? A) CNMF-E is a variant of NMF whereas the main focus of BEAR is to perform scalable and high speed RPCA. OASIS is for deconvolution of spike trains, and hence not related to our work.

There is fast and scalable RPCA paper using faster SVD methods. (https://arxiv.org/pdf/1502.00182.pdf) A) We will cite this paper in the revised version. Unfortunately, we could not make a direct comparison as the source code is not publicly available.

How about the number of iterations in time complexity O(nmr)? A) Each iteration in BEAR and SVD-based algorithms has a time complexity of O(nmr) and O(nm^2+n^2m), respectively. The required number of iterations differs for different algorithms, and therefore, the computation time should be compared as in Section 3.1.

What kind of batching is used? A) Each mini-batch is constructed by randomly sampling columns.

The time comparison should include the training of the neural net. A) All computation times reported in the manuscript include all time taken for training. We would like to clarify that even the reported computation time of BEAR with inference-only includes the time taken for training.

In Eq. 5, what does “localized” in localized sparse footprints correspond to? How is this condition enforced in the cost? A) It was reported (Yang et al 2007) that projective NMF results in high orthogonality between the learned base vectors (i.e., sparsity). Localization is a side-benefit that comes with sparsity (i.e., a non-sparse component cannot be localized).

How is non-negativity enforced? A) Non-negativity is attained by using projected gradient descent.

How can BEAR outperform some methods based on full SVD? A) The original version of RPCA relies on convex relaxation of the optimization problem which requires that L has low rank and S is sparse. This is not met in the top-right regime of the phase diagrams and therefore, fails to converge. We speculate that this left the possibility for an approximate method to outperform.

Add videos for Figure 4 and authors can show “residual (Y-L-S)” too. A) We will add multiple videos in the revised version. Adding the residual may not be particularly suited for comparing BEAR and other methods, as the residual of BEAR is always zero by design.

The experiments are limited, since the methods existed before, it’s important to show that the proposed approach existing methods in multiple common datasets. A) As the “ground truth” low rank component cannot exist in real data, the standard method for evaluating RPCA algorithms is to decompose synthetic data with known L and S, as we demonstrated in Section 3.1.

Why did authors train only on ⅓ of data for inference-only mode? How is that ⅓ selected? What if there’s a shift in the low-rank dynamics of the data? A) The portion of the data is a user-defined hyperparameter and the first ⅓ of the video was selected. The inference-only mode is not suited if a large shift in low rank component is expected.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think the authors have done a good job in addressing the reviewers’ questions. I would like to recommend to accept.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

A simple bilinear neural network is presented for efficient approximation of robust principal component analysis. It achieves an order of magnitude speed improvement with GPU acceleration compared to conventional RPCA algorithms and can perform foreground-background separation of gigabyte-scale calcium imaging data. The paper makes a solid contribution but especially one reviewer asks for clarification on many aspects of the work. This is provided by the authors in their feedback, which seems sufficiently convincing.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper presents a novel and computationally efficient approach to an important problem that is within the scope of MICCAI. Concerns in the reviews included some missing references, which the authors promised to add, and a number of detailed questions, several of which are convincingly addressed in the rebuttal. The main remaining concern is that R3 would have liked to see an even more extensive experimental comparison. However, I would agree with the other reviewers, who found the presented experiments to be convincing, especially in the light of the page limit at MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

back to top

Efficient neural network approximation of robust PCA for automated analysis of calcium imaging data