Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Javid Dadashkarimi, Amin Karbasi, Dustin Scheinost

Abstract

Functional connectomes derived from functional magnetic resonance imaging have long been used to understand the functional organization of the brain. Nevertheless, a connectome is intrinsically linked to the atlas used to create it. In other words, a connectome generated from one atlas is different in scale and resolution compared to a connectome generated from another atlas. Being able to map connectomes and derived results between different atlases without additional pre-processing is a crucial step in improving interpretation and generalization between studies that use different atlases. Here, we use optimal transport, a powerful mathematical technique, to find an optimum mapping between two atlases. This mapping is then used to transform time series from one atlas to another in order to reconstruct a connectome. We validate our approach by comparing transformed connectomes against their gold-standard'' counterparts (\textit{i.e.}, connectomes generated directly from an atlas) and demonstrate the utility of transformed connectomes by applying these connectomes to predictive models based on a different atlas. We show that these transformed connectomes are significantly similar to theirgold-standard’’ counterparts and maintain individual differences in brain-behavior associations, demonstrating both the validity of our approach and its utility in downstream analyses. Overall, our approach is a promising avenue to increase the generalization of connectome-based results across different atlases.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_28

SharedIt: https://rdcu.be/cyl4b

Link to the code repository

https://github.com/dadashkarimi/functional-optimal-transport

Link to the dataset(s)

https://github.com/dadashkarimi/functional-optimal-transport

Reviews

Review #1

Please describe the contribution of the paper

This work introduces a novel method map functional connectomes by using optimal transportation map to register the underlying atlas. The mapping is then used to transform time series from one atlas to another in order to reconstruct a connectome. The method is validated by comparing transformed connectomes against their gold-standard counterparts. The transformed connectomes are very similar to their gold-standard counterparts and maintain individual differences in brain-behavior associations. The approach is a promising avenue to increase the generalization of connectome-based results across different atlases.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength is to use optimal transportation to map the atlases and transform the time series from one atlas to the other, and map the functional connectomes in this way. This is novel and solid, effective and useful.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weakness is the choice of optimal transportation algorithm. The sinkhorn algorithm intrinsically have approximation error. It will be helpful to control the error expliciity.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The work can be reproduced. The main algorithm is based on public available Sinkhorn python library.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The manuscript is well written, some details can be further explained to help readers understand the ideas:
1. For the formulation of the problem, the probability measures are defined on the tempal-spacial domain, so the mapping seems to be defined between the time-space domains as well. But in fact, the mapping is between the atlas only, independent of time. This part is very different from the conventional setting, and need emphasis.
2. The Sinkhorn algorithm, although very popular, has intrinsic errors, it will be helpful to explicitly control the error and compare the difference of the final results. Alternative algorithms may be considered as well.
3. The cost function in the current work is L2 Euclidean distance, in this situation, the Kantorovich transportation plan is reduced to the Monge’s transportation map. This means the transportation plan T highly sparse, therefore alternative algorithm for finding the map directly may be more efficient and save much storage.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea of modeling the time series as probability measures and using optimal transportation map to register two atlases and transforming connectomes to the same atlas for comparison is very novel and solid, it has great potential for down-stream applications.

The work can be further improved by choosing more appropriate algorithms for computing the optimal transportation map. Since the transportation cost is L2 distance, the general transportation plan will waste most of storage.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The paper introduces optimal transport techniques to building a map between functional connections from different atlases. The functional maps trained from the training dataset are then applied to the validation data. The results show that the images of the original functional data out of one atlas after the mapping are highly correlated to the same data out of another atlas.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Will fill after rebuttal.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Will fill after rebuttal.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors didn’t evaluate the effect of the strength of the entropy regularization but that is a minor issue. Other than that, I think the checklist is consistent with the paper and the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The motivation of using OT to build the map is not clear. As I understand the paper, any functional maps could work. Where did OT come from? The authors didn’t put their contributions in the perspective of existing works.

Section 2 is not clear. The authors made it difficult for readers to understand optimal transport in the context of functional signals.

– 2.1 Monge problem is redundant. It doesn’t serve the story of the paper. The authors never used it. So is the equivalency of Monge and Kantorovich. Given the rest of the content of the paper, directly starting with Kantovorich is sufficient.

– 2.2 vectorizing T is redundant. vectorized T and A do not simplify the problem or introduce any new information. Equation (3) is universal and I think fits the data even better since the functional signals come as matrices.

– OT requires that both distributions (\mu and \nu) have the same total mass. Do functional connectome data in this study satisfy that? If not, how do we normalize the data? If we can’t, then unbalanced OT?

– “The mapping T represents the optimal way of transforming the brain activity”. The authors should specify the criteria for the optimality. What does optimal mean in this paper? The optimality totally depends on cost matrix. It only becomes more clear when the authors defined the cost matrix later as either the Euclidean distance or the functional distance. Since the nodes are usually defined near the cortex, why not to use geodesic distances, considering the sulci?

– Entropy regularization should be in 2.1 as part of the history of OT. In fact, the method in this paper is not new at all.

– “For bigger frames, we use the mapping we learned at the beginning of a frame for the rest time points in the window”

I don’t follow this sentence.

– “Next, we average all T_i over all … to produce a single Optimal mapping T … T = \frac{1}{|S|}\sum_{I=1}^{|S|}T_i” The authors can consider computing a Wasserstein barycenter instead of a brute Euclidean average since the paper is about OT.

Section 3, the results are not convincing.

– Probably because the motivation of the paper is not clear, the authors also didn’t compare the results from OT with others. They only evaluated two variants of their method, i.e. Euclidean vs functional distances.

– [minor] Results of null seem redundant. It doesn’t provide any information.

– [minor] The paper doesn’t include the evaluation of the impact of the strength of the entropic regularization.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea of using OT for mapping functional signals across atlases seems new. However, the motivation of using OT is not clear; the proposed method simply reiterate existing techniques; the results are not convincing.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This manuscript tackles the problem of transforming functional connectomes between different atlases (without relying on additional preprocessing of the original functional data) by finding the optimal mapping between atlases. This mapping is found by solving a modified version of Monge–Kantorovich transportation problem, with additional entropy regularization to reduce computational cost, which is available as the Sinkhorn algorithm provided in the Python Optimal Transport (POT) toolbox. Performance of the optimal transport algorithm and the predictiveness of the transformed connectomes are evaluated using HCP database where the Shen-268-atlas-based and Shen-368-atlas-bsed data are available.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This is a novel application of the optimal transport method to the problem of transforming functional connectomes across atlases. The potential impact of this type of methods is significant. With the availability of functional connectome databases likc HCP, it is promising in the future that one can train these mappings and make it available for commonly used atlases. This potentially allows better interpretability of results that are from different atlases.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Current performance of the algorithm is not enough for immediate application. For example, the correlation of the transformed values and the gold-standard values is generally within the range of 0.4~0.65.

Some results raise questions and call for potential improvement of the algorithm. For example, the current algorithm fits mappings for each time point independently, although temporal correlations usually exist in functional time series. This also possibly leads to the fact that prediction accuracy cannot be further increased by including more time points (Table 1).

Figure 3b also demonstrates a scenario where the transformed values generate better prediction accuracy than the gold-standard ones. But this scenario (368->268) creates less optimal results compared to (268->368). Conceptually, one might expect the downsampling might be an easier problem because of more available input information, but the performance contradicts this. One would worry to what extent the method can be reliably applied in real world practices.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Open source code and data source are available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It might be helpful to generalize the loss function to consider longitudinal data, and consider graphical neural networks to train the mapping between atlases.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

My recommendation is mainly based on the potential significance of the work. I cannot strongly recommend due to the currently limited performance of the algorithm.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed to apply optimal transport to study the problem of transforming functional connectomes between different atlas). The idea is novel. The work was well presented. It is a good example to adopt advanced mathematics research in medical image analysis.

One concern is that the current performance is still relatively low. Some algorithm design and presentation issues should be addressed during the rebuttal period.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Author Feedback

The reviewers found our work, using optimal transport (OT) to map connectomes generated from different atlases, to be novel, well presented, and of interest to the MICCAI community. As noted by the meta-reviewer, we address concerns related to algorithm performance, algorithm design, and presentation. We will incorporate these relatively minor clarifications into the main text for future versions.

Performance: Both R#3 and R#4 indicate that the current performance levels limited real-world use. We argue that our prediction results (e.g., extrinsic evaluation) show otherwise. At the end of the day, the real-world use of our approach is the application of transformed connectomes in downstream analyses, rather than perfectly transforming a connectome. Indeed, connectomes are noisy. As reconstructed connectomes can predict IQ in novel subjects, this shows that we are recovering the important aspects of the connectome for downstream analyses, rather than overfitting to noise or other irrelevant features. Higher correlation between the original and transformed connectomes is a goal of future work, but we show that this is not needed to establish meaningful brain-behavior associations with transformed connectomes.

This point is also shown in R#4’s question about why 268->368 transform does better than the 368->268 transform. The 268->368 only does better in the intrinsic evaluation (i.e., correlation with the gold-standard). The 368->268 does better in the extrinsic evaluation (i.e., prediction of IQ). In fact, in line with R#4’s hypothesis that downsampling is easier, the 368->268 transformed connectome (using functional distance) predicts IQ numerically better than the original 268 connectomes. Again, these results highlight that the extrinsic evaluations show significant promise for real-world use.

Design: R#2 notes that the Sinkhorn algorithm has an approximation error. This approximation does not affect our prediction results (Linear programming without approximation:0.13820.05; Sinkhorn: 0.13760.05), but does provide the most computationally feasible solution (see the paragraph before Eq. 6). R#2 incorrectly notes that the L2 distance reduces the Kantorovich problem to the Monge problem. The Kantorovich problem potentially reduces to the Monge problem when there is a one-to-one mapping between the elements to be transported. As the atlases have a different number of nodes, our problem remains the Kantorovich problem regardless of the distance used. This is echoed by R#4, who suggests we remove the description of the Monge problem. Though, we will keep the Monge description to help clarify the differences between the two problems.

Presentation: Both R#2 and R#4 note confusion over the spatial-temporal mapping. We clarify that our approach is in the spatial domain (i.e., mapping node-to-node), rather than mapping timecourse-to-timecourse or even connectome-to-connectome. Once we have a node-to-node mapping, timecourses (and resulting connectomes) naturally come for free. While our end goal is to generate the transformed connectomes, we chose the node-to-node approach as it is more general. Our method could be used to map other neuroimaging results, such as cortical thickness in the Desikan-Killiany Atlas to the Destrieux Atlas.

R3# asks if the connectome data have the same total mass. We clarify that, in our formulation (which is a spatial node-to-node mapping), the activity at each time point needs to have the same total mass, rather than the connectomes. The distributions were normalized using min-max scaling and then dividing by sum.

R#3 asks why we used OT over other functional maps. OT is one of the few methods that provides a well-defined distance metric when the support of the distributions is different. Other mappings approaches such as KL divergence do not make this guarantee.

Last, our results are robust to the entropy regularization()with prediction performance of r’s=0.13-0.14 for ’s=0.001-0.5.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposed to apply optimal transport to study the problem of transforming functional connectomes between different atlas). The idea is novel. The work was well presented.

The weakness is that some technical details are not well described. The reviewers raised interesting questions about the formulation and algorithm. The authors made good efforts to clarify them in the rebuttal. The work is a good example to adopt advanced mathematics research in medical image analysis and will inspire new research along this line. Therefore, an “Accept” recommendation is made.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think this is a very interesting and well presented paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I still didn’t get convinced by the rebuttal on the low performance part. Therefore, I recommend reject.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

16

back to top

Data-driven mapping between functional connectomes using optimal transport