Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Xiao-Xin Li, Zhijie Chen, Xin-Jie Lou, Junwei Yang, Yong Chen, Dinggang Shen

Abstract

Abstract. Medical diagnosis benefits from multimodal Magnetic Resonance Imaging (MRI). However, multimodal MRI has an inherently slow acquisition process. For acceleration, recent studies explored using a fully-sampled side modality (fSM) as a guidance to reconstruct the fully-sampled query modalities (fQMs) from their undersampled k-space data via convolutional neural networks. However, even aided by fSM, the reconstruction of fQMs from highly undersampled QM data (uQM) is still suffering from aliasing artifacts. To enhance reconstruction quality, we suggest to fully use both uQM and fSM via a deep cascading network, which adopts an iterative Reconstruction-And-Refinement (iRAR) structure. The main limitation of the iRAR structure is that its intermediate reconstruction operators impede the feature flow across subnets and thus leads to short-term memory. We therefore propose two typical Peer-layer-wise Dense Connections (PDC), namely, inner PDC (iPDC) and end PDC (ePDC), to achieve long-term memory. Extensive experiments on different query modalities under different acceleration rates demonstrate that the deep cascading network equipped with iPDC and ePDC consistently outperforms the state-of-the-art methods and can preserve anatomical structure faithfully up to 12-fold acceleration.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_32

SharedIt: https://rdcu.be/cyhVf

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper approaches the problem of MRI image reconstruction by introducing a fully-sampled side modality. A network, called DCNwPDC, is proposed based on D5C5. PDC and kEL are designed to further improve the reconstruction performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The topic is interesting and clinically significant.
2. The idea of improving dense connection and combining kspace learning to iRAR structure is novel.
3. Detailed ablation study shows the effectiveness of proposed PDC and kEL.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Experiments were conducted on MSSEG dataset that is designed for lesion segmentation Challenge. While it is difficult to collect real clinical data, there are several publicly available MRI reconstruction datasets. For example, the fastMRI dataset provides a large amount of raw complex data instead of magnitude images. Using such an uncommon dataset is hard to convince other researchers.
2. From Table 1 and 2, the performance of proposed method is quite close to the D5C5 (baseline method). It remains unclear if the presented results are statistically significantly different. Statistical significance tests with a correction for multiple comparisons should be conducted.
3. Since authors compare different deep learning-based methods, it is better to show the number of trainable parameters in different methods. Demostrating that the reported improvement results from the network design is important.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors commit to releasing the code and pre-trained models.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Adding statistical significance tests for quantitative evaluation and ablation wiil make this paper more persuasive.
2. In Fig. 3, authors can provide error maps to show the difference, which will make it easier for readers to fellow.
3. The detail of undersampling opertation should be released. It is critical for the reproducibility of this paper.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The overall rating reflects that there are three major weaknesses stated in previous section. One is about experiment design and others are the concerns of experiment results.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors include dense layers to overcome the problem of short-term memory in peer-layer-wise dense connections. They use the modified architecture to demonstrate 12 fold acceleration.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The use of dense connections to solve the issue of short-term memory is intuitive and simple. The strategy to focus on multi-modal acceleration is relevant.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. A comparison of the number of parameters and the memory requirements of the proposed architecture, with the other methods is missing
2. Ablation studies to demonstrate the effect of the dense layers one at a time would be beneficial to understand the impact of each
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have committed to the reproducibility checklist.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The authors need to compare the memory requirement (number of parameters) with respect to the other methods to inform the reader about choices and trade-offs.
2. A representative undersampling mask for the 12X case needs to be shown so that it informs the acquisition strategy. Random sampling in k-space gives rise to its own set of artifacts not accounted for in this work.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The authors propose accelerating multi-modal MRI through peer-layer wise networks with the inclusion of dense layers. This inclusion does benefit the implementation and of interest to readers in the area of multi-modal MRI reconstruction. The missing connect between the proposed reconstruction method and its effect on acquisition strategy needs to be discussed.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposed a new deep cascading network for reconstruction in MRI.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea is novel, the method is solid, and the experiments are strong.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Some of details missing.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Some details missing.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. P3 top, why whether can enlarging receptive field could be the key difference between SDC and the proposed PDC? In PDC, which part is the cause of change of receptive field? A more detailed illustration is needed for supplementary Fig.1.
2. Below eq.5, “We experimentally ﬁnd that early activation by ReLU might lead to performance degradation.” Why?
3. The purpose/function of dense consistence layer (DC) is needed. Why is it needed after Convs?
4. Are Conv_{t,d} have the same structure at layer d? If so, how do they handle the different dimensions of input? For example, Conv_{1,2}’s input is Conv_{1,1}, and Conv_{T,2}’s input is concatenation of all Conv_{t, 1}, t\in[1,T] layers.
5. The paper claim the proposed method can help acceleration of 12 folds. In experiments multiple undersampling (1/4, 1/8, 1/12) are performed and compared. What about the results of further undersampling? Say 1/16, 1/24, etc?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea is novel with solid design and experiments for validation. Although some details missing, the whole paper is good.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

8
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper addresses the problem of how to reconstruct undersampled MRI data, when another fully-sampled side modality is available. The authors design a cascading deep network and adopts reconstruction-and-refinement structure. They further use dense connection for long-term memory. Reviewers are generally positive about this paper. There are challenges, especially from R2, upon the validation part. The authors need provide more details and stats, to address concerns from the reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We have read the review comments carefully. We appreciate the reviewers’ constructive suggestions. Our response is as follows.

Reviewer #2 suggested using the fastMRI-like dataset, as the MSSEG dataset used in our experiments only provides magnitude images rather than raw complex data. Surely, it’s better to use fastMRI if it provides multiple modalities, as our work is focused on multimodal MRI acceleration. However, fastMRI only provides DP modality. Another reason for choosing MSSEG is that the landmark works [21-22] for Guidance-Based MRI Reconstruction via deep learning conducted experiments on MSSEG. For fair comparison, we also chose MSSEG. We did not find publicly available multimodal MRI dataset containing raw k-space data. We acquired our own multimodal MRI dataset by scanning ten subjects’ brains. Experiments on this dataset also demonstrate the performance boost of our method. We can later release our results in the supplementary materials.

Reviewer #2 indicated that the statistical significance of the presented results is unclear, as the performance of our method is quite close to D5C5 (the baseline method). Actually, we also noticed the limited quantitative improvement of our method when preparing this paper. However, as shown in Fig. 3, our method can visually restore more details in the Regions of Interest (ROI) and has better clinical fidelity than D5C5. This means using the quantitative metrics (PSNR/SSIM) calculated within ROI for performance evaluation might be better than using those calculated from the whole image. When calculating PSNR/SSIM from the whole image, the effect of image areas containing low information, such as the background area and the areas with less texture variations, cannot be neglected, since the contents of these image areas are easy to be recovered for most methods. As such, the quantitative differences of the compared methods will be reduced and finally leading to higher p-values. On the other hand, when calculating PSNRs/SSIMs in ROIs, the p-values can be greatly decreased. For clarity, we will add quantitative evaluations in ROIs and perform statistical significance tests in our final paper.

Reviewers #2 and #3 both suggested us comparing the parameter scale of different methods, as it can be used to demonstrate that the reported improvements are mainly due to network design rather than increased number of parameters. Actually, we implicitly reported the relationship between the performance gains and the parameter scales of some compared networks in Table 1. E.g., ~+iSDC(Cat, 4) might lead to performance degeneration against ~ although ~+iSDC(Cat, 4) uses more parameters. By contrast, ~+iPDC(Cat, 2) leads to stable performance boost over ~, although ~+iPDC(Cat, 2) has fewer parameters than ~+iSDC(Cat, 4). This demonstrates the superiority of our network design. However, as suggested by the reviewers, explicitly comparing the parameter scales of the compared networks can highlight our contribution. We’ll provide such comparisons in our final paper.

Reviewers #2 and #3 also emphasized the importance of showing the undersampling mask and the error maps. We’ll show them in the supplementary materials due to space limitation. Reviewers #3 also suggested discussing the effect of our method on different acquisition strategies. However, as indicated at the end of our conclusion, exploring different sampling patterns is beyond the scope of this work and will be left as our future work.

Reviewer #4 gave a list of constructive suggestions for some details of our work. We’ll clarify them in our final paper. However, it’s worth noting that the key difference between SDC and the proposed PDC lies in their memory lengths rather than in their receptive fields, although receptive field is also an important factor. PDC has larger receptive field than SDC because PDC breaks the limitation of the receptive-field reset caused by the intermediate reconstructions of deep cascading networks.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper addresses the problem of how to reconstruct undersampled MRI data, when another fully-sampled side modality is available. The authors design a cascading deep network and adopts reconstruction-and-refinement structure. They further use dense connection for long-term memory. Reviewers are generally positive about this paper. R2 raised critical challenges, especially upon the validation part. The authors provided rebuttal to address those concerns. The paper should become better after incorporating revisions confirmed in rebuttal by the authors.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper works on cross-modality guided image reconstruction. The proposed network is based on deep cascading network, but with Peer-layer-wise Dense Connections (PDC) between iterative subnetworks. The proposed approach is evaluated to show improvement over baseline D5C5 and some other deep networks. Overall, the proposed network blocks iPDC and ePDC aim to increase the long-term connections among the subnets, however, seems to not focus on solving how to fuse full-sampled guidance image to guide the query under-sampled image reconstruction. Therefore, I think the novel contribution specifically designed for the cross-modality fusion setting is limited. As pointed by the reviewer, from the comparisons to the baseline method of D5C5, the accuracy improvement is limited, which may show that the designed blocks have limited improvements. Based on these reasons, my accurate scoring on this work is borderline reject.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

15

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper receives mixed and rather extreme reviews. R2 is very negative, citing concerns about the experimental setup (why not using fastMRI) and insignificant performance improvements. The rebuttal has mostly answered these two concerns. Also the request for explicitly comparing the number of parameters is provided in the rebuttal. Therefore, I decide to put less emphasis on R2’s comments and recommend acceptance.

It is recommended that the authors cite DuDoRNet, CVPR2002 in the final paper as it also covers multimodal MRI acceleration via deep network.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

back to top

Multimodal MRI Acceleration via Deep Cascading Networks with Peer-layer-wise Dense Connections