Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Ce Wang, Haimiao Zhang, Qian Li, Kun Shang, Yuanyuan Lyu, Bin Dong, S. Kevin Zhou

Abstract

Computed tomography (CT) reconstruction from X-ray projections acquired within a limited angle range is challenging, especially when the angle range is extremely small. Both analytical and iterative models need more projections for effective modeling. Deep learning methods have gained prevalence due to their excellent reconstruction performances, but such success is mainly limited within the same dataset and does not generalize across datasets with different distributions. Hereby we propose ExtraPolationNetwork for limited-angle CT reconstruction via the introduction of a sinogram extrapolation module, which is theoretically justified. The module complements extra sinogram information and boots model generalizability. Extensive experimental results show that our reconstruction model achieves state-of-the-art performance on NIH-AAPM dataset, similar to existing approaches. More importantly, we show that using such a sinogram extrapolation module significantly improves the generalization capability of the model on unseen datasets (e.g., COVID-19 and LIDC datasets) when compared to existing approaches.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_9

SharedIt: https://rdcu.be/cyhUA

Link to the code repository

https://github.com/mars11121/EPNet

Link to the dataset(s)

https://www.aapm.org/grandchallenge/lowdosect/

https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI

Reviews

Review #1

Please describe the contribution of the paper

The paper proposes a deep learning approach for limited view computed tomography. Their method is based on a model-based unrolled iterative scheme, the primary innovation in the proposed method lies in a sinogram extrapolation step, that aims to increase the information content of the sinogram to provide better reconstructions.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea to include a sinogram extrapolation network to improve reconstructions within a learned model-based reconstruction is a nice idea to improve reconstruction quality. Motivated by the success of “dual domain” or “primal-dual” approaches to image reconstruction that take the measurement space into consideration.

I also appreciate that the authors evaluate their method on 3 datasets, which gives a good insight into robustness and generalization capabilities.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The authors consider a challenging limited-view problem. Sadly, they have decided to use a parallel beam geometry (instead if fan/cone-beam), which simplifies the problem (as projections are more informative and due to symmetry only an angular range of 0-180 is sufficient). But such parallel beam geometries are typically not encountered in clinical practice, at least for the imaging task.

The authors state that the approach is more generalizable than other methods. Which is correct within the set of considered approaches, but one has to acknowledge that there is a heavy reduction in reconstruction quality (quantitative and qualitative). As such, I do not fully agree with the notation of being generalizable.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

In the current form, the precise architectures are not entirely evident from the description in the paper. As details, such as number of convolutional layers, layer width, etc. are missing. But the authors have indicated that codes will be released, so this can be overcome. I suggest that the author add a link to the repository to the paper and possible add a comment along the lines: For brevity further implementation details are omitted, for details we refer to the codes […]
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As mentioned above, my main concerns are with regards to the simplified measurement geometry. Which does reduce clinical applicability and relevance of the method. A change to fan beam geometry would have improved this. I suppose higher angular sampling will be needed in those cases.

Additionally, the claim that the method is generalizable can not be fully supported by the experiments. The performance for a change of datasets is indeed better than for the comparison methods, but is nowhere close to the performance on the dataset used for training. I encourage the authors to rethink the emphasis on generalizability. Nevertheless, the reconstructions obtained from the network are still better than with FBP. Maybe there is a possibility to use this as a starting point to improve the network’s performance with an update of parameters or advanced training procedures for adaptation? In my opinion this would be a better motivation than the aim for being fully generalizable.

Some minor comments: Page 1: FBP is in fact based on the assumption of full angular sampling and low angular counts violate that assumption, thus the decline in performance. Page 2: HQS-CQ could be explained when appeared first Page 2, bottom: The authors refer to a 1995 paper as recently. I would rate this as “Classically” or established approaches. Page 4: “Heavy U-net” I am not sure what this refers to. Is it a classic U-net or an adaptation?

Lastly, I would like to draw attention to an interesting and relevant study that investigates the possibility to extend the “visible range” and use the improved sinogram for reconstruction: Bubba, Tatiana A., et al. “Learning the invisible: a hybrid deep learning-shearlet framework for limited angle computed tomography.” Inverse Problems 35.6 (2019): 064002.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

My decision to borderline reject is primarily due to my two main concerns: A simplified geometry and that the underlying claim of generablizability can not be fully supported from the results.

Borderline: Because the idea is good at its core and it can probably be improved upon in the future, but it seems there is still a need for adjustment to new datasets needed.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The ExtraPolationNetwork for limited-angle CT reconstruction is proposed. It comes with a sinogram extrapolation module to improve the generalization power of the model. Experiments show that the model achieves good performance across datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Self-contained paper and easy to understand.
2. Thorough experiments in terms of ablation study, SOTA comparison and datasets.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The paper claims the model is for limited-angle CT while it seems the results in table2 shows that the relative performance is getting better when the number of angle s increase, which is conterpart to the statement.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

All datasets are public available and the author is committed to release the code, so the reproducibility is good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. A disscusion about the phenomenon that relative performance is getting better when the number of angle s increase will be good.
2. To further improve the generalization capability, one direction might be using online adaptation and the idea of meta-learing, reference: Finn, Chelsea, Pieter Abbeel, and Sergey Levine. “Model-agnostic meta-learning for fast adaptation of deep networks.” International Conference on Machine Learning. PMLR, 2017. Dou, Qi, et al. “Domain generalization via model-agnostic learning of semantic features.” arXiv preprint arXiv:1910.13580 (2019). Yu, Hanchao, et al. “Foal: Fast online adaptive learning for cardiac motion estimation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is overall good with thorough ablation study and comparison. Only concern is the results is not very strongly support the claim.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

An ExtraPolationNetwork for limited-angle CT image reconstruction has been proposed. Dual image and projection domains learning has been used. An Extrapolation Layer module has been introduced in the sinogram pipeline.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Good experimental results have been obtained for limited-angle CT image reconstruction. Proposed framework has a good model generalization capability on unseen datasets.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I don’t see any weaknesses of this paper.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think that reproducibility checklist for this paper has been properly fulfilled.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

An ExtraPolationNetwork for limited-angle CT image reconstruction has been proposed. Dual-domain Reconstruction Pipelines for image and sinograms have been utilized. An Extrapolation Layer to extrapolate sinograms has been proposed.

Paper is well written and god experimental results have been shown.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A novel and promising approach for the challenging limited-angle CT image reconstruction problem has been proposed.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper works on the limited-angle CT reconstruction by an unrolling methods and claimed better generalizability. The paper got scores of 5, 7, 9, and the reviewers are overall positive on the basic idea, however, have concerns on the clinical applicability in fan beam setting, and the experimental support on the generalization ability. The authors are invited for rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We sincerely thank AC and all reviewers for their time and efforts to give such useful and positive comments. We below clarify the two main concerns first, and then reply to other questions raised by reviewers. We will, however, revise the paper carefully to address all comments from reviewers.

Concern1: Clinical applicability is limited (AC).

We agree with R3 that clinical applicability is essential in limited-view problems. All of our numerical experiments are conducted with Fan-Beam Geometry as described in Section 4.1 (where we clarify that our sinograms are simulated similarly as MetaInvNet [ZhangTMI2020], whose experimental geometry is Fan-Beam Geometry, and the number of detector elements is set to 800). We will describe these experimental settings more clearly in the final version.

Concern2: The experimental support on the generalization ability (AC).

The notation of being generalizable is indeed a bit stretchy in the sense of model generalization. When comparing model performance on the “seen” data distribution (AAPM-test) and “unseen” data distributions (COVID-test and LIDC-test) in Table 2, our model still drops a lot, which is not as generalizable as HQS-CG [GemanTIP1995] and FBP. However, when comparing with deep-learning-based methods, such as MetaInvNet, our method narrows the gap between them and iterative methods. Therefore, after rethinking the notation of model generalization capability, we decide to modify our paper title to “Improving generalizability in Limited-Angle CT Reconstruction with Sinogram Extrapolation”, which truly reflects the fact.

Besides, another key point to clarify is the hybrid unrolling design of EPNet. We indeed keep the original iterative reconstruction process of HQS-CG, and search in dual-domain for a better initialization of the Conjugate-Gradient (CG) module. The design incorporates expert knowledge prior and data-driven prior, which is beneficial to improve generalization ability over deep-learning-based methods. We would further explore the designing of a more generalizable model with the advised work [BubbaInverseProblem2018] in the future.

Comment1: The assumption of full angular sampling FBP (R3).

As an analytical CT reconstruction method, FBP is always used as the initialization of most iterative and deep-learning-based algorithms. Although low angular counts violate the assumption in limited-angle scenarios, we compare our method with FBP and deep-learning-based methods initialized with FBP to show the improvement on “unseen” distributions, which confirms that our method has better generalization ability over deep-learning-based methods.

Comment2: HQS-CQ algorithm (R3).

We are sorry for the unclear introduction of the HQS-CG algorithm with limited space. We will strive to provide a better introduction with proper modifications in our final version.

Comment3: Heavy U-Net (R3).

The architecture is referred to as a similar network in MetaInvNet, which is a heavy-weight U-Net. To distinguish it from SENet with light-weight parameters, we therefore call it “heavy U-net”

Comment4: A discussion about relative performance in Table 2 (R5).

In Table 2, the relative performances between EPNet and MetaInvNet/HQS-CG are similar when ${\alpha}{max}=30, 60, 90$. We think the reason why the performance reduces when ${\alpha}{max}=15$ is that the extrapolation layer gets too limited sinogram information to give effective extrapolation, which reduces the final performance. We would explore to improve the extrapolation performance with the help of different designs, with which the reconstruction results would be potentially improved.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper works on the limited-angle CT reconstruction by an unrolling method and claimed better generalization. The reviewers have some concerns on the clinical applicability in fan beam setting, and the experimental support on the generalization ability. In the response, the authors clarified on the fan/beam setting. However, the claim on the generalization seems to be not fully justified, because it can be observed from Tab 2 that the proposed approach drops dramatically when extended to other datasets. It seems that the proposed approach (Extrapolation block) introduces moderately better generalization (refer to Tab. 2) than MetaInvNet_ori and MetaInvNet, and significantly worse generalization than HQS-CG. Moreover, the paper lacks solid evidence to show the effect of EPL on the generalization ability, e.g., the proposed network with and without EPL block. The comparisons on DudoNet with/without EPL shows marginal improvement on Covid and LIDC using EPL, and both of them drop a lot with unsatisfactory generalization. DudoNet with EPL achieves even lower results than DudoNet on AAPM. From these observations, I agree with the R3’s concern on the generalization ability, and suggests the authors to provide more solid justifications on the effectiveness of EPL (and also the reasons on its effectiveness), and tune down the claim of “generalizable” on this approach.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

14

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

All reviewers agree on the value of this work. The authors address a major concern of R1 on using fan-beams, by stating in their rebuttal that they already did so (although I agree with R1 that this was really not clear from the paper, and it should be explicitly stated in the final version).

This submission can generate interesting discussions at MICCAI, so I would very much welcome it.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Authors have addressed all major concerns raised by reviewers including the fan-beam applicability and generalizability. The paper is in good quality for acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

back to top

Improving Generalizability in Limited-Angle CT Reconstruction with Sinogram Extrapolation