Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Bo Zhou, Chi Liu, James S. Duncan

Abstract

A large amount of manual segmentation is typically required to train a robust segmentation network so that it can segment objects of interest in a new imaging modality. The manual efforts can be alleviated if the manual segmentation in one imaging modality (e.g., CT) can be utilized to train a segmentation network in another imaging modality (e.g., CBCT/MRI/PET). In this work, we developed an anatomy-constrained contrastive synthetic segmentation network (AccSeg-Net) to train a segmentation network for a target imaging modality without using its ground-truth. Specifically, we proposed to use anatomy-constraint and patch contrastive learning to ensure the anatomy fidelity during the unsupervised adaptation, such that the segmentation network can be trained on the adapted image with correct anatomical structure/content. The training data for our AccSeg-Net consists of 1) imaging data paired with segmentation ground-truth in source modality, and 2) unpaired source and target modality imaging data. We demonstrated successful applications on CBCT, MRI, and PET imaging data, and showed superior segmentation performances as compared to previous methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87193-2_5

SharedIt: https://rdcu.be/cyhLx

Link to the code repository

https://github.com/bbbbbbzhou/AccSeg-Net

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper
1. The paper has presented a framework to learn unsupervised segmentation of one modality using modality where ground truth is present. Segmentation of large medical data is always challenging using Human evaluation. The AccSeg-Net Model tries to preserve anatomical structures while learning the labels in the training/testing phase.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper involved anatomical structures in the model while learning from modaltiy A to modality B.
2. Incorporated various Loss functions in the framework to take care various structural information in learning,
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The Domain A considered to always CT images in the experimental setup.
2. The images are 2D images. Will the method works same on 3D data?
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. Information about the dataset seems to be incorrect. Like, what is the size of CBCT images (3792 how these many are only considered), volume size of PET images.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The idea of paper is very good and need of an hour.
2. There are in total five loss functions are considered in this work. Involving so many functions in one task may introduce some errors.
3. The weights considered to be equal (= 1) in Eq (7). Why? Can we have some different weight vector to emphasis on particular loss function more as compared to other loss functions.
4. The contribution of work is limited as four functions are already existing in literature.
5. The better approach could be to show the effectiveness of the proposed loss function independently on various dataset. And then analysis can be presented by adding more loss functions.
6. Also, experimental set up should have different domain A, like CT only. It can be also tried out as domain A to be MRI and domain B to be CT.
7. Ref [14] seems to be more close paper to this work. The comparison with this must be address in the manuscript.
8. The author(s) should address more challenges faced during training time for such cross modality applications.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. The contribution of loss function is very limited and it’s exposure in the paper is also limited. Readers may fail to appreciate the power of loss function used in this work.
2. Dataset details are somewhat not clear in terms of any kind preprocessing and number of images used for experimental purpose.
3. Since, paper is more focused on learning domain B from domain A, But experiments are done with only one fixed domain A.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The paper discusses segmenting a liver from a target domain (e.g. MRI, PET) given training data for a different domain (CT) which has more ground truth segmentation. The authors use two subnetworks - one is a GAN and the other is a segmenter where these two subnetworks use different losses like adversarial, segmentation, contrastive and anatomy constrained losses. The paper shows some promising results.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Domain adaptation and segmentation done in one end-end system with multiple losses defined to help solve different parts of the problem.
2. Multiple target domains tested to test for effectiveness and some ablation studies discussed.
3. Paper is clear and well organized for the main part.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The work is tested mainly on the liver which typically covers a large fraction of the 3D volume. Testing on smaller organs would give more credibility in terms of whether the anatomy constraints are robust enough.
2. No ablation studies showing the impact of the different loss functions.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code will be shared and it will be possible to reproduce if all data is also available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The work is tested mainly on the liver which typically covers a large fraction of the 3D volume. Testing on smaller organs would give more credibility in terms of whether the anatomy constraints are robust enough.
2. No ablation studies showing the impact of the different loss functions. I believe the supplementary material is not exhaustive.
3. Assuming the source and target data is from different patients and the body can move between scans i.e. and scans between types are registered, it is hard to see why the anatomy-constraint loss works as claimed. Why would organs be in the same locations if not registered. How do you match the source domain image to the target domain image in the system ?
4. Fig.1, should the idt loss subimage be I_B to I_B as in eqn 2 ?
5. In the later part of the paper, it is mentioned that we have training instances of the target domain. Is this only used for prior work models or do you use it to train the 3 subnetworks. In that is so, Fig. 2 does not seem to show that clearly.
6. What is the use of Eqn 6? It does not seem to be used or explained.
7. Please bold best entries in Table 2. Also MIND and CC are not explained in the text.
8. Though the paper is mostly about segmentation when ground truth is not available, a lot of the paper discusses when there is training data for the target domain and very little on the PET data case. If the accuracy in PET can’t be verified, it should probably not be mentioned.
minor : two-step process may “be” prone Before section (2), it is mentioned 3 subnetworks, while in section 2, mentioned 2 subnetworks and then 3 later on.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper suggests a novel framework of domain adaptation and segmentation of organs.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

The paper addresses the task of cross-modality segmentation with CT source domain and CBCT, MRI, and PET target domains. The paper overcomes the lack of ground truth segmentation labels in the target domains using contrastive learning and an “Anatomy-constrained” loss while reducing the model’s size.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors claim that other models on the same task have “[…] heavy design that relies on training five different networks simultaneously […]”. I believe that statement is true since the most notable example of these models is TD-GAN, which uses several large networks to ensure cycle consistency on the source and target image. The authors overcome the stated limitation by using contrastive learning and what they call identity loss.

The experimental framework is well-designed and uses well-established metrics to evaluate the performance of medical image segmentation. The experimental framework is well-designed and uses well-established metrics to evaluate the performance of medical image segmentation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

My main concern is that the contrastive learning and the anatomy-constraint loss functions do not present a novel contribution. There is related work (some cited by the authors) in which a similar formulation has been applied to the same task (but a different source and target domains). One example is [14], which the authors cited when discussing the MIND loss but did not include it in the comparative study.

In that sense, the paper needs a better explanation of how it is different from other approaches and extended experimentation to show the model’s advantages over related work.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I believe that the paper’s results can be reproduced from what has been specified in the paper. It is clear where the input data comes from, the training recipe is in the supplementary materials, and the losses are all well-defined in section 2. More details about the architecture of the network G are required, but it is also clear that the authors are planning to release the code.

One exception, however, is that it is not clear how the patches for the PCT loss are extracted.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I believe the paper is clear at communicating the main idea, but it can be improved. Here are some examples I think can help:

1) The last paragraph of section 1 says, “We propose to use … while only using three sub-networks”, but in the first paragraph of section 2, it says that AccSeg-Net consists of two subnetworks. Later in section 2, it mentions three networks again.

2) Section 2 mentions segmenter M_B, which does not appear in Figure 1. A similar situation happens with S_B.

3) The identity loss is described in the paper with I_B, but it is depicted with I_A in Figure 1. I assume these are just typos on the notation, but they must be corrected or clarified. I would also suggest improving the caption of figure 1, so it is self-explanatory.

4) It is common in the state-of-the-art to use cross-entropy as segmentation loss. However, the authors decided to use the DSC as a segmentation loss. This needs to be adequately justified in the paper.

5) The text where the PCT loss is introduced does not describe how the patches are being computed. Are they random? Do they come in the dataset?
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper needs a more clear explanation of how it is different from other approaches of cross-modality segmentation.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths: *2. Incorporated various Loss functions in the framework to take care various structural information in learning,
1. Domain adaptation and segmentation done in one end-end system with multiple losses defined to help solve different parts of the problem.
2. Multiple target domains tested to test for effectiveness and some ablation studies discussed.
3. Paper is clear and well organized for the main part. *
Weaknesses:
- Incorporated various Loss functions in the framework to take care various structural information in learning,
- lack of ablation studies
- concern that the contrastive learning and the anatomy-constraint loss functions do not present a novel contribution
Overall:
- concerns over novelty
- concerns over lack of ablation studies
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

R1&R2 pointed out that we only consider CT as domain A and applied to different domains, and we should try on different modality applications/organs. We would like to clarify that our method is not limited to using CT as the source domain, and can be adapted to tasks with different source domains, such as MRI and other organs. In this work, since we only have CBCT and MRI liver ground-truth to objectively evaluate our results, we demonstrated the successful applications on them. We welcome the reader to try our method on different applications, as we are releasing our code on Github.

R1 asked if our method will work on 3D data. While we considered 2D data in our experiments, the framework can be easily extended to 3D data by substituting the sub-networks to the 3D version. Since our method is an open framework, the network component could also be substituted and may lead to better performance. In fact, we performed an ablation study (Table 1 in Supplementary) showing that different segmentation networks can produce reasonable segmentations.

R1 pointed out that our loss contribution is limited. We respectfully argue that, even though the individual loss is not completely new, the novelty of our work is to build a relatively lightweight network for obtaining a segmentation network without target domain ground truth. The method to properly incorporate these different losses into our network design is also one of the main contributions. On the other hand, we believe demonstrating the successful application of our new network in different target modalities, i.e. CBCT/MRI/PET, is also an important contribution.

R1&R3 pointed out that we should compare with [14]. The work in [14] proposed to add MIND loss in a cycleGAN to help MRI-to-CT synthesis, and the task is completely different from ours (segmentation without ground-truth). Thus, we didn’t compare with [14]. In fact, in Table 2, we performed an ablation study on AccSeg-Net with/without MIND loss, which demonstrated the impact of MIND loss proposed in [14].

R2 suggested there are no ablation studies on the loss functions. We want to clarify that ablation studies on the loss functions are shown in Table 1, Table 2, and Figure 2. Specifically, CC loss, MIND loss, and PCT loss are the three key losses in our method. Table 1 shows AccSeg-Net with/without PCT loss. Table 2 and Figure 2 show AccSeg-Net without or with different combinations of correlation coefficient loss and MIND loss.

R1 asked if we can adjust the weights in Eq 7. We empirically chose our weights in Eq 7 and found it is able to achieve a balanced training with the best segmentation results produced. However, we believe the weights in the objective function may need to be adjusted for different modality and organ applications.

R2 suggested that we should remove the PET results. While we do not have the ground-truth PET segmentation for quantitative evaluation, we believe showing qualitative PET results can also help the reader to appreciate the potential application of PET. We will include additional PET data details in the dataset preparation section.

R3 asked why DSC loss is used instead of CE loss. In our experiments, we consider the liver segmentation task, thus DSC loss is used. However, CE loss and other segmentation loss could be used in our AccSeg-Net. In fact, one could also use CE loss + DSC loss for potential multi-organ segmentation applications of our AccSeg-Net.

R3 asked how patches in PCT loss are selected. The patches are randomly cropped during the training.

R2 asked if unpaired training instances of the target domain are used in Fig 2. The unpaired training instances of the target domain are used by the discriminator for adversarial loss, while they are not used by the PCT loss.

In our final version, we will fix the notation errors, such as in Fig 1 & Eq 6. We will bold the best entries in Table 2 and clarify the CC (correlation coefficient) and MIND notations in our main text.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The rebuttal addressed concerns of novelty. The innovation is moderate, but the results appear effective.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

16

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper went into the rebuttal phase with relatively strong scores, with reviewers pointing out limited novelty over related work, some shortcomings in the performed experiments (no small organs, no ablation studies, only used CT as source domain), and some issues with clarity. Most of those points have been answered adequately in the rebuttal. Regarding the largest concern, the incremental novelty, the authors argue that the novelty lies not in the loss function but the combination of all components into a unsupervised DA framework. Two reviewers suggested comparison with Ref. [14] who proposed the MIND loss for image synthesis. I agree with the authors position that [14] is sufficiently different in motivation to not substantially detract from the novelty and does not warrant a direct comparison (although such a comparison could be construed in an extended version). One point where I do not follow the authors argument is why using MR as source domain would not have been possible as pointed out by R1&R2. While no GT was available for PET, switching the positions of MR and CT should have been possible. Nevertheless, the positive points overweigh for this paper, and I recommend acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

the novelty or contribution to domain adpatation from this work is limited, and no sufficient evidence in the experiment was provided to validate that their method could set the state of the art.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

14

back to top

Anatomy-Constrained Contrastive Learning for Synthetic Segmentation without Ground-truth