Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Ran Gu, Jingyang Zhang, Rui Huang, Wenhui Lei, Guotai Wang, Shaoting Zhang

Abstract

Domain generalizable model is attracting increasing attention in medical image analysis since data is commonly acquired from different institutes with various imaging protocols and scanners. To tackle this challenging domain generalization problem, we propose a Domain Composition and Attention-based network (DCA-Net) to improve the ability of domain representation and generalization. First, we present a domain composition method that represents one certain domain by a linear combination of a set of basis representations (i.e., a representation bank). Second, a novel plug-and-play parallel domain preceptor is proposed to learn these basis representations and we introduce a divergence constraint function to encourage the basis representations are as divergent as possible. Then, a domain attention module is proposed to learn the linear combination coefficients of the basis representations. The result of liner combination is used to calibrate the feature maps of an input image, which enables the model to generalize to different and even unseen domains. We validate our method on public prostate MRI dataset acquired from six different institutions with apparent domain shift. Experimental results show that our proposed model can generalizes well on different and even unseen domains and it outperforms state-of-the-art methods on the multi-domain prostate segmentation task.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_23

SharedIt: https://rdcu.be/cyl36

Link to the code repository

https://github.com/HiLab-git/DCA-Net

Link to the dataset(s)

https://liuquande.github.io/SAML/

Reviews

Review #1

Please describe the contribution of the paper

This paper describes a domain generalizable model, aiming to train a single model that generalizes well to unseen domains without the needs for model retraining like many of the existing domain adaptation or transfer methods. The paper proposes a domain composition and attention block which transfers the given (could be from unseen domain) images as a linear combination of basis domain representations. This can be taken as a plug-in block in general image-to-image approaches such as UNet presented in this paper.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of transferring of given image to a linear combination of domain basis combinations (under the deep learning paradigm) is novel and interesting. The proposed DCA module seems possibly to be easily adapted to different networks or tasks.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

One crucial topic the paper does not mention is the selection and analysis of the basis. There are experiment in ablation study but it is very limited.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Overall the approach should be reproducible. The build blocks are standard group convolutions and tensor operations. The dataset is open-public. Besides, the authors agree to open source the code which I pretty much look forward to.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Overall this paper is well organized and presented. The idea of DCA is interesting. However, a bit more clarification is needed. Here are my comments,
1. Selection of a proper N (number of basis in the bank). In the numerical experiments, the authors simply tried different N’s and use the best. Is it practical? In the introduction the authors claim that ‘DA methods require repeated model re-training …from the target domain, which is time-consuming’. Suppose given a new image from a new target domain and the DCA does not work out of the box. To retrain the DCA, one has to include all the data from all the domain, and training for different configuration of N, is this more time consuming?
2. The design of the divergence constraint seems poor. It seems that it just forces the the p vectors to be uniformly distributed on the C dimensional spherical ball. Overall it seems there are no links that connect these basis and domain knowledge. What prevents them from being just a set of regular basis (e.g. PCA when N=C, ICA when N>C)?
3. In Table 1, the numerical results are very close to each other. This makes me start questioning whether it really needs the effort to use this proposed approach which may require quite some effort on finding a proper N (which could even be task dependent making it even less practical).
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall this paper is well presented. The idea of DCA seems to be novel. The performance has been demonstrated well in the numerical experiments both quantitively and qualitatively.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors propose a domain generalization network evaluated on 2D prostate MR images which outperforms the methods benchmarked against. The network learns a basis representation, via a parallel domain preceptor, to represent a domain. Then, a divergent constraint function is used to encourage the basis representations are as divergent as possible. The authors implement the module within a Unet architecture.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper presents a new module which is integrated within a Unet architecture for domain generalization.
- The work is evaluated on 2D images of prostate images and outperforms the other domain generalization methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The authors could site some relevant important domain generalization work in the literature.

(2) The paper is not always easy to follow. The explanation of the new modules introduced is not clear. Furthermore, the intuition about the kind of features learnt in the domain representation bank is not clear.

(3) With the help of a t-sne or PCA plot one could show how the learnt features are domain invariant.

(4) The study lacks statistical analysis on the two evaluation metrics: Dice and ASSD.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have checked most of the entires as yes.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) The paper could be more complete by citing recent meta-learning based approaches in the medical imaging literature: [A] Robust White Matter Hyperintensity Segmentation on Unseen Domain (https://arxiv.org/abs/2102.06650) [B] Domain Generalizer: A Few-shot Meta Learning Framework for Domain Generalization in Medical Imaging (https://arxiv.org/abs/2008.07724) [C] van Sonsbeek, Tom, and Veronika Cheplygina. “Predicting Scores of Medical Imaging Segmentation Methods with Meta-learning.” Interpretable and Annotation-Efficient Learning for Medical Image Computing. Springer, Cham, 2020. 242-253.

(2) The authors should show with the help of a t-sne or PCA plot how the learnt features are domain invariant.

(3) The authors should provide statistical results for the two metrics (Dice and ASD) used for evaluation. It is not clear whether the method is statistically significant.

(4) The paper is not always easy to follow. The explanation of the new modules introduced is not clear. The intuition about the kind of features learnt in the filter representation bank is not clear, and could be expanded upon.

Minor: There are a few typos as mentioned below: (1) Page 3 Methods: change “frist” to “first” (2) Page 2 Introduction change “more restricts for data” to “more restrictions for data” (3) Page 2 Introduction change “images form a” to “images from a”
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

(1) The explanation is not easy to follow. (2) Statistical results are not incorporated. (3) Interpretation of the learnt feature representation bank is not clear.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This manuscript proposed a domain generalization technique for cross-site prostate MRI segmentation. This is achieved by using a channel-attention-like network structure to modulate feature maps. The proposed method is evaluated on a public multi-site prostate MRI segmentation dataset, and is shown to slightly outperforms previous domain generalization techniques.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of using domain-specific coefficients to modulate feature maps is interesting, even though this is not the first time such an idea is proposed.

The paper is well-written, with detailed explanation of proposed network structure as well as the experiment setting.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The idea of learning/predicting domain-specific feature modulation coefficients for domain generalization in medical image segmentation is not novel. It has been previously well-studied by [1]. Even though implementation details might differ, the authors unfortunately failed to provide a strong argument/experimental results showing the technical novelty and/or significant performance gain against [1].

The authors claimed that DCA block is learning domain information. However the authors failed to provide any theoretical or intuitive argument to support this argument. Given that in Table 1 DCA-Net also yields higher performance on seen domains, the slight performance gain of DCA-Net on unseen domain can be alternatively simply explained by the fact that DCA-Net is just a stronger segmentation backbone in general, compared with plain UNets, due to the introduction of channel attention [2].

With two unaddressed issues mentioned above, although the reviewer likes the idea of using domain-depended dynamic networks for domain generalization, the reviewer cannot be fully convinced by the arguments and experiments in the manuscript.

[1] Wang et al., DoFE: Domain-Oriented Feature Embedding forGeneralizable Fundus Image Segmentationon Unseen Datasets, TMI’2020

[2] Taori et al., Measuring Robustness to Natural Distribution Shiftsin Image Classification, NeurIPS’2020
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors agreed to release code and the dataset is a well-known public dataset. The reviewer therefore has no major concerns over reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As mentioned in Part 4., given that a similar idea has been well-studied by [1], the authors are expected to clarify the technical novelty compared with [1], as well as significant performance gain compared with [1].

The reviewer also suggests the authors to be more careful when claiming something like “this modules learns domain information”, unless it is backed by a theoretical analysis or intuitive explanation. Otherwise such kind of claims can be misleading. Alternatively, it is completely fine for the authors to acknowledge such arguments are just assumptions (and later validate them through experimental results in Table 1).
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A similar idea has been well-studied by [1], the authors are therefore expected to clarify the technical novelty compared with [1].
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This work proposed a block to modulate feature maps so as to combine basis domain representations. Reviewers recognized the overall contribution and results. However, as pointed out reviewers, more details are needed for clarifying the novelty against a baseline, the selection of basis, and also discussion regarding the significance of the performance gain. Please note, the aim of rebuttal is to clarify misunderstandings / rationale behind method and experiment settings. Promise of extra experiments will not be considered.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We thank the reviewers for their positive and constructive comments. They highlighted that our method is “novel and interesting” (R2&4), and “outperforms the other domain generalization methods” (R3), and described our paper as “well-written with detailed explanation and experimental setting” (R4). Here we address the main points in their reviews.

Novelty against Wang et al. TMI 2020 [1] (R4). Our DCA has several important conceptual differences from Wang et al [1]. First, we learn N basis representations from M source domains, which is flexible as the basis representations and source domains do not have one-to-one correspondence. However, [1] extracts one domain prior knowledge from each of the K source domains, where they are limited by one-to-one correspondence. Second, we proposed PDP to learn the basis representations and used a divergence constraint to encourage the basis to be different from each other, which improves the domain composition power for generalization. However, [1] simply takes an average of features from one domain as its prior knowledge, which may lead to redundancy of prior vectors when two domains are similar and limit the representation power. Thirdly, DCA block is plug-and-play and used at different levels of the decoder, leading to cascaded feature calibration, while [1]’s method is only used at the bottleneck of a network.

Selection and analysis of the basis (R2&R3). The number of basis N is a hyper-parameter. We have analyzed the performance of setting N to different values. As shown in Table 1, considering 5 source domains, when N is too small (e.g., 4) the representation power is limited, and when N is very large (e.g., 16) the model parameter will increase a lot and it does not bring extra performance gain. After selecting the hyper-parameter N, our method does not require iterative training, and a single training process is enough. However, existing methods using meta-learning need to train the model several times after setting hyper-parameters, each time simulating a different combination of seen and unseen domains. Therefore, our method is easier and more efficient to train. To analyze the basis, we have plotted t-SNE in 6 domains for comparison: https://drive.google.com/file/d/1TGQXodTEBfZHhu_2VKFHDd-aEuo9PKKl/view?usp=sharing. The image shows that the learned basis representations are indeed different from each other, and L_div can further increase intra-cluster compactness and the inter-cluster distance, making the basis more distinctive from each other, especially for BIDMC, I2CVB, ISBI1.5 and UCL. Due to the limited space, we did not show the t-SNE in the current manuscript, but will include it in a future journal version.

Performance gain compared with SOTAs (R4). We aimed to segment prostate from multi-domain MR images and used a benchmark dataset with 6 domains to fairly compare our method with the SOTAs (DeepAll, BigAug, MASF, and SAML) on the same dataset. Table 1 shows that our DCA-Net significantly outperforms DeepAll, BigAug and MASF. In addition, compared with SAML, the average ASD of DCA-Net was reduced by 20% (from 1.58 to 1.29 mm). In addition, our network only needs to be trained once, while the meta-learning-based SAML needs to be trained multiple times, as explained above and described in page 2 of the manuscript. The training time for SAML and DCA-Net was 8.1h and 5.8h, respectively. Thus, our method was superior to existing methods. We also noticed the work of [1] as mentioned by the reviewer. We have clarified the technical difference as above. In addition, that method was designed for fundus image segmentation, and adapting it for prostate segmentation is non-trivial. Meanwhile, that paper was published very shortly before our submission, and it would be good to compare our method with this recent work in a journal version of our paper.

[1]. Wang et al., DoFE: Domain-Oriented Feature Embedding for Generalizable Fundus Image Segmentation Unseen Datasets, TMI’2020

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Reviewers recognized the formulation of the proposed method. Major questions include deeper analysis of the basis proposed, how to use them for unseen domain inference, and novelty over existing works. Rebuttal reasonably address these major concerns raised in the reviews. I think the method difference as compared with the closest alternative is sufficient, and therefore, I would suggest acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Overall, this is an interesting paper studying the domain generalization in medical image segmentation. The authors have addressed most of concerns in the rebuttal and a decision of accept is recommended.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a domain adaptation approach where general domain representation banks are learnt to compose features across domains. Validation is on prostate mri images.

One reviewer finds novelty in the linear combinations of domain basis.

A second reviewer questions whether resulting features are domain invariant.

A third reviewer finds the main lines similar to an existing approach and questions whether the segmentation results are better simply due to adding a channel attention.

The general consensus is a novelty of using linear basis to represent domains. The authors have clarified in a convincing way the main differences with a recent similar work. The proposed approach appears versatile and could be well received in the community. At least by one of the reviewer, eager to test their code.

For these reasons, Recommendation is toward Acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

back to top

Domain Composition and Attention for Unseen-Domain Generalizable Medical Image Segmentation