Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xinyi Wang, Tiange Xiang, Chaoyi Zhang, Yang Song, Dongnan Liu, Heng Huang, Weidong Cai
Abstract
The recurrent mechanism has recently been introduced into U-Net in various medical image segmentation tasks. Existing studies have focused on promoting network recursion via reusing building blocks. Although network parameters could be greatly saved, computational costs still increase inevitably in accordance with the pre-set iteration time. In this work, we study a multi-scale upgrade of a bi-directional skip connected network and then automatically discover an efficient architecture by a novel two-phase Neural Architecture Search (NAS) algorithm, namely BiX-NAS. Our proposed method reduces the network computational cost by sifting out ineffective multi-scale features at different levels and iterations. We evaluate BiX-NAS on two segmentation tasks using three different medical image datasets, and the experimental results show that our BiX-NAS searched architecture achieves the state-of-the-art performance with significantly lower computational cost.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87193-2_22
SharedIt: https://rdcu.be/cyhLO
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
-The paper represents a search for a computationally efficient bi-directional architecture with forward/backward skip connections. Analysis of the bottleneck of a predecessor. -Discovery of an efficient bi-directional U-net style architecture (called EX-NET) through architecture search (with EX-NAS), which achieves state-of-the-art performance with significantly lower computational cost. -Both EX-NET and EX-NAS are novel. -Application and evaluation of the found architecture on three different medical image datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-The proposed method is comparable to state-of-the-art network architectures in terms of accuracy but with considerably fewer parameters and computational costs. -This work takes on a recently published network called BiO-Net, which incorporates bi-directional structures into a U-Net. The authors identify flaws and investigate on possible improvements. -The proposed network EX-Net is more compact compared to its predecessors. Its sparsity may represent a simpler and more principled solution to a problem. At some extent this is demonstrated in Fig. 3, where fewer noise-like mistakes are made by the proposed algorithm. -The manuscript is well-structured and the derivation of the proposed search algorithm is explicative and convincing.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-Although the authors stated that the proposed method’s accuracy scores are much higher, I am not convinced of the superiority in terms of accuracy: The Dice scores are similar to the compared methods. In my point of view, the proposed method is on par with the others in terms of accuracy, and only surpasses the performance with respect to the more effective, lower number of trainable parameters and the shorter inference computation time. This is not a weakness per se, but I was not convinced looking at those accuracy scores.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors state that the project page (which I believe includes code) will be made publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
In general, I liked reading your work, but I have a few tiny remarks/questions:
- Is it clear in Definition 1, what is meant by an evolving architecture A^i, without clearly understanding Fig 2 and Alg 1? You might want to elaborate a bit in this definition. I was wondering how easy it is to identify “correspoding level-to-level skips across different architectures”.
- Why do you use both metrics, IoU and Dice? At the end of Section 2 you write about IoU and Macs, but in the tables you also introduce Dice, which you did not motivate in text.
- Is it really necessary to present IoU as well as Dice? The IoU=Jaccard Index has a 1-to-1 correspondence to the Dice coefficient and may only be interesting in this publication because of this correspondence’s non-linearity, which reflects in the different mean values. I still think it would be enough to show either or.
- You did not introduce mIoU (although I am aware that m probably stands for “mean”)
- You are using the abbreviation SE without introducing it.
- In the caption of Fig. 3 you could indicate the type of computational overheads with e.g. params (left), comp time (right)
- The bold printed numbers in Table 1 and 2 actually are not always the highest numbers from the columns. This may be misleading.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the work may represent an incremental contribution, I very much liked reading it: it has a compelling story line, the train of thoughts is convincing, and the analysis seems coherent/reasonable. I believe, the fact that the U-Net and its bi-directional variations can be compressed/sparsified in such a manner, may be very much interesting to the community.
Is there a typo in some numbers in Table 1? If I understood correctly, Figure 1 suggests, that BiO-Net++ should have many more trainable parameters compared to BiO-Net, yet we constantly have 0.43 for the last four entries in this column.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
2
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
Authors present the EX-NAS that is a novel, efficient, and two-phase NAS algorithms that searches for bi-directional skip connections between encoder and decoder by averaging (fusing) over multiscale features and bilinear resizing across inconsistent spatial dimensions. To achieve this, they construct an adaptive learnable selection matrix to search for optimal set of skip connections with the highest relaxation parameters followed by complementary searching scheme for finding an evolved sub-set skips for better network compactness in order to reduce computational cost. The proposed method has been properly validated against existing methods using three different segmentation datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Both selection and progressive evolutionary search schemes are novel, well designed, well presented, and supported with complementary analysis like skip fairness (Section 2.3).
- The proposed network achieves comparative results against existing networks but with less significant complexity. This is shown with thorough validations in Table 1&2 over three different datasets representing micro- and macro-organ segmentation tasks.
- The need for compact networks without compromising performance is very well appreciated especially for clinical applications where applications are still designed to be executed on premise and CUPs.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
No weakness.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
authors have mentioned “Our project page will be made available to foster any further research.” I also encourage authors to make the codes publicly available so the community can benefit.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
One minor comment. As authors have already pointed out “dense connections bring a mere marginal improvement…” Looking at Table 1, for example, the Bio-Net++ outperforms the rest but the corresponding performances have not been bolded. Although the overall evaluation is performed by including complexity but each section needs to be evaluated separately and then ultimately the Ex-Net can be bolded under Methods section by including both performance+complexity.
- Please state your overall opinion of the paper
strong accept (9)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Very well written paper, presenting a novel approach for an important topic supported with thorough validations. The topic is of high interest as the need for optimal networks, without compromising performance, is increasing rapidly and such method can foster implementing for example segmentation tasks in clinical environments.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
This paper proposes to use a hybrid-differentiable-evolution mechanism to learn a multi-scale fusion strategy, which improves the segmentation results from the BioNet on three public datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Present a new searching scheme by combining both differentiable searching methods and evolution mechanism to search how to aggregate multi-scale features. Evaluate on three different medical datasets and comparison with top methods
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1.It is interesting to search for how to aggregate multi-scale information for medical image segmentation, however, the novelty is basically while has been widely explored, including many that has not been discussed/cited/compared such as:
Ji, Y., Zhang, R., Li, Z., Ren, J., Zhang, S., & Luo, P. (2020, October). UXNet: Searching Multi-level Feature Aggregation for 3D Medical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 346-356). Springer, Cham.
He, Y., Yang, D., Roth, H., Zhao, C., & Xu, D. (2021). DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation. arXiv preprint arXiv:2103.15954.
Yu, Q., Yang, D., Roth, H., Bai, Y., Zhang, Y., Yuille, A. L., & Xu, D. (2020). C2fnas: Coarse-to-fine neural architecture search for 3d medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4126-4135).
Kim, S., Kim, I., Lim, S., Baek, W., Kim, C., Cho, H., … & Kim, T. (2019, October). Scalable neural architecture search for 3d medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 220-228). Springer, Cham.
Bae, W., Lee, S., Lee, Y., Park, B., Chung, M., & Jung, K. H. (2019, October). Resource optimized neural architecture search for 3D medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 228-236). Springer, Cham.
-
The search method consists of two phases: gradient-based (phase 1) and evolutionary algorithms (phase 2), whether it is worthwhile to add more complex multi-evolutionary algorithms to bring an average gain of 0.2% (see Table 2), and considering the total consumption of search and retraining models, such kind of auto ml method has significant limitations in practice.
-
The intro section is well written. However, I believe Method section could be highly improved, I found quite difficult to understand the details of the approach. Beside, the figures are not well assist the text to illustrate the detail of method.
-
The experiments are rather inadequate and lack details. For example, in Table 1, the original bionet has 14.99M. After adding dense hopping connections, why does BiO-Net++’s parameter decrease to 0.43M, which should be equal to or more than 14.99M. And why does EX-Net occupy the same parameter 0.43M? It is a subnet of BiO-Net++, and the parameter should be less than 0.43M? Meanwhile, ablation study is missing. 5.The author did not provide comparison experiments or other quantitative metrics to support the claim that the proposed search method is efficient than other method in computation.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Can be reproduced
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- There is much room for improvement in the method section and in the figure.
- limited novelty and not fully discussed with highly relevant papers. For example, UXNet in miccai2020, c2fnas in cvpr2020.
- The experimental setup is confusing and lacks analysis of the ablation study of each component (see the weakness section ).
- Please state your overall opinion of the paper
reject (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The novelty of idea.
- The wirting and organization of the paper.
- What is the ranking of this paper in your review stack?
5
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths: -The proposed network EX-Net is more compact compared to its predecessors. Its sparsity may represent a simpler and more principled solution to a problem. At some extent this is demonstrated in Fig. 3, where fewer noise-like mistakes are made by the proposed algorithm. -The manuscript is well-structured and the derivation of the proposed search algorithm is explicative and convincing.
- Both selection and progressive evolutionary search schemes are novel, well designed, well presented, and supported with complementary analysis *
Weaknesses:
- not convinced of the superiority in terms of accuracy
- clarification of novelty would be helpful
Overall: * * *
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
8
Author Feedback
We sincerely thank all reviewers and ACs for their time and positive remarks: novel, efficient with SOTA performance, interesting to the community.
Not convinced of the accuracy superiority: (AC, R1) Our EX-Net was searched on one dataset only (MoNuSeg) and designed for all 3 universal tasks. We will revise our claims: “for all metrics, EX-Net obtained higher scores than all SOTA NAS counterparts on all tasks (and achieved on-par results with our proposed BiO-Net++ with lower complexity)”. Two tailed paired t-tests demonstrate that the p-values between EX-Net and NAS counterparts are < 0.05 for both nuclei datasets and 0.1 for CHAOS dataset, yielding the significance of our method. We will add the results to the final version.
Novelty concerns: (AC, R4) Thanks for bringing up more recent works. We carefully inspected all mentioned literatures and there are 4 major differences in our method: (1) Our searching algorithm extends NAS studies upon the recurrent bidirectional paradigm, which ALL existing works are currently unable to deal with. Bidirectional connectivity triggers multiple encoding and decoding phases with an escalation of computational cost (BiO-Net), thus discovering such a light architecture is in high demand. (2) We designed a novel hierarchical searching method rather than any direct use of existing gradient-based (UXNet, DiNTS, SCNAS), reinforcement-based (RONASMIS), and one-shot (C2FNAS) strategies. (3) Our EX-NAS is the first to consider multi-scale feature aggregations in a resource-aware searching paradigm, while the single-scale methods proposed in SCNAS, RONASMIS, C2FNAS are not straightforward to be extended to the multi-scale settings. Our hierarchical search requires only 0.46 day with a single GPU that is much more efficient than C2FNAS (>5 days) and UXNET (>1 days). (4) Differing from the evolution strategy in C2FNAS that trains each architecture instance independently, our phase2 search speeds up such process via head network sharing. As a similar method to UXNet, MS-NAS (MICCAI 2020) also constructs a multi-scale search space, which was directly compared in Table 1 and our EX-Net surpasses MS-NAS on all metrics.
Parameter number in Table 1: (R1, R3, R4) We first reduced the computational complexity in BiO-Net by optimizing the feature fusion scheme and feature channel numbers at each level. Such optimizations are adopted in the design of both BiO-Net++ and EX-Net yielding only 0.43 M trainable parameters. Our Phase1 search algorithm provides a further 8.6% MACs reduction, and our EX-Net eventually achieves a 20% MACs reduction compared to BiO-Net++. EX-Net follows the recurrent bi-directional paradigm with repeated use of the same building blocks at different iterations. In our final searched architecture, there is no building block that has been skipped at all iterations (Suppl. Fig. 1), resulting in no reduction in the total network parameter number.
Saliency of our hierarchical search: (R4) We disagree that ‘such method has significant limitations in practice’. BiO-Net++ is already highly optimized and our EX-Net requires 20% fewer MACs than BiO-Net++ without compromising accuracy, which validates the effectiveness and efficiency of EX-NAS. Also, our idea that shrinks the search space for a more efficient search is insightful for other general auto ML applications in practice. Noteworthy, both R1:“compressed/sparsified in such a manner, may be very much interesting to the community” and R2: “compact networks without compromising performance is very well appreciated especially for clinical applications where applications are still designed to be executed on premise and CUPs” agree that our method provides a novel and practical solution to real-world tasks.
Unclearness in the definition, description, and presentation: (R1, R3, R4) We will revise the entire manuscript again for any unclearness and consider re-organizing all sections to be more comprehensible as the reviewers suggest.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors clarified the relevance of the baselines and the statistical assessment.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
15
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The idea presented in this work is novel. Thanks to the proposed network architecture search strategy, the authors are able to discover a more compact and efficient architecture. While, I agree with the reviewers that the gains in accuracy are not as outstanding as the authors claim, I consider that the value of this work lies mostly in the gained efficiency. However, the authors are advised to lower the claims regarding accuracy in a potential final version of this work.
One of the reviewers considered that the method was not novel enough and justified this statement with a large body of literature. The authors have provided a satisfactory answer to this point by clearly explaining which are the differences of the proposed method w.r.t. previous works.
I therefore recommend acceptance of this work to MICCAI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
8
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The context of network structure architecture search in this paper is interesting. Selection and progressive evolutionary search are novel. In the rebuttal, the author’s response to the concerns on the novelty and accuracy have been addressed well. The authors promised to soften the claim on the accuracy and have successfully clarified the differences from recent relevant works. They also clarified more details of the computational complexity.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
3