Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Li Chen, Wenjin Liu, Niranjan Balu, Mahmud Mossa-Basha, Thomas S. Hatsukami, Jenq-Neng Hwang, Chun Yuan

Abstract

Vessel tracing by modeling vascular structures in 3D medical images with centerlines and radii can provide useful information for vascular health. Ex-isting algorithms have been developed but there are certain persistent prob-lems such as incomplete or inaccurate vessel tracing, especially in complicat-ed vascular beds like the intracranial arteries. We propose here a deep learn-ing based open curve active contour model (DOST) to trace vessels in 3D images. Initial curves were proposed from a centerline segmentation neural network. Then data-driven machine knowledge was used to predict the stretching direction and vessel radius of the initial curve, while the active contour model (as human knowledge) maintained smoothness and intensity fitness of curves. Finally, considering the non-loop topology of most vascu-latures, individually traced vessels were connected into a tree topology by applying a minimum spanning tree algorithm on a global connection graph. We evaluated DOST on a Time-of-Flight (TOF) MRA intracranial artery da-taset and demonstrated its superior performance over existing segmentation-based and tracking-based vessel tracing methods. In addition, DOST showed strong adaptability on different imaging modalities (CTA, MR T1 SPACE) and vascular beds (coronary arteries).

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_56

SharedIt: https://rdcu.be/cyhWf

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The present paper describes a novel method for vessel tracing in 3D medical images. The method, denominated DOST, begins by applying a neural network to identify vessel curves used as initialization for an active contour model. The stretching direction of the active contour model and the radius of the vessel being traced are predicted by using a deep neural network. Additionally, the vessel curves are connected by creating a vascular tree, which is refined according to predefined criteria.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

In general, the paper is well written, and it presents innovative solutions to problems found in vessel tracing, such as the initialization of the active contour with curves instead of seeds; the evolution of the contour according to parameters predicted by neural networks; and the connection of individual results using a tree, refined according to some predefined criteria. The literature review is extensive and clearly identifies the main problems associated to vessel tracing. The main steps of the method are described in detail, and it is compared to other state-of-the-art methods for vessel tracing using datasets from different imaging modalities, reaching promising results. Finally, an ablation study is also included, to evaluate the contributions of each individual component of the proposed method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

One suggestion for improvement would be to discuss some of the limitations of the proposed method. For example, how much data is required to train the neural networks it requires and how easy is it to obtain that data. It would also be great if the authors could explain how they estimated the loss threshold of 0.05 and minimum distance of 10mm used to remove edges in the vascular tree, which are mentioned on the last paragraph of page 5.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is described in detail in the paper, even when no code is provided. The formulas used for the loss functions and evolution of the active contour are provided together with the formulas of the metrics used to evaluate the method.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Comments were already provided on section 4.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well organized and the method is clearly described. The contributions are novel and the method was evaluated in detail with comparison to other state-of-the-art methods and an ablation study.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper describes a method to extract vessels from 3D medical images, while taking topological and geometric properties of the vessel trees into account.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This method guarantees the topological properties of the vessel trees, i.e. vessels are continuous lines without gaps, and the vessel trees do not contain loops. Also, a geometric property of vessels (smoothness of the vessels) is taken into account.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Description of parameter sensitivity: How are the parameters alpha and beta for the open snake selected? Are they fixed for all images in the experiment, or do they need to be adjusted depending on the image?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Datasets are clearly described. One important step is not described in necessary detail.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- In Sec. 2.3: The stretching directions are determined using a deep neural work. But how is this network trained? How does the training data look like, i.e. were manually annotated D+1-dimensional (D=500) vectors used? This is a very significant step, but not mentioned in either “Methods” nor “Experimental settings and results”.
- In Sec. 2.3: The curves are stretched in a (D+1)-dimensional space, with spatial dimension D=500, and an additional dimension for radius. Is there any intuition about the 500 dimensions of the spatial stretching?
- In Table 1: The method “OCS” achieves the best values for three metrics, while the proposed method achieves the best values only for two metrics. What is then the advantage of the proposed method compared to “OCS”?
- In Table 2: Why does the results for OV and AI decrease after adding the global tree constraint? Examples should be provided to show this.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The main steps are illustrated well. Results are convincing (except the comparison with OCS). Ablation study clearly shows the effects of each components of the method.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors propose a method to segment vasculature in 3D images and to obtain the complete topology of the centerline vasculature tree. The first step is a regular encoder-decoder network to segment the vasculature followed by thresholding and skeletonization in order to obtain initial curves. The second step is the innovative part where the curve is fit and stretch using the deep open snake tracing (DOST: stretching direction is trained by a neural network). Then the final step links all the curves using minimum spanning tree. The methods is evaluated against state-ot-the-art methods on the MRA BRAVE dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The main strength is to propose a pipeline combining robustness of nnet segmentation and smoothness/continuity of snake curve.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The method is not fully clear especially the deep learning part during the snake tracking (what is exactly trained there (direction, stretching magnitude) and how the ground truth is computed). This is the innovative part so it should be more detailed. Also, I don’t fully understand the training masked by the non-zero regions with the segmentation network (see detail in constructive comments).
- Same for the new evaluation metrics, more details should be brought (figure, more rigorous formulation) because now it is difficult to understand exactly what is computed (see detail in constructive comments).
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Source code is made available and dataset can be asked on reasonable request
- For the experiments reproducibility is hampered by missing parameters and optimizers used (by reading only the paper): What are the parameters/optimizers used for the DOST minimization? What are the parameters/optimizers used for the two neural networks: image segmentation and stretching magnitude computation. How the latter neural network is trained (data selection, ground truth)?
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Page 1 Abstract
- “TOF MRA intracranial artery” -> please explain the acronym
Page 3
- Figure 2: I don’t understand the example with artificial vessels. Why after the Deep snake tracing, there is so long green parts, as it is outside the vasculature part (blue)?
- Figure 2: On the TOF MRA image, after minimum spanning tree, there are less snake curve than after Deep snake tracing. Why it is the case? Is there a mechanism to remove some curve too far from the ‘main’ graph?
Page 4
- “2 blocks in encoder/decoder, each with 2 3D convolutional layers+RELU followed by 3D max pooling/up sampling layers” -> I find it too vague. Why do you mean by 2 blocks? 2 downsampling/upsampling layers? Is there any sigmoid at the end? The output is the distance transform map, right?
- “The L2 loss for training the segmentation network is masked by the non-zero regions” -> So all the regions where there is no vasculature, do not influence the training? How is it possible that during inference, the network manages to put zero everywhere in those regions if no loss function trained it to do so? Is this because you use pre-trained network? If it is the case, it could be nice to mention it to clarify.
- Equation 4: Do you want to minimize this function?
- Equation 6: I is only the image intensity? of the raw image? or the distance transform map? So you want here to maximize bright intensity? Does it work as well for CT scan?
- “E_{str} is the energy for stretching directions at both ends of the snake” -> Why do you need stretching after the vasculature segmentation+skeletonization? To fill potential gap (false negative segmentation) between curves?
Page 5
- “6 blocks followed by a convolution layer with D + 1 dimensions (D = 500)” -> Do you mean “fully connected layer” instead of “convolution layer”?
- “for prediction of the 1-dimensional radius r(P(c(s)))” -> So the radius is computed only at the end of the snake? How do you compute the remaining radius along the curve?
- “and D-dimensional stretching direction” -> How the direction are encoded in the network? Why 500? How is made the stretching magnitudes ground truth for the training?
- “Background intensities for the snake pair were sampled from points at twice the radius” -> Why twice the radius? why not one or closer like 1.1?
Page 6
- Datasets: Training between each dataset are independent? Also, snake energy E_{DOST} ( in particular I(x(s),y(s),z(s)) ) is unchanged between MR and CT dataset? (as the vessel intensity is different bright vs black)
- Equation 10, 11, 12: It could be nice to have a figure to explain the metrics as it is difficult to grasp the idea. What do you mean by {t_j}, should it be T? the set of all snake curve? But after the minimum spanning tree, the snake curve does not mean anything anymore? I don’t understand what count and match ( min_ij( pt norm < r^ij) ?) mean w.r.t to curves.
- Evaluation on BRAVE: What are the parameters/optimizers used for the DOST minimization? What are the parameters/optimizers used for the two neural networks: image segmentation and stretching magnitude computation. How the latter neural network is trained (data selection, ground truth)?
- “For segmentation-based approaches, Frangi vesselness filter, U-Net and Deep Distance Transform (DDT) were selected” -> How was done the skeletonization and the creation of the tree topology after the segmentation?
Page 7
- “DOST showed higher performance than most methods.” -> When I look at Table 1, it is not clear what DOST brings compare to OCS.
- Figure 3: DOST snake curve are based on a segmentation method similar to U-net. So, why there is no noisy outliers as well on DOST? Do you filter the snake curve at some point in your method?
Page 8
- “Tracing results are shown in Fig. 4” -> Is there any manual part here to select the right artery? Same question for figure 5.
- “In addition, DOST, as an adaptive hybrid (segmentation and tracking-based) tracing method, is able to identify complete vascular trees from multiple vascular beds and modalities.” -> Would it work as well with the other methods?
- “DOST is not purely post processing/smoothing on CNN based tracking results. The snake constraint was applied during the trace stretching and the tracing was initiated from segmentation-based curve proposal instead of seed points.” -> This paragraph is not really clear to me. What do you want to discuss/emphasize here?
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The method is not clear/detailed enough
- The implementation of the experiments are not detailed enough (see reproducibility)
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents a new method for vessel tracking in 3D image volumes. The method uses deep learning to initialize the active contour model and predict the vessel radii. R1 comments that it is an innovative solution to vessel tracing and suggests mentioning the method’s weaknesses. R2 finds that the results are convincing and has some questions regarding parameter sensitivity and network training. R3 has some concerns about the paper’s clarity on the method and its implementation. After reading the comments raised by the reviewers, my recommendation is given in Q3.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

R1.1limitations

The requirement for labeled traces is a limitation. We used 117 training cases from BRAVE. However, we have also shown DOST works with 5/6 cases in CAT08/Harborview data. R1.2parameter for tree constraint

0.05 for loss threshold and 10mm for minimum distance were empirically chosen. Lower threshold and higher distance cause more false/noise connections, while the opposite leads to broken connections/miss tracings on real arteries. R2.1alpha beta

The alpha, beta control the membrane and thin-plate terms in snake. Larger beta/alpha makes snake more stiff and harder to adjust to sharp direction changes. They are fixed. R2.2How the stretching network trained? R3.1 deep learning during the tracking is unclear.

We followed the training in CNN tracker/ref13 for a fair comparison. The network takes a 3D patch and predicts 1 of the 500 directions (one-hot) for stretching and 1 magnitude for radius. Semi-automated traces (ref23) were used to identify the most matching direction for each patch. 500 was validated and chosen in ref13. Same training set as the segmentation. R2.3/R3.17OCS has better results?

As it was stated following table1, ground truth was generated by manually correcting traces from OCS, which inevitably causes smaller mean distance (AI) to OCS. OCS also has the problem of missing many real arteries although arteries are usually consistent (lower ID switch). R2.4OV/AI decrease after tree constraint.

After tree constraint, some broken gaps were connected. Some connected arteries might not be traced by the ground truth or they are noise branches, decreasing the overlap OV. R3.2mask in the segmentation?

There are two outputs for the segmentation network: distance map (continuous variable) and probability map (artery voxels are 1). The loss was the L2 loss calculated only in the predicted mask region. R3.3More details for evaluation metrics

Due to the many contents and page limit, we cannot describe them in more figures or formulas. However, more intuitive descriptions were added. T, same as {t_i}, meaning the set of all snakes. Even after minimum spanning tree, the data structure remains T (only difference: snakes are stretched further). Points (as the unit) on each snake were matched/compared with points on the ground truth for evaluation. R3.4experiments parameter

Both networks used Adam optimizer with learning rate of 0.001. R3.5Fig2

We did not draw weak artery signals in green parts. Updated. R3.6mechanism to remove traces? Why no noisy outliers? There is a mechanism to remove outlier traces from graph by 〖Loss〗_con and distance.
R3.7segmentation structure

2 down/upsampling layers in each block. Output: distance and probability map trained together. No pretrain. R3.8snake loss function

to minimize this function R3.9image intensity in snake energy

”I” is the distance map. No matter which modality, “I” is always bright for arteries. R3.10 Why stretching needed?

Segmentation+skeletonization is not robust. Curve proposals are partial broken, which are fixed in the deep snake framework. R3.11”fully connected layer” instead of “convolution layer”?

This is a fully convolutional network so that it can be used for images with arbitrary size. R3.12Radius Radius was estimated iteratively at each stretching points. R3.13Background signal

Signal was extracted at twice (not 1.1) the radius to avoid the transition region from foreground. R3.14skeletonization

All methods used the same skeletonization Ref19. R3.15any manual part in fig4/5 to select the right artery?

Yes, we followed the same selection criteria in the challenge from the given landmarks. Arteries were manually selected in fig5 so that they can be evaluated. For intracranial artery, selection was automated with largest connected components in the graph. R3.16confusion in discussion

to emphasize the difference of DOST with tracking based methods after smoothing.

back to top

Deep Open Snake Tracker for Vessel Tracing