Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Iram Shahzadi, Annika Lattermann, Annett Linge, Alexander Zwanenburg, Christian Baldus, Jan C. Peeken, Stephanie E. Combs, Michael Baumann, Mechthild Krause, Esther G. C. Troost, Steffen Löck

Abstract

Radiomics has shown great potential for outcome prognosis and presents a promising approach for improving personalized cancer treatment. In radiomic analyses, features of different complexity are extracted from clinical imaging datasets, which are correlated to the endpoints of interest using machine learning approaches. However, it is generally unclear if more complex features have a higher prognostic value and show a robust performance in external validation. Therefore, in this study, we developed and validated radiomic signatures for outcome prognosis after neoadjuvant radiochemotherapy in locally advanced rectal cancer (LARC) using computed tomography (CT) and T2-weighted magnetic resonance imaging (MRI) of two independent institutions (training/validation: 94/28 patients). For the prognosis of tumor response and freedom from distant metastases (FFDM), we used different imaging features extracted from the gross tumor volume: less complex morphological and first-order (MFO) features, more complex second-order texture (SOT) features, and both feature classes combined. Analyses were performed for both imaging modalities separately and combined. Performance was assessed by the area under the curve (AUC) and the concordance index (CI) for tumor response and FFDM, respectively. Overall, radiomic features showed prognostic value for both endpoints. Combining MFO and SOT features led to equal or higher performance in external validation compared to MFO and SOT features alone. The best results were observed after combining MRI and CT features (AUC=0.76, CI=0.65). In conclusion, promising biomarker signatures combining MRI and CT were developed for outcome prognosis in LARC. Further external validation is pending before potential clinical application. Keywords: Rectal Cancer, Tumor Response, Distant Metastases, Biomarkers

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_73

SharedIt: https://rdcu.be/cyl9k

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

Radiomics holds an important promise for outcome prognosis and improving personalized cancer treatment. The authors design a study to answer a research question, whether using higher-order features has higher prognostic value “Do we need complex image features to personalize treatment of patients with locally advanced rectal cancer?”. They consider MFO (first order) and SOT (second order), then add MFO+SOT, experiment with both CT, MRI and both, and finally include external validation to be able to gain more certainty on the results. The external validation ended up being an important factor, since results actually differed. Another important detail was that the authors used cross-validation and some statistical significance was included in the study as well.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

A solid protocol/study design, using exams and data obtained for the study from two sites is a very useful part. Also, I was able to notice that, in the details, the team dealt adequately with pressing issues, including how to deal with many features, feature selection and other details, cross validation and so on. In summary, the study seems to be well designed and implemented, and the results are relevant. The discussion also references and discusses other results from other researches.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

A more detailed distinction and specification of MFO and SOF would be useful.

The authors correctly indicated that the quality of the features is very dependent on the implementation of the actual feature extraction. The authors do not detail it, they apply a publicly available Python module MIRP [22], no further details. Still, it seems adequate to do this in order to explore the research question that was posed in this paper.

As the authors stated, it would be great to have a lot more data and there is some degree of variability due to the lack of more data.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

it is reproducible, although the actual dataset used is in-house, but that is because it is a clinically-backed study.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I do not have many comments, the work seems ok. You could try to add more information regarding the algorithms used to extract MFO and SOF.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The clarity of the formulation of the research question. The solid retrospective study work, also the fact that reading throughout the details of the experimental protocol, it seemed correct and interesting.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors investigate the use of radiomic features at different complexity levels derived from CT and MRI from locally advanced rectal cancer, with the clinical endpoints of response to nRCT and freedom from distant metastases. The authors conclude that using both simple and more complex imaging features from both modalities leads to a better performance. Furthermore, the study applies extensive robustness testing of the radiomics features used, and validates the results on data from a different center.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This study is very carefully performed, clearly described, evaluated following best practices and complemented by extensive supplementary material. The authors clearly outline prior work and motivate their experimental setup. The figures are very well prepared and give the reader a good overview. While the method is mainly an application of good practices in radiomics, the setup and rigor is in my opinion among the best in the radiomics literature. The authors clearly evaluate and discuss their results and candidly list limitations.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The novelty of the study is not very high.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility is excellent, the authors clearly list crucial information and cite relevant tools.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I would like to congratulate the authors for this solid radiomic study. The validation on an external dataset alongside the robustness testing elevates this contribution over many other publications in the field. The paper is clearly written, well-motivated, and the experiments thoroughly planned and selected to answer the question at hand. The scope of this contribution goes beyond the typical conference contribution, and should in my opinion be submitted to a journal (including a comparison to other robustness studies for other cancer applications and externally validated studies).
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I strongly recommend this contribution to be accepted, due to its clarity, experimental rigor, and very good execution. Since the novelty is not very high for a technical conference such as MICCAI, this contribution may be better suited for publication in a journal.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors have developed a method to predict responsiveness to neoadjuvant radiochemotherapy and the development of distant metastases from CT and MRI data of patients with locally advanced rectal cancer. This is expected to be another building block to further individualize cancer treatment. They extracted MFO features and SOT features from image data and calculated the performance and AUC of the models for CT, MRI, and combined data. The combined model yielded the best results
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This work has a strong clinical relevance, because especially in colorectal carcinoma the therapy could and should be much more individualized.
- The authors only use data for prediction which are always collected as standard in routine care. This is an important aspect, since there is usually no time for the additional collection of data not normally collected in everyday clinical practice.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Unfortunately, the authors could only work with retrospective data, which are usually somewhat unreliable and sometimes incomplete. The number of patients included is also relatively small. However, this is sufficient as a proof of concept
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

II’m actually not an expert in feature extraction and radiomics, but I understood the procedure and found it very comprehensible. It should therefore also be reproducible in my opinion
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
In fact, I just have a few questions about statistics:
1. Cox regression was used. This assumes that the hazard ratio is constant over time (therefore also called “proportional hazards regression”). This is the case as soon as the event risk (hazard) of group 2 is proportional to that of group 1 (assumption of proportional hazards). At each time point, the event risk (hazard) may be different, but the differences over time should be the same in both groups over time. This assumption is not always justified, but can be roughly assessed using the Kaplan-Meier curves. Has this been examined for the variables for which Cox regression was used?
2. the Chi^2 test and the Mann-Whitney-U test were also used. Was there a correction for multiple testing (e.g., Bonferroni or Benjamini-Hochberg) for all of these tests?
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The work is very relevant. The authors describe their approach very comprehensibly and accurately. The structure and of the paper and the language are very good and the paper reads very well as a result. The results are very promising.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors developed and validated biomarker signatures for outcome prognosis after neoadjuvant radio-chemotherapy in locally advanced rectal cancer (LARC) for improving personalized cancer treatment. The study used multimodal images, namely T2-w MRI and CT imaging and has been externally validated to establish certainty on the obtained results. In total, the study has a strong clinical relevance. The paper is well written and organized, experiments are well-setup and results are very promising. A few suggestions for the final version are to add short description of the SOT and MFO features, use high resolution images for Figures 1 and 2 and enlarge Fig.2. Moreover, what kind of second-order features were used? Rigid vs. non rigid registration choice for MRI-CT alignment is not well explained, so please clarify. Please list the rationale behind the specify choice of 0.8 ICC to remove features, is it experimentally, empirically, or observationally chosen (biased). More clarification about the statistical analysis as raised by R3 should be given.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

Reviewer #1 1) You could try to add more information regarding the algorithms used to extract MFO and SOF. Response: Due to the page limit of this proceeding we only referred to the literature for the specific definition of imaging features and for the algorithms used to compute features of the two classes. However, we now mention in section 2.2 that a publicly available Python module MIRP extracts features according to Image Biomarker Standardization Initiative (IBSI) and the definitions of the formulas used to calculate the features can be found in reference [19] of the manuscript. Reviewer #3: 1) Cox regression was used. This assumes that the hazard ratio is constant over time (therefore also called “proportional hazards regression”). This assumption is not always justified, but can be roughly assessed using the Kaplan-Meier curves. Has this been examined for the variables for which Cox regression was used? Response: For each covariate used for model building in Table S2, scaled Schoenfeld residuals were correlated with time, to test for independence between residuals and time. No significant test results were obtained, and we could not show that the proportional hazard assumption was violated. We now include relevant information in the manuscript. 2) The Chi^2 test and the Mann-Whitney-U test were also used. Was there a correction for multiple testing (e.g., Bonferroni or Benjamini-Hochberg) for all of these tests? Response: We used Chi^2 tests and Mann-Whitney-U tests to compare clinical variables between training and validation data. The p-values in Table S1 were presented without correction for multiple testing. The only statistically significant variable was radiotherapy dose (p<0.001). Applying Bonferroni correction for multiple testing with n=11 variables would not affect the significance of this test. We now mention in the heading of Table S1 that a multiple testing correction was not applied. Meta Reviewer #2: 1) Add short description of the SOT and MFO features. Response: In the introduction section (Page 2, Lines 29-34) we shortly described morphological, first order (FO) and second order texture (SOT) features. We have added reference [19] (Page 2, Line 34), which contains detailed information on all features, so that readers can refer to it for further information on different feature classes. 2) Use high resolution images for Figures 1 and 2 and enlarge Fig.2. Response: We will improve the resolution of Figure 1 and 2 and try to enlarge Figure 2 as proposed. 3) Moreover, what kind of second-order features were used? Response: A variety of second order texture features were extracted from the 3D ROI. We now added relevant information to the manuscript (Page 4, Lines 22- 26). 4) Rigid vs. non rigid registration choice for MRI-CT alignment is not well explained, so please clarify. Response: The main aim of image registration in this work was to transfer tumour contours from MR images to CT images. Since the registration was done using MR and CT data for the same patient and imaging was acquired within a limited amount of time before the start of treatment, anatomical changes were generally small. Thus, rigid registration was seen as sufficient for the task. 5) Please list the rationale behind the specify choice of 0.8 ICC to remove features, is it experimentally, empirically, or observationally chosen (biased). Response: In general, ICC values above 0.75 indicate a high resemblance among units in the same group (in our case features extracted from baseline and perturbed images) [1]. In a previous radiomic study, we used a threshold of 0.8 for the lower boundary of the 95% CI of the ICC for selecting robust features [2]. In the present study, we used the same approach. References: [1] Koo, T. K., et al., Journal of chiropractic medicine, 15(2), 155-163 (2016). [2] Zwanenburg, A., et al., Scientific reports 9(1), 1-10 (2020).

back to top

Do we need complex image features to personalize treatment of patients with locally advanced rectal cancer?