No ethical approval and informed consent were obtained for this retrospective study because all the patients' deidentified data included in this study have been publicly and freely available for scientific purposes.

The clinical, histopathologic, and imaging data of the patients were obtained from The Cancer Genome Atlas–Kidney Renal Clear Cell Carcinoma (TCGA-KIRC) database and The Cancer Imaging Archive [24–26]. Currently, the TCGA-KIRC database includes 537 patients with clear cell RCC. Nonetheless, only 267 patients' imaging data can be accessed using The Cancer Imaging Archive. The results shown here are, in whole or in part, based on the data generated by TCGA Research Network [27]. The authors of this study acknowledge that they previously used this database (TCGA-KIRC) a few times in different contexts [28, 29].

To extract the preoperative unenhanced CT images, all imaging datasets were independently reviewed by a radiologist (with 3 years of experience) and a radiology resident (with 2 years of training) to minimize the chance of omission. The study inclusion and exclusion criteria are presented in Figure 1 in detail. Patients with multiple tumors were excluded because of the uncertainty in the database about which tumor had high and low nuclear grade. Imaging sources with fewer than five imaging studies were excluded from the study to minimize the heterogeneity of the imaging protocol further. Only 81 patients with 81 clear cell RCCs (56 high and 25 low nuclear grade) met our eligibility criteria. Patients' demographics are presented in Table 1.

The reference standard for statistical classifications was the nuclear grade of the clear cell RCCs, as reported in TCGA-KIRC [24, 25]. The nuclear grades I and II were grouped as low grade, and grades III and IV were grouped as high grade.

The CT acquisition parameters were as follows: slice thickness, 5 mm; tube voltage, 120–140 kV; tube current, 95–575 mA; and pixel size, 0.562 × 0.562 mm² to 0.976 × 0.976 mm². The TCGA-KIRC database includes various data sources. Therefore, achieving a completely uniform imaging protocol was not possible, so that all images underwent some preprocessing steps to minimize possible consequences of the protocol heterogeneity.

To give a simplified overview to the readers, a general technical study pipeline is given in Figure 2.

Before texture feature extraction, all images underwent the following image processing steps: gray-level normalization using a ± 3σ technique (scale, 100) [12, 30], pixel resampling and rescaling using cubic B-spline interpolation (resultant pixel size, 1 × 1 mm²) [31], and gray-level discretization (bin-width, 0.3) [31].

The tumors were manually segmented from the unenhanced CT images using 3D Slicer software (version 4.8.1). The largest cross-sectional area of the tumors was selected for the tumor segmentation (Fig. 3). While the segmentation was being performed, available contrast-enhanced CT or contrast-enhanced MRI studies were also used in all cases to ascertain the tumor boundaries.

Texture features were extracted from original, filtered, and wavelet-transformed images using PyRadiomics software (version, 2.0.1.post26+g8cedd8f) [32]. A Laplacian of Gaussian (LoG) filter was used for image filtration with values of 2, 4, and 6 mm, which represent fine, medium, and coarse patterns, respectively. We generated the wavelet-based texture features using Py-Wavelet (version 0.5.2) [33]. The wavelet transformation is an image-processing technique to decompose images using high- and low-frequency band-pass filter combinations to obtain possible hidden and significant information from the images [34].

The extracted texture features were as follows: 72 first-order, 56 gray-level dependence matrix, 96 gray-level cooccurrence matrix, 64 gray-level run-length matrix, 64 gray-level size zone matrix, 20 neighboring gray-tone difference matrix, and 372 wavelet-based texture features using four wavelet-transformed images (high-high, high-low, low-high, and low-low frequency bands) along with the other six feature groups. The total number of the features extracted was 744 per lesion. Detailed definitions and mathematic formulas for these features have been described elsewhere [33, 35].

To evaluate the interobserver reproducibility of the texture features in the tumor segmentation process, two radiologists (with 3 years and 1 year of experience) independently segmented randomly selected 25 tumors regardless of their nuclear grade. The segmentations were done on the same image slice determined by the radiologist with 3 years of experience. Both radiologists were blind to the nuclear grade of the clear cell RCCs. Intra-class correlation coefficients (ICCs) were calculated for each texture feature using SPSS (version 20, IBM). Only the features with ICC ≥ 0.9, which indicates excellent reproducibility, were included in the further dimension reduction steps [36].

The level of collinearity among the features was assessed using Pearson correlation coefficient (r). The threshold for collinearity was r = 0.7 [37]. The features with high collinearity were excluded from the analysis. In case a feature pair had high collinearity, the one with the lowest collinearity with the other features remained in the analysis.

The feature selection for artificial neural network (ANN) was performed using the Waikato Environment for Knowledge Analysis toolkit (version 3.8.2, The University of Waikato) [38, 39]. For feature selection, a wrapper attribute evaluator along with an incremental wrapper-based subset search method was used [38, 39]. In the search method, the texture features were ranked by their probabilistic significance computed as a two-way function (attribute-class and class-attribute association) [40]. The feature selection was performed using a nested cross-validation approach with 10-fold inner and 10-fold outer loops. The measures used to evaluate the performance of the attribute combinations in the feature selection process were accuracy for discrete class and root mean squared error for numeric class.

The feature selection for binary logistic regression analysis was performed with univariate analysis using SPSS. The variables with p < 0.05 in the univariate analysis were included in the multivariate binary logistic regression.

In addition to quantitative texture features, age, sex, and maximum 3D tumor size were also included as clinical variables in the feature selection process for both ANN and binary logistic regression.

ML-based classifications were performed using the Waikato Environment for Knowledge Analysis toolkit. ANN and binary logistic regression were used for model development. For creating ANN, we used multilayer perceptron, which is also known as the simplest form of deep neural network. Because our data had an imbalance between classes, we also used the synthetic minority oversampling technique (SMOTE) [41]. To balance the number of low nuclear grade clear cell RCCs against high nuclear grade clear cell RCCs, we applied 124% of SMOTE to the low nuclear grade clear cell RCCs, resulting in 56 labeled cases for each class (low vs high nuclear grade clear cell RCCs). The SMOTE algorithm creates synthetic instances that are not exact replications [41]. Hence, the method increases the representation of the minority group, while preserving the structure of the actual data [42, 43]. Ten-fold cross-validation was performed for the validation of the models. The performance evaluation was done using the AUC. Accuracy, sensitivity, specificity, precision, F-measure (weighted-harmonic mean of precision and recall), and Matthews correlation coefficient were also calculated. For sensitivity, specificity, and precision, overall weighted scores were also calculated. Comparison of the 10-fold cross-validated models created with and without SMOTE was made using the Wilcoxon signed-rank test [44]. A two-tailed p < 0.05 indicated statistical significance.

Among 744 texture features, 653 had excellent reproducibility (ICC ≥ 0.9) in the analysis conducted by two radiologists. Therefore, these features were included in the further dimension reduction steps.

By excluding the features with high col-linearity, the number of features was further reduced to 47. These features were included in the algorithm-based feature selection process.

Using the wrapper-based classifier-specific algorithm, the number of the selected features decreased to five for the ANN classifier. None of the selected texture features was from the original image. Four of the selected features were from the LoG filtered–images. The remaining feature was originated from the wavelet-transformed image. The dominant feature classes were the gray-level cooccurrence matrix and gray-level dependence matrix. The selected features for ANN and their respective ICCs are presented in Table 2. Distribution of the normalized texture feature values in the actual data (without SMOTE) is shown in graphs and plots in Figures 4A and 4C. The distribution of the normalized texture feature values with SMOTE is shown in graphs and plots in Figures 4B and 4D. Col-linearity status of the selected features for ANN is given in a matrix in Figure 5.

For the binary logistic regression, the number of the selected features decreased to six after a univariate analysis. None of the selected texture features was from the original image. Five of the selected features were from the LoG filtered–images. The remaining feature originated from the wavelet-transformed image. The dominant feature class was the gray-level cooccurrence matrix. The selected features for the logistic regression and their respective ICCs are presented in Table 3. Distribution of the normalized texture feature values in the actual data (without SMOTE) is shown in graphs and plots in Figures S1A and S1C, which can be viewed in the AJR electronic supplement to this article (available at www.ajronline.org). The distribution of the normalized texture feature values with SMOTE is shown in graphs and plots in Figures S1B and S1D. Collinearity status of the selected features for logistic regression is given in a matrix in Figure S2, which can be viewed in the AJR electronic supplement to this article (available at www.ajronline.org).

After the feature selection, three features were common in the subsets selected for ANN and logistic regression. For both ANN and binary logistic regression, none of the clinical variables was selected as valuable for the model development.

The ANN algorithm alone correctly classified 81.5% of the clear cell RCCs regarding nuclear grade (low vs high), with an AUC value of 0.714. Overall weighted sensitivity, specificity, and precision were 81.5%, 65.2%, and 81.4%, respectively.

The ANN with SMOTE correctly classified 70.5% of the clear cell RCCs, with an AUC value of 0.702. Overall weighted sensitivity, specificity, and precision were 70.5%, 70.5%, and 71.1%, respectively.

The logistic regression algorithm alone correctly classified 75.3% of the clear cell RCCs, with an AUC value of 0.656. Overall weighted sensitivity, specificity, and precision were 75.3%, 53.5%, and 74.2%, respectively.

The logistic regression algorithm with SMOTE correctly classified 62.5% of the clear cell RCCs, with an AUC value of 0.666. Overall weighted sensitivity, specificity, and precision were 62.5%, 62.5%, and 62.5%, respectively. Detailed performance metrics are presented in Table 4.

To address the concerns regarding the data imbalance between classes and their potential influence on the classifications using ANN and logistic regression algorithm, no statistically significant difference was found between the model performances (10-fold cross-validated AUCs) created with actual data (without SMOTE) and those created with SMOTE (p > 0.05).

With or without SMOTE, the performance (10-fold cross-validated AUCs) of the ANN and binary logistic regression in predicting nuclear grade (low vs high) were statistically significantly different (p < 0.05). The ANN performed better than the logistic regression algorithm.

In reviewing the literature, no data were found regarding the use of unenhanced CT in the texture analysis of clear cell RCCs for predicting their nuclear grade. Hence, the current study was designed to determine the potential value of the method using an ML-based analysis using ANN and logistic regression. The results of this study indicate that the ML-based unenhanced CT texture analysis using ANN and logistic regression can correctly classify about three- to four-fifths of the clear cell RCCs regarding their nuclear grade (low vs high grade). Overall, the ANN performed better than the binary logistic regression algorithm.

To our best knowledge, there have been only two studies evaluating the potential value of the CT texture analysis in discriminating nuclear grade of the clear cell RCCs [13, 14]. The authors of both studies only used contrast-enhanced phases. Despite several technical and methodologic differences, both studies reported high predictive performance for the nuclear grade of the clear cell RCCs.

Apart from texture analysis, the prediction of nuclear grade has also been studied by rather conventional methods, such as apparent diffusion coefficient maps [45]. In a recent meta-analysis, the predictive performance of the technique was moderate, with a pooled sensitivity of 78%, specificity of 86%, and hierarchic summary ROC AUC of 0.850 [45].

The nuclear grade of RCCs is one of the most important prognostic factors associated with survival [46–48]. Percutaneous biopsy is a widely accepted method to predict nuclear grade. In most cases, the accuracy of the biopsy is within the range of 75% and 85% [49]. In a meta-analysis, the percutaneous biopsy showed a moderate concordance (87%) with surgical specimens [8]. However, nuclear grading with biopsy is an invasive method and is susceptible to significant sampling bias [8, 9]. At some institutions, the percutaneous biopsy failure rate may be as high as 37% [50]. Because there are expanding treatment options, including active surveillance and ablative therapies, it has been essential to determine the biologic behavior of the tumors in vivo. On the basis of our current study and the other two studies [13, 14], ML-based CT texture analysis, with its noninvasive nature, showed a predictive performance comparable to that of biopsy in predicting nuclear grade of the clear cell RCCs. Also, it may play a key role in active surveillance of renal masses by allowing repeated noninvasive assessment of the nuclear grade during follow-up [7, 51]. In the future, the combined use of unenhanced and contrast-enhanced CT images might also be useful in creating better models for this purpose.

Several limitations need to be acknowledged for this study. First, the retrospective nature of the study was inherently disadvantageous. Second, our patient population was rather small and relatively imbalanced between classes, which might lead to over- or underfitting in ML-based classifications. To avoid over- or underfitting, we considered a few measures. We used the simplest form of ANN or deep learning, which is multilayer perceptron, as well as a conventional method, the logistic regression. A convolutional neural network would be an option, but considering its complexity, it would have almost certainly led to over- or underfitting using such a small labeled data. We used a cross-validation algorithm that separates the feature selection and model validation. Otherwise, the model would have been too optimistic. We used SMOTE to avoid the data imbalance problem, which also could have led to overly optimistic results in ML-based classifications [41]. Third, 2D tumor segmentation was used. Three-dimensional segmentation would have been much more representative for tumor texture, but it would not be useful for clinical practice because of the excessive segmentation duration. For the future, integration of semiautomated or fully automated segmentation techniques with texture analysis software programs might provide an opportunity to overcome this limitation. Fourth, the use of a rather thick image slice of 5 mm might be considered a limitation. Nonetheless, rather than using thinner slices, we think that consistency of the slice thickness is much more important so that we constructed our inclusion and exclusion criteria on the basis of this issue. Fifth, the imaging database of TCGA-KIRC in The Cancer Imaging Archive includes patients from various centers and sources with different image acquisition protocols. To minimize inter-scanner variabilities and effects, all image datasets in our study underwent some preprocessing steps before texture analysis, such as normalization, discretization, and pixel rescaling [30, 31]. Sixth, we did not include contrast-enhanced CT images in the study because of inconsistent contrast administration protocols in the database. Seventh, we did not use an independent validation set for ANN because we have a small patient population. On the other hand, we used a nested cross-validation algorithm with 10-fold inner and 10-fold outer loops for the ANN. This method uses all the data by splitting it into training, test, and validation sets, which then simulates an external validation process by reducing the bias and giving an estimate of the error similar to that of independent validation [52, 53]. Nevertheless, independent external validation is also needed for clinical use of the method. Eighth, we did not consider using a nested cross-validation method for the logistic regression analysis to present a rather conventional analysis. Nonetheless, we used at least a simple 10-fold cross-validation method to avoid overly optimistic results. Finally, a fundamental problem with the texture field is the interpretation of the texture features in a context, even if they were somehow validated [15]. In the context of the nuclear grade of clear cell RCCs, the selected features might represent some inherent heterogeneity differences associated with patient survival. Although every feature has been built on very detailed mathematic models, establishing meaningful clinical and pathologic correlates seems problematic at this stage of the developing field of texture analysis [15].