The study cohort included 1073 patients who underwent partial or total thyroidectomy between January 2012 and June 2014 at Jiangsu Institute of Nuclear Medicine. The decision to undergo surgery was based on any one of the following criteria: first, abnormal results of ultrasound-guided fine-needle aspiration, including malignancy, suspicion of malignancy, and follicular lesion of undetermined significance; second, suspicious malignant ultrasound findings, including hypoechogenicity, microcalcifications, a taller-than-wide shape and associated cervical lymphadenopathy with round shape, intranodal cystic components, or microcalcifications; and third, pressure symptoms [12]. Patients' medical records, including age, sex, ultrasound features of the dominant nodule, and histopathologic results, were collected retrospectively. The dominant nodule was defined as the nodule most likely to be malignant among all the nodules observed on ultrasound. The largest nodule was designated as the dominant nodule when none of the nodules showed suspicious ultrasound features. Nodule identification was achieved via ultrasound imaging and gross pathologic examination records. We excluded 103 nodules in 103 patients because the ultrasound data or images for these nodules were incomplete, or there were coalescent thyroid lesions not clearly distinguishable and matched with the histopathologic results. This retrospective study was approved by the institutional review board of Jiangsu Institute of Nuclear Medicine, and the need for informed patient consent for inclusion in this study was waived.

Thyroid ultrasound examination was performed by a radiologist using an ultrasound system (iU22, Philips Healthcare) with a 5-12–MHz transducer. Static images were archived as jpeg image files for subsequent evaluation. The ultrasound images of the dominant nodule were presented in a random fashion by the study coordinator (an author with 6 years of ultrasound experience). During interpretation, two radiologists with differing experience in ultrasound examination of thyroid nodules (17 and 3 years, respectively) were blinded to any subsequent cytologic or histologic diagnosis as well as the assessments of the other radiologist. In the measurement of nodules, lengths corresponded to the long-axis measurements on the longitudinal scan, and widths and thicknesses corresponded to the short-axis measurements on the transverse scan. The following ultrasound features were documented for each nodule: location (left, right, or isthmus), position (upper pole, medium, or lower pole), shape (ovoid-to-round, taller-than-wide, or irregular), margin (ill-defined or well-defined), internal contents (solid, mixture, or cystic), echogenicity (hyperechoic, isoechoic, hypoechoic, or marked hypoechoic), calcification (microcalcification, macrocalcification, or rim calcification), echogenic foci in solid portion (absent or present), halo sign (usual or unusual), infiltration and extracapsular invasion (absent or present), multifocal (absent or present), increased intranodular vascularity (absent or present), and abnormal lymphadenopathy (absent or present). The definitions and categorizations of shape, echogenicity, and calcification were consistent with those used in the literature [6]. Partly interrupted rim calcification corresponded to incomplete peripheral calcification surrounding the lesion. Ill-defined margin included spiculated and microlobulated margin. The internal content was categorized in terms of the ratio of the cystic portion to the solid portion as solid (≤ 10% of the cystic portion), mixture (> 10% of the cystic portion and ≤ 90% of the cystic portion), and cystic (> 90% of the cystic portion) [6]. Echogenic foci were bright spots with comet tails caused by reverberation artifacts. When a lesion was surrounded by an irregular or thick anechoic halo, it was considered an unusual halo. Infiltration was defined as an interruption of the hyperechogenicity of the thyroid capsule. Extracapsular invasion was considered if the tumoral tissue extended beyond the contours of the thyroid gland and invaded into adjacent structures [13]. A multifocal lesion was defined by the presence of two or more isolated or noncontiguous lesions with similar suspicious malignant ultrasound features in one or both thyroid lobes [14]. Increased intranodular vascularity was defined by an increased predominance within the nodule in comparison with the surrounding parenchyma. Lymph nodes were considered abnormal when a suspicious sonographic finding (calcifications and cystic change) was present, whereas round shape, hyperechogenicity, and abnormal vascularity of lymph nodes were excluded for low specificity or positive predictive value [15, 16].

In accordance with our experiences, combined with the data available in the literature [16–18], we adopted a five-tier sonographic scoring system for stratifying the risk of nodule malignancy in routine clinical practice. Ultrasound features, including marked hypoechogenicity, taller-than-wide shape, microcalcifications, multifocal, infiltration or extracapsular invasion, and abnormal lymphadenopathy, were regarded as indications for malignancy. Borderline ultrasound features included hypoechogenicity, an irregular shape, an ill-defined margin, an unusual halo, increased intranodular vascularity, macrocalcifications, and partly interrupted rim calcifications. The ultrasound features of a benign nodule included isoechogenicity, hyperechogenicity, an ovoid-to-round shape, a well-defined margin, echogenic foci in the cystic portion, a usual halo (regular and thin), and a spongiform and pure cystic nodule. The criteria for ultrasound-based diagnosis of a thyroid nodule were as follows: If a nodule had three or more ultrasound features of malignancy, regardless of borderline or benign ultrasound features, the nodule was considered malignant. If a thyroid nodule had at least one ultrasound feature of malignancy, regardless of borderline or benign ultrasound features, it was considered as being suspicious for malignancy. If a thyroid nodule had one or more borderline ultrasound features without malignant features, regardless of benign ultrasound features, it was considered borderline. If a thyroid nodule had two or more benign ultrasound features suggesting no malignancy or borderline ultrasound features, it was considered as likely benign. Finally, if a thyroid nodule was a spongiform or pure cystic nodule with no features of malignancy or borderline ultrasound features, it was considered benign [18].

We built the models using Matlab (version 8.1, MathWorks). The observations of radiologist 1 (i.e., the experienced radiologist) were evaluated using the machine learning algorithms to obtain the model diagnoses for the same case. The standard data-mining process was adopted in the classifier development and consisted of the following steps: data cleaning and integration, data selection and transformation, and data mining and pattern evaluation. All variables describing clinical information and ultrasound features were selected as input variables for the classifier models. Subsequent attribute optimization was used to optimize variable selection and input node confirmation. A fivefold cross-validation using a train-and-test method was performed to guarantee the validity of the results. Unrelated and confounding variables were then pruned from the final models. A random number was assigned to each nodule. Nodules with an odd number (n = 485, 50%) were assigned to a training cohort, and those with an even number were assigned to a validation cohort (n = 485, 50%).

In the first training phase, we had applied three classification algorithms on the obtained significant attributes. For naïve Bayes classifier, we set the first evidence on the input nodes, and then the output nodes could be queried using Bayesian network inference. The links in the naïve Bayes classifier were directed from output to input, which made the classifier simplified, because there were no interactions between the inputs. The SVM classifier was trained with a learning algorithm from optimization theory. The training instances closest to the maximum margin hyperplane were called support vectors, which were then used to build an optimal linear separating hyperplane. The RBF-NN contained three layers of nodes but with only a single hidden layer. The classifier-adopted RBF-NN used a clustering method to determine the center of the RBF, and the least-mean square method was used to determine the connection weights between the hidden layer and output layer. Training was terminated when the error between the actual output and the desired output was minimized or the given maximum number of epochs to train was obtained. Cross-validation was used to reduce the risk of overtraining, which will lead to good performance in a training cohort but poor performance in a validation cohort. The data were split via stratified sampling to ensure the same class distribution in the subset to generate comparison results with lower bias.

Subsequently, the classifiers were used for classification in the second validation phase. The performance was analyzed with ROC curves. Ultimately, once we add evidence to a trained and cross-validated classifier given prior knowledge, it will generate case-specific malignant prediction through predicting the probability of classification.

Continuous variables are presented as mean ± SD and were compared with the two-sample t test. Categoric variables are presented as percentages, and chi-square or Fisher exact tests were used to determine the statistical significance of the differences between the two groups. For evaluation of the performance of machine learning algorithms and the two radiologists in the task of predicting the probability of malignancy, the AUC values were calculated and compared.

The mean age of the 970 patients was 46.71 ± 12.14 (SD) years (range, 13–83 years); 756 (77.94%) were female and 214 (22.06%) were male. There were 507 cases of malignant nodules (52.27%) and 463 cases of benign nodules (47.73%). Diagnoses of malignancy included papillary thyroid carcinoma (n = 487), follicular thyroid carcinoma (n = 12), medullary thyroid carcinoma (n = 4), well-differentiated carcinoma (n = 3), and clear cell carcinoma (n = 1).

The sex distribution was not statistically significantly different between the patients with malignant and benign nodules (male-to-female ratio, 103:404 vs 111:352; p = 0.1702), but a statistically significant difference in age was observed between the patients with malignant and benign nodules (44.76 ± 12.15 years vs 48.92 ± 12.14; p < 0.01). Benign nodules were statistically significantly larger than the malignant nodules (p < 0.01), and a taller-than-wide shape was found more frequently in malignant nodules (39.05%) than in benign nodules (10.37%). More malignant nodules had an ill-defined margin compared with the benign nodules (91.72% vs 57.24%). Solid nodules were more frequently detected among the malignant nodules (96.45% vs 67.82%). Hypoechogenicity was an ultrasound feature found in most malignant nodules (94.87%), and the frequency of microcalcifications in malignant nodules was statistically significantly higher than that in benign nodules (44.38% vs 8.64%). When echogenic foci presented in the solid portion, the prevalence of malignancy was 24.4% (10/41) (Fig. 1). The presence of unusual halo sign, infiltration and extracapsular invasion, and multifocal and abnormal lymphadenopathy were statistically significantly different between malignant and benign groups (p < 0.01). No statistically significant differences were found with respect to patient sex, left and right lobe location, cystic contents, hyperechogenicity and marked hypoechogenicity, macrocalcification, partly interrupted rim calcification, and increased intranodular vascularity between benign and malignant nodules (p > 0.05). Thus, these features were regarded as suspicious invalid redundant features for malignancy.

We assessed the predictive accuracy of machine learning algorithms, both including (Table 1) and excluding (Table 2) the suspicious invalid redundant features. After excluding the suspicious invalid redundant features, the accuracies of the naïve Bayes classifier and SVM for predicting malignancy of thyroid nodules (83.30% and 83.89%, respectively) were better than those achieved with inclusion of the suspicious invalid redundant features (82.99% and 83.71%, respectively; p < 0.05). After excluding the suspicious invalid redundant features, the accuracy of the RBF-NN was slightly less than that with inclusion of the suspicious invalid redundant features (83.61% vs 83.92%; p = 0.6302), whereas the AUC was higher with suspicious invalid redundant features excluded versus included (0.8951 vs 0.8937; p < 0.05). Therefore, we considered that the suspicious invalid redundant features were unrelated to the construction of the classifier and pruned them from the final models.

The performances of the proposed models were evaluated using the validation cohort (Table 3). The most favorable sensitivity, specificity, positive predictive, negative predictive, and accuracy values were achieved when we defined the indeterminate as benign. In the final evaluation of the validation cohort, radiologist 1 (the experienced radiologist) achieved the highest prediction accuracy of 88.66% with a sensitivity of 91.54% and a specificity of 85.33%. Radiologist 2 (the less experienced radiologist) achieved a prediction accuracy of 81.03% with a sensitivity of 85.38% and a specificity of 76.00%. For comparison, the respective corresponding values were 83.30%, 89.62%, and 76.00% for the naïve Bayes classifier; 83.09%, 89.23%, and 75.11% for the SVM; and 84.74%, 92.31%, and 76.00% for the RBF-NN. The AUC value for radiologist 1 was higher (AUC = 0.9135) than those for radiologist 2, naïve Bayes classifier, SVM, and RBF-NN (AUC = 0.8492, 0.8811, 0.9033, and 0.9103, respectively; p < 0.05). In addition, the AUC for the RBF-NN was statistically significantly higher than those for the other machine learning algorithms and radiologist 2 (all p < 0.05).