Introduction	Choose Top of pageABSTRACTIntroduction <<Materials and MethodsResultsDiscussionAPPENDIX 1: Logistic Regr...APPENDIX 2: Model 3ReferencesCITING ARTICLES

Mammography, accepted as the most effective screening method in the detection of early breast cancer, still has limited accuracy

and significant interpretation variability that decreases its effectiveness [1–6]. The use of computer models can help by detecting abnormalities on mammograms [7–10]; estimating the risk of breast cancer for improved sensitivity and specificity of diagnosis [11–16]; and identifying high-risk populations for screening, genetic testing, or participation in clinical trials [17–22]. This study focuses on the second goal: the use of a computer-aided diagnosis (CADx) model for risk estimation to aid radiologists in breast cancer diagnosis.

CADx models can quantify the risk of cancer using demographic factors and mammography features already identified by a radiologist or a computer-aided detection model. CADx models estimate the probability (or risk) of disease that can be used for improved decision making by physicians and patients [23–25]. Previous studies on CADx tools use either small subsets of data, suspicious mammograms, or mammograms recommended for biopsy [11–15]. Although most of these studies show that CADx tools are efficient in predicting the outcome as benign or malignant disease, none shows the effectiveness of CADx models when applied to mammography data collected during daily clinical practice. In addition, previous studies used biopsy results as the reference standard, whereas we use a match with our state cancer registry. To our knowledge, our study is the first one to develop and test a logistic regression–based CADx model based on consecutive mammograms from a breast imaging practice incorporating BI-RADS descriptors.

As the variables that help predict breast cancer increase in number, physicians must rely on subjective impressions based on their experience to make decisions. Using a quantitative modeling technique such as logistic regression to predict the risk of breast cancer may help radiologists manage the large amount of information available, make better decisions, detect more cancers at early stages, and reduce unnecessary biopsies. The purpose of this study was to create a breast cancer risk estimation model based on demographic risk factors and BI-RADS descriptors available in the National Mammography Database using logistic regression that can aid in decision making for the improved early detection of breast cancer.

Materials and Methods	Choose Top of pageABSTRACTIntroductionMaterials and Methods <<ResultsDiscussionAPPENDIX 1: Logistic Regr...APPENDIX 2: Model 3ReferencesCITING ARTICLES

The institutional review board determined that this retrospective HIPAA-compliant study was exempt from requiring informed consent. We used variables collected in the National Mammography Database [26] to develop a CADx model. The National Mammography Database is a recommended format for collecting practice-level mammography audit data to monitor and standardize performance nationally. The National Mammography Database includes Breast Imaging Reporting and Data System (BI-RADS) descriptors [27, 28].

Subjects

We collected data from all screening and diagnostic mammography examinations that were per formed at the Medical College of Wisconsin, Milwaukee, an academic, tertiary care medical center, between April 5, 1999 and February 9, 2004. Our database included 48,744 mammography examinations (477 malignant and 48,267 benign) performed on 18,270 patients (Table 1) having the mean age of 56.8 years (range, 18–99 years). Our data set consisted of 65,892 records; each record represents a mammography lesion (benign or malignant) observed on the mammogram or a single record of demographic factors only, if nothing is observed on the mammogram. The data were entered using the PenRad mammo graphy reporting and tracking data system (struc tured reporting software, PenRad) by technologists and radiologists. There were a total of eight radiologists, four of whom were general radiologists with some mammography background, two who were fellowship-trained, and two who had lengthy experience in breast imaging. The experience of the eight radiologists ranged between 1 and 35 years, and the number of mammograms interpreted by them ranged from 49 to 22,219. All mammography observations were made by radiologists; all demographic factors were recorded by technologists. This facility used a combination of digital and film mammography (∼ 75% film mammo graphy). No computer-aided detection tool was used for lesion detection. Mean glandular dose was not available at the time of our study.

View Larger Version

TABLE 1: Distribution of Study Population

The clinical practice we studied routinely converts screening examinations to diagnostic mammo graphy examinations when an abnormality is identified; therefore, practice performance para meters were calculated in aggregate because these examinations could not be accurately separated. Specifically, we measured recommended per formance parameters (cancer detection rate, early-stage cancer detection rate, and abnormal interpretation rate) for all mammograms in our data set.

In contrast to our practice performance audit, based on mammograms, the analysis of the classification accuracy of the logistic regression model and radiologists was conducted at the record level. Because breast cancer classification actually occurs at the record level (i.e., each finding on mammography will require a decision to recall or to biopsy), we target this level of level of detail to help improve radiologists' performance. We clearly indicate when analyses in this article are based on mammograms rather than on records.

We used cancer registry matching as the reference standard in this study. All newly diagnosed cancer cases are reported to the Wisconsin Cancer Reporting System. This registry collaborates with several other state agencies to collect a range of data, including demographic information, tumor characteristics, treatment, and mortality. Data exchange agreements with 17 other state cancer registries yield data for Wisconsin residents receiving care in other states. We sent 65,892 records in the database to the cancer registry and received back 65,904 records after their matching protocol. An additional 12 records were returned to us because of duplication of records for patients diagnosed with more than one cancer. We developed an automated process that confirmed whether the cancer matched the assigned abnormality. This process ensured that the record indicated the same side and the same quadrant and that the diagnosis was made no longer than 12 months after the mammography date. If more than one record indicated the same side and quadrant, the matching was done manually. We used a 12-month follow-up period as the reference standard because it has been recommended as an interval sufficient to identify false-negatives in mammography prac tice audits [27, 28]. We removed 299 records belonging to 188 mammograms from 124 women because they could not be matched due to missing laterality or quadrant information from either the cancer registry (117 records) or the mammography structured report (182 records) (Table 2). Of the unmatched 299 records, 183 records represented a second record identifying a finding in women who already had a cancer matched to the registry. The remaining 116 records consisted of 38 BI-RADS category 1, 24 category 2, 22 category 3, 21 category 0, four category 4, and seven category 5. We then removed 101 duplicates. Finally, we removed 3,285 records that had BI-RADS assessment categories 0, 3, 4, and 5 (indicating a finding) that did not have descriptors recorded in the record. The final sample consisted of 62,219 (510 malignant, 61,709 benign) records.

View Larger Version

TABLE 2: Data Processing

View larger version (25K)

Fig. 1 —Descriptors of National Mammography Database [26] entered to build logistic regression model for breast cancer prediction. aBinary variable with categories “Present” or “Not Present.” bClass 1, predominantly fatty; class 2, scattered fibroglandular; class 3, heterogeneously dense; and class 4, extremely dense tissue.

Statistical Analysis

Model construction—Logistic regression, a statistical approach to predicting the presence of a disease based on available variables (symptoms, imaging data, patient history, and so forth), has been successfully used for prediction and diagnosis in medicine [29, 30]. To build a breast cancer risk estimation model, we mapped the variables collected by physicians in their daily clinical practice (based on BI-RADS descriptors in the National Mammography Database) to 36 discrete variables. Figure 1 shows the schema of these variables used to build the model. We constructed two risk estimation models. Both models used the presence or absence of breast cancer as the dependent variable, and these 36 discrete variables were used as independent variables to build the model. Model 2 included these same variables plus the BI-RADS assessment categories assigned by the radiologists. More than 600 two-way interaction effects are possible in each model. We did not include any interaction term in our models.

Before model construction, we grouped BI-RADS categories 1 and 2 as “BI-RADS 1 or 2” because these cases had a low frequency of malignancy. The logistic regression model was built using R statistical software (The R Foundation for Statistical Computing) [31]. We used forward selection based on the chi-square test of the change in residual deviance. We used a cutoff of p < 0.001 for adding new terms. This stringent criterion was used to avoid including terms that, although statistically significant because of the large sample size, are not clinically important. The p values listed in Tables 3 and 4 are from chi-square tests of the significance of each term entered last. The importance of each term in predicting breast cancer can be assessed using the odds ratios provided in the tables. The details of logistic regression (including the interpretation of odds ratios) are discussed in Appendix 1.

View Larger Version

TABLE 3: Model I, Multivariable Model with BI-RADS Categories Excluded

View Larger Version

TABLE 4: Model 2, Multivariable Model with BI-RADS Categories Included

A number of sources of correlation are possible in these data. Findings from a particular radiologist may be more similar than findings from different radiologists, findings within a patient may be more similar than those from different patients, and findings during the same mammography visit of a patient may be more similar than those during other mammography visits of the same patient. We investigated models in which the radiologist is included as a random effect and compared it with our models in which the radiologist is excluded from the models. We found no substantial differences in the coefficients for the other terms in the model due to including the radiologist as a random effect. Thus, we chose the simpler model without the radiologist. We were unable to test random effects for patient or for mammogram within patient because the expected number of cancers for each patient is very small. Random effects models tend to be biased in these circumstances [32]. Instead, we relied on our stringent criterion of p < 0.001 for inclusion in the model to avoid the overly optimistic p value that occurs when the variance of the parameters is reduced by positive correlation induced by clustered data. The parameter estimates themselves are unbiased regardless of the form of the variance.

To show that BI-RADS descriptors substantively contribute to prediction accuracy in Model 2, we also constructed a secondary model (Model 3). Model 3 omits these descriptors and includes only patient demographic factors (age, history of breast cancer, family history of breast cancer, history of surgery, breast density, and hormone therapy) and BI-RADS assessment categories as independent variables to test whether performance declines. The details of Model 3 are provided in Appendix 2.

Model evaluation—We used a 10-fold cross-validation technique to evaluate the predictive performance of the two models. This methodology avoids the problem of validating the model on the same data used to estimate the parameters by using separate estimation and evaluation subsets of the data. Specifically, we divided the data set into 10 subsets (with approximately one tenth of benign abnormalities and one tenth of malignant abnormalities in each subset or “fold”) so that all abnormalities associated with a single patient were assigned to the same fold. This ensured that all folds are independent of each other. We started with the first nine folds (omitting the 10th fold) to estimate the coefficients of the independent variables (training) and predicted the probability of cancer on the 10th fold (testing). Then we omitted the 9th fold (used as the testing set) and trained the model using the other nine folds. Similarly, we tested on each fold. Finally, we combined all test sets to obtain a full-test set and evaluated the overall performance of the model using the full-test set. Note that for inclusion of variables in the final model, we used the whole data set (62,219 records), which gave us the best possible estimates of the variables from the available data.

Performance measures—We measured the performance of the two models using the outcome (i.e., the probability of cancer) of the full-test set obtained by 10-fold cross-validation. We plotted and measured area under the receiver operating characteristic (ROC) curve of Model 1 and Model 2 using the probability of cancer. We measured the performance of radiologists using BI-RADS assessment categories assigned to each mammography record. We first ordered BI-RADS assessment categories by likelihood of breast cancer (1, 2, 3, 0, 4, and 5), generated an ROC curve, and measured its area (A_z) using a nonparametric method [33]. We compared the performance of the two models with that of radiologists using the nonparametric method of DeLong et al. [34] for comparing two or more areas under ROC curves obtained from the same data set.

For the purpose of assessing the sensitivity and specificity of radiologists, we classified BI-RADS categories 1, 2, and 3 as negative; and BI-RADS categories 0, 4, and 5 as positive [28]. We compared the sensitivity of the two models with the radiologists' sensitivity at 90% specificity, and the specificity of the two models with the radiologists' specificity at 85% sensitivity, with the corresponding CIs estimated using the efficient score method corrected to continuity [35]. Note that the points “sensitivity at 90% specificity” and “specificity at 85% sensitivity” on the radiologists' ROC curve were not observed in practice; they were obtained from the linear interpolation of the two neighboring discrete points. We used these levels of sensitivity and specificity because they represent the minimal performance thresholds for screening mammography [36]. We also estimated the number of true-positive and false-negative records at 90% specificity by multiplying the sensitivity (of radiologists, Model 1 and Model 2) by the total number of malignant records. Similarly, we estimated the number of false-positive and true-negative records at 85% sensitivity by multiplying the specificity (of radiologists, Model 1 and Model 2) by the total number of benign records. Finally, we identified the most important predictors of breast cancer using the odds ratio given in the

Discussion	Choose Top of pageABSTRACTIntroductionMaterials and MethodsResultsDiscussion <<APPENDIX 1: Logistic Regr...APPENDIX 2: Model 3ReferencesCITING ARTICLES

We constructed two breast cancer risk estimation models based on the National Mammography Database descriptors to aid radiologists in breast cancer diagnosis. Our results show that the combination of a logistic regression model and radiologists' assessment performs better than either alone in discriminating between benign and malignant lesions. The ROC curve of Model 1, which includes only demographic factors and mammography observations, overlaps and intersects with the radiologists at certain points in the curve, showing that one is not always better than the other. On the other hand, Model 2, which also includes radiologists' impressions, clearly dominates the other two ROC curves, indicating better sensitivity and specificity at all threshold levels. Adding radiologists' overall impressions (BI-RADS category) in Model 2, we could identify more malignant lesions and avoid false-positive cases as compared with the performance of Model 1 and radiologists alone.

Our computer model is different in various ways when compared with the existing mammography computer models in the literature. The existing models can be categorized in the following ways: for detecting abnormalities present on the mammograms, for estimating the risk of breast cancer based on the mammographic observations and patient demographic information, and for predicting the risk of breast cancer to identify high-risk individuals. The first category of models is used to identify abnormalities on the mammograms, whereas our model provides the interpretation of mammography observations after they are identified. The models in the second category, in which we classify our model, have used suspicious findings recommended for biopsy for training and evaluation or biopsy results as the reference standard. For example, one study constructed a Bayesian network using 38 BI-RADS descriptors; by training the model on 111 biopsies performed on suspicious calcifications, they found an A_z of 0.919 [37]. Another study developed linear discriminant analysis and artificial neural network models using a combination of mammographic and sonographic features; they found an A_z of 0.92 [16]. In contrast, our computer model was trained and evaluated on consecutive mammography examinations and used registry match as the reference standard. The third category of models (risk prediction models) has been built using consecutive cases, but they included only demographic factors and breast density in their model [19, 21, 22] and cannot be directly compared with our model.

In addition, our model differs from these risk prediction models by estimating the risk of cancer at a single time point (i.e., at the time of mammography) instead of over an interval in the future (e.g., over the next 5 years). In contrast to their findings, our model did not find breast density to be a significant predictor of breast cancer. This could be because the risk of breast cancer is explained by more informative mammographic descriptors in our logistic regression model. Our model reinforces previously known mammography predictors of breast cancer—irregular mass shape; ill-defined and spiculated mass margins; fine linear calcifications; and clustered, linear, and segmental calcification distributions [38]. In addition, we found increasing mass size and high mass density to be significant predictors, which has not been shown in the literature to our knowledge. Note that our results reflect a single practice and must be viewed with some caution with respect to their generalizability because significant variability has been observed in the interpretive performance of screening and diagnostic mammography [5, 6].

We developed two risk estimation models by excluding (in Model 1) and including (in Model 2) BI-RADS assessment categories. Although Model 2 performed significantly better than Model 1 in discriminating between benign and malignant lesions, Model 2 may have weaknesses as a stand-alone risk estimation tool if the assessed BI-RADS category is incorrect. If the BI-RADS assessment category does not agree with the findings, Model 1 and Model 2 used jointly will show a high level of disagreement in the prediction of breast cancer (as in example case 2) and will potentially indicate this error. When the radiologist's BI-RADS category is correct (i.e., when there is an agreement between the predictions of Model 1 and Model 2), Model 2 will be a better model for breast cancer prediction. In future work, we plan to estimate the level of disagreement between the two models and investigate the possible use of these models as complementary tools.

Our secondary model (Model 3) showed that the exclusion of the BI-RADS descriptors significantly impairs the performance of the logistic regression model, underscoring the need for the collection of these variables in a clinical practice.

It is common for clinical data sets to contain a substantial number of missing data. Although complete data are ideally better, that situation is rarely encountered in the real world. There is no perfect way to handle missing data, but there are two possibilities: to impute the missing descriptor depending on the fraction of various possible values of the descriptor or to assume that the missing descriptor was not observed by radiologists and mark it as “not present.” When building the model, we made the decision to label all of the missing data as not present; therefore, when testing and applying the model on a new case, the missing descriptors should be treated as not present. Our approach to handling missing data is appropriate for mammography data, where radiologists often leave the descriptors blank if nothing is observed on the mammogram.

To our knowledge, no prior studies discuss a logistic regression–based CADx model incorporating mammography descriptors from consecutive mammograms from a breast imaging practice. The use of a logistic regression model has some attractive features when compared with artificial intelligence prediction tools (e.g., artificial neural networks, Bayesian networks, support vector machines). Logistic regression can identify important predictors of breast cancer using odds ratios and can generate confidence intervals that provide additional information for decision making.

Our models' performance depends on the ability of radiologists to accurately identify findings on mammograms. Therefore, based on the literature, performance may be higher in facilities where most mammograms are read by mammography subspecialists as compared with general radiologists [39]. However, with appropriate training [40], general radiologists in combination with the model may approach the accuracy of subspecialty-trained mammographers. Decreasing variability in mammography interpretation, one of the underlying motivations of this research, can only be realized with further development of tools such as our model and with research to validate accuracy, effectiveness, and generalizability. We consider this work to be only a first step toward this goal.

We could not compare practice parameters directly with the literature because screening and diagnostic examinations could not be separated for this database. Our prediction Model 2 shows a significant improvement over radiologists' assessment in classifying abnormalities when built on a mix of screening and diagnostic data. The model's performance may differ when built separately on screening and diagnostic mammograms. For screening mammograms, the incidence is low and descriptors are less exact because of general imaging protocols and so may result in less accurate model parameters. In contrast, for diagnostic mammograms, the model parameters may be more accurate because more descriptors can be observed as a result of additional specialized views. In addition, the performance of our existing model may differ when tested on screening and diagnostic mammograms separately. The model may perform better when tested on the diagnostic examinations but worse when tested on the screening examinations.

Our risk estimation models are designed to aid radiologists, not to act as a substitute. The improvement in the model's performance by adding BI-RADS assessments indeed suggests that the radiologist's integration of the imaging findings summarized by the BI-RADS assessment categories does augment predictions based on the observed mammographic features. However, the logistic regression model contributes an additional measure of accuracy over and above that provided by the BI-RADS assessment categories, as evidenced by the improved performance compared with that of the radiologists alone.

The objective of our model is to aid decision making by generating a risk prediction for a single point in time (at mammography). As we were designing the study, we did not want to influence the probability of breast cancer based on future events but only on variables identified at the time of mammography. For this reason, we excluded unmatched BI-RADS 1 cases from our analyses, which represented either undetected cancer (present on the mammogram but not seen) or an interval cancer (not detectable on the mammogram). The inclusion of these cases may have erroneously increased the probability of malignancy by considering future risks rather than making a prediction at a single time based on mammography features alone. However, the exclusion of these cases may have erroneously decreased the estimated probability of malignancy, given that at least some of the false-negative cancers were likely present at the time of the mammogram, especially those in women with dense breasts, which is a limitation of our model.

Our models provide the probability of cancer as the outcome that can be used by radiologists for making appropriate patient management decisions. The use of such models has a potential to reduce the interpretive variability of mammography across practices and radiologists. Our models also facilitate shared decision making by providing the probability of cancer, which can be better understood by patients than BI-RADS categories. In the future, we will test our models' performance on other mammography practices to evaluate their generalizability. We will also include potentially important interaction effects that deserve particular attention. Note that including interaction effects will further improve the performance of our models.

In conclusion, we found that our logistic regression models (Model 1 and Model 2) can effectively discriminate between benign and malignant lesions. Furthermore, we have found that the radiologist alone or the logistic regression model incorporating only mammographic and demographic features (Model 1) are inferior to Model 2, which incorporates the model, the features, and the radiologist's impression as captured by the BI-RADS assessment categories. Our study supports that further research is needed to define how radiologists and computational models can collaborate, each adding valuable predictive features, experience, and training to improve overall performance.

APPENDIX 1: Logistic Regression	Choose Top of pageABSTRACTIntroductionMaterials and MethodsResultsDiscussionAPPENDIX 1: Logistic Regr... <<APPENDIX 2: Model 3ReferencesCITING ARTICLES

Binomial (or binary) logistic regression is a form of regression that is used when the dependent variable is dichotomous (e.g., present or absent) and the independent variables are of any type (discrete or continuous). The independent (observed) variables, X_i, X₂,... X_n, are related to the dependent (outcome) variable, Y, by the following equation: where β1 is the regression coefficient of X₁, p = probability {Y = 1}, and . The value of p can be calculated by taking the inverse of the Logit (p) as shown in the following equation: where p is the probability of the presence of disease (e.g., probability of cancer) when the findings X₁, X₂,... X_n, (e.g., calcification types, breast density, and age) are identified. β_i is the coefficient of the independent variable X_i that is estimated using the available data (training set). Only significant variables (p values ≥ a) are included in the model. Variables can be added by stepwise, forward, or backward selection methods. Odds ratio is commonly used to interpret the effect of independent variables on the dependent variable, which is estimated by exp (β_i). For example, if β₁ is the coefficient of variable “prior history of breast surgery,” then exp is the odds ratio corresponding to the history of surgery—that is, the odds that the patient has a malignant lesion increases by the factor of exp (β₁) if the patient has ever had breast surgery and all other independent variables remain fixed. More details of logistic regression and its application to the medical field can be found in other sources [29, 41, 42].

APPENDIX 2: Model 3	Choose Top of pageABSTRACTIntroductionMaterials and MethodsResultsDiscussionAPPENDIX 1: Logistic Regr...APPENDIX 2: Model 3 <<ReferencesCITING ARTICLES

In order to assess the contribution of mammography descriptors in estimating the risk of breast cancer, we constructed Model 3, which included patient demographic factors (age, history of breast cancer, family history of breast cancer, history of surgery, breast density, and hormone therapy) and BI-RADS assessment categories, and excluded mammography descriptors. Only three variables were found significant in predicting the risk of cancer in Model 3 (Table 6); BI-RADS assessment categories were the most important predictor.

View Larger Version

TABLE 6: Model 3, Multivariate Model with Patient Demographic Factors and BI-RADS Categories Only

We measured the performance of our model using receiver operating characteristic (ROC) curves and precision–recall curves (Figs. 3 and 4). We used precision–recall curves in addition to ROC curves to gain more insights into the performance of our model because precision–recall curves have higher discriminative power than ROC curves in cases of skewed data [43, 44]. “Precision” measures the positive predictive value and “recall” measures the sensitivity of a test. We plotted and measured the area under the precision–recall curve (A_PR) of the three models (Model 1, Model 2, and Model 3) and radiologists using the probability of cancer and BI-RADS assessment categories, respectively [43].

Model 3 achieved an A_z (area under the ROC curve) and A_PR that were significantly higher than that of Model 1 and radiologists (all p < 0.001). More important, Model 3 excluding descriptors performed significantly worse (p < 0.001) than Model 2 including descriptors in terms of A_z and A_PR (Table 7). Thus, the inclusion of mammographic descriptors significantly contributes to the superior performance of Model 2.

View Larger Version

TABLE 7: Comparison of Area Under Receiver Operating Characteristic (A_z) and Precision–Recall (A_PR) Curves

View larger version (14K)

Fig. 3 —Graph shows receiver operating characteristic curves constructed from output probabilities of Model 1, Model 2, and Model 3, and radiologist's BI-RADS assessment categories. AUC = area under curve.

View larger version (24K)

Fig. 4 —Graph shows precision–recall curves constructed from output probabilities of Model 1, Model 2, and Model 3, and radiologist's BI-RADS assessment categories. AUC = area under curve, PPV = positive predictive value.