| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1
Department of Radiology, CB# 7515, RRL, 106 Mason Farm Rd., University of
North Carolina at Chapel Hill, Chapel Hill, NC 27599-7515.
2
Lineberger Comprehensive Cancer Center, CB# 7295, Chapel Hill, NC,
27599-7295.
3
Presbyterian Breast Imaging Center, 1718 E. 4th St., Charlotte, NC
28204.
4
Seaboard Radiology, 630 E. 11th St., Washington, NC 27889.
Received January 23, 2001;
accepted after revision March 16, 2001.
Supported by grant U01-CA-70040 from the National Cancer Institute.
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. All screening cases for 1997 and 1998 were identified from the Carolina Mammography Registry. Mammographic assessments from community mammography practices were linked with population-based cancer outcomes. The findings of four community-based radiologists who reviewed the mammograms of 339 asymptomatic women were 93 false-negatives, 180 true-negatives, and 66 false-positives. The percentage of false-negative, true-negative and false-positive findings on breast films that reviewers evaluated was determined. The findings of the reviewers were compared with the original interpreting radiologists' assessments.
RESULTS. The overall breast-specific workup rate by the reviewing radiologists was 21%. The average workup rate for the false-negative findings was 42% (range, 35-51%). Adjusting for the 13% workup rate in the cancer-free breasts, the percentage of false-negative findings that were detectable was estimated to be 29%.
CONCLUSION. This peer review of screening mammograms from a population-based screening registry estimated a missed detectable cancer rate of 29%. Thus, 71% of cancers missed at screening would not have been worked up by peers in the same community.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
In a community-based practice, the radiologists may see only a few cancers in an entire year. Thus, they have a more limited exposure to mammograms that are positive for cancer. This fact, added to the imperfection of screening mammography, contributes to missed cancers in the practice of mammography. This study had as its objective to investigate the proportion of missed cancers in community practice that may be due to human error. What is a reasonable expectation for the sensitivity of screening mammography in these settings? A reference of peer-reviewed missed cancers from a community would allow radiologists who practice screening mammography in the same community to fairly evaluate their own performance.
Delayed diagnosis of breast cancer accounts for the highest proportion of medical malpractice cases in the United States [4,5,6]. Establishing a community standard for usual experience with missed cancers from community-based screening might be helpful in defense of some screening malpractice cases. Our objective was to determine the percentage of these false-negative findings that might be considered detectable. We estimated this by measuring the percentage of false-negative findings for which the experienced peer interpreting radiologists recommended further diagnostic workup, adjusted for the review workup rates for true-negative findings in breasts.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Breast pathology data were received in three ways: weekly reporting of newly diagnosed cancers from the North Carolina Central Cancer Registry from the northeastern quarter of the state in which most of our facilities are located, annually from the North Carolina Central Cancer Registry with complete state cancer data, and directly from facilities that report data from pathology reports they receive for their own patients. One pathology file is created from these three reporting sources. The registry links mammography assessments from community mammography practices with population-based cancer outcomes from the pathology database for all mammographic examinations performed in participating practices.
Our study had a full review and was approved by the University of North Carolina School of Medicine Institutional Review Board. All data were linked and used as de-identified patient data.
Study Methods
This study was conducted over two consecutive years, 1997 and 1998. For a
facility to be included, it had to participate in the registry for a minimum
of 1 year. The mammograms of missed cancers for this study consisted of all
available screening cases in which a cancer diagnosis was confirmed within 1
year of negative findings on a screening mammogram (false-negative). All
mammograms identified as false-negative for cancer were reviewed for
documentation that the initial interpretation was negative with no further
workup recommended, and all pathology reports were reviewed to ensure that the
finding was breast cancer.
For a screening mammogram to be included as false-negative for cancer, it had to meet the following criteria: it had to be a screening study with a negative assessment and a cancer diagnosis date within the following year. The cancer diagnosis could result from an interval cancer detected symptomatically or from a screen-detected cancer revealed when the woman returned for an early annual screen before the end of the year. The following specific definitions were used for classification and analysis.
Definitions
Screening mammography study.—A bilateral mammography
examination recorded as an asymptomatic screening study by the technologist or
radiologist, with no record in the registry of a previous screening mammogram
for that woman in the preceding 9 months.
Cancer outcome.—A cancer diagnosis that included all invasive breast cancers, metastatic cancer to the breast, and ductal carcinoma in situ. Lobular carcinoma in situ was not considered a cancer diagnosis for this analysis
Negative screening mammography study.—The initial assessment for the screening study being normal or benign (BI-RADS code 1 or 2) or probably benign with short-term follow-up advised (BI-RADS code 3), as recommended by the BI-RADS manual [8]. In the experience of our registry, we found that many times a code 3 is assigned with a recommendation for an immediate workup. These specific cases with a recommendation for an immediate workup and a code 3 were classified as positive for cancer.
False-negative findings on screening mammograms (also referred to as "missed cancer").—All screening mammograms negative for cancer that had a diagnosis of breast cancer within a year, including both interval and screen-detected cancers. They were included as a screen-detected cancer only if the cancer was detected at an early (before the 365-day cutoff) annual screening examination.
True-negative screening mammogram.—A screening mammogram negative for cancer that did not have a cancer diagnosis within 1 year of the mammogram with negative findings.
Positive screening mammography study.—The initial assessment for the screening being abnormal (BI-RADS code 0, 4, or 5). In addition, any case in which a BI-RADS code 3 was associated with a recommendation for further immediate workup was classified as a study with positive findings, for reasons described previously.
False-positive mammogram.—A mammogram with positive findings that did not have a cancer diagnosis date within 1 year of the screening mammogram positive for cancer.
Detectable missed-cancer rate.—The proportion of missed cancers that experienced peer mammographers could identify from screening films, correcting for baseline recall in breasts with true-negative findings.
All false-negative mammograms that met our inclusion criteria were requested from the practices. To have a mix of studies, we requested true-negative and false-positive studies among women similar in age (within 5 years) and seen on the same date as the women with false-negative screening mammograms. We also requested an additional set of randomly selected true-negative and true-positive mammograms. The final study film set approached three controls (true-negative, false-positive, or true-positive) per false-negative. The original films, not copies, were used for the study. Over the 2 years of the study, 25% of the requested films were not located by the practice and were not included.
The initial group indentified for study included the screening studies for 364 individual women. We excluded 25 women from the study for the following reasons: 12 because we could not determine which breast had the abnormality leading to the recommendation for further workup (these women had a diagnostic procedure with normal findings performed the same day as a positive screening, leaving no record of which breast led to the diagnostic study), seven false-positive cases in which both breasts were recommended for further study, four identified as having implants after the films arrived, one with cancer in both breasts, and one with a missed cancer inadvertently included in both study years (the second interpretation was excluded). After the exclusions, 339 women with a total of 93 cancers remained in the study group. The 93 cancers included 85 invasive (67 invasive ductal, 15 invasive lobular, three metastatic) and eight cases of ductal carcinoma in situ. Insufficient numbers prohibited analyzing the data by cancer type.
These women had mammograms of 678 breasts. Ninety-three women had one breast with a missed cancer (false-negative screening study) and the contralateral breast with normal or benign findings. Figure 1A,1B,1C,1D,1E is an example of a woman with a false-negative study who was diagnosed with cancer within the following year when she appeared with symptoms. Of the 93 cancers that were false-negative by our definition, 88 were interval and 5 were detected at the next early annual screen. A total of 180 women were assessed as negative for cancer in both breasts and were not recommended for workup by the original practice radiologist (true-negative screening study). There were also 66 women with findings negative for cancer, but one breast was recommended for workup (false-positive screening study). These women had one false-positive mammogram with the contralateral breast being true-negative.
|
|
|
|
|
Rereading Exercise
Four community-based radiologists, experienced in mammography, participated
in the review exercises. The four radiologists were representative of the
larger practices in our area. In their practices in 1997, they interpreted, on
average, 500, 766, 867, and 1260 mammograms a month. Two of the radiologists
were 100% dedicated to mammography; the other two were general diagnostic
radiologists with a significant portion of their clinical practice comprising
mammography. In both years of the study, the reviews were performed in 1 day
with multiple viewing stations. The radiologists rotated among the stations,
interpreting the same set of films under the same conditions. The films were
masked to keep the radiologist from knowing the identity of the patient or
practice. Only the standard two views of the screening study were used.
There were four interpretations for each breast, one from each radiologist. The radiologists scored their impression for each breast as follows: "Negative" (BI-RADS 1 or 2) or "needs further assessment" (BI-RADS 0) [8]. We also asked them to specify the view on which they assessed the abnormality, craniocaudal or mediolateral.
We analyzed individual mammograms of 678 breasts. They were divided into three categories: 93 false-negative, 66 false-positive, and 519 true-negative mammograms, with each mammogram being of one breast. We expected overcalling of abnormal findings on screening interpretations as a result of the experimental situation because the interpreting radiologists knew that the mammographic studies included missed cancers. The reviewers did not know what percentage of the total films or women had cancer. Our analysis adjusted for this potential for increased recall.
The initial analysis was to estimate how often the reviewing radiologists would have assessed the specific breast as needing further workup. We evaluated the workup rates by radiologist and by film category (whether false-negative, false-positive, or true-negative). In our study, the cancers occurred in 14% of mammography films reviewed.
To estimate the detectable missed-cancer rate, we corrected the average false-negative workup rate for the review by subtracting the average workup rate for the true-negatives. By this subtraction, we produced a rate that corrected for the recall rate in true-negative breasts, acknowledging that some fraction of the recall in the false-negative breasts would have occurred if the breast had been true-negative. When adjusting the workup rates for the baseline among true-negatives, we used averages of the rates of the four reviewers. No routine statistical method could obtain an accurate 95% confidence interval (CI) for the percentage of missed cancers that were worked up on average by the four reviewers. We used a conservative worst case scenario with standard error calculations based on the assumption that the percentages were obtained from a single radiologist.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
|
Table 2 displays the percentage of the 678 breasts that was recommended for further workup by individual radiologist and film category. Overall, the workup rate for all breasts was 21%. The recommended workup rate for the reviewing radiologists differed significantly, ranging from 16% for reviewer 1 to 26% for reviewer 3, (chi-square test, p < 0.0001).
|
The four radiologists recommended further workup in the false-negative breasts an average of 42% of the time (range, 35-51%) and 13% of the time for true-negative mammograms (range, 9-17%), a statistically significant difference (p < 0.0001). Similar workup rates were found for true-negative mammograms regardless of whether they occurred in a woman in whom both breasts were assessed correctly as negative (true-negative women), both were assessed as negative with one incorrect (false-negative women), or one breast was assessed as negative and one as positive (false-positive women). The average workup rates were 12%, 15%, and 15%, respectively.
Correcting for the control workup rate among the true-negative breasts, we obtained an adjusted estimate of the proportion of detectable cancers among the missed cancers, which was 29% (95% CI, 20-40%). Among the participating mammography facilities, the screening workup rates ranged from 4% to 10%. We applied these background workup rates as the control and estimated that our reviewing radiologists would have worked up from 33-39% (29%, + 4% or +10%) of these missed cancers.
The number of radiologists who would have worked up the false-negative breasts (n = 93) was distributed as follows: 33% of the films would have not been worked up by any of the four reviewing radiologist; 20%, by one; 15%, by two; 12%, by three; and 20%, by all four. By contrast, these percentages for the true-negative breasts were 68%, 20%, 8%, 2%, and 2%, respectively. The majority (at least three of the four radiologists) recommended workup for 32% of false-negative breasts and 4% of true-negative breasts. The difference, 28%, is nearly identical to the estimated detectable missed-cancer rate of 29%.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The films were interpreted without the benefit of previous studies for comparison for practical reasons in a study that was time-intensive. The expected bias from this approach would be exaggerated recall in the experimental setting. Thus, our results in terms of agreement not to work up a screening mammogram are conservative at best. Another source of bias may result from inclusion of a few studies from each of the practices of the reviewers. For the analysis, we categorized agreement on the basis of the majority of assessments, thus avoiding a major effect on the results. If they had missed working up a cancer initially and now recommended working it up, that change would have resulted in raising the percentage of missed cancers that we classified as detectable.
The sensitivity of mammography has a wide range of reported values. In a review of 13 reports from the current literature for 1990-1999, the range for sensitivity was 68-92% [9,10,11,12,13,14,15,16,17,18,19]. The sensitivity of the Carolina Mammography Registry is 74%. It is difficult to compare sensitivities as reported in the literature for several reasons. Sensitivity estimates may differ on the basis of the distribution of patient characteristics (age and breast density in particular), proportion of the population that was symptomatic, length of screening interval, definition of a positive mammogram, accuracy of identifying (or estimating) false-negative cases, method of calculating sensitivity, and sample size of the population studied. In some published reports, it is not clear whether a recommendation for short-term follow-up is considered positive or negative when classifying a mammographic assessment. The Carolina Mammography Registry considers a recommendation for short-term follow-up to be a negative mammogram as suggested by the American College or Radiology BI-RADS manual [8].
Several studies have evaluated the reasons, both technical and interpretive, for missed cancers [20,21,22,23,24,25,26,27,28,29]. The technical reasons most often relate to radiographic technique, processing, and positioning but constitute only a small proportion of the missed lesions. Interpretation errors can be due to either failed perception or misinterpretation of an identified abnormality. Failed perception most often relates to the nature of the parenchyma. Misinterpretation is generally due to identification of a lesion on only one view, lack of change when compared with a prior study, location of a lesion at the site of a previous biopsy, or lesion characteristics that appear benign.
Two other studies, similar to ours, have reported the percentage of missed cancers that would be classified as detectable in an independent second review. Moberg et al. [30] reviewed a set of missed cancers from the Swedish screening program in Stockholm. There were 104 cases from two screening units. Sixty cases were studied two ways: one group with only the missed cancers and the second group with cancers in a 1:8 mix with noncancers. The radiologists noted the abnormality seen and the location as well as whether they would recommend further workup on the basis of the screening mammogram. Reviewing the missed cancers in the mixed set (1:8 mix with noncancers) resulted in workup rates of 15%, 15%, and 12% of the films for the three reviewers; the workup rates in the cancer-only grouping were higher at 27%, 27%, and 25%. These rates are similar to our rates for our true-negative films. Our mix based on the breast-specific analysis was approximately 7:1.
Vitak [31] had two external and two internal reviewers scrutinize 544 missed cancers to assess the estimates of these cancers that should have been detected among a group that formed 37% of the films for review. With two external reviewers, only 25% of the original missed cancers were identified for further workup. This result is close to our adjusted rate. We corrected the potential for excessive recommendations for workup from the experimental setting by using the mammograms of all normal breasts as controls. If the experimental situation led to excessive recommendations, we would see this excess in the noncancer breasts. We did see more workup recommendations in the review than is usual from the practices whose films were used.
In a representative busy practice in which a radiologist interprets about 6000 screening mammograms a year on average, there would be approximately 18-24 cancers. Cancer may be missed because cancers in any one practice are rare. If the sensitivity in that setting is 80%, then on average only 3.6-4.7 of the cancers would be missed. Of these few missed cancers in this hypothetic representative practice, applying the detectable rate of 29%, we believe that on average, 1.0-1.4 (fewer than 2) of the missed cancers in this practice would have been worked up by most peers who reviewed the films. At least three radiologists recommended further workup of the false-negative breasts in 30 (32%) of 93 cases compared with only 4% of the true-negative breasts. The difference of these two rates is 28%, nearly identical to the estimated detectable missed-cancer rate of 29%.
We found no study in the United States that reviewed missed cancers for the purpose of estimating the agreement in assessment among a group of peer radiologists from the same community. Most studies have been conducted for the purpose of understanding why cancers are missed. The estimates of cancers that were detectable among the missed cancers in our study (29%) were higher than those in the two cited previously [30, 31]. In our study, possibly the expertise and experience of the interpreting radiologists had a greater differential from the original interpreting radiologist than those in the European studies.
Our work had some limitations. We have tested only whether the woman would have been recommended for further workup. We do not know whether further workup would have led to a recommendation for biopsy. Findings on further evaluation, for some women, may have been benign, and the lesion, not recommended for biopsy. Thus, this detectable rate of missed cancers may be an overestimate of the actual rate of breasts with cancer that would have been diagnosed. Our comparison of the original and reviewing radiologist was based solely on the screening film, so the agreement analysis still is valid. Further studies could look at the completed workup. We also did not provide comparison films. The workup rates may have been lower with comparison films, and this potential difference makes our estimates of detectable cancers that were missed higher than may be the case.
We used experienced peer mammographers for our reviewing interpreting radiologists. The community of radiologists performing mammography will most often have varied experience, both in training and in amount of time spent interpreting mammograms. Establishing a detectable missed-cancer rate, however, seems best done by experienced community radiologists, whom we expected would have the greatest ability to discriminate between mammograms with normal findings and those that needed further evaluation. Had we used reviewing radiologists with more varied levels of experience, the estimate for the detectable rate for missed cancers could have been lower.
If the majority interpretations are used for defining who is worked up, then our study suggests that false-negatives might be reduced by about 28%, the percentage of false-negative mammograms recommended for workup by all, or all but one, of our reviewing radiologists. The literature has had mixed reviews on the value of double reading, though many studies have suggested that double reading increases sensitivity while lowering the recall rate [32,33,34,35,36]. Further research is necessary, perhaps as a randomized trial. In this study, the task was to blindly interpret the films, some of which revealed missed cancers; this work was not designed to comment on double reading of films. The message, rather, is that an estimated 71% of the missed cancers remain undetectable when interpreted independently by experienced peers.
It is important for both referring physicians and women undergoing mammography to understand that neither mammography nor the professionals who interpret mammograms are perfect. The bottom line is that a negative finding on mammography has a high probability of being correct, but occasionally will be incorrect. The odds are still in favor of the value of screening mammography.
Acknowledgments
We thank Brian Koon, Cheryl A. Viglione, and Claire Poyet for giving their
time and expertise in this review and Robert McLelland for his guidance and
expert assistance. We also thank Shen Zhuang and Yuhua Lin for their
programming assistance and Kara Gasink, Molly Blackley, Molly Heinzen,
Renée Kemske, Ann Williams, and Elsa
Desrochers for their administrative, organizational, and editorial assistance.
In addition, we thank the practices for their willingness to share their
mammograms and personnel who collated and sent the mammograms for this
research.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  H. Singh, S. Sethi, M. Raber, and L. A. Petersen Errors in Cancer Diagnosis: Current Understanding and Future Directions J. Clin. Oncol., November 1, 2007; 25(31): 5009 - 5018. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. J. Morton, D. H. Whaley, K. R. Brandt, and K. K. Amrami Screening Mammograms: Interpretation with Computer-aided Detection--Prospective Evaluation Radiology, May 1, 2006; 239(2): 375 - 383. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Mouchawar, S. Taplin, L. Ichikawa, W. E. Barlow, A. M. Geiger, S. Weinmann, J. Gilbert, M. M. Manos, and M. U. Yood Late-Stage Breast Cancer Among Women With Recent Negative Screening Mammography: Do Clinical Encounters Offer Opportunity for Earlier Detection? J Natl Cancer Inst Monographs, November 1, 2005; 2005(35): 39 - 46. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. B. Kopans Be Careful to Not Willingly Suspend Disbelief Radiology, December 1, 2004; 233(3): 645 - 647. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
