Interpretive Error in Radiology
Abstract
OBJECTIVE. Although imaging technology has advanced significantly since the work of Garland in 1949, interpretive error rates remain unchanged. In addition to patient harm, interpretive errors are a major cause of litigation and distress to radiologists. In this article, we discuss the mechanics involved in searching an image, categorize omission errors, and discuss factors influencing diagnostic accuracy. Potential individual- and system-based solutions to mitigate or eliminate errors are also discussed.
CONCLUSION. Radiologists use visual detection, pattern recognition, memory, and cognitive reasoning to synthesize final interpretations of radiologic studies. This synthesis is performed in an environment in which there are numerous extrinsic distractors, increasing workloads and fatigue. Given the ultimately human task of perception, some degree of error is likely inevitable even with experienced observers. However, an understanding of the causes of interpretive errors can help in the development of tools to mitigate errors and improve patient safety.
Error in diagnostic interpretation is an unfortunate and common occurrence exacerbated by the increasing demands placed on radiologists. In this article, we discuss how humans search scenes, the definition and cause of interpretative error, mitigating factors, and potential solutions.
Definition
When discussing error in diagnostic interpretation, it is important to distinguish between an error and observer variation. Radiology often involves decision-making without clear clinical presentations and imaging appearances that are not always diagnostic of a particular diagnosis. Therefore, the interpretation of a radiologic study is usually not a binary process (normal or abnormal) [1].
Interpretive error can be defined as an incorrect interpretation. However, because of the subjective nature of radiology, the definition of what is erroneous is established by expert opinion. In a true error, the discrepancy is substantially different from the consensus of one's peers [2].
Medical Error
In a 1999 report, the Institute of Medicine (IOM) estimated that 44,000–98,000 people die in U.S. hospitals each year as a result of preventable errors [5]. The IOM is considered to have underestimated the severity of the problem. More recent analysis based on subsequent studies calculated a mean rate of death from medical error of more than 251,000 patients per year, which suggests that medical error is the third most common cause of death in the United States after cancer and heart disease [6]. Radiologists' diagnostic interpretations have a tremendous impact on diagnostic error rates resulting from delayed or erroneous diagnosis [7].
Radiologic Interpretative Error
In 1949, Garland [8] found a 33.3% error rate in the interpretation of positive films based on group consensus opinion and an 8% intrareader variation (when a reader disagrees with him- or herself when rereading a study). This rate of error has remained virtually unchanged. In a mix of abnormal and normal studies representative of a typical clinical practice, the error rate is approximately 4% [7, 9]. Because 1 billion radiographic examinations are performed worldwide annually, a 4% error rate translates into approximately 40 million interpretive errors per year [2].
Fortunately, most errors are minor or, if significant, are found and corrected quickly enough to avoid serious harm to the patient. However, many interpretation errors do result in harm to patients and can be a humbling and frustrating experience for radiologists [10].
Despite advances in imaging technology, there is no evidence of improvement in the perceptual abilities of the human eye and brain [11]. Given this human limitation, many of the solutions developed to decrease interpretive error are technology based.
Cause of Interpretive Error
Perceptual errors account for 60–80% of interpretive errors, with variation depending on the modality and specific scenario (e.g., outpatients vs emergency patients) [12, 13]. Because the first step in image interpretation is detection, an error in perception can prematurely end the diagnostic process and lead to missed diagnoses [14]. For an interpretive error to be considered a perceptual error, the finding should be sufficiently conspicuous and detectable in retrospect by interpreting radiologists or in the consensus of their peers [2]. Not all subtle and insubstantial findings subsequently found to represent a pathologic process are therefore considered perceptual errors [2].
The intrinsic characteristics of lesions influence their detectability or conspicuity and depend on the physical properties of the lesion and its surrounding structures. Contrast is a composite property of the lesion and its surroundings. The greater the difference between the two, the better the lesion visibility [15]. Small nodule size, low lesion attenuation, and ill-defined margins are lowconspicuity characteristics on chest radiography [16].
In addition to intrinsic lesion characteristics, there are a number of both intrinsic and extrinsic factors that can contribute to the likelihood of an interpretative error (Fig. 1).
Scene Processing
Knowledge of how humans analyze and process scenes is required to understand perceptual and interpretive error. When scanning the environment, the eyes make jerky movements called “saccades” interleaved with fixation periods [17]. Frequent saccades are made to capture detailed snapshots with the fovea (the central part of the retina), which has sufficient photoreceptor density to provide high-resolution vision [17]. Vision depends on information obtained during fixation pauses between saccades because no useful visual information is obtained while the eyes undergo saccades [18].
It is postulated that when analyzing a radiologic study, there is rapid identification of abnormalities using peripheral vision with subsequent scrutiny utilizing central vision [19]. Radiologists compare this “gist” impression with information contained in long-term memory that forms the viewer's cognitive schema (or expectations) of what information is in an image. This rapid response is shown when a radiologist identifies subtle abnormalities on mammography and chest radiography in only 250 ms [20–23].
A second systemic scan then occurs, allowing specific object recognition using central foveal vision [22]. Features are examined closely and are tested against the cognitive schema to determine whether a finding is suspicious. Decisions are made once image features sufficiently match the cognitive scheme of the viewer [23]. This pathway, lasting seconds to minutes, is capacity-limited and is the bottleneck of attention [19, 22].
Omission Errors
Eye-tracking technology has been used to classify omission or false-negative errors into three categories on the basis of fixation times on missed lesions: search, recognition, and decision-making.
Search Error
This type of error occurs when the observer never fixates on the lesion with high-resolution foveal vision precluding processing [24–26] (Figs. 2 and 3).
Fig. 2A —70-year-old man with no significant medical history.
A, Search-related omission error. Routine chest radiograph was correctly interpreted as normal.
Fig. 2B —70-year-old man with no significant medical history.
B, Search-related omission error. Patient presented 8 months later with complaints of left arm pain. Chest radiograph was interpreted as normal; however, left apical mass (arrow) was missed. Apices are common blind spot in interpretation, and radiologist reported not looking in this region.
Fig. 2C —70-year-old man with no significant medical history.
C, Discrimination error. Patient presented 1 month after B with persistent symptoms. Chest radiograph interpreted as “unchanged left apical pleural-based opacity, likely pleural thickening.” However, opacity (arrows) is larger compared with size on prior examination (B), is markedly asymmetric, and has convex borders suggestive of malignancy. This is example of discrimination error rather than omission error because finding was noted but was incorrectly interpreted. Imaging performed 3 months later (not shown) showed further interval growth with bone destruction. Pathologic diagnosis was squamous cell carcinoma, Pancoast tumor.
Recognition Error
A recognition error is a failure of the basic mechanism of object recognition; the radiologist fixates on the target for a duration shorter than the threshold dwell time considered sufficient to recognize lesion features [25]. The threshold for lesion detection depends on the imaging modality and ranges from 500 to 1000 ms [25, 27]. Both search and recognition errors are considered perceptual errors [28].
Decision-Making Error
Attention and Perception
Failure of attention is one source of error in radiology. Because humans cannot process everything in the visual field, attention is drawn to objects that have prominent intrinsic features. This is referred to as bottom-up processing. Color, motion, orientation, and size are among the attributes that guide visual search and are called “preattentive,” meaning that they are perceived rapidly without any conscious effort [29, 30]. In top-down guidance, the searcher has a representation of a target in mind and directs attention to items that have those features. Learning, memory, attention, and expectation shape this perceptual process. [29]. Scene guidance is learned and is based on the understanding of a scene's contents and layout.
Differences in attentional processing explain the variations in search patterns between experts and novices. Expert radiologists know where to look for nodules, which limits inspection of many irrelevant areas [29]. Other regions may not be scrutinized because they lack concerning preattentive attributes.
Observers can fail to note even seemingly obvious findings that they are not actively paying attention to. To determine whether this phenomenon, termed “inattentional blindness,” occurs in expert searchers, Drew et al. [31] created a pixelated gorilla (48 times the size of a 5-mm nodule) with a white outline and inserted it into consecutive CT images while radiologists performed a nodule detection task. Twenty of 24 (83%) radiologists failed to notice the gorilla even though 60% directly fixated on it [31]. Although radiologists outperformed naïve observers (0% detection rate), their 83% miss rate received significant media attention [32, 33]. Inattentional blindness has also been implicated in the failure of 60% of radiologists to notice a missing clavicle on a chest radiograph when denied relevant history. When given the indication that the chest radiograph was part of a metastatic survey, 83% of the observers found the abnormality [34].
Focused attention is considered a fundamental feature of the human brain. It is regarded as an inherent limitation of the human “search engine”; therefore, inattentional blindness cannot be entirely prevented [30].
Cognitive Biases
Cognitive biases are logic fallacies that are often triggered by a particular situation that can lead to error. These biases are largely the result of nonanalytic reflexive thinking and can affect diagnostic interpretation [24, 35–39] (Table 1 and Figs. 4–7).
| Cognitive Bias | Description | Corrective Strategies |
|---|---|---|
| Anchoring | Failing to adjust initial impression in light of contrary information. Associated with confirmation bias in which clinicians modify interpretation of subsequent information to suit initial predictions to support their hypothesis (Fig. 4). | Seek to disprove initial diagnosis rather than to confirm it. Avoid early guesses and reconsider diagnosis with worsening symptoms. |
| Framing | Radiologists are influenced by the way a problem is worded or framed. The diagnostic possibilities are mentally restricted by the referral situation (Figs. 4 and 5). | Initially read study objectively without reviewing clinical history. Remember that clinicians may have biases and that even a strongly held clinical interpretation may be incorrect. |
| Availability | The tendency to consider some diagnoses to be more likely if they readily come to mind. This bias may be especially frequent after a known diagnostic miss (Fig. 6). | Be aware of the tendency to overestimate the frequency of previously missed or memorable cases. Objective information of the true base rate of disease should be used when possible. |
| Alliterative | Results from the influence radiologists have on each other. Previous interpretations (even from the same reader) influence the interpretation of the current examination (Fig. 7 and 8). | Attempt to increase diagnostic possibilities to break tendency to simply repeat what was previously reported. When possible, read old reports after new interpretation. |
Note—These errors occur secondary to mental shortcuts inherent in the heuristic intuitive method of diagnosis. Because these biases are unconscious, they are notoriously difficult to recognize and avoid; however, awareness may decrease susceptibility.
Fig. 4A —Cognitive error due to anchoring and framing biases.
A, Axial CT images of cervical spine (A) and head (B and C) of trauma patient (72-year-old man). Emergency department provider insisted patient was asymptomatic (anchoring bias), which prompted interpreting radiologist to attribute fracture of right lamina (circle, A) to sequela of degenerative disease (framing bias). However, on reviewing head CT, which showed interhemispheric subdural hematoma (ovals, B and C), radiologist corrected his report to reflect that this fracture was acute fracture.
Fig. 4B —Cognitive error due to anchoring and framing biases.
B, Axial CT images of cervical spine (A) and head (B and C) of trauma patient (72-year-old man). Emergency department provider insisted patient was asymptomatic (anchoring bias), which prompted interpreting radiologist to attribute fracture of right lamina (circle, A) to sequela of degenerative disease (framing bias). However, on reviewing head CT, which showed interhemispheric subdural hematoma (ovals, B and C), radiologist corrected his report to reflect that this fracture was acute fracture.
Fig. 4C —Cognitive error due to anchoring and framing biases.
C, Axial CT images of cervical spine (A) and head (B and C) of trauma patient (72-year-old man). Emergency department provider insisted patient was asymptomatic (anchoring bias), which prompted interpreting radiologist to attribute fracture of right lamina (circle, A) to sequela of degenerative disease (framing bias). However, on reviewing head CT, which showed interhemispheric subdural hematoma (ovals, B and C), radiologist corrected his report to reflect that this fracture was acute fracture.
Fig. 6A —Availability bias.
A, Axial (A) and sagittal (B) CT images of 50-year-old man with history of nodule seen at left base on chest radiography (not shown). CT study was interpreted as showing left lower lobe subpleural nodule “concerning for neoplasm.” However, this nodule shows classic characteristics of round atelectasis. Radiologist reported missing lung cancer day before this interpretation, predisposing her to diagnose lung cancer.
Fig. 6B —Availability bias.
B, Axial (A) and sagittal (B) CT images of 50-year-old man with history of nodule seen at left base on chest radiography (not shown). CT study was interpreted as showing left lower lobe subpleural nodule “concerning for neoplasm.” However, this nodule shows classic characteristics of round atelectasis. Radiologist reported missing lung cancer day before this interpretation, predisposing her to diagnose lung cancer.
Fig. 7A —Prevalence effect and alliterative error.
A, Posteroanterior (A) and lateral (B) chest radiographs of 20-year-old man who presented with cough, fever, and chest pain. Official interpretation was right-sided ventriculoperitoneal (VP) shunt (arrows).
Fig. 7B —Prevalence effect and alliterative error.
B, Posteroanterior (A) and lateral (B) chest radiographs of 20-year-old man who presented with cough, fever, and chest pain. Official interpretation was right-sided ventriculoperitoneal (VP) shunt (arrows).
Fig. 7C —Prevalence effect and alliterative error.
C, Repeat posteroanterior chest radiograph obtained weeks after A and B was interpreted as “unchanged right-sided VP shunt.” However, structure is too medial, lacks lumen, and is not in subcutaneous tissues (A) as would be expected with shunt. Findings are consistent with retained guidewire. Interventional radiologist retrieved guidewire, confirming diagnosis. In most cases, thin line overlying patient on frontal view is VP shunt and is therefore first consideration (low prevalence effect). Cognition needed to note correct nature of this finding may not be given in time-pressured environment. Interpretation of repeat chest radiograph deferred to opinion of prior reader (alliterative error).
Satisfaction of Search
Initially characterized as the discontinuation of a visual search once the searcher finds an abnormality and becomes “satisfied” with the “meaning” of an image, satisfaction of search (SOS) occurs when interpreting a number of different imaging modalities [40–43] (Fig. 8).
Fig. 8A — Satisfaction of search (SOS) and alliterative error.
A, 78-year-old woman presented to emergency department with nonlocalized chest pain and history of lung cancer. Reading of initial radiograph was “bilateral perihilar and right midlung opacities, suggestive of underlying mass and hilar adenopathy.” Significant left-sided pneumothorax (arrows) was missed. On review, radiologists thought that pneumothorax would not have been missed without presence of distracting right-sided opacity; interpretive error in this case is consistent with SOS error.
Fig. 8B — Satisfaction of search (SOS) and alliterative error.
B, Interpretation of portable postintubation radiograph obtained 3 hours after A also failed to note left-sided pneumothorax (arrow). Cognitive failure identified is alliterative error. In this case, dominant abnormality (right lung mass) was mentioned by prior radiologist; therefore, subsequent readers only looked for changes such as lines and tubes.
Since the original 1990 study showing the SOS phenomenon, radiologic imaging has become computer-based to allow greater flexibility in adjusting window and level settings to optimize search. Furthermore, readers are now more aware of the SOS phenomenon. A recent study by Berbaum et al. [44] using modern computed radiography of the chest noted no SOS effect. It is not certain whether there have been similar changes in this effect on other imaging modalities, and further research in this area is needed.
Prevalence Effect
The prevalence effect refers to the relationship between the prevalence of a particular abnormality and observer performance. For example, there may be an increase in false-positives readings on the chest radiographs of known smokers given their increased prevalence of nodules.
Reed et al. [45] found that as the prevalence expectation for nodules increased radiologists had an increase in the number of fixations per image and the overall analysis time increased. These results show a change in visual search in response to prevalence expectations. When Evans et al. [46] manipulated the prevalence levels of breast cancer on mammograms, omission errors increased in the low-prevalence scenario. In conditions of low prevalence, observers are more likely to reject ambiguous findings and terminate a search more quickly [29] (Fig. 7).
External Factors Contributing to Interpretive Error
Clinical History
The presence of clinical history may affect a radiologist's level of interpretive error. Conceptually, clinical information can improve accuracy during both perception and interpretation, focusing perception on an area of interest or changing the level of interpretive suspicion [47]. An argument against providing clinical information is that history could bias radiographic interpretation and cause readers to perceive abnormalities that are not present [47].
There have been a number of studies analyzing the effect of clinical history on interpretive accuracy with conflicting results depending on the particular experimental condition. Doubilet and Herman [48] and Berbaum et al. [49] found increased interpretative accuracy when radiologists were given relevant history. On the contrary, Good et al. [50] found that clinical history had no effect on the accuracy of chest radiograph interpretations. In an analysis of 16 articles comparing the accuracy of tests with and those without clinical information, Loy and Irwig [47] found that clinical information improves interpretive accuracy through improved sensitivity without a loss of specificity consistent with readers being alerted to additional imaging features rather than merely altering their level of suspicion. Leslie et al. [51] found that providing clinical history resulted in changes in 19% of CT reports with more than half being major.
Accurate clinical information also ensures that the appropriate study is performed, helps radiologists avoid spending time and effort searching for findings irrelevant in the clinical context, and facilitates optimal diagnostic workup [4, 37]. These factors alone outweigh any potential bias from knowledge of the clinical scenario [4]. Moreover, no proven deleterious effect has been documented by providing an accurate clinical history [47].
Fatigue and Error
Krupinski et al. [52, 53] found increased subjective fatigue and decreased accuracy on both conventional radiography and CT interpretation after a day of reading. Specific work shifts may exacerbate interpretive error rates. Although there is minimal research regarding the effects of after-hours work (usually defined as between 5 pm and 8 am) on interpretative accuracy, there is concern that circadian misalignments during these shifts can result in sleepiness and chronic fatigue and adversely affect performance [54].
Workload, Interpretative Speed, and Error
McDonald et al. [55] found that the average number of images requiring interpretation per minute has increased nearly sevenfold per radiologist from 1999 to 2010. There is a concern that this drastic increase in imaging content can increase error secondary to radiologists' fatigue and stress because they may take fewer breaks to keep up with the increased workload [52, 55, 56].
Rapid film interpretation speed can also be a source of error. According to Oestmann et al. [19], the accuracy in lung cancer detection decreases significantly with viewing times of less than 4 seconds. Sokolovskaya et al. [57] reported that when radiologists were asked to interpret studies at twice their baseline speed, the number of major misses increased from 10% to 26.6%. Because reimbursement in many practices is based on productivity, there is a strong financial incentive to read as many studies—without breaks—as possible. A sustained artificially high interpretative rate can result in additional interpretive errors from both general and oculomotor fatigue.
Distractions and Error
Radiologists spend a significant amount of time toggling between medical tasks such as examination interpretation, consulting with referring physicians, and nonmedical functions such as answering telephone calls and returning pages [58]. Multitasking has the potential to introduce errors [59]. Correlating the number of telephone calls received by an overnight on-call resident with resident–attending discordant interpretations, Balint et al. [58] found that in the 1 hour preceding a discordant preliminary report, a single additional telephone call above the baseline increased the odds of a major discrepancy by 12% (Fig. 9).
Fig. 9A —Distraction error.
A, Posteroanterior (A) and oblique (B) radiographs of wrist and hand of 32-year-old man show subluxation of fifth carpometacarpal articulation (oval, B) missed on initial interpretation. Radiologist recalled being on telephone with clinician while interpreting this study.
Fig. 9B —Distraction error.
B, Posteroanterior (A) and oblique (B) radiographs of wrist and hand of 32-year-old man show subluxation of fifth carpometacarpal articulation (oval, B) missed on initial interpretation. Radiologist recalled being on telephone with clinician while interpreting this study.
Role of Specialization
The interplay of the particular imaging modality and the specialty of the interpreter is an additional factor to consider when developing mechanisms to reduce error rates. Several studies show that subspecialists have improved accuracy in their relative fields of expertise compared with general radiologists [60–62].
Potential Solutions
Many of the solutions proposed to reduce interpretative error are technology-based because computers are not subject to human limitations. Other solutions attempt to optimize conditions to facilitate interpretative accuracy. Because diagnostic errors have differing causes, proposed solutions will have different efficacies depending on the particular situation and interpreter.
Nontechnologic Solutions
Structured Reports
Structured reporting was developed to enhance communication with referring physicians and improve interpretative efficiency. A well-constructed structured report template can serve as a checklist that facilitates systematic inspection of relevant anatomy and potentially decreases inattention-related errors [63].
Optimization Ergonomics
Optimal lighting and reduction of physical stressors by optimizing the ergonomics of PACS stations can improve the reading experience and can potentially decrease interpretation errors [64].
Rohatgi et al. [54] describe strategies for radiologists to minimize associated adverse health effects when reading after hours. Suggestions included naps, appropriate light exposure, social interaction, and caffeine [54]. Additional radiology-specific research is needed to understand the role of after-hours coverage on interpretive error and attrition.
Interruption Reduction
Not all interruptions are harmful. Therefore, when designing methods to minimize them, it is important to remember that the goal is not to eliminate all interruptions but to eliminate those that are uninformative and that occur during error-prone scenarios such as during active imaging interpretation [59]. Interruptions may be avoided by providing radiologists access to clinical data at the PACS workstation and portals for referring physicians [59]. Clinical assistants and caller ID systems can further ensure that radiologists attend to distractions that warrant immediate attention.
Double Reading
A long-recognized method to reduce error in interpretation is to have “films interpreted independently by two readers” [8]. Double reading is not practiced consistently in the United States because it is time-consuming and the second read is not reimbursed [65]. Because of the time commitment and lack of financial compensation, double reading should be reserved for complex cases in which a second opinion will provide a substantial benefit. Furthermore, the process must be rapid, and mechanisms to reconcile discrepancies between readers should be clearly defined [66].
Peer Review and Quality Improvement
The traditional punitive approach to quality improvement is to treat all interpretive errors as unique events attributable to individual radiologists. Unfortunately, this model captures only a small fraction of errors and does not result in substantial improvement in overall performance [67]. In an excellent review, Larson et al. [68] outline a modern nonpunitive approach to performance improvement with the expectation that this approach will enable a shift in culture from self-protection to shared responsibility. In this model clinicians may readily disclose even personal mistakes and view feedback as a learning experience [68].
Rosenkrantz and Bansal [13] showed that analysis of report addenda may serve as an additional easily searchable source of error characterization, although it is limited by the fact that these errors were detected and corrected.
Technologic Solutions
Oculomotor dynamics and visual search characteristics can be quantitated and analyzed using eye-tracking technology. Once technologies such as perceptual feedback, attentional guidance, and search pattern analysis become more clinically practical, they can play a prominent role in error reduction. Computer-aided diagnosis and methods of improving lesion conspicuity are other currently available technologic solutions.
Perceptual Feedback
Eye-tracking technology has shown that the average dwell time (i.e., the total fixation time) on false-negative nodules is almost as long as the average dwell time for nodules reported as positive. This suggests that an active decision was made to call the region negative [69, 70]. This understanding led to the development of a perceptual-based method of enhancing nodule recognition in which areas of interest receiving dwell times more than a specified temporal threshold (e.g., > 1 second) are circled, and radiologists are permitted a second look [71]. A 14–16% increase in true-positive nodule detection is seen in readers who received this feedback with no change in specificity [71–73].
Attentional Guidance
Because humans are highly sensitive to other peoples' eye movements, there is interest in determining whether presenting the eye movement patterns of one observer to a subsequent observer improves performance in a range of visual search and problem-solving tasks [74]. Litchfield et al. [74] found that novice radiologists improved performance after watching the search patterns of both experts and novices when they searched for nodules. Subsequent experiments showed that viewing unrelated eye movement patterns did not lead to performance improvement. Therefore, better performance was secondary to guiding the observer toward task-relevant areas. Expert radiologists, however, did not consistently show improvement on nodule search tasks after eye movement previews suggesting that this guidance may be most useful during the early stages of learning before the development of an effective visual search [74].
Search Strategies
Compared with experts, novices have more fixations, saccades, and greater coverage of images despite worse accuracy. Also, a recent study of radiologists at different levels of training showed that improvements in search did not parallel diagnostic accuracy [75]. Additional research can further elucidate optimal training methods and reveal clinically practical interpretation methods to reduce error rates.
Bone Subtraction Techniques
Overlying bones are a major contributor to anatomic noise on chest radiographs and are an important cause of missed lung nodules [76]. Several techniques (software- and hardware-based) have been proposed to suppress overlying bone structures, thereby increasing lesion conspicuity by reducing surrounding anatomic noise [76–78]. Digital removal of bone shows promise because no additional equipment is required and there is no additional radiation exposure. Several studies show a statistically significant increase in observer sensitivity for pulmonary nodules and infections of up to 17% [76, 79, 80] (Fig. 10).
Fig. 10A —50-year-old woman with no significant medical history. (Courtesy of Riverain Technologies, Miamisburg, OH)
A, Routine chest radiograph. Study was read as normal; however, there is low-conspicuity poorly marginated density (arrow) in right mid upper lung.
Fig. 10B —50-year-old woman with no significant medical history. (Courtesy of Riverain Technologies, Miamisburg, OH)
B, Bone suppression image. Lack of superimposed ribs on this image shows lesion (arrow) more clearly.
Fig. 10C —50-year-old woman with no significant medical history. (Courtesy of Riverain Technologies, Miamisburg, OH)
C, Combined computer-aided detection (CAD) and bone suppression image. CAD placed true-positive mark on nodule (circle) for radiologist to review. This nodule proved to be early-stage adenocarcinoma at pathology.
Computer-Assisted Detection
Computer-aided detection (CAD) refers to pattern recognition software that flags suspicious features on an image in an attempt to decrease false-negative readings [76, 81]. The radiologist reviews the examination and the CAD-marked areas of concern before issuing a final report [81] (Fig. 10). CAD systems do not mark all actionable findings; therefore, the absence of a CAD mark on a finding should not preclude evaluation. In addition, current CAD systems generate more false findings than true findings. The radiologist must determine whether a CAD mark warrants further evaluation [81]. The difficulty in discriminating between true- and false-positive marks is the biggest current challenge in CAD software. CAD is currently used and studied most widely in mammography but is also used in chest imaging and other modalities.
Chest imaging—CAD in chest imaging has improved significantly since its inception. The number of false-positive marks on chest radiographs has decreased from 15 per image in the 1970s to 1.3 per image in 2015 [76, 82].
There have been fewer studies assessing the utility of CAD for chest CT interpretation. White et al. [83] found a 1.9% increase in accuracy for readers using CAD on chest CT examinations.
CAD has been advanced as a “second reader” for acute clinical scenarios when the focus of attention may be on more critical findings and for high-demand readings such as lung cancer screening [83–85].
Mammography—There is a long history of CAD use in mammography with mixed results. Initial studies were promising and led to U.S. Food and Drug Administration approval in 1998. More recent studies, however, question its clinical utility. In 2007, Fenton et al. [86] found no improvement in sensitivity in facilities using CAD and a significant decrease in positive predictive value. Lehman et al. [87] in 2015 corroborated those findings and noted that diagnostic accuracy did not improve with CAD on any performance metric (sensitivity, specificity, or overall cancer detection rate). Therefore, CAD in mammography is associated with an increased rate of harms associated with screening including higher recall and biopsy rates and is of unclear clinical benefit [86].
Summary
Radiologists use visual detection, pattern recognition, memory, and cognitive reasoning to synthesize final interpretations of radiologic studies. This synthesis is performed in an environment in which there are numerous extrinsic distractors increasing workloads and fatigue. Given the ultimately human task of perception, some degree of error is likely inevitable even with experienced observers. However, an understanding of the causes of interpretive errors can help in the development of tools to mitigate errors and improve patient safety.
References
1.
Brady A, Laoide RO, McCarthy P, McDermott R. Discrepancy and error in radiology: concepts, causes and consequences. Ulster Med J 2012; 81:3–9
2.
Bruno MA, Walker EA, Abujudeh HH. Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction. RadioGraphics 2015; 35:1668–1676
3.
Fitzgerald R. Radiological error: analysis, standard setting, targeted instruction and teamworking. Eur Radiol 2005; 15:1760–1767
4.
Robinson PJ. Radiology's Achilles' heel: error and variation in the interpretation of the Röntgen image. Br J Radiol 1997; 70:1085–1098
5.
Kohn LT, Corrigan JM, Donaldson MS, eds. To err is human: building a safer health system. Washington, DC: National Academy Press, 2000
6.
Makary MA, Daniel M. Medical error: the third leading cause of death in the US. BMJ 2016; 353:i2139
7.
Sabih DE, Sabih A, Sabih Q, Khan AN. Image perception and interpretation of abnormalities; can we believe our eyes? Can we do something about it? Insights Imaging 2011; 2:47–55
8.
Garland LH. On the scientific evaluation of diagnostic procedures. Radiology 1949; 52:309–328
9.
Berlin L. Accuracy of diagnostic procedures: has it improved over the past five decades? AJR 2007; 188:1173–1178
10.
Ropp A, Waite S, Reede D, Patel J. Did I miss that: subtle and commonly missed findings on chest radiographs. Curr Probl Diagn Radiol 2015; 44:277–289
11.
Kim YW, Mansfield LT. Fool me twice: delayed diagnoses in radiology with emphasis on perpetuated errors. AJR 2014; 202:465–470
12.
Funaki B, Szymski G, Rosenblum J. Significant on-call misses by radiology residents interpreting computed tomographic studies: perception versus cognition. Emerg Radiol 1997; 4:290–294
13.
Rosenkrantz AB, Bansal NK. Diagnostic errors in abdominopelvic CT interpretation: characterization based on report addenda. Abdom Radiol (NY) 2016; 41:1793–1799
14.
Gray JE, Taylor KW, Hobbs BB. Detection accuracy in chest radiography. AJR 1978; 131:247–253
15.
Kundel HL, Revesz G. Lesion conspicuity, structured noise, and film reader error. AJR 1976; 126:1233–1238
16.
Wu MH, Gotway MB, Lee TJ, et al. Features of non-small cell lung carcinomas overlooked at digital chest radiography. Clin Radiol 2008; 63:518–528
17.
McCamy MB, Otero-Millan J, Di Stasi LL, Macknik SL, Martinez-Conde S. Highly informative natural scene regions increase microsaccade production during visual scanning. J Neurosci 2014; 34:2956–2966
18.
Ibbotson M, Krekelberg B. Visual perception and saccadic eye movements. Curr Opin Neurobiol 2011; 21:553–558
19.
Oestmann JW, Greene R, Kushner DC, Bourgouin PM, Linetsky L, Llewellyn HJ. Lung lesions: correlation between viewing time and detection. Radiology 1988; 166:451–453
20.
Evans KK, Georgian-Smith D, Tambouret R, Birdwell RL, Wolfe JM. The gist of the abnormal: above-chance medical decision making in the blink of an eye. Psychon Bull Rev 2013; 20:1170–1175
21.
Kundel HL, Nodine CF. Interpreting chest radiographs without visual search. Radiology 1975; 116:527–532
22.
Drew T, Evans K, Vo ML, Jacobson FL, Wolfe JM. Informatics in radiology: what can you see in a single glance and how might this guide visual search in medical images? RadioGraphics 2013; 33:263–274
23.
Krupinski EA. The role of perception in imaging: past and future. Semin Nucl Med 2011; 41:392–400
24.
Krupinski EA. Current perspectives in medical image perception. Atten Percept Psychophys 2010; 72:1205–1217
25.
Krupinski EA, Nodine CF, Kundel HL. Enhancing recognition of lesions in radiographic images using perceptual feedback. Opt Eng 1998; 37:813–818
26.
Nodine CF, Kundel HL. Using eye movements to study visual search and to improve tumor detection. RadioGraphics 1987; 7:1241–1250
27.
Carmody DP, Nodine CF, Kundel HL. An analysis of perceptual and cognitive factors in radiographic interpretation. Perception 1980; 9:339–344
28.
Tourassi G, Voisin S, Paquit V, Krupinski E. Investigating the link between radiologists' gaze, diagnostic decision, and image content. J Am Med Inform Assoc 2013; 20:1067–1075
29.
Wolfe JM, Evans KK, Drew T, Aizenman A, Josephs E. How do radiologists use the human search engine? Radiat Prot Dosimetry 2016; 169:24–31
30.
Wolfe JM, Horowitz TS. What attributes guide the deployment of visual attention and how do they do it? Nat Rev Neurosci 2004; 5:495–501
31.
Drew T, Vo ML, Wolfe JM. The invisible gorilla strikes again: Sustained inattentional blindness in expert observers. Psychol Sci 2013; 24:1848–1853
32.
National Public Radio website. Spiegel A. Why even radiologists can miss a gorilla hiding in plain sight. www.npr.org/sections/health-shots/2013/02/11/171409656/why-even-radiologists-can-miss-a-gorilla-hiding-in-plain-sight. Published February 11, 2013. Accessed December 9, 2016
33.
The Washington Post website. Kliff S. Study: most radiologists don't notice a gorilla in a CT scan. https://www.washingtonpost.com/news/wonk/wp/2013/02/19/study-most-radiologists-dont-notice-a-gorilla-in-a-ct-scan/. Published February 19, 2013. Accessed December 9, 2016
34.
Potchen EJ. Measuring observer performance in chest radiology: some experiences. J Am Coll Radiol 006; 3:423–432
35.
Lee CS, Nagy PG, Weaver SJ, Newman-Toker DE. Cognitive and system factors contributing to diagnostic errors in radiology. AJR 2013; 201:611–617
36.
Croskerry P. From mindless to mindful practice: cognitive bias and clinical decision making. N Engl J Med 2013; 368:2445–2448
37.
Berlin L. Malpractice issues in radiology: alliterative errors. AJR 2000; 174:925–931
38.
Kanza RE, Allard C, Berube M. Cardiac findings on non-gated chest computed tomography: a clinical and pictorial review. Eur J Radiol 2016; 85:435–451
39.
Lim KY, Kligerman SJ, Lin CT, White CS. Missed pulmonary embolism on abdominal CT. AJR 2014; 202:738–743
40.
Fleck MS, Samei E, Mitroff SR. Generalized “satisfaction of search”: adverse influences on dual-target search accuracy. J Exp Psychol Appl 2010; 16:60–71
41.
Ashman CJ, Yu JS, Wolfman D. Satisfaction of search in osteoradiology. AJR 2000; 175:541–544
42.
Berbaum KS, Franken EA Jr. Satisfaction of search in radiographic modalities. Radiology 2011; 261:1000–1001; author reply, 1001–1002
43.
Berbaum KS, Schartz KM, Caldwell RT, et al. Satisfaction of search from detection of pulmonary nodules in computed tomography of the chest. Acad Radiol 2013; 20:194–201
44.
Berbaum KS, Krupinski EA, Schartz KM, et al. Satisfaction of search in chest radiography 2015. Acad Radiol 2015; 11:1457–1466
45.
Reed WM, Ryan JT, McEntee MF, Evanoff MG, Brennan PC. The effect of abnormality-prevalence expectation on expert observer performance and visual search. Radiology 2011; 258:938–943
46.
Evans KK, Birdwell RL, Wolfe JM. If you don't find it often, you often don't find it: why some cancers are missed in breast cancer screening. PLoS One 2013; 8:e64366
47.
Loy CT, Irwig L. Accuracy of diagnostic tests read with and without clinical information: a systematic review. JAMA 2004; 292:1602–1609
48.
Doubilet P, Herman PG. Interpretation of radiographs: effect of clinical history. AJR 1981; 137:1055–1058
49.
Berbaum KS, el-Khoury GY, Franken EA Jr, Kathol M, Montgomery WJ, Hesson W. Impact of clinical history on fracture detection with radiography. Radiology 1988; 168:507–511
50.
Good BC, Cooperstein LA, DeMarino GB, et al. Does knowledge of the clinical history affect the accuracy of chest radiograph interpretation? AJR 1990; 154:709–712
51.
Leslie A, Jones AJ, Goddard PR. The influence of clinical information on the reporting of CT by radiologists. Br J Radiol 2000; 73:1052–1055
52.
Krupinski EA, Berbaum KS, Caldwell RT, Schartz KM, Kim J. Long radiology workdays reduce detection and accommodation accuracy. J Am Coll Radiol 2010; 7:698–704
53.
Krupinski EA, Berbaum KS, Caldwell RT, Schartz KM, Madsen MT, Kramer DJ. Do long radiology workdays affect nodule detection in dynamic CT interpretation? J Am Coll Radiol 2012; 9:191–198
54.
Rohatgi S, Hanna TN, Sliker CW, Abbott RM, Nicola R. After-hours radiology: challenges and strategies for the radiologist. AJR 2015; 205:956–961
55.
McDonald RJ, Schwartz KM, Eckel LJ, et al. The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload. Acad Radiol 2015; 22:1191–1198
56.
Vertinsky T, Forster B. Prevalence of eye strain among radiologists: influence of viewing variables on symptoms. AJR 2005; 184:681–686
57.
Sokolovskaya E, Shinde T, Ruchman RB, et al. The effect of faster reporting speed for imaging studies on the number of misses and interpretation errors: a pilot study. J Am Coll Radiol 2015; 12:683–688
58.
Balint BJ, Steenburg SD, Lin H, Shen C, Steele JL, Gunderman RB. Do telephone call interruptions have an impact on radiology resident diagnostic accuracy? Acad Radiol 2014; 21:1623–1628
59.
Grundgeiger T, Sanderson P. Interruptions in healthcare: theoretical views. Int J Med Inform 2009; 78:293–307
60.
Briggs GM, Flynn PA, Worthington M, Rennie I, McKinstry CS. The role of specialist neuroradiology second opinion reporting: is there added value? Clin Radiol 2008; 63:791–795
61.
Bell ME, Patel MD. The degree of abdominal imaging (AI) subspecialization of the reviewing radiologist significantly impacts the number of clinically relevant and incidental discrepancies identified during peer review of emergency after-hours body CT studies. Abdom Imaging 2014; 39:1114–1118
62.
Sickles EA, Wolverton DE, Dee KE. Performance parameters for screening and diagnostic mammography: specialist and general radiologists. Radiology 2002; 224:861–869
63.
Marcovici PA, Taylor GA. Structured radiology reports are more complete and more effective than unstructured reports. AJR 2014; 203:1265–1271
64.
Siewert B, Sosna J, McNamara A, Raptopoulos V, Kruskal JB. Missed lesions at abdominal oncologic CT: lessons learned from quality assurance. RadioGraphics 2008; 28:623–638
65.
Onega T, Aiello Bowles EJ, Miglioretti DL, et al. Radiologists' perceptions of computer aided detection versus double reading for mammography interpretation. Acad Radiol 2010; 17:1217–1226
66.
Imaging Technology News website. Yates T. The antidote for radiology errors. www.itnonline.com/article/antidote-radiology-errors. Published June 1, 2012. Accessed December 9, 2016
67.
Kaewlai R, Abujudeh H. Peer review in clinical radiology practice. AJR 2012; 199:[web]W158–W162
68.
Larson DB, Kruskal JB, Krecke KN, Donnelly LF. Key concepts of patient safety in radiology. RadioGraphics 2015; 35:1677–1693
69.
Krupinski E. Perceptual factors in reading medical images. In: Samei E, Krupinski E, eds. The handbook of medical image perception and techniques. Cambridge, UK: Cambridge University Press: 2010:81–90
70.
Kundel HL, Nodine CF, Krupinski EA. Searching for lung nodules: visual dwell indicates locations of false-positive and false-negative decisions. Invest Radiol 1989; 24:472–478
71.
Krupinski EA, Nodine CF, Kundel HL. A perceptually based method for enhancing pulmonary nodule recognition. Invest Radiol 1993; 28:289–294
72.
Kundel HL, Nodine CF, Krupinski EA. Computer-displayed eye position as a visual aid to pulmonary nodule interpretation. Invest Radiol 1990; 25:890–896
73.
Nodine C, Kundel H. A visual dwell algorithm can aid search and recognition of missed lung nodules in chest radiographs. In: Brogan D, ed. Visual search: proceedings of the First International Conference on Visual Search. London, UK: Taylor & Francis, 1990:399-406
74.
Litchfield D, Ball LJ, Donovan T, Manning DJ, Crawford T. Viewing another person's eye movements improves identification of pulmonary nodules in chest x-ray inspection. J Exp Psychol Appl 2010; 16:251–262
75.
Kelly BS, Rainford LA, Darcy SP, Kavanagh EC, Toomey RJ. The development of expertise in radiology: in chest radiograph interpretation, “expert” search pattern may predate “expert” levels of diagnostic accuracy for pneumothorax identification. Radiology 2016; 280:252–260
76.
Schalekamp S, van Ginneken B, Karssemeijer N, Schaefer-Prokop CM. Chest radiography: new technological developments and their applications. Semin Respir Crit Care Med 2014; 35:3–16
77.
Kuhlman JE, Collins J, Brooks GN, Yandow DR, Broderick LS. Dual-energy subtraction chest radiography: what to look for beyond calcified nodules. RadioGraphics 2006; 26:79–92
78.
Szucs-Farkas Z, Schick A, Cullmann JL, et al. Comparison of dual-energy subtraction and electronic bone suppression combined with computer-aided detection on chest radiographs: effect on human observers' performance in nodule detection. AJR 2013; 200:1006–1013
79.
Schalekamp S, van Ginneken B, van den Berk IA, et al. Bone suppression increases the visibility of invasive pulmonary aspergillosis in chest radiographs. PLoS One 2014; 9:e108551
80.
Freedman MT, Lo SC, Seibel JC, Bromley CM. Lung nodules: improved detection with software that suppresses the rib and clavicle on chest radiographs. Radiology 2011; 260:265–273
81.
Castellino RA. Computer aided detection (CAD): an overview. Cancer Imaging 2005; 5:17–19
82.
Li F, Engelmann R, Armato SG 3rd, MacMahon H. Computer-aided nodule detection system: results in an unselected series of consecutive chest radiographs. Acad Radiol 2015; 22:475–480
83.
White CS, Pugatch R, Koonce T, Rust SW, Dharaiya E. Lung nodule CAD software as a second reader: a multicenter study. Acad Radiol 2008; 15:326–333
84.
Wormanns D, Ludwig K, Beyer F, Heindel W, Diederich S. Detection of pulmonary nodules at multirow-detector CT: effectiveness of double reading to improve sensitivity at standard-dose and low-dose chest CT. Eur Radiol 2005; 15:14–22
85.
Zhao Y, de Bock GH, Vliegenthart R, et al. Performance of computer-aided detection of pulmonary nodules in low-dose CT: comparison with double reading by nodule volume. Eur Radiol 2012; 22:2076–2084
86.
Fenton JJ, Taplin SH, Carney PA, et al. Influence of computer-aided detection on performance of screening mammography. N Engl J Med 2007; 356:1399–1409
87.
Lehman CD, Wellman RD, Buist DS, Kerlikowske K, Tosteson AN, Miglioretti DL. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern Med 2015; 175:1828–1837
Information & Authors
Information
Published In
Copyright
© American Roentgen Ray Society.
History
Submitted: June 17, 2016
Accepted: August 17, 2016
Version of record online: December 27, 2016
Keywords
Authors
Metrics & Citations
Metrics
Citations
Export Citations
To download the citation to this article, select your reference manager software.