HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Hillman, B. J. Search for Related Content PubMed PubMed Citation Articles by Hillman, B. J. Social Bookmarking                      What's this?

Critical Thinking

Deciding Whether to Incorporate the Recommendations of Radiology Publications and Presentations into Practice

¹ Department of Radiology, Box 170, The University of Virginia Health System, Charlottesville, VA 22908.

Received November 8, 1999; accepted after revision November 17, 1999.

 
This is the third in a series on noninterpretive skills for residents in diagnostic radiology from the American College of Radiology and the Association of Program Directors in Radiology. Editor: Jannette Collins.

Address correspondence to B. J. Hillman.

Introduction

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
Medical practice is largely based on clinical anecdotes, uncontrolled investigations, and expert opinion. The demand for scientific evaluation to guide how physicians conduct patient care is increasing because the financial resources that pay for health care are limited, and because practice based on such influences may be inappropriate. Pressure from payers, employers, and ultimately society encourages physicians to practice—in the vernacular—evidence-based medicine.

In radiology, the situation is especially problematic. Over the past 30 years, perhaps more than any other specialty, radiology has experienced the introduction of many revolutionary technologies that have dramatically changed imaging practice. The diffusion of these technologies and the introduction of new applications for their use have far outpaced clinical researchers' determination of their appropriate roles in clinical practice. Instead, use has been guided by methodologically poor reports that tend to exaggerate the performance of new techniques [1,2,3], further spurring diffusion and use. The end result is the suspicion that expensive technologies such as CT and MR imaging are being used more frequently than is necessary for cost-effective patient care. This suspicion is widely held among policymakers and the payers of health care and is primarily responsible for the targeting of medical imaging as an opportunity for cost reduction.

At the root of the problem is the naivete of most radiologists as they read radiology literature and listen to meeting or postgraduate course presentations. Few radiology training programs emphasize research in their curriculum, and few programs address and encourage critical thinking skills. More critical radiologists would better dissect the methods supporting the results and conclusions of studies, adopting into their practices only valid information and discarding studies with biased results.

Comprehensive instruction in study design, biases, and hypothesis testing is beyond the scope of this presentation. Rather, I present a framework that can guide interested readers to continue their study of this subject to become better radiologists who can more appropriately care for patients. Additionally, I hope to influence radiology training programs and institutions such as the Radiology Residency Review Committee and the American Board of Radiology to consider the incorporation of critical thinking skills in the education of residents as a means of improving radiology.

Characteristics of Believable Publications and Presentations

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
Believable studies detail applications that are appropriate for incorporating into clinical practice. They have several important features, including pertinence, validity, reliability, and generalizability. A pertinent study describes a clinical situation that is relevant to the subject of interest. The evaluation of pertinence is generally subjective, based on the opinion of individuals whose clinical practice involves the area detailed in the study.

Validity refers to the methods used in a study. Were the methods relatively free of biases (almost every study has some biases)? Were the measurement methods generally accepted and susceptible to comparisons with other studies? Could the results be tested by accepted statistical techniques? There are numerous types of validity. First, evaluation of validity considers "face validity," in other words, on the surface, does the study seem appropriate? Other types of validity, such as criterion and construct validity in survey research, and how to test for them, pertain more directly to the type of study and methodology. Assuring validity is frequently an arcane science, requiring experts from other disciplines as part of the research team. Readers should feel more assured about the validity of the study if appropriate methodologists and statisticians have been involved in the study design.

Reliability means that if the study were conducted a second time in the same setting with the same population of interpreters, a similar result would likely be obtained. Reliability is frequently an issue in interpreter studies of imaging efficacy (defined later in this article). One method of determining reliability is to have the observers of a study reinterpret a sample of cases that they interpreted at an earlier time. If the observers are able to produce similar interpretations (intraobserver agreement), the study is reliable.

Generalizability is perhaps the most important consideration for a radiologist trying to determine whether a result would be obtained in his or her own practice. To what extent do the setting, radiologists, and patient population reflect his or her practice? Did the technology reflect clinical practice or did it reflect a research setting? If the latter, how much of a stretch is it to imagine using the methodology in practice? Health policy experts generally discuss the differences between studies of efficacy and studies of effectiveness. Studies of efficacy examine a technology as it is used by subspecialized experts—often those involved in developing and testing the technology in idealized settings. Studies of effectiveness examine how a technology performs in general practice. For obvious reasons, studies of effectiveness are of much greater interest to payers and policymakers than studies of efficacy. Generalizability is enhanced when studies are multiinstitutional, large, and effectively include a broad spectrum of possible patients and practitioners in general practice. Unfortunately, such studies are rare in the radiology literature.

Purpose of the Study and Study Design

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
Clinical research in radiology generally focuses on the development of new technology, description of clinical applications, detailing of individual cases and case series, and evaluation of one or more technologies [4]. Case reports and case series (no matter how large) are inevitably influenced by biases that should alert the critical reader to question the veracity of the conclusions, not because the conclusions are wrong, but because it is usually impossible to determine whether they are correct or incorrect. Such descriptive studies constitute most of the radiology literature [4].

I equate scientific clinical research that involves patients with a term that has become trendy in medical literature—technology assessment. A simple nomenclature of technology assessment (in ascending order of relevance to patient care) [5] categorizing studies according to their end point includes imaging efficacy, diagnostic efficacy, therapeutic efficacy, patient health outcomes (including quality of life outcomes), and cost-effectiveness.

Imaging efficacy details the performance of a technology—radiologist combination in determining whether disease is present or absent. It uses familiar measurements such as sensitivity, specificity, accuracy, positive and negative predictive value, and receiver operating characteristic curve analysis. Diagnostic and therapeutic efficacy reflect how the information from imaging impacts the referring physician's diagnosis and treatment plan, respectively. Both compare before-test with after-test considerations and estimate the difference between the two [6]. Patient health outcomes include both traditional measures of health (physiologic health), such as morbidity and mortality, and quality-of-life health outcomes that consider how health care affects constructs such as physical, mental, social, and role functioning; sense of well-being; and health-seeking behaviors [5]. Intuitively, medical imaging might play an important role in improving these aspects of health, but little research sustains this belief. Cost-effectiveness research requires the measurement of both cost and benefit, whereby the result is generally expressed in terms of incremental dollars spent per incremental improvement in health. For a number of reasons, largely related to important misapprehensions about the term, the frequency of the use of the term, and how to apply this research, I treat this as a special case to be discussed in a separate section of this presentation.

For the remainder of this section, I focus on the design of studies of imaging efficacy, but the general observations hold true for other genres of technology assessment. I have chosen to focus on imaging efficacy because these studies are, by far, the most frequently published scientific clinical research in the radiology literature.

Imaging efficacy research is susceptible to a prospective randomized controlled trial design (the generally accepted standard of clinical research). Randomized controlled trials (trials that ensure that patient populations in experimental and control groups are similar) greatly reduce the likelihood that biases will influence results. However, few randomized controlled trials have been performed in radiology because randomized controlled trials are difficult to design, time-consuming to implement and complete, impractical because of the number of patients required, and expensive. For a fast-moving technology such as diagnostic imaging, there are reasonable concerns that by the time randomized trials are completed the technologies tested will be obsolete (sometimes referred to as the "moving target" problem). Much more frequently, diagnostic imaging studies use a paired design whereby each patient is examined with two or more technologies. This design allows smaller sample sizes, less time, and less expense because it enables each patient to act as his or her own control and enables pairing of the analysis.

Both paired and unpaired designs are subject to a number of important biases that can severely affect the result (usually in a positive direction) and incorrectly influence the practice of the unwary reader. Such biases include verification (workup) bias, test interpretation (test review or diagnostic review) bias, casemix (selection or spectrum) bias, bias in reporting uninterpretable results, lack of a definitive reference standard, investigator bias, misapprehensions that result from inappropriately reporting certain measures (e.g., accuracy and positive predictive value), and inappropriate conclusions based on sample size [7,8]. A recent publication evaluating 11 metaanalyses of diagnostic technologies, including three imaging technologies, revealed that the literature is rife with some of these biases [9]. The study reported that biases systematically resulted in the exaggeration of the performance of imaging technologies. The most frequently encountered biases and the ones that led to the most severe skewing of the results were case mix and verification biases.

Common Biases and Sources of Misapprehension

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
Verification (workup) bias occurs when the succeeding workup is based on the result of the technology being evaluated, such that the reference or gold standard (the truth by which the correct diagnosis is established) differs among cases included in the study. Most commonly, positive cases receive a more rigorous workup than negative cases so that both the true-negative and false-negative rates are incorrectly estimated. Frequently in the radiology literature, authors simply assume that a negative test result means no disease [7]. Obviously, the ideal solution is to design the study so that all patients receive the same diagnostic verification. However, the reference standard is frequently the result of an invasive test or even pathologic evaluation—a practical and ethical impossibility. In such cases, the best the investigators can do is to assiduously follow the clinical course and subsequent testing of patients for a length of time compatible with the disease course to determine the true diagnosis. However, all subjects, whether their diagnoses were adequately verified or not, need to be included in the report. Statistical correction for unverified diagnoses is feasible [7].

Test interpretation (test review or diagnostic review) bias results from observers participating in an imaging efficacy study with foreknowledge that biases their interpretations. Sources of knowledge include remembering retrospectively accumulated cases, knowing the results of other tests, or having heard of the reference test result. The best solution for this problem is to have observers interpret cases "off-line" (not in the clinical setting) and to blind them to all such information. When two or more technologies are involved in the study, the order in which they are presented to each observer should be randomized to equalize any recall effect that might be present. Controversy continues as to whether observers should be provided clinical information in such designs. Clinical information influences performance [10, 11], so the choice should depend on whether the effort is intended to replicate the clinical setting [7]

Case-mix (selection or spectrum) bias occurs when cases are selected that inaccurately reflect the range of cases that occur in the general population. This is especially problematic for the evaluation of new technologies. For new technologies, we know in which cases we find disease, but we know less well which cases are missed. Thus, either retrospective or prospective case selection will generally favor selecting cases with the most advanced disease. In essence, the study becomes an evaluation of the wellest of the well and the sickest of the sick. An interpreter study using these easier cases is more likely to show a favorable result than when the technology is used in general practice.

A common practice in the radiology literature is to not report cases that show an uninterpretable result. Dealing with uninterpretable results can be problematic; however, fair and honest reporting requires that all cases be included in a publication or presentation. Failure to do so will give an unreasonably optimistic picture of the technology studied. If an uninterpretable result can be shown, using the reference standard, to be associated with either the absence or presence of disease (non-random pattern), it may be reasonable to include the test results in either of these categories with explanation. However, if the pattern is random, it is better to include noninterpretable results as a separate category [7].

Lack of a definitive reference standard is the sole bias that tends to make the performance of a technology appear more negative than it really is. Incorrectly classifying some of the results as correct or incorrect based on a reference standard that is sometimes wrong can lead to errors in categorization that understate accuracy. This is especially problematic when the technology being tested in reality performs better than the fallible reference standard. An example of such a situation is the use of biopsy as the reference standard for sporadically occurring diseases, such as autoimmune conditions, in which the more global view of an imaging technology might better reflect the patient's condition. When no adequate reference standard is available, assiduous clinical and imaging follow-up of patients to improve on classification (similar to the recommendations for avoiding verification bias) is the best approach.

The developers and initial evaluators of a new technology have investigator bias. Their experiences with their technology have been salutary and the career benefits they have derived from working with it may engender an exaggerated view of its capabilities. In addition to unconsciously allowing other biases detailed in this section to creep into their designs, affected investigators may also unconsciously fall prey to interpreting results in ways that confirm their preconceived views.

Accuracy is frequently used to report the performance of imaging tests. Accuracy is usually defined as all patients correctly diagnosed divided by the total number of patients in a study. Now imagine an extreme example in which a test is terrible at identifying patients with the disease—0% sensitivity—but very good at being negative when no disease is present. And imagine that the population sample used to evaluate this test has 99 healthy people and one person with disease. Accuracy will be 99%, but in reality the test is functionally useless for its purpose. Examples of the misuse of accuracy, perhaps not quite so extreme but still misleading, abound in the radiology literature. Similarly, although the use of positive predictive value is generally recommended by most methodologists, its use can also be problematic for interpreting imaging efficacy studies. The case samples in imaging efficacy studies are typically heavily weighted toward cases with positive findings so that the number of cases required to achieve a statistically significant result can be decreased. However, positive predictive value is heavily influenced by the a priori probability of disease being present, or the representation of positive cases in a sample. That the disease prevalence in the sample exceeds the prevalence of disease in the general population causes the study results to be an exaggeration of positive predictive value beyond that which could be expected in practice [8].

Studies in the radiology literature—even well designed ones—tend to be small in the number of patients and observers they report. The reasons for this are expense, available case material, and influence of career incentives to complete research quickly and publish frequently. When a positive difference between two technologies is established, little harm is done, so long as the other dictums of pertinence, validity, reliability, and generalizability are accommodated. However, a negative result—one showing no difference—is suspect. It may be that a real difference exists, even a clinically important one, but that the sample size is too small to reveal it significantly. In designing a study, I believe it is important to first consider what would be considered a practically significant difference. The study should be sized to have a high reliability for revealing such a difference. Smaller studies are inadequate and larger studies are wasteful.

Cost-Effectiveness Research

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
Cost-effectiveness research involves collecting data on both patient health outcomes and costs. It is research that addresses populations of people. Cost-effectiveness results are not intended for application to specific patients. Rather, the results are intended to guide more rational policymaking, particularly in circumstances when decisions must be made between competing claims on limited resources. Therefore, cost-effectiveness research is in great demand by payers, regulatory agencies, and health enterprises.

The frequency with which improper claims are made for cost-effectiveness in the radiology literature is disturbing. That this is the case reflects an imperfect understanding of the meaning of the term. As pointed out by Doubilet et al. [12], the term "cost-effectiveness" is meaningless in an isolated context. Rather, a technology can be considered cost-effective only in comparison with another technology, and the technology is both less costly and of greater benefit than the technology to which it is compared (technologic dominance) [12]. This is a relatively rare circumstance. More often, a technology is either more costly and of greater benefit (e.g., nonionic contrast material) or less costly and of lesser benefit (e.g., stereotactic core breast biopsy) than the established technology it seeks to replace. In such cases, it is a societal decision (usually carried out through innumerable local political processes) whether the trade-off in cost and benefit are worthwhile.

In 1997, Blackmore and Magid [3] detailed their review of the 1989-1995 radiology literature to evaluate the quality of studies claiming to be cost-effectiveness research. These researchers assessed 44 articles for six major and four minor generally accepted criteria of methodologic rigor. The six major criteria included an explicit statement about which technologies were being compared, a perspective of analysis (e.g., payer, patient, societal, etc.—this is critical because the costs and benefits that need to be studied will vary according to perspective), the cost data, the health outcome data, the use of a summary measure that used cost and benefit, and sensitivity analyses to test the effect of uncertainties. The four minor criteria included the source of cost data, the consideration of longterm (downstream) costs; discounting for opportunity costs; and the use of incremental computation (e.g., looking at marginal, rather than average, costs and benefits). All 10 criteria were present in only three of the 44 studies; the median number of major and minor criteria per study were three and one, respectively.

Critically Reading a Research Article

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
While reading a journal article or listening to a presentation, I look for a number of different qualities that naturally flow from the typical organization of presentation. The presence of these qualities makes me feel more comfortable about the quality of the research, and hence, whether I consider the conclusions suitable for integration into clinical practice.

The introduction should have a succinct statement of what was studied and why it is an important subject. The subject or problem should be well defined. Ideally, an implicit or explicit hypothesis should be stated.

The methods section should provide sufficient detail so that the work could be replicated. It should indicate the specific technologies studied; the patient and radiologist populations, including sample size, inclusion and exclusion criteria, and other important descriptions; the outcomes or end points that were studied and the justification for their being chosen; the reference standard (if an imaging efficacy study); and the details of the analysis and statistical hypothesis testing.

The results should report what was promised in the introduction and methods—nothing more and nothing less.

The discussion should provide a restatement of the one or two most important findings, a statement about how the findings should be placed in the context of other related studies, an explanation of possible reasons why the authors' results differ from those of previous work, a detailing of possible biases in the work and why they could not be avoided, and a conclusion that indicates the authors' view of how the research should influence further research or clinical practice.

Recommendations

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 
Assimilation of the principles detailed in this presentation and their application to reading the radiology literature and listening to presentations would represent an advance in critical thinking by radiologists. Widespread improvement in critical thinking will lead to an improvement in the appropriateness of care delivered by radiologists with resultant decreased patient costs and increased health benefits. For readers interested in further improving their critical thinking skills, the references listed will further define the principles detailed in this presentation. Programs should establish, at a minimum, a basic series of lectures delivered to each resident class to help trainees develop critical thinking skills. However, the only way to become an accomplished critical thinker is to practice. This can be difficult for the neophyte unaccustomed to thinking about what he or she reads in this way. I have found a critical journal club to be a terrific mechanism for learning this art. Ideally, a critical journal club would meet once a month for about an hour. No more than four articles should be discussed by the group members. Articles should be chosen less for content and more for the opportunities they afford to teach the group what was good and bad about the research. Each article should be presented according to its adherence to the principles I have discussed (and beyond for the more advanced). The group should then discuss each article and debate whether the article is methodologically strong enough to warrant the results and the conclusion.

References

Top Introduction Characteristics of Believable... Purpose of the Study... Common Biases and Sources... Cost-Effectiveness Research Critically Reading a Research... Recommendations References

 

1. Kent DL, Larson ED. Magnetic resonance of the brain and spine: is clinical efficacy established after the first decade? Ann Intern Med 1988;108: 402 -424
2. Beam CA, Sostman HD, Zheng J-Y. Status of clinical MR evaluations 1985-1988: baseline and design for further assessments. Radiology 1991;180: 265 -270[Abstract/Free Full Text]
3. Blackmore CC, Magid DJ. Methodologic evaluation of the radiology costs-effectiveness literature. Radiology 1997;203: 87 -91[Abstract/Free Full Text]
4. Holman BL. The research that radiologists do: perspective based on a survey of the literature. Radiology 1990;176: 329 -332[Abstract/Free Full Text]
5. Hillman BJ. Outcomes research and costs-effectiveness analysis for diagnostic imaging. Radiology 1994;193: 307 -310[Free Full Text]
6. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991;11: 88 -94
7. Begg CB, McNeil BJ. Assessment of radiological tests: control of bias and other design considerations. Radiology 1998;167: 565 -569[Abstract/Free Full Text]
8. Michael M III, Boyce WT, Wilcox AJ. Biomedical bestiary: an epidemiological guide to flaws and fallacies in the medical literature. Boston: Little, Brown, 1984: 161
9. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999;282: 1061 -1066[Abstract/Free Full Text]
10. Berbaum KS, Franken EA, Dorfmann DD, et al. Tentative diagnosis facilitates the detection of diverse lesions in chest radiographs. Invest Radiol 1986;21: 532 -539[Medline]
11. McNeil BJ, Hanley JA, Funkenstein HH, Wallman J. Paired receiver operating characteristic curves and the effect of history on radiographic interpretation. Radiology 1983;149: 75 -77[Abstract/Free Full Text]
12. Doubilet P, Weinstein MC, McNeil BJ. Use and misuse of the term cost-effectiveness in medicine. N Eng J Med 1986:314: 253 -256[Medline]

CiteULike

Complore

Connotea

Del.icio.us

Digg

Reddit

Technorati

This article has been cited by other articles:

				C. T. Thomas, P. T. Bradshaw, B. H. Pollock, J. E. Montie, J. M.G. Taylor, H. D. Thames, P. W. McLaughlin, D. A. DeBiose, D. H. Hussey, and R. L. Wahl In Reply: J. Clin. Oncol., January 15, 2004; 22(2): 380 - 381. [Full Text] [PDF]

				M. E. Spieth and R. B. Gunderman Radiologic Research and Residency * Dr Gunderman responds: Radiology, February 1, 2003; 226(2): 593 - 593. [Full Text] [PDF]

				S. Chan The Clinical Relevance and Scientific Potential of Ultra High-Field-Strength MR Imaging AJNR Am. J. Neuroradiol., October 1, 2002; 23(9): 1441 - 1442. [Full Text] [PDF]

				R. B. Gunderman, J. M. Nyce, and J. Steele Radiologic Research: The Residents' Perspective Radiology, May 1, 2002; 223(2): 308 - 310. [Full Text] [PDF]

				R. B. Gunderman Is Technical School a Good Model for Radiology Residency? Am. J. Roentgenol., November 1, 2001; 177(5): 1005 - 1007. [Full Text] [PDF]

				R. C. Brunken, D. R. Neumann, B. Bybel, C. C. Blackmore, and C. A. Beam Beyond Radiology Consultant Am. J. Roentgenol., November 1, 2001; 177(5): 1216 - 1217. [Full Text] [PDF]

				T. The Evidence-Based Radiology Working Group Evidence-based Radiology: A New Approach to the Practice of Radiology Radiology, September 1, 2001; 220(3): 566 - 575. [Abstract] [Full Text] [PDF]

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Hillman, B. J. Search for Related Content PubMed PubMed Citation Articles by Hillman, B. J. Social Bookmarking                      What's this?