April 2007, VOLUME 188
NUMBER 4

Recommend & Share

April 2007, Volume 188, Number 4

Neuroradiology

Original Research

Specificity of MR Angiography as a Confirmatory Test for Carotid Artery Stenosis: Is It Valid?

+ Affiliations:
1Department of Radiology, Mayo Clinic, Rochester, MN.

2Present address: Department of Radiology, Baylor University Medical Center, 3500 Gaston Ave., Dallas, TX 75246.

Citation: American Journal of Roentgenology. 2007;188: 1114-1116. 10.2214/AJR.06.0414

ABSTRACT
Next section

OBJECTIVE. We believe that many studies in the literature show a falsely elevated specificity for carotid MR angiography (MRA) in the detection of high-grade stenosis. The purpose of this study was to test the hypothesis that inclusion of a substantial proportion of normal carotid arteries in a study population will falsely elevate the specificity of MRA for confirming a high-grade carotid artery stenosis.

MATERIALS AND METHODS. Seventy-seven carotid arteries were evaluated in 63 patients suspected of having a high-grade carotid stenosis, and all vessels were evaluated with contrast-enhanced MRA. Two subgroups were created, and the specificity of MRA was calculated for each group using digital subtraction angiography (DSA) as the gold standard. Group 1 included 44 vessels classified as high-grade stenosis on sonography and all were evaluated with DSA. To test our hypothesis, group 2 included the 44 carotid arteries from group 1 plus 33 carotid arteries classified as normal or minimally narrowed on sonography and MRA.

RESULTS. In group 1, the specificity of MRA for accurately confirming a high-grade stenosis was 29% for contrast-enhanced maximum-intensity-projection (MIP) images alone and 75% for contrast-enhanced axially reformatted source images as compared with DSA. When the 33 normal arteries from group 2 were added to the data set, the specificities increased to 70% and 89%, respectively.

CONCLUSION. The calculated specificity of MRA as a confirmatory test for high-grade carotid stenosis is highly dependent on the proportion of normal carotid arteries included in the calculation. Based on our results, the specificity of MRA reported in the literature has likely been overstated because of spectrum bias.

Keywords: biomedical statistics, cardiovascular imaging, carotid arteries, carotid artery stenosis, MR angiography, neuroimaging

Introduction
Previous sectionNext section

The concept of spectrum bias in medical studies is well documented [1]. Briefly, spectrum bias occurs when a diagnostic test has different sensitivities or specificities in patients with different clinical manifestations of the disease for which the test is intended. The reported specificity of MR angiography (MRA) for carotid artery stenosis has typically included both clinically normal and significantly narrowed carotid arteries, leading to results appropriate for use as a screening test. However, MRA has been adopted in many practices as a confirmatory test after a sonography examination. When MRA was used as a confirmatory test, these previous articles may have overstated the specificity, or true-negative rate, of MRA for accurately identifying clinically significant carotid stenoses if substantial numbers of nondiseased carotid arteries were included in those analyses.

As a screening test, the sensitivity of carotid Doppler sonography is generally optimized at the expense of specificity. If screening sonography reveals a carotid stenosis, additional confirmatory imaging is often performed to improve the specificity of the diagnostic testing. Because it is noninvasive, MRA has replaced digital subtraction angiography (DSA) as the confirmatory test for diagnosing high-grade carotid stenoses in many practices [2-4]. The prime objective of implementing MRA in this clinical scenario is to confirm that vessels identified on screening sonography as severely stenosed are, in fact, severely stenosed. In other words, MRA is used to identify vessels falsely identified on sonography as severely stenosed. In that clinical scenario, the true-negative rate—that is, specificity—should be determined on the basis of a population of vessels previously identified as abnormal on screening sonography.

The common use of MRA as a confirmatory test is based on previous reports that suggest a high specificity for accurately excluding a suspected high-grade carotid artery stenosis [5-11]. However, the most typical study design in previous reports includes both carotid arteries from all patients, irrespective of whether the carotid artery was identified as normal or abnormal on screening sonography. We suspect that this practice of including both carotid arteries is done to increase the number of vessels to satisfy statistical goals, but we submit that such a practice does not reflect the common clinical scenario and falsely elevates the apparent specificity, or true-negative rate, of MRA as a confirmatory test. The addition of normal or nearly normal vessels (true-negative cases) can be expected to falsely elevate the specificity of MRA as a confirmatory test because calculation of specificity is heavily dependent on the number of true-negative cases [12-17].

In the current study, we tested the hypothesis that inclusion of a substantial proportion of normal carotid arteries in a study population will falsely elevate the specificity of MRA for confirming a high-grade carotid artery stenosis. On the basis of those findings, we make recommendations regarding appropriate reporting of studies of the sensitivity of MRA or of any noninvasive confirmatory test reported in the future.

Materials and Methods
Previous sectionNext section

Institutional review board approval was obtained for the study. Between July 2002 and October 2004, 63 patients were evaluated with DSA before carotid endarterectomy (CEA) or carotid stenting. All patients underwent DSA and gadolinium-enhanced MRA studies that were performed within 28 days of each other. Before undergoing contrast-enhanced MRA and DSA, most of the patients were initially screened with carotid sonography. DSA was used as the standard of reference, and the results were interpreted using criteria from the North American Symptomatic Carotid Endarterectomy Trial (NASCET) [18].

Three experienced neuroradiologists retrospectively reviewed the MRA studies and were blinded to sonography and DSA results. Each reviewer separately evaluated the contrast-enhanced maximumintensity-projection (MIP) images alone and the contrast-enhanced source images. The contrast-enhanced source images were acquired in the coronal plane and reformatted in the axial plane. The reviewers recorded the percent diameter of stenosis for both contrast-enhanced MRA data sets in accordance with NASCET criteria after reviewing the images on a computer workstation. The final value was the mean value from the three reviewers. A cut point of 70% or greater was used to designate a high-grade stenosis.

To determine what effect the addition of normal carotid arteries has on the calculation of specificity, the carotid arteries from these 63 patients were divided into two groups. Group 1 was the clinically relevant vessels that were being evaluated before an anticipated intervention, either CEA or stenting. Group 1 included only those vessels for which the initial screening carotid sonography findings suggested a high-grade (70-99%) carotid artery stenosis. The 44 carotid arteries in group 1 were evaluated on sonography, contrast-enhanced MRA, and DSA.

Group 2 was created to test our theory that including a large number of clinically irrelevant normal carotid arteries in a study population will skew the results for calculated specificity. In group 2, we added 11 vessels that were designated as normal or mildly stenosed (0-39%) on sonography, contrast-enhanced MRA, and DSA to the carotid arteries in group 1. Also added to group 2 were 22 contralateral carotid arteries that were normal or minimally narrowed on contrast-enhanced MRA. Although DSA was not performed in these 22 contralateral carotids arteries, they were assumed to be normal or mildly stenosed because no false-positives were encountered in our study population with contrast-enhanced MRA findings that suggested a normal or mildly stenosed vessel.

Specificity calculations were performed on both groups for the contrast-enhanced MRA MIP images and contrast-enhanced axially reformatted source images.

Results
Previous sectionNext section

In contrast to group 1, group 2 contained a large proportion of normal and presumably normal vessels. In group 1, three vessels (7%) were in the 0-39% stenosis range; 21 (48%), in the 40-69% range; and 20 (45%), in the 70-99% range based on DSA results. In group 2, 35 (46%) vessels were in the 0-39% stenosis range; 21 (27%), in the 40-69% range; and 21 (27%), in the 70-99% range.

The specificity of the contrast-enhanced MIP images was 29% (7/24) in the clinically germane group 1 and 70% (40/57) in group 2; these results show significantly elevated apparent specificity when normal vessels are included in the calculation. The specificity of contrast-enhanced axially reformatted source images was 75% (18/24) in group 1 and was 89% (51/57) in group 2. In both groups, the contrast-enhanced axially reformatted source images were more specific than the MIP images alone for confirming high-grade carotid stenosis.

Of the 59 carotid arteries that were evaluated using carotid sonography (44 in group 1 and 15 in group 2), there were 24 false-positives, 20 true-positives, 13 true-negatives, and two false-negatives compared with the reference standard of DSA. The calculated sensitivity and specificity of sonography in our population was therefore 91% and 35%, respectively.

Discussion
Previous sectionNext section

For our study, we evaluated the specificity of contrast-enhanced MRA as a confirmatory test for carotid arteries suggested to have a high-grade stenosis on sonography to reflect the typical clinical pathway in which patients with positive sonography results are referred to MRA for confirmation. The falsely elevated specificity encountered in our study when normal or mildly stenotic carotid arteries were included in calculations confirms the theoretic conclusions suggested by previous authors [12]. Our results indicate that the specificity of contrast-enhanced MRA when used as a confirmatory test for carotid artery stenosis has likely been overstated in the medical literature.

Based on our results, the specificity of contrast-enhanced MRA MIP images is very poor when applied to carotid arteries screened as positive for stenosis on sonography. Indeed, for arteries identified as high-grade stenosis on sonography, the specificity of contrast-enhanced MIP images for confirming high-grade stenosis was worse than a flip of a coin (i.e., < 50%). Among 24 vessels falsely identified as severely stenosed on sonography when using DSA as the reference standard, contrast-enhanced MIP images correctly showed only seven of these vessels as less than 70% stenosed. The contrast-enhanced MRA axially reformatted source images performed better than the MIP images, but still showed falsely elevated specificity when normal and near-normal arteries were included in the calculation.

In previous studies, researchers have reported MRA as a highly specific test for evaluating high-grade cervical carotid artery stenosis [10-17]. In a previous study by Huston et al. [10] evaluating 2D time-of-flight MRA, a technique that is widely known to overestimate carotid artery stenosis, the specificity decreased from 80% to 18% when the normal carotid arteries were removed from the calculation. Similarly, the specificity of MRA decreased from 82% to 57% in reports by Anson et al. [11] and Kallmes et al. [12] after the normal carotid arteries had been excluded. The results from these studies are in line with our data and confirm that the specificity of MRA is falsely elevated if a large number of normal carotid arteries are used to calculate the true-negative rate.

Our study addresses the particular question of MRA as a confirmatory test for carotid artery stenosis after an initial positive result on sonography. This clinical scenario is different from evaluating MRA as an initial screening examination. In this regard, we do not dispute the specificity calculations from previous articles that evaluated MRA as a screening test. However, we disagree with the subsequent application of these results when using MRA as a confirmatory test. The most appropriate method for assessing the utility of MRA as a confirmatory test is to apply it only to those carotid arteries determined to have a high-grade stenosis on initial screening sonography.

Screening tests should have high sensitivity to limit the false-negative cases, even at the expense of false-positive results [19]. A second, or confirmatory, test should have high specificity to accurately identify false-positive cases resulting from the screening test. Given this logic, we caution the wide application of MRA as a confirmatory test because the tendency for MRA to “overcall” stenoses is widely known [20]. We submit that the current state-of-the-art MRA may function well as a screening test but that its value as a confirmatory test remains suspect.

There are several limitations to our study. This study was retrospective and evaluated only patients suspected of having a highgrade carotid stenosis. DSA was considered the gold standard for this study even though 22 vessels in group 2 did not undergo DSA evaluation. The ability of contrast-enhanced MRA to accurately show a carotid artery as normal is widely accepted; therefore, we are confident that those 22 vessels added to group 2 were indeed normal. To prove our hypothesis was correct, it was necessary to add these clinically irrelevant vessels to group 2 to show specificity would become falsely elevated.

Although the commonly accepted threshold for a high-grade stenosis is 70%, it is possible that stenoses in the 50-70% range are important in certain situations [18]. We designated 70% and greater stenoses as positive and those less than 70% as negative. If future investigations suggest a definitive role for CEA or stenting in patients with stenoses less than 70%, the specificity of MRA will need to be reevaluated in this subset of patients. It was not the purpose of our analysis to determine the accuracy of MRA but rather to confirm a suspected source of bias in the MRA evaluation of carotid stenosis. Although it is true that DSA is an invasive test that is associated with complications, DSA is a relatively safe examination when performed by experienced operators and still has a place in the evaluation of patients with suspected carotid artery stenosis [21].

As MRA and CT angiography techniques continue to improve, these techniques may prove to be more specific noninvasive confirmatory tests for evaluating carotid artery stenosis. In addition, 64-MDCT angiography with bone-removing techniques has shown considerable promise [22]. Advances in MRA continue with 3.0 T and novel sequences offering higher spatial resolution [23]. Prospective studies comparing the sensitivity and specificity of CT angiography versus MRA should be performed to investigate whether CT angiography is a more specific confirmatory test. If confirmation of a high-grade stenosis suspected on sonography is necessary before intervention, practitioners should consider these results when deciding whether to proceed with DSA.

Address correspondence to K. F. Layton ().

References
Previous sectionNext section
1. Lachs MS, Nachamkin I, Edelstein PH, Goldman J, Feinstein AR, Schwartz JS. Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection. Ann Intern Med 1992; 117:135-140 [Google Scholar]
2. De Marco JK, Schonfeld S, Wesbey G. Can noninvasive studies replace conventional angiography in the preoperative evaluation of carotid stenosis? Neuroimaging Clin N Am 1996; 6:911-929 [Google Scholar]
3. U-King-Im JM, Hollingworth W, Trivedi RA, et al. Cost-effectiveness of diagnostic strategies prior to carotid endarterectomy. Ann Neurol 2005; 58:506-515 [Google Scholar]
4. Buskens E, Nederkoorn PJ, Buijs-Van Der Woude T, et al. Imaging of carotid arteries in symptomatic patients: cost-effectiveness of diagnostic strategies. Radiology 2004; 233:101-112 [Google Scholar]
5. Huston J 3rd, Fain SB, Wald JT, et al. Carotid artery: elliptic centric contrast-enhanced MR angiography compared with conventional angiography. Radiology 2001; 218:138-143 [Google Scholar]
6. DeMarco JK, Huston J 3rd, Bernstein MA. Evaluation of classic 2D time-of-flight MR angiography in the depiction of severe carotid stenosis. AJR 2004; 183:787-793 [Abstract] [Google Scholar]
7. Polak JF, Kalina P, Donaldson MC, O'Leary DH, Whittemore AD, Mannick JA. Carotid endarterectomy: preoperative evaluation of candidates with combined Doppler sonography and MR angiography—work in progress. Radiology 1993; 186:333-338 [Google Scholar]
8. Huston J, Nichols DA, Luetmer PH, et al. MR angiographic and sonographic indications for endarterectomy. Am J Neuroradiol 1998; 19:309-315 [Google Scholar]
9. Lustgarten JH, Solomon RA, Quest DO, Khanjdi AG, Mohr JP. Carotid endarterectomy after noninvasive evaluation by duplex ultrasonography and magnetic resonance angiography. Neurosurgery 1994; 34:612-619 [Google Scholar]
10. Huston J 3rd, Lewis BD, Wiebers DO, Meyer FB, Riederer SJ, Weaver AL. Carotid artery: prospective blinded comparison of two-dimensional time-of-flight MR angiography with conventional angiography and duplex US. Radiology 1993; 186:339-344 [Google Scholar]
11. Anson JA, Heiserman JE, Drayer BP, Spetzler RF. Surgical decisions on the basis of magnetic resonance angiography of the carotid arteries. Neurosurgery 1993; 32:335-343 [Google Scholar]
12. Kallmes DF, Omary RA, Dix JE, Evans AJ, Hillman BJ. Specificity of MR angiography as a confirmatory test of carotid artery stenosis. Am J Neuroradiol 1996; 17:1501-1506 [Google Scholar]
13. Litt AW, Eidelman EM, Pinto RS, et al. Diagnosis of carotid artery stenosis: comparison of 2DFT time-of-flight MR angiography with contrast angiography in 50 patients. Am J Neuroradiol 1991; 12:149-154 [Google Scholar]
14. Mittl RL Jr, Broderick M, Carpenter JP, et al. Blinded-reader comparison of magnetic resonance angiography and duplex ultrasonography for carotid artery bifurcation stenosis. Stroke 1994; 25:4-10 [Google Scholar]
15. Blatter DD, Bahr AL, Parker DL, et al. Cervical carotid MR angiography with multiple overlapping thin-slab acquisition: comparison with conventional angiography. AJR 1993; 161:1269-1277 [Abstract] [Google Scholar]
16. Laster RE Jr, Acker JD, Halford HH 3rd, Nauert TC. Assessment of MR angiography versus arteriography for evaluation of cervical carotid bifurcation disease. Am J Neuroradiol 1993; 14:681-688 [Google Scholar]
17. Spartera C, Morettini G, Marino G, et al. Detection of internal carotid artery stenosis: comparison of 2D-MR angiography, duplex scanning, and arteriography. J Cardiovasc Surg 1993; 34:209-213 [Google Scholar]
18. [No authors listed]. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis: North American Symptomatic Carotid Endarterectomy Trial Collaborators. N Engl J Med 1991; 325:445-453 [Google Scholar]
19. Herman CR, Gill HK, Eng J, Fajardo LL. Screening for preclinical disease: test and disease characteristics. AJR 2002; 179:825-831 [Abstract] [Google Scholar]
20. Duran-Ferreras E, Schoeck AP, Barest GD, Viereck JC, Babikian VL, Kase CS. Overestimation of the degree of stenosis in the intracranial internal carotid artery determined by magnetic resonance angiography compared to conventional arteriography. Rev Neurol 2003; 37:730-732 [Google Scholar]
21. Cloft HJ, Joseph GJ, Dion JE. Risk of cerebral angiography in patients with subarachnoid hemorrhage, cerebral aneurysm, and arteriovenous malformation: a meta-analysis. Stroke 1999; 30:317-320 [Google Scholar]
22. Lell M, Anders K, Klotz E, Ditt H, Bautz W, Tomandl BF. Clinical evaluation of bone-subtraction CT angiography (BSCTA) in head and neck imaging. Eur Radiol 2005; 3:1-9 [Google Scholar]
23. Nael K, Michaely HJ, Villablanca P, Salamon N, Laub G, Finn JP. Time-resolved contrast enhanced magnetic resonance angiography of the head and neck at 3.0 Tesla: initial results. Invest Radiol 2006; 41:116-124 [Google Scholar]

Recommended Articles

Specificity of MR Angiography as a Confirmatory Test for Carotid Artery Stenosis: Is It Valid?

Full Access, , , , , ,
American Journal of Roentgenology. 2007;188:W359-W366. 10.2214/AJR.06.0627
Abstract | Full Text | PDF (7804 KB) | PDF Plus (662 KB) 
Full Access, , , ,
American Journal of Roentgenology. 2007;188:W367-W369. 10.2214/AJR.05.0762
Abstract | Full Text | PDF (859 KB) | PDF Plus (903 KB) 
Full Access,
American Journal of Roentgenology. 2011;197:1309-1321. 10.2214/AJR.11.7420
Abstract | Full Text | PDF (1195 KB) | PDF Plus (1215 KB) 
Full Access, , , , ,
American Journal of Roentgenology. 2007;188:W355-W358. 10.2214/AJR.05.1466
Abstract | Full Text | PDF (2281 KB) | PDF Plus (357 KB) 
Full Access, , , , , ,
American Journal of Roentgenology. 2007;188:1109-1113. 10.2214/AJR.06.1175
Abstract | Full Text | PDF (128 KB) | PDF Plus (199 KB) 
Full Access, , , ,
American Journal of Roentgenology. 2016;206:595-600. 10.2214/AJR.14.14156
Abstract | Full Text | PDF (839 KB) | PDF Plus (868 KB) | Supplemental Material