| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Original Research |
1 Department of Radiology, New York Presbyterian Hospital, Weill Medical College
of Cornell University, 520 E 70th St., Starr 630, New York, NY 10021.
2 Department of Neurology, New York Presbyterian Hospital, Weill Medical College
of Cornell University, New York, NY 10021.
Received December 20, 2005;
accepted after revision June 28, 2006.
Supported in part by a GE-AUR Research Award.
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. Twenty CT perfusion data sets were postprocessed by a neuroradiologist using an automated postprocessing program and by five observers (neuroradiology attending, neurology attending, radiology resident, senior and junior CT technologists) who received a brief training session in use of software for semiautomated postprocessing. For assessment of intraobserver variability, each observer repeated postprocessing of 10 CT perfusion data sets. Standard regions of interest were placed on identical locations for each observer's cerebral blood flow (CBF), cerebral blood volume (CBV), and mean transit time (MTT) maps of three brain regions: an ischemia-infarct region, normal cortical gray matter, and white matter.
RESULTS. The variability in mean quantitative values of CBF, CBV, and MTT was 2.5-9.5% among all observers. Greater variability (20.4%) was introduced with the automated program. High correlation was found among all possible pairings of observers (r = 0.87-0.99). Low correlation was observed between automated postprocessing and postprocessing by all observers. Intraobserver variability in quantitative CT perfusion data ranged from 0.29% to 10.8%. High intraobserver correlation (r = 0.91-0.99) was found for the observers.
CONCLUSION. Quantitative CBF, CBV, and MTT data obtained from postprocessing of CT perfusion data sets are reproducible among observers with varying levels of skill and experience. Observer interaction with the software is an important component for correct identification of user-defined parameters. Establishing a uniform and standard postprocessing technique is essential for maintaining good reproducibility.
Keywords: brain • cerebrovascular disease • CT • perfusion CT
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
A previous study [2] revealed variability in quantitative CT perfusion maps postprocessed by several CT technologists. However, the performance of radiologists and that of CT technologists was not directly compared. With the increasing number of CT perfusion examinations being performed in radiology departments, the responsibility for postprocessing of the CT perfusion data sets has shifted to radiology fellows, residents, and CT technologists. The purpose of this study was to assess interobserver and intraobserver variability by evaluating the reproducibility of CT perfusion maps postprocessed automatically and by observers with different levels of skill.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CT Perfusion Scanning Protocol
Patient preparation included removal of metallic hardware from dental and
hair prosthetics to minimize distortion artifact and placement of an 18- or
20-gauge peripheral IV catheter for contrast injection. Central venous
catheters were used in selected patients according to departmental policy only
if peripheral access was not available
[3]. The standard protocol for
performing CT perfusion at our institution is a deconvolution-based method in
cine scanning mode (cine 4i) on a LightSpeed or Pro-16 scanner (GE
Healthcare). The kVp/mA is set at 80/190, and power injection of 45 mL of
nonionic contrast medium (300 mg I/mL) at 4.0 mL/s is performed with a
5-second delay. The gantry angle is parallel to and above the orbital roof to
avoid radiation exposure of the lens. The total scanning time is 45 seconds.
Low or isoosmolar contrast medium is preferred to minimize the risk of adverse
reactions. The acquired source images are transferred to a workstation
(Advantage Windows, GE Healthcare) for postprocessing of parametric maps of
CBF, CBV, and MTT with the software package CT Perfusion version 3.0.
Study Design
Interobserver reproducibility—Five observers with varying
levels of skill participated in the study: a junior CT technologist, a senior
CT technologist, a third-year radiology resident, a neurology attending
physician, and a neuroradiology attending physician. At their current
positions, the junior CT technologist had 3 years of experience; senior CT
technologist, 23 years; radiology resident, 3 years; neurology attending
physician, 4 years; and neuroradiology attending physician, 2 years. Only the
neuroradiology attending physician had experience postprocessing CT perfusion
data, for a total of 3 years, including 1 year during fellowship training. To
maintain anonymity, each observer was randomly assigned an identification
number to use for all data collection and analysis. All observers received the
same 30-minute training session from a neuroradiology attending physician, who
was not one of the observers. The training session was conducted to help the
observers gain working knowledge of the postprocessing software program and to
help them understand selection of user-defined parameters based on previously
published guidelines [1].
Emphasis was placed on selection of the arterial and venous input functions
and on choosing the cutoff values for unenhanced and enhanced images according
to the following specific guidelines: first, avoid partial volume averaging by
selecting the largest vessel perpendicular to the imaging plane for the
arterial and venous regions of interest (ROIs), choosing an ROI pixel size to
fit the lumen of the arterial and venous structures, and avoiding use of
small, deep veins for the venous ROI; second, limit truncation of
time-attenuation curves by selecting the cutoff for enhanced images at a time
point after the venous time-attenuation curve has completed its downslope and
reached a plateau (but before recirculation occurs) and selecting the
unenhanced cutoff at a time point immediately preceding the upslope of the
arterial time-attenuation curve. The observers were instructed to avoid
occluded or diseased arterial and venous structures when selecting the
location for the arterial input and venous functions.
To limit performance bias, each observer had 10 practice cases to postprocess in a single day that were not included in the data analysis. After completion of the practice cases, a 14-day waiting period was instituted to ensure that there would be no bias based on time from training. The observers were then given the standard 20 CT perfusion data sets to individually postprocess and complete in a 5-day period. During postprocessing of the CT perfusion data set, the observers recorded the following information: presence of motion on axial source images, location of the arterial and venous input functions selected, unenhanced and enhanced cutoff values chosen, and whether these values were adjusted from the automatic thresholds defined by the software. The observers also used a standard stopwatch to record the time it took to postprocess each CT perfusion data set. The start point was defined as selection of the CT perfusion data set from the browser window and the end point as closing the software program.
Intraobserver reproducibility—For evaluation of intraobserver reproducibility, each observer repeated postprocessing of a standard subset of 10 cases from the original 20 data sets. To limit recall bias, each observer waited at least 14 days before postprocessing the second data set. The instructions for this section of the study protocol were the same as those for the assessment of interobserver reproducibility.
Correlation of automated postprocessing and postprocessing by observers—A neuroradiologist who was not one of the five observers postprocessed the 20 CT perfusion data sets using the automated software function. The locations of the arterial and venous structures for the time-attenuation curves and the cutoff values for unenhanced and enhanced images are selected automatically. After computation of CBF, CBV, and MTT maps with these parameters, the neuroradiologist applied the designated ROI template for each patient to derive the quantitative values in the ischemia-infarct region and the contralateral cortical gray matter and white matter. The automatically selected user-defined parameters were recorded, as was the time in minutes to complete this process in accordance with the observers' protocol. The additional standard subset of 10 cases was postprocessed as a measure of intraobserver variability for the software program.
Data Analysis
All observers used the same software program (CT Perfusion version 3.0) for
postprocessing the parametric maps of CBF, CBV, and MTT. A single
neuroradiologist who was not one of the five observers created a template for
ROI placement unique to each patient at a single slice location on the CT
perfusion axial source images. The ROI template for each patient was saved and
labeled in the computer's ROI and templates folder for later use by all
observers during postprocessing of CT perfusion data sets. These templates
were created in the program to yield identical ROI size (157 mm2)
and location on each observer's postprocessed maps for standard quantitative
analysis of CBF, CBV, and MTT. The ROI templates were designed to sample areas
of the brain with varying perfusion properties, including an abnormal region
(ischemia-infarct) and the normal contralateral cortical gray matter and white
matter (Fig. 1). The location
of the ROI was carefully selected for sampling of only the specific brain
region being studied without including other areas. Each observer was
instructed to place the appropriate ROI template for that patient on the
postprocessed CT perfusion maps.
|
Mean quantitative values of CBF, CBV, and MTT for all patients were used in the statistical analysis. The statistical software program Stata 9.0 (StataCorp) was used for all data analysis. Descriptive statistics, including mean, SD, and range, were calculated. Coefficient of variation and 95% confidence limits were used to assess the degree of variability in the quantitative data generated from the different observers and the automated program, representing the measurement error made solely on the basis of postprocessing of CT perfusion data sets by different observers. Coefficient of variation is calculated as SD divided by mean. This value is multiplied by 100 for a percentage. One-way analysis of variance statistical analysis was used to detect significant differences in the mean quantitative values CBF, CBV, and MTT among the observers' results for the ischemia-infarct and normal cortical gray matter and white matter regions in the brain. Further analysis was performed with Pearson's correlation coefficient to assess agreement in quantitative values among the varying skill levels of the observers at all possible observer pairings and between postprocessing done by the observers and automated postprocessing.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Interobserver Reproducibility
The CBF, CBV, and MTT quantitative measurements were analyzed separately
for the three selected ROI placements in the ischemia-infarct region and the
contralateral normal cortical gray matter and white matter. Each observer's
mean and SDs of CBF, CBV, and MTT were calculated for the three brain regions
(Table 1). Comparison of all of
observers simultaneously with one-way analysis of variance revealed no
significant difference between the five observers' mean CBF, CBV, and MTT
values for the ischemia-infarct region, cortical gray matter, and white matter
(F score < 2.45).
|
The coefficient of variation and 95% confidence limits for CBF, CBV, and MTT among all observers are shown in Table 2 for the ischemia-infarct region, cortical gray matter, and white matter. In further analysis, Pearson's correlation coefficients were calculated for all possible pairings between individual observers. The range was r = 0.91-0.97 for CBF, r = 0.87-0.99 for CBV, and r = 0.95-0.99 for MTT. Overall, the highest correlation (r = 0.97-0.99) occurred in the pairing of the neuroradiology attending and the radiology resident. The lowest correlation (r = 0.87-0.95) occurred in the pairings of the neurology attending with the neuroradiology attending and the neurology attending with the radiology resident.
|
Intraobserver Reproducibility
The coefficient of variation and 95% confidence limits were calculated for
the five observers (Table 3),
representing the measurement error introduced from each observer repeating the
postprocessing technique. Pearson's correlation coefficients were calculated
for each observer. The neuroradiology attending, radiology resident, and the
senior CT technologist had the highest correlation (r = 0.99) between
their two data sets. Slightly lower correlations were found for the junior CT
technologist (r = 0.95) and the neurology attending (r =
0.91).
|
Correlation Between Automation and Observers
The coefficient of variation and 95% confidence limits for automated
postprocessing compared with postprocessing by all observers is shown in
Table 2. The correlation of the
CT perfusion quantitative data derived automatically was assessed in relation
to data derived by each observer separately. The correlation coefficient
ranged from r = 0.65-0.73 for CBF, r = 0.04-0.32 for CBV,
and r = 0.86-0.87 for MTT. The lowest correlation was found in the
pairing of automated postprocessing with that by the neuroradiology attending
and the highest correlation between automation and the junior CT technologist
and automation and the neurology attending. Repeated analysis of the subset of
10 CT perfusion data sets revealed consistent selection of all user-defined
parameters and identical quantitative values, representing no intraobserver
variability for the automated program.
Software Adjustments
For the original 20 CT perfusion data sets postprocessed, the concordance
rate among all observers for selection of the arterial input function was 75%
(15/20) and of the venous function was 95% (19/20). The anterior cerebral
artery was the most frequently chosen arterial input function among all
observers, and the posterior aspect of the superior sagittal sinus was the
most frequently chosen venous function. A total of 150 CT perfusion data sets
were processed by all of the observers in both the interobserver and
intraobserver reliability sections of the study. The cutoff parameters for
unenhanced and enhanced images were not adjusted by the observers in 65.3%
(98/150) of the data sets, indicating that adequate unenhanced and enhanced
parameters were selected automatically according to the observers. However, in
the 34.7% (52/150) of the data sets that observers needed to adjust the
user-defined parameters, the unenhanced cutoff value was changed in 86.5%
(45/52), the enhanced parameter was changed in 9.6% (5/52), and both
parameters were changed in 3.8% (2/52) of the data sets.
In selection of arterial input function and venous function, the automated program incorrectly identified an arterial or venous structure for computation of the CT perfusion maps in 65% of cases. The automated program selected a venous structure for the arterial input function in 55% (11/20) of cases. In the cases in which the software program correctly selected an artery, the anterior cerebral artery (A2 segment) was chosen in 89% (8/9) of cases. The automated program selected an arterial structure for the venous function in 15% (3/20) of cases. When the program correctly selected a venous structure, the superior sagittal sinus was chosen in 82% (14/17) of cases.
Postprocessing Time
The mean time for the observers to postprocess the CT perfusion data sets
was 6.5-21.3 minutes. The shortest time was recorded for the neuroradiology
attending and the longest time for the senior CT technologist. Comparison of
mean times for all observers by one-way analysis of variance showed a
significant difference (F score > 2.49). However, each observer had a trend
toward improvement in postprocessing time from first to last data set. Each
observer's times for the repeated 10 CT perfusion data sets were compared for
assessment of intraobserver improvement. A statistically significant
difference (p < 0.05) was found for the neuroradiology attending physician
(12.9% improvement), radiology resident (33.2% improvement), and senior CT
technologist (33.8% improvement).
The mean time for the automated program to postprocess CT perfusion data sets was 4.0 minutes, statistically different from the observers' mean times, except for the neuroradiology attending physician (one-way analysis of variance with Bonferroni correction). Repeated analysis showed no improvement in postprocessing time for the automated program.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The software for postprocessing of CT perfusion data sets relies on selection of user-defined parameters, such as arterial and venous input functions and the cutoff values for unenhanced and enhanced images. These parameters are used in a mathematic model for generation of quantitative CBF, CBV, and MTT data for interpretation. Although there are fully automated and semiautomated programs for selection of these user-defined parameters, an observer must determine the most appropriate parameters for a patient's data set. Observer participation is necessary because the automated program may not correctly select an arterial or venous structure, as occurred in approximately 65% of cases in our study. Greater variability in the quantitative values of CBF, CBV, and MTT was introduced with automated postprocessing (Table 2). Improved reproducibility can be maintained with observer input in the postprocessing technique, and observers must understand how to select parameters in the software program.
We routinely use a semiautomated postprocessing mode at our institution. The locations of the arterial input and venous functions that yield the most appropriate arterial and venous time-attenuation curves are selected manually by the observer. The observer incorporates the clinical information needed to avoid an occluded or diseased vessel. The software then is used to define the unenhanced and enhanced cutoff values based on the arterial and venous time-attenuation curves chosen. Finally, the observer accepts or adjusts these parameters before generation of the CT perfusion maps. With this method, quantitative CT perfusion values can vary owing to observer interaction with the software program.
An early reproducibility study [8] showed that the quantitative values CBF, CBV, and MTT are reproducible when CT perfusion data sets are postprocessed by two experienced radiologists. Another study [2] revealed a high degree of correlation among CT technologists' postprocessing of CT perfusion data sets, the coefficients of variation being as high as 31%. However, the quantitative differences between the findings for technologists in that study manifested as significant differences in the qualitative appearance of the CBF maps, raising concern about the clinical use of quantitative CT perfusion data. Therefore there is a need to establish the reproducibility of postprocessed CT perfusion data sets between experienced and inexperienced radiologists and between radiologists and technologists.
In our study, there was a low degree of variability, 2.5-9.5% (Table 2), in the quantitative values CBF, CBV, and MTT generated by different observers postprocessing the same CT perfusion data sets. The greatest variability occurred in CBF values, particularly for the ischemia-infarct region. Even though minor variation existed among the observers, the CBF differences did not manifest as clinically significant differences in misclassification of ischemia-infarct tissue. An important finding was that clinical interpretation of CT perfusion data would not depend on an observer's postprocessing performance because the mean quantitative values of CBF, CBV, and MTT were not statistically different among all observers.
Closer inspection of the performance of the individual observers in relation with one another, however, revealed an overall high degree of correlation (r = 0.87-0.99). Slightly stronger association was consistently found between the neuroradiology attending physician and the radiology residents, possibly reflecting their similar backgrounds and familiarity with the use of imaging software. Slightly lower correlation was consistently found when the neurology attending physician was paired with the neuroradiology attending physician, radiology resident, and CT technologists. These findings may be explained by the differences in training in computer technology. This difference may have had an effect on overall understanding of selection of the most appropriate arterial and venous time-attenuation curves for avoiding partial volume averaging and of determination of the cutoff parameters for unenhanced and enhanced images to limit truncation of the time-attenuation curve and eliminate excess noise in the baseline data. Selection of the appropriate arterial input and venous functions with motion-degraded data may have been more challenging for the neurology attending physician to postprocess than for the observers trained in radiology. These differences are manifested in the intraobserver variability results. There was a slightly lower degree of correlation for the neurology attending physician compared with the neuroradiology attending physician, radiology resident, and senior CT technologist. However, overall intraobserver variability was low for all observers, 0.7-10.8% (Table 3).
Although there is less than 10% variability in postprocessed CT perfusion data among observers, the source of this variation is most likely observers' selection of user-defined parameters. A previous report [2] indicated that selection of the cutoff parameter for enhanced images is the most important source of variability and that this parameter is the only one that differs significantly among observers in postprocessing of CT perfusion data. In one third of the cases in our study, the observers did not agree with the software recommendations for unenhanced and enhanced parameters. The observers changed the unenhanced parameter in 86.5% of the data sets adjusted. Previous studies [1, 2] have shown that adjustments in the unenhanced and enhanced parameters may lead to variability in the quantitative values CBF, CBV, and MTT.
Comparison of the postprocessing times of all observers showed a significant difference. The most experienced observer (neuroradiology attending physician) had the shortest postprocessing time, and the two CT technologists had the longest postprocessing times. More important, there was a trend among all observers toward improvement in postprocessing time from the first CT perfusion data set to the last data set processed. Three observers (neuroradiology attending physician, radiology resident, and senior CT technologist) statistically improved their postprocessing times 12.8-33.8%. These findings imply that not only experience but also repetition and practice improve CT perfusion postprocessing time.
A limitation of the study was that postprocessing of all CT perfusion data sets was performed with a single program in which a deconvolution method is used to generate quantitative CT perfusion maps. It is possible that other software may yield different degrees of variability depending on the sensitivity of the program for user-defined parameters in generation of quantitative results. Future studies can be performed to determine the variability of postprocessing programs in which other mathematic methods are used.
Quantitative CBF, CBV, and MTT data obtained from postprocessing of CT perfusion data sets are reproducible among observers with varying levels of skill and experience. Observer interaction with the program is an essential component in postprocessing of CT perfusion data sets for correct selection of the arterial and venous input parameters and for limitation of the variability of quantitative data. Establishing good reproducibility of quantitative CT perfusion maps has important implications in the clinical use of this technique in the evaluation of cerebrovascular disease. However, reproducibility rests on the use of a uniform and standard postprocessing technique, particularly for selection of user-defined parameters. After a brief training session, reproducible postprocessing of CT perfusion data sets can be performed by radiologists, neurologists, residents, and CT technologists.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  V. Goh, S. Halligan, A. Gharpuray, D. Wellsted, J. Sundin, and C. I. Bartram Quantitative Assessment of Colorectal Cancer Tumor Vascular Parameters by Using Perfusion CT: Influence of Tumor Region of Interest Radiology, June 1, 2008; 247(3): 726 - 732. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P.C. Sanelli, G. Nicola, R. Johnson, A.J. Tsiouris, I. Ougorets, C. Knight, B. Frommer, S. Veronelli, and R.D. Zimmerman Effect of Training and Experience on Qualitative and Quantitative CT Perfusion Data AJNR Am. J. Neuroradiol., March 1, 2007; 28(3): 428 - 432. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
