| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1 Department of Radiology, MC 2026, University of Chicago, 5841 S Maryland Ave.,
Chicago, IL 60637.
2 Department of Health Studies, University of Chicago, Chicago, IL 60637.
3 GE Healthcare, Chicago, IL 60605.
Received February 5, 2003;
accepted after revision April 20, 2004.
Address correspondence to A. H. Dachman
(ahdachma{at}uchicago.edu).
Abstract
 Top Abstract IntroductionSubjects and MethodsResultsDiscussionReferences |
|---|
SUBJECTS AND METHODS. CT colonography of patients with proven polyps, masses, or polypoid folds was performed on an MDCT scanner, and the images were reconstructed using the standard, soft, lung, and detail algorithms. Two experiments were performed. The first used four patient data sets of a short segment of colon (30-60 images), each reconstructed using all four algorithms and then viewed on a workstation in a four-on-one format that allowed all four reconstructions to be viewed simultaneously. The second used six sets of cut-film images (four or eight images each); images within each set differed only in the reconstruction algorithm used to generate them (eight-image sets were prepared with two different level settings). Twenty-one reviewers with varying levels of experience who were unaware of the purpose of the study were asked to rank the images within each set according to their value in the detection of either polyps or masses.
RESULTS. Reviewers showed statistically significant differences in preference for the four algorithms (p = 0.037 in the computer-based experiment; for the cut-film experiment, p = 0.029 for the four-image sets and p = 0.041 for the eight-image sets). In the computer-based experiment, reviewers preferred the detail algorithm to the standard algorithm with an estimated probability of 0.67 (95% confidence interval [CI], 0.57-0.75) and the soft algorithm over the standard algorithm with an estimated probability of 0.59 (95% CI, 0.51-0.66). However, reviewers with the most experience (having interpreted at least 250 cases) preferred the soft algorithm over the standard algorithm by the same two-to-one margin as observed for the detail algorithm. In contrast, the standard and detail algorithms were ranked similarly in the cut-film experiment, with the soft and lung algorithms ranked worst.
CONCLUSION. To our knowledge, ours is the first observer study on the effect of the reconstruction algorithm on conspicuity of folds and polyps in CT colonography. Our results indicate significant differences in the reconstruction algorithms, with the soft and detail algorithms being preferred over the standard algorithm by experienced reviewers when interpreting images on a workstation. These results indicate the need for further research into the effect of reconstruction algorithms on CT colonography.
|
TopAbstractIntroductionSubjects and MethodsResultsDiscussionReferences |
|---|
|
TopAbstractIntroductionSubjects and MethodsResultsDiscussionReferences |
|---|
Interpreters
The reviewers for this study were recruited at an international meeting of
abdominal imaging radiologists, and the study was conducted during the course
of the meeting. All reviewers were board-certified radiologists who had at
least some experience in interpreting virtual colonoscopy studies. Two
different experiments are detailed in this article. Fifteen reviewers
participated in both computer-based and cut-film experiments; an additional
six reviewers participated in the cut-film experiment only. Reviewers were not
told the purpose of the study or the subject being evaluated. Reviewers were
asked their level of experience in CT colonography (i.e., the number of
colonoscopically proven cases they had interpreted).
Computer-Based Experiment
Four CT data sets were used, each corresponding to a 30- to 60-slice
section of the colon that was well distended. Three of these sets were
obtained from a scan of a single patient acquired at 60 mA (these are referred
to as sets A, C, and D, one of which contained the image of a 7-mm polyp).
These three data sets were of different segments of colon in that patient. The
fourth data set (set B) came from a scan of a second patient acquired at 100
mA that contained the image of a 10-mm polyp. Each of the four reconstruction
algorithms was used in each data set to acquire the CT data, and the resulting
four windows (one for each algorithm) were arranged on the screen in a
two-by-two layout (Fig. 1).
Preferences for the imaging software were set so that the names of the
algorithms used were hidden, and the only information visible on the screen
was a series number and the window and level settings for each image. The
order in which the four windows were arranged (i.e., upper left, upper right,
lower left, and lower right) was rotated for each reviewer across the four
sets of images, and the order for each set was rotated for each successive
reviewer. Thus, each reviewer was presented with the four algorithms in a
different order for each set, and each set was presented with one of four
different orderings depending upon the reviewer. The four sets of images were
presented one at a time in alphabetical order (i.e., A, B, C, and D).
|
All images were initially set at a display field of view of 30 cm, a window setting of 2,000 H, and a level setting of -450 H; a function key was provided that permitted the reviewer to return to these window/level settings. For each data set, the reviewer was asked to first optimize the window and level setting for each of the four images and then to rank the images from best to worst according to their relative value for detecting either polyps or masses. Each reviewer was given a score card on which he or she copied down the series number for each of the four images according to his or her ranking. After the reviewer had finished with a data set, the persons administering the study copied down the final window and level settings for each image.
Cut-Film Experiment
Six sets of single colonographic images (each cut from a larger film) were
prepared, three of which contained just four images each, whereas the other
three contained eight. All images were taken from patients scanned at 100 mA
and were prepared using a window setting of 2,000 H. Within each set, the
images differed only in the algorithm used to generate them. Each set of four
images was prepared at a single level setting, whereas each set of eight
images was prepared at two different level settings (i.e., four algorithms
times two level settings each). The three sets of eight images were prepared
using levels of -450 and -600 H (set A), -450 and -700 H (set B), and -600 and
-700 H (set C). The three sets of four images were prepared using levels of
-450 (set D), -600 (set E), and -700 H (set F). A unique number was placed at
the bottom on the back of each image so that it could be identified. All
reviewers were presented with the six sets in the following order: B, E, A, F,
C, D.
Before ranking the images in each set, the person administering the study shuffled the images and laid them out on a viewbox. The images were placed face up so that the reviewer was unable to see the identification number on the image. The reviewer was then asked to arrange the images from best to worst in order of their value in detecting a polyp or mass. As soon as the reviewer was finished with a set, he or she moved to the other side of the viewbox to work on the next set while the administrator recorded the order in which the previous set had been ranked. This arrangement permitted each reviewer to move quickly through the six sets of films.
Data Analysis
The data from the computer-based experiment were used to compute the mean
ranking for each algorithm, both separately for each set of images and for the
experiment as a whole. The Hotelling T2 test
[14] was used to test the null
hypothesis that the mean rankings were the same for all four algorithms. The
final window and level settings were plotted separately for each algorithm.
The mean rankings for each algorithm (across the four sets of images) were
also plotted against the logarithm of the number of cases previously
interpreted by the reviewer to examine whether there was any relationship
between a reviewer's level of experience and the ranking assigned by him or
her to a particular algorithm. Data from the cut-film experiment were analyzed
in the same manner.
The Plackett-Luce model [15, 16] (also known as the rank-ordered logit model in the economics literature and the exploded logit model in the marketing literature) was used to estimate the probability of preferring one algorithm over another and to determine whether this probability was affected by the characteristics of the images (i.e., window and level settings) or of the reviewers (i.e., level of experience). The model expresses the probability of a particular ordering of items as a function of a set of parameters (one for each item), each of which indicates the degree to which a reviewer prefers that particular item relative to the others. Using the difference between the parameters for a pair of items, we can estimate the probability that a reviewer prefers one item in the pair to the other. The model was fitted using the method of maximum likelihood, and the variance of the parameters was estimated using the bootstrap method [17] by resampling 1,000 times from the set of reviewers. Hypothesis tests were conducted using Wald statistics constructed from the bootstrapped variance matrix. All p values given are two-sided. All analyses were performed using the Stata statistical software package (release 7.0, StataCorp).
|
TopAbstractIntroductionSubjects and MethodsResultsDiscussionReferences |
|---|
|
Table 2 shows the results of fitting the Plackett-Luce model to the data for all sets combined. Overall, reviewers preferred the detail to the standard algorithm with an estimated probability of 0.67 (95% confidence interval [CI], 0.57-0.75). Reviewers also preferred the soft to the standard algorithm, although by a somewhat smaller margin of 0.59 (95% CI, 0.51-0.66). The difference between the preference for the detail and soft algorithms was itself statistically significant (p = 0.038), with reviewers estimated to prefer the detail to the soft algorithm 58% of the time. The lung algorithm was rated as somewhat less preferable than the standard algorithm, although this difference was not statistically significant (p = 0.511).
|
The 15 reviewers who participated in the computer-based experiment ranged in experience from those having interpreted only 10 virtual colonoscopies to those having interpreted roughly 1,000 (median, 85). Some evidence indicated that the variation in the algorithm that the reviewers preferred depended on their level of experience (p = 0.059 for a joint test of the three interaction terms between algorithm and log cases). These differences are listed in the bottom two panels of Table 2, which show the estimated probabilities of preferring each algorithm to the standard algorithm for reviewers who had interpreted 25 cases and for those who had interpreted 250 cases. More experienced reviewers were less likely to prefer the lung algorithm and more likely to prefer the soft algorithm than those with less experience. In fact, those who had interpreted 250 cases were just as likely to prefer the soft to the standard algorithm as they were to prefer the detail to the standard algorithm. We found it interesting that preference for the detail algorithm over the standard algorithm did not appear to be related to the reviewer's level of experience.
Final window and level settings (recorded after each reviewer had finished his or her ranking) are plotted in Figure 2 and Figure 3. The reviewers deviated considerably from the initial settings of 2,000 H for window and -450 H for level, tending to settle on a final window setting of just slightly below 2,000 H and were equally likely to choose a final level setting above -450 H as a level below that value. The final window settings were somewhat more spread out for the soft and detail algorithms; however, the medians for all four algorithms were similar, and the distributions of the final level settings for the four algorithms were nearly identical. The final level setting was related to image preference (p = 0.044 for a joint test of both linear and quadratic terms), and this relationship appeared to be nonlinear (p = 0.033 for the quadratic term). Specifically, a reviewer's relative preference for an image was estimated to increase with increasing level settings up to -358 H and then to decrease again. Using the bootstrap percentile method, we obtained a 95% CI for this "ideal" range of level settings of between -633 and 180 H, indicating that these data were insufficient to estimate the ideal level setting with much precision. We found no evidence that the final window settings were associated with image preference.
|
|
With regard to the difference in rankings between sets (previously noted), a joint test of the nine interaction terms between the set and the algorithm yielded a p value of 0.017, suggesting that these differences (especially in the way the detail algorithm was ranked between set D and the other sets) were not merely due to chance. In contrast, there was no evidence that the position in which an image was presented on the screen (i.e., upper left, upper right, lower left, or lower right) affected a reviewer's preference for that image.
Cut-Film Experiment
In Table 3, the mean
rankings for the cut-film experiment are presented separately for the four-
and eight-image sets. In both cases, the standard algorithm was ranked best
overall, followed by the detail algorithm. As in the computer-based
experiment, however, there were some notable differences in preferred
algorithm in the different sets of images. For example, the standard algorithm
was ranked worst (on average) in set E and second to worst among the
-700-level-setting images in set B. Using the Hotelling
T2 test to compare the mean rankings yielded
p values of 0.029 for the four-image sets and 0.041 for the
eight-image sets.
|
Fitting the Plackett-Luce model to the cut-film data (both four-image and eight-image sets combined) showed that reviewers preferred the standard to the lung algorithm, with an estimated probability of 0.65 (95% CI, 0.49-0.78) and preferred the standard to the soft algorithm, with an estimated probability of 0.59 (95% CI, 0.55-0.62). Reviewers made little distinction between the standard and detail algorithms, preferring the standard to the detail algorithm with an estimated probability of 0.53 (95% CI, 0.49-0.57). Overall, no evidence indicated a relationship between experience level and algorithm preference (p = 0.219 for a joint test of the interaction terms between the algorithm and log cases). However, as in the computer-based experiment, those with more experience tended to dislike the lung algorithm more than did those with less experience. Within each of the eight-image sets, reviewers preferred the images generated at the higher level setting (this is evident from a comparison of the mean rankings in the last column of Table 3), and this effect was linear over the range of levels studied.
|
TopAbstractIntroductionSubjects and MethodsResultsDiscussionReferences |
|---|
A limitation of this study is the fact that only four CT data sets were included, making it impossible to know whether these results could be generalized to the larger population of CT colonographic cases. Our data do suggest that algorithm preference may differ across cases, implying that one algorithm may not be uniformly the best. More work is necessary to determine how specific features of a data set might affect algorithm preference. Also, we did not test all possible algorithms. This was an intentional decision because most of the other algorithms available to us caused an unacceptable degradation of the 3D endoluminal view when surface rendering was tested in a pilot study. The routine reconstruction of CT colonographic studies into multiple algorithms is not practical; thus, we selected only algorithms that would be adaptable to both 2D and surface-rendered 3D images.
It is perhaps surprising that although the reviewers preferred the detail to the standard algorithm when interpreting a case on the computer, they preferred the standard algorithm when interpreting the cut-film images. The cut-film experiment represented an admittedly contrived situation that is unlikely to occur during the course of normal diagnostic work. Still, this difference may reflect one or more important differences in the way radiologists interpret information on a single film versus information on a series of images displayed on a computer screen.
Using the data from the computer-based experiment, we estimated the ideal level setting (i.e., that associated with the highest relative preference) to be -358 H, although the width of the CI indicates that this estimate has little precision. Moreover, it is likely that the ideal level setting may be different for different cases. Because we instructed each reviewer to select an optimal level setting for each reconstructed image, the fact that this level setting was associated with image preference even after controlling for the reconstruction algorithm implies that reviewers did not always succeed in finding the optimal level setting or that reviewers who disliked a particular reconstruction tried to improve it by shifting the level farther away from the initial setting (which was relatively near -358 H). In either case, more research should be done to determine how radiologists select a level setting for a particular case and whether this selection affects their ability to make an accurate diagnosis.
Acknowledgments
We thank GE Healthcare for providing the workstation used in this project
and the reviewers for participating in the study: Dennis Balfe, Dina Caroline,
Ronnie DuBrow, Marc Gollub, Peter Hahn, Bob Halvorsen, Amy Harris, Sunan
Hilton, R. Brooke Jeffrey, C. Dan Johnson, Bob Koehler, Michael Macari, Frank
Miller, Paul Silverman, MaryAnn Turner, Judy Yee, Kenyon Kopecky, Martina
Morrin, Bob MacCarty, Eric Paulson, and Vassilios Raptopoulos.
|
TopAbstractIntroductionSubjects and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  T. Mang, A. Maier, C. Plank, C. Mueller-Mang, C. Herold, and W. Schima Pitfalls in Multi-Detector Row CT Colonography: A Systematic Approach RadioGraphics, March 1, 2007; 27(2): 431 - 454. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. A. Taylor, S. Halligan, A. Slater, V. Goh, D. N. Burling, M. E. Roddie, L. Honeyfield, J. McQuillan, H. Amin, and J. Dehmeshki Polyp Detection with CT Colonography: Primary 3D Endoluminal Analysis versus Primary 2D Transverse Analysis with Computer-assisted Reader Software Radiology, June 1, 2006; 239(3): 759 - 767. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
