| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Original Research |
1 Department of Intestinal Imaging, St. Mark's and Northwick Park Hospitals,
Watford Rd., Harrow HA1 3UJ, United Kingdom.
2 Present address: Department of Specialist X-ray, University College Hospital,
235 Euston Rd., London NW1 2BU, United Kingdom.
3 Department of Radiology, Charing Cross Hospital, London, United Kingdom.
4 Medicsight PLC, London, United Kingdom.
Received December 31, 2004;
accepted after revision February 7, 2005.
Address correspondence to S. A. Taylor
(csytaylor{at}yahoo.com).
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. A library of colonoscopically validated CT colonography cases were collated and separated into training and test sets according to the time of accrual. Training data sets were annotated in consensus by three expert radiologists who were aware of the colonoscopy report. A subset of 45 training cases containing 100 polyps underwent batch analysis using ColonCAR version 1.2 software to determine the optimum polyp enhancement filter settings for polyp detection. Twenty-five consecutive positive test data sets were subsequently interpreted individually by each expert, who was unaware of the endoscopy report, and before generation of the annotated reference via an unblinded consensus interpretation. ColonCAR version 1.2 software was applied to the test cases, at optimized polyp enhancement filter settings, to determine diagnostic performance. False-positive findings were classified according to importance.
RESULTS. The 25 test cases contained 32 nondiminutive polyps ranging
from 6 to 35 mm in diameter. The ColonCAR version 1.2 software identified 26
(81%) of 32 polyps compared with an average sensitivity of 70% for the expert
reviewers. Eleven (92%) of 12 polyps 
10 mm were detected by ColonCAR
version 1.2. All polyps missed by experts 1 (n = 4) and 2 (n
= 3) and 12 (86%) of 14 polyps missed by expert 3 were detected by ColonCAR
version 1.2. The median number of false-positive highlights per case was 13,
of which 91% were easily dismissed.
CONCLUSION. ColonCAR version 1.2 is sensitive for polyp detection, with a clinically acceptable false-positive rate. ColonCAR version 1.2 has a synergistic effect to the reviewer alone, and its standalone performance may exceed even that of experts.
Keywords: colon • colonography • computer-assisted imaging • CT colonography • polyps • software
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
However, it is increasingly clear that individual variation in the interpretive capabilities of the reviewers plays a significant role. A definite, as yet undefined, learning curve exists for interpreting CT colonography examinations [11], and perceptual errors perhaps account for more than 50% of missed lesions [12].
By highlighting potential polyp candidates for the reviewer, computer-aided detection (CAD) holds considerable promise as a means by which observer sensitivity can be improved, assuming it can show adequate sensitivity itself. It has been shown, however, that the performance of the CAD algorithm need not exceed the performance of the individual reviewer to be effective (Giger M et al., presented at the 2004 annual meeting of the Radiological Society of North America). Several researchers have reported promising data using CAD algorithms, mainly as a second reviewer [13-19]. In contrast, computer-assisted reader (ColonCAR version 1.2, Medicsight PLC) software highlights polyp candidates for the radiologist at the time of the primary interpretation, rather than being used subsequently as a second reviewer, and has had recent regulatory approval. Furthermore, whereas CAD programs use a fixed-detection algorithm, ColonCAR version 1.2 software is interactive, allowing the observer to alter polyp detection filters to match the clinical requirements. For example, these filters may trade off decreased sensitivity for smaller polyps in exchange for improved specificity and may also account for acquisition parameters, use of tagging agents, adequacy of distention, and so forth. The intention is that the radiologist can ultimately calibrate the sensitivity/specificity profile according to individual and local requirements. However, for this interactive calibration to be viable in a clinical setting, the software must have demonstrably adequate baseline performance characteristics at default settings. The purpose of this study was to determine the default sensitivity of recently developed ColonCAR version 1.2 software for polyp detection, and to compare this with the performance of unassisted reviewers who are expert in CT colonography.
|
|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Training Data Set
A total of 242 training cases (159 men, 83 women; median age, 62 years;
range, 34-89 years) were collected. Of the cohort, 147 were average-risk
screening patients, and the rest were high risk or symptomatic. All patients
had undergone full bowel preparation using standard cleansing regimens.
Technical parameters for CT colonography were as follows: 1.25-mm collimation
(38 cases), 2.5-mm collimation (193 cases), 3-mm collimation (11 cases); 120
kV (235 cases), 140 kV (seven cases); 35 mA (five cases), 50 mA (30 cases),
100 mA (181 cases), 120 mA (21 cases), 200 mA (one case), 240 mA (one case),
280 mA (two cases), and 320 mA (one case). Fecal and fluid tagging, IV
contrast material, and IV hyoscine butylbromide were used in 146, 1, and 40
cases, respectively. Prone and supine data sets were obtained in all
patients.
All cases were loaded onto a computer workstation equipped with CT colonography software (MedicColon, version 1.1, Medicsight PLC). This software allowed side-by-side viewing of 2D supine and prone axial images, supplemented by multiplanar reformations and a surface-shaded 3D cut surface for problem solving. Each case was interpreted in consensus by two of three radiologists with expertise in CT colonography who also had full knowledge of the colonoscopy report for the case in question. All three radiologists had previously interpreted at least 200 endoscopically confirmed CT colonography cases and had audited detection rates in-line with published literature. Each case was interpreted fully by the observers using a primary 2D interpretation with multiplanar reconstruction; 3D views were reserved for problem solving. Although the reviewers paid special attention to locating the polyps documented by colonoscopy, a search for endoscopic false-negative findings was also made by careful examination of the whole colon by both reviewers. Any presumed endoscopic false-negative finding was indicated as a true polyp only if both reviewers had high confidence that the lesion was real when judged by generally accepted criteria (e.g., homogeneous, well defined, present on both supine and prone data sets). Disagreement was resolved by face-to-face discussion. If the colonoscopy suggested more than one polyp in a particular colonic segment, size was used to determine the best match for the CT colonography-detected polyp with its endosopic equivalent [16]. Cases in which the reviewers could not locate endoscopically verified polyps were not later reanalyzed after application of the ColonCAR version 1.2 software. If detected, these cases would, by definition, be identified as false-positives for the ColonCAR version 1.2 software [16] (see the following text).
|
|
Software Development
The annotated training data sets were analyzed to determine geometric
features differentiating true polyps (as indicated by the expert reviewers)
from surrounding structures. Initial observations revealed that most polyps
appeared as spherical or semispherical objects protruding into the colonic
lumen, in contradistinction to folds, which appeared as elongated, raised
objects. These observations are in-line with those from other researchers
[13-15,
18,
20-22].
On the basis of these observations, a mathematic algorithm was developed that
first segmented the colon from the CT data set and then determined the
inherent sphericity of all raised objects in the colonic lumen. The algorithm
was then applied to a subset of 45 of the annotated training data sets that
contained 100 polyps of 3 mm or more. Values of object sphericity and also
height (or flatness) were systematically altered across the available range,
and each new setting combination was applied to the data set. The algorithm
constructed regions of interest around each polyp candidate, and the
coordinates of these highlighted regions were compared with the coordinates of
true polyps (contained in the binary image files created by the prior data set
annotation). In this way, the number of true- and false-positives could be
determined for each combination of parameter settings. A receiver operating
characteristic (ROC) curve was subsequently generated and optimal settings
determined (based on 95% polyp detection with the fewest false-positive
annotations). The optimal values of sphericity and flatness were deemed the
default setting (see following text).
Once developed, the algorithm was incorporated into a colon visualization program (ColonCAR version 1.2, Medicsight PLC) designed to highlight potential polyp candidates for the reviewer. The interface allows the user to exert control over the degree of flatness and sphericity of detected objects via slider bars with an arbitrary scale between 0.5 and 1. Once the settings of sphericity and flatness are set, the user applies this polyp enhancement filter to each of the supine and prone data sets in turn. Detected objects were highlighted for the radiologist via red and yellow rings superimposed on the axial 2D image (Fig. 1B). The corresponding non-filtered 2D image was displayed adjacent to the filtered image. The polyp enhancement filter takes, on average, 0.3 sec per slice.
Test Data Set
To document the diagnostic performance of expert radiologists, the test
data sets were initially interpreted independently by each of the three
experts (see previous definition) using the MedicColon version 1.1 viewing
software (i.e., primary 2D interpretation with 3D for problem solving). The
polyp enhancement filter was not applied, and reviewers were unaware of the
endoscopy findings, indication for the study, and overall prevalence of
abnormality in this test data set as a whole. Reviewers annotated the position
of each polyp they detected using the manual drawing tool described
previously, with each annotation again saved as a binary image file. Slice
numbers and maximum polyp size were also documented exactly as for the
training set annotation. Once the three experts had interpreted all available
test cases, the reference standard for this set was determined. This reference
standard was based on a repeat interpretation of the test cases by all three
expert radiologists, this time in face-to-face consensus and with full
knowledge of the colonoscopy report for each individual patient. Polyps were
annotated and documented in the same way as for the training set. As before,
presumed endoscopic false-negatives were annotated only if all three reviewers
had high confidence that the lesion truly represented a polyp. The reference
standard size for a polyp was determined in consensus, with reference to the
endoscopic size and the CT colonography 2D measurement.
The study sheets for the individual expert interpreters were then marked by comparison with the reference standard study sheets by one of the reviewers. Polyps were marked as true-positives if the documented slice numbers on either supine or prone data sets (or both) matched (within two slices either way), and if the polyp size was within 3 mm. If a reviewer had documented a polyp but slice numbers or size were mismatched with the reference standard, the case was recalled on the workstation. By use of the saved binary image files, the site of the reference standard annotation could be recalled, along with the annotation made by the individual reviewer for comparison. In this way it was possible to categorize reviewers' observations as true-positive or false-positive with certainty. This procedure also allowed accountancy for transcription errors in slice numbers or size mismeasurement.
Assessment of ColonCAR Version 1.2 Software
The first 25 consecutive positive test cases were used for the present
study. These 25 data sets contained a total of 32 nondiminutive polyps (> 5
mm). Thirty of these had been identified on colonoscopy (two were presumed
endoscopic false-negatives of 6 and 7 mm). All endoscopically detected polyps
were identified and annotated via the consensus interpretation. Thirteen cases
used fecal and fluid tagging, two cases received IV contrast material, and no
smooth-muscle spasmolytic was administered in any case. The acquisition
parameters were as follows: 120 kV (all cases); 100 mA (13 cases), 50 mA (nine
cases), 200 mA (three cases); and collimation, 1.25 mm (one case), 3.2 mm (two
cases), and 2.5 mm (22 cases).
The cases were loaded on a workstation with proprietary software (ColonCAR version 1.2) by one of the expert reviewers. The annotated position of polyps from the consensus reference standard interpretation was recalled by simultaneous loading of the saved binary image file. The reviewer then applied the polyp enhancement filter to the supine and prone data sets in turn using the default settings (see previous description) for sphericity and flatness. Circled regions of interest were classified as either true-positive (if they corresponded to all or part of a known polyp identified and annotated by the reference interpretation) or false-positive (if no polyp was present). The reviewer then classified false-positives into one of four categories that were defined by the ease with which they could be dismissed, as follows: category E, immediately dismissed, extracolonic; category 1, immediately dismissed, intracolonic (e.g., obviously the rectal tube, ileocecal valve); category 2, quickly dismissed (i.e., only a few seconds required for analysis and decision making) using 2D axial scrolling only (e.g., obvious fecal residue, indentation by extracolonic structures, normal fold); and category 3, dismissed only after problem-solving maneuvers (e.g., supine-prone correlation, use of multiplanar reconstructions, or 3D cube view).
Analysis
The CAR software was deemed to have detected a polyp if the polyp was
highlighted on either the supine or prone data set or both. Comparison of
detection rates between tagged and nontagged studies was made using the
Fisher's exact test. The overall number of false-positives was also compared
between tagged and nontagged data sets using the Mann-Whitney U test
statistic. Differences in sensitivity for the detection of polyps among the
three experts and the polyp enhancement filter were assessed using Fisher's
exact test. A per-patient analysis was not performed because all cases were
positive. Significance was inferred at a probability level of 5%.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The performance of individual experts compared with the polyp enhancement filter alone is shown in Table 1. No significant differences were seen for detection between any expert and polyp enhancement filter, although the polyp enhancement filter detected 26 of 32 polyps of 6 mm or over, compared with 18 of 32 for reviewer 3 (p = 0.06). Of the false-negative polyps for the experts, 10, four, and one were missed by one, two, or all three reviewers, respectively. Furthermore, of the polyps missed by the three experts, the polyp enhancement filter detected four of four for reviewer 1, three of three for reviewer 2, and 12 of 14 for reviewer 3 (Figs. 3A and 3B), and highlighted all polyps missed by at least two reviewers. Therefore, assuming all reviewers had correctly reacted to and classified objects highlighted by the polyp enhancement filter, polyp sensitivity would have been 100%, 100%, and 94% for the three reviewers.
|
|
|
Of the 26 polyps of 6 mm or more highlighted by the polyp enhancement filter, nine (35%) were detected on the supine scan, seven (27%) were detected on the prone scan, and 10 (38%) were detected on both data sets. No significant difference was seen in polyp detection between tagged (12/14) and non-tagged (14/18) data sets (p = 0.67).
The median number of false-positives per two-view data set was 13 (range, 3-28) (supine: median, 6; range, 0-15; and prone: median, 7; range, 1-16). Table 2 shows the breakdown of false-positives according to ease of dismissibility. Only 9% of false-positives (median, 1 per case) required further investigation either with supine-prone correlation, use of multiplanar reconstructions, or 3D problem solving: most of these eventually proved to be fecal residue or bulbous haustral folds. The remaining false-positives were either extracolonic or intracolonic but were rapidly dismissible with 2D axial scrolling only.
|
Significantly more false-positives were generated by tagged data sets than by nontagged data sets (median, 16 vs 8; p = 0.03). Many false-positives in tagged data sets were generated at the fluid-air interface and thus were quickly dismissible (category 2) (Fig. 4).
|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The polyp enhancement filter that we evaluated detected 81% of all polyps of 6 mm or larger and 92% of polyps measuring 10 mm or more, figures that compare favorably with previous data published for CAD algorithms [14, 16, 19]. We found no significant difference in the detection rates of polyps 6 mm or greater when the polyp enhancement filter was compared with three unaided expert reviewers. Indeed, the software was slightly better than the average performance of the reviewers and considerably better than that of reviewer 3. The study was powered to detect a difference in performance of 20%, which was deemed to be a clinically problematic performance gap. Although the polyp enhancement filter was theoretically underpowered to detect smaller differences, its actual levels of detection were high and indeed outperformed one of the expert reviewers.
It is interesting that the polyp enhancement filter detected all polyps missed by reviewers 1 and 2, and all but two polyps missed by reviewer 3, suggesting the software is complementary even to expert reviewers. Indeed, assuming all reviewers had correctly classified objects highlighted by the polyp enhancement filter, the potential contribution of ColonCAR version 1.2 to the sensitivity of reviewers in this study was 10%, 13%, and 38%, respectively for reviewers 1, 2, and 3. Overall polyp sensitivity would have been 100%, 100%, and 94% for the three reviewers had all true-positive ColonCAR version 1.2 marks been noted. The data also suggest that because the polyp enhancement filter increased the performance of even expert reviewers, the performance of nonexpert reviewers might also be significantly improved by the use of the ColonCAR version 1.2 software.
Although the polyp enhancement filter correctly highlighted most polyps of 6 mm or more, this filter will translate into improved reviewer performance only if these highlighted objects are correctly interpreted as polyps by the reviewer. Therefore, any CAD or ColonCAR version 1.2 system in its current form is intended to work synergistically with, not instead of, the reviewer, and therefore cannot replace the fundamental need for adequate reviewer training in the interpretation of CT colonography data sets. Studies investigating the impact of ColonCAR version 1.2 software on actual reviewer performance are clearly required.
The polyp enhancement filter highlighted a relatively large number of false-positives compared with some CAD systems [14], although the findings were comparable with other studies [16, 18]. However, more than 90% of these false-positives were classified as easily dismissed by the adjudicator and not a productivity hindrance. A median of one per case required more detailed problem solving. Because ColonCAR version 1.2 filtration is displayed simultaneously with the unfiltered data, the impact of false-positives is likely to be less than with second-interpretation CAD systems. In the ColonCAR version 1.2 paradigm, the colon is interpreted once, with the highlighted regions of interest intended merely to direct reviewers' attention to the corresponding regions of the adjacent unfiltered data. An obvious false-positive thus does not materially interfere with interpretation because it can be dismissed quickly during the course of the primary interpretation. In contrast, a second-interpretation CAD paradigm requires that each highlighted region must be reviewed in turn only after a complete and unaided primary interpretation. The result is that any false-positive prompt must draw the reviewer back to areas of the colon previously deemed normal on the primary interpretation. Furthermore, because the reviewer can interactively alter the ColonCAR version 1.2 filters, it is possible to reduce the number of false-positives and therefore increase specificity (e.g., by increasing sphericity) while accepting that ColonCAR version 1.2 sensitivity may be correspondingly reduced.
We examined a default setting derived from a training set of 100 polyps; we made no attempt to assess the effect on detection of changing the two available parameters (sphericity and flatness) in this study. Furthermore, we used a mixture of cases from several institutions—for example, with differing acquisition parameters and with and without fecal tagging. It is likely that the polyp enhancement filter can be altered to take account of these other factors and possible that performance could have been improved further still.
For example, the single missed polyp larger than 10 mm was submerged in tagged fluid on both the supine and prone scans, suggesting a change in the filter algorithm may be required to deal with this type of situation. Further research on the optimal parameters for certain classes of cases is required. It may be that subtraction software will be required in such cases, but the well-documented artifacts induced by subtraction [23] might increase the number of false-positives. In our study, tagged fluid itself resulted in significantly more false-positive annotations, again suggesting a change in filter settings will be required for these cases, or better preprocessing or segmentation for this class of cases.
Our study has some weaknesses. We used an indirect design and did not directly assess reviewer performance with and without the polyp enhancement filter. Nor did we investigate any effect on interpretation time. However, this is an early iteration of the software, and these aspects were not the primary aim of our study. Rather, we wished to document the baseline sensitivity of the software when used at default settings. One strength of our protocol was that the test cases were entirely novel to the software because it had been developed using other data (i.e., the training set). Although individual cases were not shared between the training and test data sets, the same institutions contributed cases to both sets. It could therefore be argued that an inevitable degree of similarity exists between them. However, the test sets contained data from five centers using a wide range of scanning protocols, which was clearly a more demanding test for the software than using data obtained from a single institution. Our study was powered to detect a 20% difference in sensitivity between the software and the unaided expert reviewers, with the hypothesis that the software would fare best. In retrospect, the difference between the reviewers and the software was less than we had anticipated, but consensus of three expert reviewers plus optical colonoscopy correlation is clearly a demanding reference standard. It would have been interesting to determine how the software might affect the performance of less expert reviewers (who might be the most appropriate target for this type of software), prospectively or in a clinical rather than a test setting; further work in this area is required. An expert adjudicator classified the importance of false-positive prompts, and it is possible that some of these might have been more troublesome to a nonexpert reviewer. The impact on clinical workflow is a further area of study.
In conclusion, the sensitivity of the Colon-CAR version 1.2 software tested at default settings is complementary to the performance of expert reviewers of CT colonography and showed that expert reviewers have the potential to improve as much as 38% with computer assistance. The ColonCAR version 1.2 false-positive rate is acceptable in a clinical setting because most false-positive findings are quickly dismissible. Further work is required to optimize the polyp enhancement filter settings according to individual case specifics and to assess directly the clinical impact of the software on reviewer performance and productivity.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  S. A. Taylor, C. Robinson, D. Boone, L. Honeyfield, and S. Halligan Polyp Characteristics Correctly Annotated by Computer-aided Detection Software but Ignored by Reporting Radiologists during CT Colonography Radiology, September 29, 2009; (2009) radiol.2533090356v1. [Abstract] [Full Text]
|
|
|
|
 |
|
 S. A. Taylor, J. Brittenden, J. Lenton, H. Lambie, A. Goldstone, P. N. Wylie, D. Tolan, D. Burling, L. Honeyfield, P. Bassett, et al. Influence of Computer-Aided Detection False-Positives on Reader Performance and Diagnostic Confidence for CT Colonography Am. J. Roentgenol., June 1, 2009; 192(6): 1682 - 1689. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. M. Summers, L. R. Handwerker, P. J. Pickhardt, R. L. Van Uitert, K. K. Deshpande, S. Yeshwant, J. Yao, and M. Franaszek Performance of a Previously Validated CT Colonography Computer-Aided Detection System in a New Patient Population Am. J. Roentgenol., July 1, 2008; 191(1): 168 - 174. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. A. Taylor, R. Greenhalgh, R. Ilangovan, E. Tam, V. A. Sahni, D. Burling, J. Zhang, P. Bassett, P. J. Pickhardt, and S. Halligan CT Colonography and Computer-aided Detection: Effect of False-Positive Results on Reader Specificity and Reading Efficiency in a Low-Prevalence Screening Population Radiology, April 1, 2008; 247(1): 133 - 140. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S A TAYLOR, D BURLING, M RODDIE, L HONEYFIELD, J MCQUILLAN, P BASSETT, and S HALLIGAN Computer-aided detection for CT colonography: incremental benefit of observer training Br. J. Radiol., March 1, 2008; 81(963): 180 - 186. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. A. Taylor, S. C. Charman, P. Lefere, E. G. McFarland, E. K. Paulson, J. Yee, R. Aslam, J. M. Barlow, A. Gupta, D. H. Kim, et al. CT Colonography: Investigation of the Optimum Reader Paradigm by Using Computer-aided Detection Software Radiology, December 19, 2007; (2007) 2461070190. [Abstract] [Full Text]
|
|
|
|
|
|
M. E. Baker, L. Bogoni, N. A. Obuchowski, C. Dass, R. M. Kendzierski, E. M. Remer, D. M. Einstein, P. Cathier, A. Jerebko, S. Lakare, et al. Computer-aided Detection of Colorectal Polyps: Can It Improve Sensitivity of Less-Experienced Readers? Preliminary Findings Radiology, October 1, 2007; 245(1): 140 - 149. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. G. Fletcher, F. Booya, R. M. Summers, D. Roy, L. Guendel, B. Schmidt, C. H. McCollough, and J. L. Fidler Comparative Performance of Two Polyp Detection Systems on CT Colonography Am. J. Roentgenol., August 1, 2007; 189(2): 277 - 282. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. Mang, A. Maier, C. Plank, C. Mueller-Mang, C. Herold, and W. Schima Pitfalls in Multi-Detector Row CT Colonography: A Systematic Approach RadioGraphics, March 1, 2007; 27(2): 431 - 454. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
