AJR 2003; 181:1487-1490
© American Roentgen Ray Society
Using Receiver Operating Characteristic Methodology to Evaluate the Diagnostic Quality of Radiography on Paper Prints Versus Film
T. A. Bley1,
E. Kotter,
U. Saueressig,
O. S. Springer,
D. Fisch,
N. A. Ghanem and
M. Langer
1 All authors: Department of Diagnostic Radiology, University Hospital Freiburg,
Hugstetter Stra. 55, Freiburg 79106, Germany.
Received January 22, 2003;
accepted after revision June 2, 2003.
Address correspondence to T. A. Bley
(bley{at}mrs1.ukl.uni-freiburg.de).
Presented at the annual meeting of the American Roentgen Ray Society, San
Diego, CA, May 2003.
Abstract
OBJECTIVE. The aim of this study was to compare the diagnostic
quality of paper prints with film copies in a sample of observers who were
trying to detect small coin lesions on radiographs of a phantom.
MATERIALS AND METHODS. The phantom consisted of 60 high-contrast and
60 low-contrast test objects, half of which had holes in them. Diameter and
depth of the holes varied from 0.5 mm to 2 mm. Fifteen radiographs were
obtained from different areas of the test objects. Film copies and paper
prints were made using high-quality printers. Five observers independently
evaluated 1,800 high-contrast and 1,800 low-contrast images. Data were
evaluated using the well-established receiver operating characteristic
methodology.
RESULTS. The mean area under the curve rated 0.863 for paper prints
(0.859 for high contrast and 0.860 for low contrast) and 0.926 for laser films
(0.937 for high contrast and 0.913 for low contrast). The difference between
the two imaging techniques was statistically significant for both high- and
low-contrast lesions (p < 0.05).
CONCLUSION. Detection of small coin lesions on radiographs of a
phantom was significantly less sensitive on paper prints than on film. We
found paper prints less acceptable for the diagnosis of small-sized
lesions.
Introduction
Since the introduction of digital radiography, radiologists have tended to
view digital radiographs, CT, and MRIs on digital monitors at work stations.
Film copies are no longer used by radiologists. To reduce costs, increasing
numbers of radiologic institutions and departments are changing from film
copies to lower-priced paper prints for documenting radiologic findings
[1]. Depending on leasing costs
for the equipment, normal 14 x 17 inch paper prints cost approximately
10¢ each. A film copy of the same size, including processing, costs
approximately $3. Because the operating costs of a paper print system are
comparatively low, the cost per print decreases as the total number of prints
increases.
Under the circumstances, paper prints are given to the clinician for
documentary purposes. However, radiologists may be asked to compare findings
with previous external examinations documented on paper prints. Radiologists
may be asked to spontaneously make a diagnosis using paper prints when no
digital workstation is at hand.
In a previous study using CT scans, we found that paper prints have
sufficient quality for documentation of radiologic findings for most purposes
but not for primary diagnosis
[2]. The aim of this study was
to compare the diagnostic value of high-quality paper prints with film copies
for detecting small coin lesions in a contrast-detail radiography phantom.
Materials and Methods
Phantom
A commercially available contrast-detail radiography phantom was used
(TRG-Phantom, Alvim, R&D, Jerusalem, Israel)
[3]. It consisted of a
polyvinylchloride body with two groups of six columns each, in which ten
circular plates (test objects meant to represent small coin lesions) were
arranged (Fig. 1). Columns
1–6 included polyvinylchloride plates for high contrasts; columns
7–12 included polymethyl-methacrylate plates for low contrasts. Half of
the plates had a small hole somewhere in the middle. The size of the hole
varied from 0.5 mm to 1.0 mm in the polyvinylchloride plates and from 0.9 mm
to 2.0 mm in the polymethylmethacrylate plates. The depth of each hole equaled
its diameter. The plates were arranged with increasing diameter of the hole
from column 1 to 6 and from column 7 to 12. The arrangement of plates in each
column was randomized for each radiograph. The probability of a hole being
present at a particular location was 0.5. Fifteen different arrangements of
the plates were randomized in the phantom. The setup of our study was similar
to the one used in a previous study using the same type of phantom
[4].
View larger version (122K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 1. —Radiograph of phantom with six columns of 10 high-contrast
polyvinylchloride test objects and six columns of 10 low-contrast
polymethylmethacrylate test objects. Holes are 0.5–2 mm deep and wide
and appear in half of test objects. For our study, test objects were arranged
randomly.
|
|
Radiographs and Processing
Fifteen digital radiographs of different areas of the plates were obtained
with a Optitop 150/40/0 radiography tube (Siemens, Erlangen, Germany) with the
following parameters: 44 kV; 3.2 mAs; focus–detector distance, 115 cm.
No grid or filter was used. In each radiograph, the arrangement of the test
objects was randomized in each column. The ADC HR cassettes MD30 were
processed by an ADC compact plus (Agfa, Mortsel, Belgium). Quality of the
radiographs was optimized with the following processing: collimation
configuration, 1.00; musica contrast, 2.00; edge contrast, 2.00; latitude
reduction, 2.00; noise reduction, 5.00; dose indicator, 1.97; window/center,
1.07/2.05.
Paper prints and film copies were produced from these digital radiographs
of the phantom. Center and window for the film copies (center 2.05, window
1.07) and paper prints (center 450, window 2,300) were optimized. Paper prints
were printed with a paper printing system (ConVis System Integrations, Mainz,
Germany; Röntgen Bender, Baden-Baden, Germany). A high-quality laser
printer specially calibrated for radiography prints, Xerox DC12, Series 50
(Xerox, Stanford, CA) was used with 600 dots per inch and 8-bit gray scale.
Film copies were obtained by a laser camera AP 400 (Agfa). No magnification
was used between true hole size and perceived hole size.
Image interpretation
Five observers (one fellow and four residents, with 1–8 years of
experience in radiology, mean 3.5 years) interpreted film copies and paper
prints of the phantom radiographs independently of each other in a quiet
atmosphere. Paper prints were carefully examined under good light. The laser
prints were examined on a tested light box with maximal aperture. For the
receiver operating characteristic methodology, each observer had to define the
probability of presence of a hole for each of the test objects according to a
5-point rating scale: 1, definitely no presence of a hole; 2, probably no
presence of a hole; 3, indeterminate; 4, probably a hole present; and 5,
certainly a hole present.
Statistics
Multiobserver receiver operating characteristic methodology was used to
analyze the rating data
[5–9].
The LABMRMC [10] program was
used to perform the calculations.
Diagnostic accuracy was measured using the area under the binomial receiver
operating characteristic curve. A total of 18,000 interpretations (9,000 high
contrast, 9,000 low contrast) were analyzed (6 sizes x 10 rows x 2
contrasts = 120 per radiograph x 15 arrangements x 2 modalities
x 5 observers). The total of 18,000 interpretations equals 1,500
interpretations per hole size (120 test objects per radiograph vs 10 test
objects of each hole size per radiograph). The significance level was set at
p < 0.05. The overall performances of both modalities and both
contrast levels were analyzed. Receiver operating characteristic curves of all
observers combined were obtained by computing mean values of the individual
observers.
Results
The mean area under the receiver operating characteristic curve was 0.926
(standard error [SE], 0.015) for film copies and 0.863 (SE, 0.024) for paper
prints. In high-contrast test objects, the mean area under the curve was 0.937
(SE, 0.016) for film copies and 0.859 (SE, 0.031) for paper prints. In
low-contrast test objects, the area under the curve was 0.913 (SE, 0.016) for
film copies and 0.861 (SE, 0.023) for paper prints. All the differences were
statistically significant, with p < 0.05.
The holes representing coin lesions were detected more accurately on film
copies than on paper prints. The observers found film copies easier to
interpret for high-contrast and for low-contrast test objects and for the
total of all test objects (Figs.
2,3,4).
When segregating our results by hole size, we observed a significant
difference for the hole size of 0.7 mm and a random error for the hole size of
0.5 mm in high-contrast test objects (Figs.
5 and
6).
View larger version (14K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 2. —Graph shows area under receiver operating characteristic
curve for paper prints and film copies for all test objects. On film copies
(solid line), observers detected significantly more holes than on
paper prints (broken line) (smaller false-positive fraction with
higher true-positive fraction).
|
|
View larger version (14K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 3. —Graph shows area under receiver operating characteristic
curve for paper prints (broken line) and film copies (solid
line) for high-contrast test objects. Difference in observer's
performance between paper prints and film copies is significant and more
pronounced in high-contrast test objects.
|
|
View larger version (15K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 4. —Graph shows area under receiver operating characteristic
curve for paper prints (broken line) and film copies (solid
line) for low-contrast test objects. Difference in observer's performance
between paper prints and film copies is statistically significant but less
pronounced in low-contrast test objects.
|
|
View larger version (12K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 5. —Graph shows receiver operating characteristic data of
high-contrast test objects segregated by hole size and by paper ( )
versus film (•). No significant difference in detection of holes was
found between paper and film except for holes of 0.7 mm
(asterisk).
|
|
View larger version (11K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 6. —Graph shows receiver operating characteristic data of
low-contrast test objects segregated by hole size and by paper ()
versus film (•). No significant difference in detection of holes was
found between paper and film.
|
|
Discussion
Modern paper printing systems produce high-quality images sufficient for
documentation of radiologic findings in digital radiography
[11,
12]. Our earlier study
[2] showed that the image
quality of paper prints is adequate to document the findings in 94% of CT
scans. We concluded that a modern paper printing system can replace the
expensive laser hard copies for documenting radiologic findings in most CT
examinations. Similar results were obtained in a study by Ibbott et al.
[13] in which paper prints of
CT scans were rated as acceptable documentation in 95% of findings.
Compared with film copies, the contrast and detail resolution of paper
prints seems too limited, especially for examining the lung parenchyma; small
coin lesions might be overlooked. Some fine details of the lung parenchyma
itself may no longer be differentiated
[2]. Using phantoms, Warren
[14] showed that laser hard
copies are superior to paper prints for showing detail in the presence of
image noise. On the other hand, Lyttkens et al.
[15] found no difference
between laser hard copies and paper prints for detecting coin lesions on
conventional radiographs of the chest that were simulated using a phantom.
In modern filmless radiology departments, radiologists work with images on
digital work-stations instead of hard copies. Paper prints instead of film
copies are given to the clinician to reduce costs. In this setting,
radiologists may be asked to compare findings with previous external
examinations documented on paper prints. The aim of the current study was to
compare the diagnostic value of paper prints with film copies for detection
rather than documentation of coin lesions in a radiography phantom.
The phantom we used was specially designed to test the precision of the
complete imaging chain. The size of the lesions was adjusted to perceptibility
rather than to match coin lesions in conventional chest radiography or CT.
Therefore, one must not directly transfer results to clinical applications but
use them to quantify the precision of different imaging techniques.
Our results indicate that small coin lesions in a radiography phantom are
significantly better detected on film copies than on high-quality paper
prints. Further analysis of patients' radiographs is necessary to evaluate
detectability of coin lesions in a clinical setting. For daily routine in
clinical practice, we conclude that paper prints are of less value for
detecting small contrast details and are therefore less acceptable for the
diagnosis of small lesions.
References
- Denslow S. Desktop publishing and medical imaging: paper as
hardcopy medium for digital images. J Digit Imaging1994; 7:140
–145[Medline]
- Bley T, Burger D, Ghanem N, et al. Image quality of paper prints
compared to film copies in CT investigations. Rofo Fortschr Geb
Rontgenstr Neuen Bildgeb Verfahr2002; 174:423
–425[Medline]
- Gurvich VA. Statistical approach for image quality in daily medical
practice. Med Phys 2000;27
: 94–100[Medline]
- Fischbach F, Ricke J, Freund T, et al. Flat panel digital
radiography compared with storage phosphor computed radiography: assessment of
dose versus image quality in phantom studies. Invest
Radiol 2002;37:609
–614[Medline]
- Dorfman DD, Alf E. Maximum likelihood estimation of parameters of
signal detection theory and determination of confidence intervals: rating
method data. J Math Psychol1969; 6:469
–487
- Hanley JA, McNeil BJ. The meaning and use of the area under a
receiver operating characteristic (ROC) curve.
Radiology1982; 143:29
–36[Abstract/Free Full Text]
- Metz CE. Some practical issues of experimental design and data
analysis in radiological ROC studies. Invest Radiol1989; 24:234
–245[Medline]
- Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC
curves obtained from partially paired datasets. Med Decis
Making 1998;18:110
–121[Abstract/Free Full Text]
- Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic
rating analysis: generalization to the population of readers and patients with
the jackknife method. Invest Radiol 1992;27
: 723–731[Medline]
- Kurt Rossman Laboratories web site. Available at:
xray.bsd.uchicago.edu/krl/index.htm.
Accessed August 4, 2003
- Combs M, Snell J, Cail W, Maier T, Buck A. The gray-scale ink-jet
printer: value in making hard copies of digital images.
AJR 1995;164:225
–227[Abstract/Free Full Text]
- Kirkhorn T, Kehler M, Nilsson M, Lyttkens K, Andersson B, Holmer N.
Demonstration of digital radiographs by means of ink jet-printed paper copies:
pilot study. J Digit Imaging1992; 5:246
–251[Medline]
- Ibbott GS, Zhang Y, Mohiuddin M, Adams E. Reproduction of
radiological images on plain paper. RadioGraphics1998; 18:755
–760[Abstract]
- Warren R. Contrast and latitude of CT hard copy: a ROC study.
Radiology1981; 141:139
–145[Abstract/Free Full Text]
- Lyttkens K, Kirkhorn T, Kehler M et al. Evaluation of the image
quality on ink-jet printed paper copies of digital chest radiographs as
compared with film: a receiver operating characteristic study. J
Digit Imaging 1994;7:61
–68[Medline]
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
Top
Abstract
Introduction
