Cross-sectional imaging studies play an important role in evaluation of patients with tumors because they offer a method of measuring tumor size, which is one of the major methods of assessing tumor progression and response to therapy. In 1990, rigorous criteria for categorization of brain tumor response were suggested that were based on change in the largest cross-sectional diameter of the enhancing portion of the tumor on serial CT or MR images [1]. These criteria are now generally accepted for brain tumors and are in widespread use. These criteria (hereafter referred to as the Macdonald criteria) figure importantly in decisions about whether a brain tumor has responded to therapy and, thus, whether therapy is to be continued or altered. It seems reasonable to consider the test performance characteristics of a test that carries such important weight in determination of therapeutic changes. To our knowledge, such an assessment has seldom been performed. In one such evaluation [2], consensus agreement among five readers was found for only 35% of CT studies and 45% of MRI studies of brain tumors. However, the study was limited by a mixed reader sample consisting of radiologists and nonradiologists and by performance of measurements on hard-copy film, which is subject to inaccuracies based on the fact that measuring devices external to the image must be placed on the image.

With the popularization of digitization in medical imaging, image interpreters at medical institutions typically review electronic images and measure tumors using software programs that allow placement of electronic calipers on lesions. Measurements on electronic images would be expected to be both more accurate and more reproducible than those on hard-copy film, although to our knowledge, such a comparison has not been made.

We set out to study interobserver variability in the use of digital images of tumors in a homogeneous sample consisting solely of radiologists at the same level of training. We hypothesized that use of digital images and a homogeneous reader group would have lower interobserver variability than the consensus agreement of 45% in MR studies of high-grade brain tumors in a previous study [2] in which the subjects were readers in different disciplines using hard-copy film. In that study, consensus agreement was 45% on MRI studies of high-grade brain tumors.

We enrolled into the study 22 patients who were treated at our brain tumor center. The study was approved by the institutional review board at our hospital, and informed consent was obtained from each patient. Patients were randomly selected in the following manner. The study research assistant reviewed the list of patients referred from our brain tumor center who were to undergo MRI in our radiology department. To have been considered for the study, patients had to have undergone a previous MRI examination that showed contrast-enhancing tumor and had to be competent to provide informed consent. Patients provided consent for analysis of the images from the initial and subsequent MRI examinations. Thus the data set consisted of two sets of MR images of each patient. Identifying information was removed from the MR images with the workstation software (eFilm, Merge Healthcare). In all cases, the initial and subsequent MRI examinations were performed within 12 weeks of each other.

All tumors were high-grade gliomas (World Health Organization grade 3 or grade 4), and the patients had undergone biopsy alone or biopsy in combination with surgery. Five tumors had central nonenhancing regions consistent with necrosis, and 11 tumors had central nonenhancing regions representing resection cavities. Seven tumors had ill-defined borders consistent with infiltrative tumor. Two tumors had multiple enhancing foci interspersed with nonenhancing tissue. In both cases, only the largest tumor was measured.

Tumor measurements for determining treatment response according to the Macdonald criteria are based on the product of orthogonal diameters on the image with the largest contrast-enhancing tumor area. In the event that multiple lesions are present, the sum of products of individual measurable lesions is calculated. The Macdonald criteria for assessment of tumor response are as follows. A complete response is determined by resolution of all contrast-enhancing tumor. A partial response is defined by an at least 50% decrease in the product of two orthogonal diameters. Progressive disease is defined by at least a 25% increase in the product of orthogonal diameters. Confirmation of all three of these determinations with a repeated imaging study 4 weeks after the first is suggested. All other interval changes are defined as stable disease.

Image acquisition parameters were as follows: unenhanced axial T1-weighted images, contrast-enhanced T1-weighted images (TR/TE, 480/14; 5-mm slices with 2.5-mm interslice gap), T2-weighted axial MR images (2,200/80–30, 5-mm slices with 2.5-mm interslice gap), FLAIR axial images (8,000/120; inversion time, 2,000 milliseconds; 5-mm images with 2.5-mm interslice gap), and contrast-enhanced coronal T1-weighted spoiled gradient-recalled echo images (12/5, 1.2-mm slices). Images were reconstructed in the sagittal plane with 1.6-mm contiguous slices from the coronal spoiled gradient-recalled echo data set.

Based on the Macdonald criteria, complete agreement was found among readers in 10 cases, all indicating stable disease; 12 patients were judged by at least one reader to have tumor measurements indicating progressive disease (Table 1). A total of 176 readings were made of image sets from two serial examinations (eight readers, 22 patients). Tumor measurements consistent with tumor progression were made in a total of 24 instances (14%).

All readers recorded tumor measurements consistent with progressive disease in at least one patient. For any one reader, the number of patients in whom progressive disease was considered present ranged from one patient (4.5% of patients assessed) to five patients (22.5%) (Table 2). Two readers considered tumor progression present in five patients; three readers, three patients; two readers, two patients; and one reader, one patient. The postoperative changes did not appear to influence subsequent tumor measurements. The Kendall correlation value of 0.44 (p = 0.01) between the reader average progression call and the reader average initial tumor size judgment indicated a statistically significant relation.

The major findings in this study are as follows. First, we found that the rate of consensus agreement among all readers with regard to tumor progression was relatively low (45%), which was identical to that found in a prior study using hard copy film and a heterogeneous set of readers that was smaller in size than our study [2]. Next, when assessing interreader variability with regard to tumor progression using kappa analysis, we found weak agreement among readers. The analysis of reproducibility of tumor measurements showed that for each of the three imaging planes, only moderate agreement among readers was found and that variability in the measurements in any one plane did not substantially differ between imaging planes. Variability in measurements among larger tumors tended to be greater than for smaller tumors. Thus our hypothesis that use of electronic calipers placed on digital images by a homogeneous group of readers would result in improved interreader variability was disproved.

Measurement of tumor size on images figures importantly in clinical care and therapeutic trials because tumors in internal organs (e.g., brain tumors) are not palpable or otherwise amenable to physical examination, and change in size cannot be estimated from functional changes (e.g., findings at neurologic examinations or assessments of function, such as Karnofsky score). The importance of initial tumor size measurements of brain tumors (i.e., before therapy) has been emphasized. Results of one study [3] indicated that the size of recurrent glioblastoma multiforme managed with brachytherapy is a significant predictor of survival. Another study showed that the response of malignant glioma to chemotherapy is related to tumor size [4].

Despite published findings indicating the importance of initial tumor size to prognosis, the importance of serial changes in tumor size is a matter of debate. Furthermore, interobserver variability confounds the issue for brain tumors and non-CNS tumors alike. The medical literature on measurement of lung tumors is informative on these issues. For instance, one recent study [5] showed substantial interobserver variability in manual measurements of lung tumor size on CT scans. Another recent study of lung tumors [6] showed no correlation between early changes in tumor size (measured with Response Evaluation Criteria in Solid Tumors) and survival. The results of that study suggested that response criteria other than size would be more appropriate for measuring response. Although those studies involved a different tumor type and a different response criterion than those used in our project, the findings raise concerns about the utility of tumor size measurements in making decisions about therapy.

The Macdonald criteria are one of the more common methods of measuring brain tumor size [1]. These criteria have been used in a large number of clinical trials since their introduction in 1990. Although some investigators have noted that, in the future, assessment of tumors would optimally combine anatomic measurements with assessments of physiologic change by techniques such as perfusion imaging and molecular imaging, at present at most large brain tumor centers these techniques are not routinely used to evaluate therapeutic response [7]. Conventional contrast-enhanced MRI remains the standard imaging technique for determination of tumor response. It therefore is appropriate to consider the question of test performance of measurements made on electronic images.

In one study of interobserver variation in brain tumor measurements on electronic images (similar to our study), the investigators found substantial interobserver variation [8]. Unlike in our study, in that investigation tumors were measured at one time point. Furthermore, tumor dimensions were not measured directly in three planes, but the measurement in the third dimension was inferred from the number of slices on which contrast-enhancing tumors were seen. In that study, readers differed in 2D measurements of contrast-enhancing tumors 29% of the time. As those authors noted, this finding is likely to affect determinations of tumor progression in many instances if different readers measure tumor dimensions on different serial images.

Authors of a previous study [1] indicated the limitations of brain tumor measurements on CT and MR images, including irregular shape of lesions and variations in experience and skill of observers. In that study, interobserver variability regarding CT and MRI tumor measurements in the assessment of progression according to the Macdonald criteria was measured for a diverse group of five clinicians that included two neuroradiologists, a neurosurgeon, a neurologist, and a radiation oncologist. Consensus agreement about tumor response was reached for only 35% of CT studies and 45% of MRI studies. We attempted to limit variation in degree of experience among readers by having only neuroradiology fellows, all with the same level of training, perform measurements. We attempted to take irregularity of shape of enhancing tumors into account by using images obtained in three orthogonal planes. In the previous study [1], readers measured tumors on printed film by referring to the centimeter scale on the hard-copy images. Such a technique would be expected to involve some measure of imprecision, depending on the method by which diameters are compared against the centimeter scale. To eliminate this confounding feature, rather than relying on the printed scale, we used electronic images and electronic calipers that generated measurements when the calibers were placed. We performed only measurements of interobserver reproducibility. Assessment of repeated measurements by the same observers might have shown improved performance characteristics compared with measurements performed by different observers.

On the basis of the study design, we hypothesized that a higher rate of consensus agreement would be reached in our study compared with a previous study [2]. Instead, the consensus among all readers in our study as to tumor stability or progression (45%) was the same as in the previous study. Full consensus was reached only on the cases in which the measurements indicated the presence of stable disease and not in cases in which the measurements indicated disease progression. Furthermore, in the cases of approximately 20% of the patients, the readers were almost evenly divided between judgments, three readers disagreeing with the other five readers. This lack of improvement in degree of consensus agreement might have been due to factors in study design but also may represent inherent difficulties in measurement of tumors according to the Macdonald criteria. With that in mind, we analyzed not variance due to reader but residual variance.

The results with our variance components model indicated that variance due to reader was small but that the residual variance component was large. In a previous study of brain tumor measurements using Macdonald criteria [2], the investigators reported an intra class correlation coefficient of 0.64, compared with a value of 0.80 in our study. Although we found a higher intraclass correlation coefficient, these findings nonetheless indicate that repeated measurements in the same case are variable even with the same reader measuring the tumor dimensions. In essence, these findings indicate a limitation of the precision of the Macdonald criteria for measuring tumor size and for indicating tumor progression. The measurement variability in our study suggests a 14% false-positive rate for indicating progression when brain tumors are, in fact, stable.

Because of the relatively high false-positive rate we found inherent in the measuring technique and the various morphologic features assessed, we estimated various false progression rates associated with two other criteria for tumor progression. Based on the fitted model used in our analysis, changing the criterion for progression from a 25% increase to a 30% increase in bidimensional product would result in a much lower false-positive rate of 6%. Furthermore, redefinition of progressive disease as a 50% increase in maximum pair product would be expected to result in a false progression rate of 2%. Careful analysis of the effect on sensitivity and specificity, with a valid reference standard, clearly is needed before any such change can be considered.

In our study, we found only moderate reproducibility between readers in the measurements of diameters and weak agreement on determination of lesion progression even though the readers were at a similar level of relatively advanced training. The variance analysis of our study did not indicate tendencies of individual readers to consistently measure tumors as either larger or smaller than the others did. Thus the moderate degree of reproducibility in our study likely cannot be ascribed primarily to attributes of the specific readers.

That many tumors in our series were irregularly contoured and some had infiltrative margins likely added to difficulty in reproducibly measuring tumor size. It was difficult to determine the degree to which postoperative changes, which also contributed to irregular tumor margins, influenced tumor measurements. The diameter method of tumor measurement is based on the assumption that a solid ellipsoid accurately characterizes tumor shape in most cases. However, the complex shape of many tumors and the presence of a necrotic region do not make them amenable to consideration as a solid ellipsoid, thus the tumors are not easily represented by a single set of diameter measurements. For instance, in one study of the use of diameter measurements of tumors [9], 85% of tumors were best characterized by more than one set of diameter measurements. The second set of cross-sectional measurements was most commonly needed because of the presence of a necrotic portion the size of which had to be subtracted from the larger set of cross-sectional diameters for accurate representation of tumor volume. In that study, however, even in the case of solid tumors, the assumption of an ellipsoid shape appeared to increase the variance in tumor size measurements.

Our findings indicate that in our patient sample, the very nature of the tumor measurement method produced a high rate of false-positive readings on stable tumors, indicating that other techniques of measurement would be preferable. A number of methods of tumor measurement other than the Macdonald criteria have been proposed. For instance, measurement of tumor volume by tracing tumor margins with manual or semiautomated methods is an alternative to the cross-sectional diameter technique [10]. In one study [9], investigators found that use of a computer-assisted planimetric technique of measuring tumor perimeters decreased intrareader and interreader variability compared with computation of a volume based on three orthogonal diameter measurements by readers. Because we did not have ready access to a volumetric software program, we did not compare manual linear measurements with semiautomated volumetric measurements. It is possible that computational methods will be valuable in improving tumor measurements. To be truly valuable, however, those techniques will have to be inexpensive, easy to use, and widely available to clinical radiologists and oncologists.

We thank the following individuals for contributions to this study: David Reardon, Robert Case, Christopher Cepeda, William Delfyett, Geentali Kapoor, Ian Kurth, Ronaldo Lessa, Felix Lin, and Timothy Waltner.

Address correspondence to J. M. Provenzale.

This is a Web exclusive article.