| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Fundamentals of Clinical Research for Radiologists |
1 Department of Radiology, Medical College of Wisconsin, 8701 Watertown Plank Rd., Milwaukee, WI 53226.
Received November 15, 2001;
accepted after revision January 22, 2002.
Series editors: Craig A. Beam, C. Craig Blackmore, Stephen J. Karlik, and
Caroline Reinhold.
Introduction
 Top Introduction Minimizing BiasPower in ComparisonsPrecision in EstimationReferences |
|---|
When reading the scientific literature or designing studies, every clinical radiologist should be aware of and concerned about three main considerations of modern experimental design that apply to research in clinical radiology (Fig. 1). The first consideration is the extent to which the findings of the study might mislead ("bias"). Another consideration is the ability of the study to reveal something important ("power"). The final consideration is the desire to create useful information ("precision") from the research. The deceptively simple statistical concept of the "average" will be shown to be central to many of these considerations.
|
In this article, I will review these three key considerations, each of which needs to be adequately appreciated and addressed by investigators seeking to design a successful diagnostic radiology study. Because successfully engineering the scientific study requires drawing heavily on both clinical and statistical sciences, interdisciplinary collaboration should be encouraged and nurtured. In this way, research in clinical radiology will mature into a modern scientific discipline. Motivating such collaborations is a goal of this series of articles.
|
TopIntroductionMinimizing BiasPower in ComparisonsPrecision in EstimationReferences |
|---|
Statistically, bias is a property of averages. A statistical measure is said to be biased if, on average, it does not equal what it is intended to estimate. To say that a study is biased is to say that it was conducted in such a fashion that, on average, the measurements from the study are biased.
What Is the Weight of a I-Oz Marble?
Suppose that a group of researchers had a reliable spring scale with which
to measure the weight of marbles. Reliable means that the researchers
generally get the same value each time they weigh the same marble. Now suppose
that the researchers have a marble that they know weighs exactly 1 oz and thus
that marble becomes the gold standard. They weigh this marble five times and
get the following values: 1.1, 1.2, 1.2, 1.1, and 1.1 oz. The values are
always slightly more than the marble's true weight of 1.0 oz. Sometimes the
"error" is 0.1 oz, and other times it is 0.2 oz. The average of
these errors is 0.14, and so, on average, the scale errs by 0.14 oz.
Statistically, this measurement would be described as biased: it tends to overestimate true weight by 0.14 oz. Knowing this bias, the researchers could correct the scale by advising users to always subtract 0.14 from the reading. Then, although individual measurements may be a little off, on average, the users will get the correct value. Thus, the corrected measuring device would be said to be "unbiased" for the weight of marbles.
The previous case is an example of measurement bias. Studies can be affected by other biases as well. Studies of diagnostic technologies have their own special biases [1] with which the reader of the literature of diagnostic radiology should be familiar. These specific biases will be the subject of a subsequent article on the clinical assessment of diagnostic technologies in this series. For the present discussion, however, I will focus on two biases that affect every type of clinical study. These biases come about by the way subjects are selected for, and participate in, a study.
Selection Bias
The article in this series by Ella Kazerooni
[2], "Population and
Sample," makes it very clear that subjects selected for a study must be
representative of some clinically relevant population. One of several
statistical motivations for this notion has to do with bias: We want the
measures from our study to reflect the value of the measures in the general
population. We do not want to be off the mark, so to speak. To accomplish this
objective, we must have a sample that in some way reflects the population
being studied.
Recalling that the statistical meaning of bias involves averages, we can restate our consideration as seeking to sample from the study population in such a way that our measurements on average equal the value in the population. Luckily, this goal can be accomplished by the well-known mechanism of random sampling.
By randomly sampling, we follow a procedure that guarantees that every sample has the same chance of being selected for our study. If we decided to do a study with a sample of 100 randomly selected subjects from our study population, we would have to follow a method of sampling so that every possible sample of 100 subjects would be equally likely to be selected. How does random sampling ensure that our results will not be biased? The answer to this question requires the logic of statistical science. However, an intuitive answer is that measures that are simple averages will be unbiased for the population average when the measures are based on random samples.
Are simple averages relevant to clinical radiology research? Thankfully, the answer in many cases is yes. Many published clinical studies report means (which, of course, are averages) of measurements. Measures of diagnostic accuracy such as sensitivity and specificity, which are frequently reported, are averages as well. Other commonly used measures in clinical radiology are not simple averages but do enjoy the property of being unbiased when based on random samples. Examples of these are the slope in linear regression and the nonparametric receiver operating characteristic curve area.
Participation Bias
When conducting research that compares groups of subjects, care should be
taken to ensure that the group assignments are free of bias. In other words,
the way in which subjects participate should not bias the findings of the
study. The mechanism by which this bias is typically eliminated is
randomization. In the valuable reference book Statistics in Medicine,
Theodore Colton [3] writes,
"Randomization ensures that the personal judgment and prejudices of the
investigator and of the patient do not influence treatment
allocation."
Randomization, in fact, has become the gold standard for the clinical trial. For example, popular guidelines for evaluating the quality of research are based on the assumption that the controlled randomized trial is the epitome of study design. Some scientists advise using randomization simply because it is a good strategy for success in publication: "Without proper randomization, the investigator is immediately on the defensive and increases his vulnerability to the critical onslaught of his peers." [3].
What actually is randomization? First, let us specify what it is not. Colton [3] admonishes:
It is worthwhile to point out that one should not confuse randomization... with haphazard assignment.... The pattern of assignment to treatment may appear to be haphazard, but this arises from the haphazard nature with which digits appear in a table of random numbers, and not the haphazard whim of the investigator in allocating patients.
Randomization is an objective process that takes group assignment out of the hands of humans and gives the responsibility to the random number generator. Once the human factor in group assignment is eliminated, we can make the important assertion that the process of allocation was unbiased. The statistical significance of this step is that each possible allocation had an equal chance of occurring so that, on average, the findings from the study are not affected by the way the subjects participated in the study.
It is widely held that randomization "averages out" the effect of influencing factors that are unknown to the investigator. This tenet is true and provides another example of how the concept of the average is fundamental to our modern understanding of experimental design. However, the benefit of randomization is realized only if the averaging is performed across the many different ways of allocating subjects to treatment. In any one study, which can have only one such allocation, an imbalance of factors could influence the findings. Randomization does not guarantee an equitable allocation in any particular study; its benefits accrue as we consider the process of averaging across studies.
Consider the following study: Six subjects are selected for a clinical study of gadolinium enhancement of breath-hold T2-weighted MR imaging of hepatic lesions. Suppose that enhancement will be measured with a contrast-to-noise ratio (CNR) determined by dividing the difference between the lesion and liver signal intensities by the standard deviation of the background noise. Now suppose that the researchers wish to compare the CNR in the unenhanced section of the liver with the CNR in the enhanced section of the liver. However, the institutional review board requires the use of separate groups of subjects. Therefore, the subjects must be assigned to one of two "treatment" groups. How should the assignments be made?
If the investigators were to use randomization in this study, they would have to apply a mechanism that would give each possible allocation of three subjects the same chance to be in the enhanced MR imaging group. Note that randomization does not mean assigning individuals to treatments according to no discernible plan or pattern. For example, randomization would not occur if the first three patients who showed up at clinic were assigned to the gadolinium-enhanced imaging group and the next three to the unenhanced imaging group. That is not randomization because the researchers have not ensured that every allocation of three individuals to the enhanced imaging group was equally likely. The researches cannot feign ignorance either. Perhaps those three individuals who were assigned to the enhanced imaging group always show up early in the morning, and so the others would never have a chance to be in the group that undergoes enhanced MR imaging. In sum, to say that subjects were randomly assigned to treatments is to say that complete control had been exercised over the allocation mechanism in a quite definite way.
Randomization controls the bias of allocating individuals to treatments by the same averaging seen with random sampling. To say that randomization averages out the influence of unknown effects is to say that, on average, the values resulting from a study will equal the average of the values resulting from every possible experimental allocation of subjects to treatments.
Suppose that randomization was followed, and the data in Table 1 were observed. One would probably conclude from this study that the use of gadolinium does not improve the CNR because the mean CNR of the two treatment groups are equal. Because randomization was used, researchers would trust that any effects that might have biased the findings have been averaged out. "Trust" is the operative word: Randomization does not guarantee that the group allocation actually realized in this particular instance was equal with respect to characteristics that might be important. Randomization is only a property of averages. Any one particular randomization can, by chance, lead to severe disparities between the two groups in some characteristic.
|
Actually, the principal investigator of this supposed study was wise enough to design into it the collection of extra information about the subjects. One extra (or concomitant) variable measured was whether the subject had cirrhosis of the liver (determined independently of the measurement of the CNR). Table 2 presents the raw data from this study categorized by the treatment received (i.e., enhanced or unenhanced MR imaging) and by the presence of cirrhosis in the six subjects.
|
Examination of this table shows that three of the subjects selected for the study had cirrhosis and that two of these subjects were assigned by the process of randomization to the treatment (gadolinium-enhanced MR imaging) group. Conversely, two of the subjects without cirrhosis were assigned to the "control" (unenhanced MR imaging) group. Obviously, the occurrence of cirrhosis was not equally represented in the two groups. Did randomization fail? No. The allocation used in this study was just one possible allocation of the six subjects to the two treatment groups. There are, in fact, 20 different ways to assign these six subjects to the two groups. The investigators used a method that picked one of these assignments at random—that is, in a way that each assignment was equally likely (one in 20) to be picked. Thus, they randomly assigned subjects to the groups. This time, randomization just happened by chance to come up with the assignment of two subjects with cirrhosis to the treatment group and two subjects without cirrhosis to the control group.
The investigators are concerned because they believe that the presence of cirrhosis is likely to have dampened the enhancement of the gadolinium. What can they do? They consult their statistician who then generates Table 3. From this analysis, it becomes obvious that there is no benefit for subjects with cirrhosis but a big benefit for other subjects.
|
The need to be cautious with the results from even the most carefully planned randomized trial is appreciated by experienced researchers. Colton [3], for example, observes:
Randomization achieves a balance in the long run. However, with a small series of patients, randomization may not always produce groups that are alike in every respect.... [A]s a general rule, a report of a clinical trial should include among its first tables one in which the treatment and control groups are compared on the several important characteristics relating to the disease under study.
In sum, the gadolinium-enhanced imaging example shows that successful study design requires collection of data that could plausibly influence the outcome of the study, good statistical methods by which to adjust the outcomes for these concomitant variables, and randomization of subjects to average out the possible influence of unrecognized factors.
|
TopIntroductionMinimizing BiasPower in ComparisonsPrecision in EstimationReferences |
|---|
To be clinically useful, a successful study must have the power to detect the smallest difference that is deemed clinically important. If a difference in sensitivity as small as 1% leads to clinically important differences in patient outcome, we then are required to design a study that has adequate power to detect a difference as small as 1% in the sensitivities of the two modalities. If, however, our study was able to detect only a larger difference—for example, 20%—and gave negative results, we could not say with confidence that no clinically significant difference exists between the modalities. The difference might, indeed, lie between 10% and 20%, a range we consider clinically important. We would have to regard our study as unsuccessful.
Statistically, power is expressed as the probability of rejecting the hypothesis of no difference (the "null" hypothesis) when, in fact, a specific, clinically important difference does exist. The concept of power depends on specification of hypotheses and definition of a specific, clinically important difference. To assess the power of a study, it is not enough to say that the sensitivity of the new test is greater than that of the standard. A definite value for this difference must be specified.
Two important aspects of study design determine the power of a study. One is sample size, and the other is the design itself. A successful study is one that has sufficient power to detect the smallest clinically significant difference. The sample size that ensures this power is thus a requirement for the successful study. Determination of the sample size is the purview of statistical science, and so the required sample size for a study is often the contribution of the collaborating statistician. However, determination of sample size and power also requires specification of the smallest clinically important difference for the problem at hand. This determination is the purview of clinical medicine. Thus, statistically engineering the study for power should be a collaborative undertaking between clinical and statistical scientists.
Although the role of sample size and power is well known in medical circles, I do not think the role of experimental design and power is as well appreciated. The graph in Figure 2 illustrates the importance of the relationship. This graph depicts sample size requirements for two basic study design types that one might consider when comparing the diagnostic accuracies of two modalities.
|
Our scenario is that a clinical radiologist seeks to compare the sensitivity of a new diagnostic modality against that of an established modality. Based on her understanding of the medical literature, and of the costs and benefits to her patients in testing for this particular condition, the clinical researcher has determined that the smallest clinically relevant difference in sensitivities for this diagnostic problem is 5%.
The two basic study designs for this sort of clinical trial are the "independent groups" design and the "paired groups" design. The independent groups design specifies that the assignment of each of the study's subjects to one of two groups should be randomized. One group will be imaged using the reference modality, and the other group will be imaged using the new modality. In the paired group design, each of the subjects is imaged using both of the modalities being studied. Preferably, the interpretation of each modality is done independently of the result of the other modality, and the order in which the subjects are imaged with each modality is also randomized.
Figure 2 shows the total sample size required to achieve various levels of statistical power for the two designs. In fact, there are two sets of points for the paired groups design because the power of this design also depends on the extent to which the two modalities disagree (i.e., the proportion of patients for whom one modality is positive and the proportion for whom one is negative and vice versa). One set of points shows sample size required when the disagreement between the modalities is minimal, and the other set shows the power of the study when the modalities disagree as much as possible. (More details about these considerations and computations can be found in an earlier article that I wrote for ARJ [4].)
Figure 2 provides confirmation of the intuitive realization that greater power in a study requires a larger total sample size, or, conversely, the intuitive realization that a larger sample size means greater power. This relationship between power and sample size is true regardless of which study design is chosen.
However, note that the paired groups design requires a smaller total sample size for any power we may wish to achieve. For example, to achieve 90% power requires approximately 200 subjects with the independent groups design but only approximately 50 subjects when the paired groups design is used and the measures of the two modalities under study have the lowest level of disagreement possible. Even in the worst-case scenario, in which there is maximal disagreement between the modalities, the paired groups design requires only approximately 80 subjects.
The previous example illustrates that study design can greatly increase power for a given sample size or, conversely, that the study design can reduce the sample size needed to obtain a specific power. Successful studies are ones that achieve the desired power economically. Knowledge of study design is, therefore, essential to the engineering of powerful and economical scientific studies.
|
TopIntroductionMinimizing BiasPower in ComparisonsPrecision in EstimationReferences |
|---|
The biggest objection to a statistical test is that it answers with a "yes" or a "no" an overly simplistic question: Is there some difference? The emphasis on significant differences...distracts from the real (issue), which is how big is the difference....
Given this recent trend, the design of the modern study in clinical radiology research must ensure success in estimation. The phrase "success in estimation" means that, statistically, the study has been designed to achieve sufficient precision in estimation with a desired level of confidence. Understanding how to design a study to be successful in estimation requires, then, an understanding of the statistical concepts of precision and confidence.
To estimate the sensitivity of a new diagnostic technology, we would do well to follow the direction given by Kazerooni in "Population and Sample" [2] and perform the test on a random sample from the study population. Because our sample is, of necessity, not the complete population of interest, we would expect imprecision in our estimate of sensitivity from this one sample. Being scientifically sophisticated, we are not satisfied in reporting only the estimated sensitivity but also want to assess the probable error in our estimate. The standard way to both report an estimate and provide an assessment of probable error is through the use of statistical confidence intervals (CIs).
A statistical CI of a quantity is a range of values along with a statement of the level of confidence. Usually, the CI accompanies a single value (or point) estimate of the quantity. For example, if the sample previously discussed yielded an estimated sensitivity of 75% with an accompanying 95% CI for values ranging from 67% to 83%, how should we interpret these values?
The observed sensitivity is 75%, so that is our point estimate. However, we estimate the value might be within the range of values from 67% to 83% with 95% confidence. The adjective "confidence" in the phrase "confidence interval" is not an assertion of personal belief. The term has an explicit statistical meaning that, not surprisingly, is related to the long-term process of sampling. To say that the interval is a 95% CI means that the interval was formed by a statistical method in such a way that if a large number of random samples were taken from the study population and an interval were computed for each sample, 95% of these intervals would contain the true value of the sensitivity of the test. Figure 3 is a graph depicting this concept using a computer-simulated experiment.
|
Another important feature of a confidence interval is its width. Wide confidence intervals are less informative than narrow ones. For example, to say that the sensitivity of a test falls between 68% and 72% is much more informative than saying the sensitivity falls somewhere between 0% and 100%.
The width of a confidence interval is its precision. Successful studies provide precise estimates. Therefore, engineering the successful study requires first specifying the precision the investigators wish to obtain. As in considering power, specification of precision should in some way reflect a clinically relevant definition of precision. For example, if the researchers want to estimate sensitivity, it might be relevant clinically to require the precision of estimation to be within 5% of the true value if the researchers conclude that sensitivities this similar are virtually equal for clinical purposes.
Having specified the precision to be achieved by the study, the researchers have basically two design considerations by which to achieve this goal. One consideration is sample size. As expected, the precision of a confidence interval increases (i.e., its width decreases) with a larger sample size. Therefore, when designing a study for estimation, one must select a sample size large enough to achieve a desired precision in the confidence intervals.
Another way by which to achieve greater precision is by manipulating the level of confidence. Although the standard, by and large, for CIs is the 95% level, there is nothing sacred about this number. Other levels of confidence could be considered. The problem is, however, justifying this break from tradition.
Figure 4 shows the impact of changing the level of confidence on the precision of CIs based on the same sample size. Precision (interval width) is greatest for smallest confidence. In other words, precision and level of confidence exist in a trade-off relationship. Precision can be increased by decreasing confidence. In most cases, choosing precision at the expense of confidence will probably not be an acceptable trade-off. To alter the confidence level, one has to argue effectively that not following the status quo 95% level was appropriate. Generally, however, people set the level at 95% and find the sample size required to obtain adequate precision in estimation.
|
In this article, I have reviewed some of the key considerations in modern experimental design as they apply to diagnostic radiology. Each of these considerations—bias, power, and precision—should be addressed by investigators who want to design a successful study in diagnostic radiology. Because engineering a successful scientific study requires the expertise of both the clinical and statistical sciences, collaboration between these disciplines should be nurtured. In this way, research in clinical radiology will mature into a modern scientific discipline.
|
TopIntroductionMinimizing BiasPower in ComparisonsPrecision in EstimationReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  G. T. Sica Bias in Research Studies Radiology, March 1, 2006; 238(3): 780 - 789. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. C. Blackmore and P. Cummings Observational Studies in Radiology Am. J. Roentgenol., November 1, 2004; 183(5): 1203 - 1208. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
= independent groups study, 
= minimal disagreement in paired
groups study, 
= maximal disagreement in paired groups study.
