| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Fundamentals of Clinical Research for Radiologists |
1 Department of Radiology, LUCAS Center P267, Stanford University, Stanford, CA 94305.
Received November 17, 2004; accepted after revision November 23, 2004.
Address correspondence to S. K. Plevritis.
The emergence of evidence-based medicine has handed radiologists the challenge of evaluating the impact of a diagnostic imaging test on a patient's long-term outcome, often measured by overall survival and total health care expenditures [1]. This new challenge represents a significant departure from traditional evaluations of diagnostic examinations in which the main end points are intermediate ones—namely, test sensitivity and specificity. The shift from intermediate technology-specific to long-term patient-specific outcomes is being driven by the fact that a test with the highest diagnostic accuracy may not necessarily be the test of choice in clinical practice [2]. When making the decision to order a diagnostic imaging test, a clinician considers the health outcomes downstream from the imaging examination. For example, the health risks of interventions resulting from false-positive (FP) and false-negative (FN) findings should be compared with the health benefits associated with true-negative (TN) and true-positive (TP) findings. Increasingly, the cost of the diagnostic test, including the downstream costs generated as a result of imaging, are also factored in the decision-making process.
Few radiologists would argue against the importance of measuring the impact of diagnostic tests on long-term outcomes, but many are concerned with the feasibility of evaluating long-term outcomes through traditional clinical trials. Except when evaluating the impact of an imaging test for an acute state that is life-threatening, evaluating the impact of an imaging test in the adult population in terms of overall survival can require follow-up of 10 years or more. In children, even longer follow-up periods could be required. Long follow-up times compete with demands to diffuse promising technologies quickly and increase the risk of delaying technologic innovations. For a disease with a low risk of death, an economically unfeasible sample size may be required to detect a survival benefit due to diagnostic testing.
Linking the intermediate outcomes (such as TPs, TNs, FPs, and FNs) to long-term outcomes (such as survival) without requiring a clinical trial is sometimes possible. This link is often made when existing clinical data (usually collected for different purposes) can be extrapolated to address the problem of interest. Often the data are extrapolated through a number of assumptions that are formulated into a mathematic model in which the link between intermediate and long-term outcomes is expressed in terms of probabilistic events [3]. The Markov model, described later in this article, is an example of the methods commonly used in this extrapolation process. When reliable models can be generated, the opportunity arises to evaluate a variety of hypothetical clinical paradigms that would not be economically feasible or practical to analyze experimentally via traditional clinical trials. The process of choosing from a number of hypothetical clinical paradigms by comparing them in terms of model-based probabilistic outcomes is often referred to as decision analysis.
|
|
|
|
Decision Analysis
Decision analysis is a deductive reasoning process that enables a decision maker to choose from a well-defined set of options on the basis of the systematic model-based analysis of all the probable outcomes [4-6]. Every outcome has a known probability of occurrence and a numeric value (i.e., life expectancy). The purpose of decision analysis is to quantify each option in terms of its expected (or average) value. A rational decision maker would choose the option that provides the greatest expected value. For example, if the outcome of the decision is measured in terms of life expectancy, the decision maker would choose to maximize the expected value; if the outcome is measured in costs, the decision maker would choose to minimize the expected value.
The critical components underlying decision analysis include clarifying the decision and the value used to measure the success of the decision, identifying the options, formulating every possible outcome from every possible decision, assigning a probability to each possible chance event, and assigning a value to each possible outcome. Once these components are determined, computing the expected values for each option can be straightforward.
Consider a generic clinical problem that involves the optional use of a diagnostic imaging test:
A patient presents with clinical symptoms of a life-threatening disease that requires surgery. What should the clinician recommend, knowing that surgery carries a risk of death? If the patient's probability of having the disease is low relative to the risk of surgery-related fatality, the clinician may recommend "Do Nothing" to avoid the risk of death due to the surgery. If the patient's probability of disease is high, the clinician may recommend "Surgery" immediately on the premise that the risk of death from the disease is higher than the risk of death from surgery. Now suppose a diagnostic imaging test becomes available with known sensitivity and specificity for the disease of interest. The clinician may choose to order the imaging test and then recommend Do Nothing if the imaging finding is negative or Surgery if the imaging finding is positive. Should the clinician order the imaging test, or make the recommendation of Do Nothing or Surgery, without the findings from the imaging test?
To answer this question, the decision maker, who is the clinician in the our example, needs to define a value on which to base the decision. If the value is life expectancy, then the clinician would want to know if ordering the diagnostic test will increase the patient's life expectancy. Decision analysis reveals how the patient's life expectancy depends on the choice made by the decision maker and events that are governed by chance—namely, the patient's probability of having the disease before getting the imaging results (i.e., the pretest probability or disease prevalence), the patient's life expectancy if the disease is present and untreated, the expected survival gain from a successful surgery, the risk of death from surgery, and the sensitivity and specificity of the imaging test.
Decision Trees
Decision analysis is aided by the use of a decision tree
[4,
5]. A decision tree is a
graphic model that represents the consequences for each possible decision
through a sequence of decision and chance events
[7]. A decision tree is
constructed with three types of nodes: decision nodes, chance nodes, and
terminal nodes, commonly represented as squares, circles, and triangles,
respectively. A decision node is a branching point in the tree where several
options are available to the decision maker for his or her choosing. A chance
node is a branching point from which several outcomes are possible, but they
are not available to the decision maker for his or her choosing. Instead, at a
chance node, the outcome is randomly drawn from a set of possible outcomes
(this is equivalent to saying that they are governed by chance). A chance
event could be, for example, that a patient presenting with symptoms for a
disease actually has the disease. At a chance node, every outcome is assigned
a probability of occurrence, which is often estimated from a clinical trial or
observational data. The decision tree is typically drawn by starting at the
far left with a decision node and continuing from left to right through a
sequence of decision and chance nodes. Every possible pathway through the
decision tree ends at the far right with a terminal node. Every terminal node
is assigned a value.
A simple decision tree associated with the clinical problem described previously is given in Figure 1A. This decision tree has one decision node that illustrates three possible options: Do Nothing, meaning the patient is sent home; Surgery, meaning the patient undergoes immediate surgery; and Imaging, meaning the patient undergoes a diagnostic imaging test and then surgery if the imaging findings are positive.
The Do Nothing option yields two chance events: the patient has the disease, with probability P(D+), and is assigned a life expectancy of L1 years; or the patient does not have the disease, with probability P(D-), and is assigned a life expectancy of L2 years.
The Surgery options yields four chance events: the patient has the disease with probability P(D+), experiences fatal surgical complications with probability P(O+), and has a life expectancy of L3 years; the patient has the disease with P(D+), undergoes successful surgery with probability P(O-), and has a life expectancy of L4 years; the patient does not have the disease with probability P(D-), experiences complications due to surgery with probability P(O+), and has a life expectancy of L5 years; and the patient does not have the disease with probability P(D-), does not experience complications due to surgery with probability P(O-), and has a life expectancy of L6 years.
The Imaging option yields six chance events. In four chance events, the imaging findings are positive, with probability P(T+), and the patient undergoes surgery; then the patient who has the disease, with conditional probability P(D+|T+), experiences fatal surgical complications with probability P(O+) and is assigned a life expectancy of L7 years; the patient who has the disease, with conditional probability P(D+|T+), has a successful surgery, with probability P(O-), and is assigned a life expectancy of L8 years; the patient who does not have the disease, with conditional probability P(D-|T+), but experiences fatal surgical complications, with probability P(O+), and is assigned a life expectancy of L9 years; the patient who does not have the disease, with conditional probability P(D-|T+), has successful surgery, with probability P(O-), and is assigned a life expectancy of L10 years. In two chance events the imaging findings are negative, with probability P(T-); and either the patient has the disease, with probability P(D+|T-), and is assigned a life expectancy of L11 years; or the patient does not have the disease, with probability P(D-|T-), and is assigned a life expectancy of L12 years.
All the probabilities populating the decision tree are summarized in Table 1.
|
To evaluate the Imaging option, the probabilities P(D+|T+),
P(D-|T+), P(D+|T-), P(D-|T-), P(T+), and P(T-) must be
evaluated. The probability that the patient has the disease given a positive
imaging finding, denoted as P(D+|T+), is commonly referred to as the
positive predictive value of the test. The probability that the patient does
not have the disease given a negative imaging finding, denoted as
P(D-|T-), is commonly referred to as the negative predictive value of
the test. These probabilities can be derived from the pretest probability of
the disease and the sensitivity and specificity of the test using Bayes'
theorem:
 |
where P(T+|D+) is defined as the sensitivity and P(T-|D-) is
defined as the specificity [8].
The probability of a positive and negative test can be computed as:
 |
Therefore, incorporating an imaging test into the decision tree simply requires knowledge of the test's sensitivity and specificity and the patient's pretest probability of disease.
Expected Value Decision Making
Decision analysis operates on the principle that a rational choice from a
set of options is the one with the greatest expected value
[4,
5]. It is possible that a
"good" decision leads to a "bad" outcome because
chance is involved. The likelihood of a bad outcome is minimized when the
decision is the one with the greatest expected value. This principle is often
referred to as Bayes' Decision Rule and is credited to the Reverend Thomas
Bayes, an 18th century minister, philosopher, and mathematician who formulated
Bayes' theorem.
Computing the expected value of each option is accomplished by "rolling back" or "averaging" the decision tree. The process of rolling back the decision tree for the clinical example illustrated in Figure 1A is shown in Figures 1B, 1C, 1D. In each figure we progressively roll back the right-most layer of terminal branches to their originating node and assign an expected value to that node, in effect turning the originating node into a new terminal node. If the originating node is a chance node, then the expected value is calculated as the weighted average of the expected values of its possible outcomes, where the weights are the probabilities that each outcome will occur. If the originating node is a decision node, the outcome is the one with the best expected value. This process is continued until the decision node at the far left-most part of tree is the only remaining decision node in the tree. The decision tree is said to be "fully collapsed." An example of a fully collapsed tree is given in Figure 1D.
For the Do Nothing option, the life expectancy is P(D+) x L1 + P(D-) x L2 years.
For the Surgery option, the life expectancy is P(D+) x [P(O+) x L3 + P(O-) x L4] + P(D-) x [P(O+) x L5 + P(O-) x L6].
For the Imaging option, the life expectancy is P(T+) x {P(D+|T+) x [P(O+) x L7 + P(O-) x L8] + P(D-|T+) x [P(O+) x L9 + P(O-) x L10]} + P(T-) x [P(D+|T-) x L11 + P(D-|T-) x L12].
To evaluate and compare the life expectancies for each of the three options, the probabilities and life expectancies L1 through L12 must be assigned. Consider the example in which a 60-year-old patient presents with symptoms indicative of a specified disease that has poor prognosis. Probability values for the chance events are given in Table 1. These values can be derived from the following three assumptions: patient's pretest probability for the disease is 0.10; the probability of surgery-related death is 0.05; and the diagnostic test has a sensitivity of 0.90 and specificity of 0.80. To compute the expected value of each option, we also need to assign a value to each possible outcome. Table 2 lists all the possible intermediate outcomes (column 1) and their associated life expectancies (column 6). For example, if the clinician recommends Do Nothing and the intermediate outcome is that the patient has the disease (D+), then the patient's life expectancy is L1 = 65 years. If the patient does not have the disease, his or her life expectancy is 80 years. If the patient experiences operative death, then his or her life expectancy is 60 years. We assume successful surgery is not curative but extends the patient's life expectancy to 72.5 years. Later we will show how these life expectancies can be estimated with a Markov model that links the intermediate health outcomes to overall survival.
|
|
Factors that were not considered in the decision process that could change the clinician's recommendation include the invasiveness of the imaging test; quality of life while living with the symptoms; utilities derived from TP, TN, FP, and FN findings on imaging; and the possibility of delaying the surgery. However, the general ideas presented here can be extended to include these factors. In addition, this general approach can be used to consider more complex decisions that may involve more than one imaging test ordered sequentially or in parallel.
Sensitivity Analysis
Sensitivity analysis is a necessary component of decision analysis that is
used to evaluate the robustness of the decision to variations in model
assumptions. In decision trees, probabilities of chance events and the values
at terminal nodes may not be known. Under these circumstances, the values
assigned may be reflective of an expert's best guess. The possibility exists
that by varying the dubious model inputs, the expected values will not be
affected greatly or will be affected but not enough to change the ranking of
the options in order of expected value. Under either of these scenarios, a
decision maker would be more confident in implementing the option with the
greatest expected value. However, when changing an input affects the ranking
of the options, the decision maker would be less certain about proceeding
without clarifying the value of that input.
N-way (or multivariate) sensitivity analyses refer to the process of varying N parameters in a model simultaneously while all other parameters remain constant. The most simple and common example is one-way (or univariate) sensitivity analysis, in which one model parameter is varied in a range between an upper and lower bound, while all the other parameters are kept constant. A series of one-way sensitivity analyses is the easiest way to identify which parameters have the strongest effect on the optimal decision. For the example just given, a one-way sensitivity analysis on the pretest probability is shown in Figure 2 using the remaining parameters in Tables 1 and 2. For P(D+) less than or equal to 0.03, the Do Nothing option has the greatest life expectancy; for P(D+) greater than 0.03 but less than 0.54, the Imaging option has the greatest life expectancy; and for P(D+) greater than or equal to 0.54, the Surgery option has the greatest life expectancy. The point at which the decision shifts from one alternative to another is often referred to as the crossover point or the threshold.
|
|
|
Markov Models for Estimating Life Expectancy
Markov models are commonly used in medical decision analysis for estimating life expectancy [10, 11]. In the previous example, the life expectancy for every possible outcome was known, but this information is usually not available. As discussed in the introduction, the challenge in basing decisions on maximizing life-years lies in finding a model that links the known intermediate health states to survival. A Markov model may be an appropriate tool for establishing this link when it is possible to represent a patient's life history from a known intermediate health to death through a series of transitions among a finite set of health states that have been observed elsewhere.
A simple Markov model for the clinical example described is composed of four health states: "Alive," "Disease-Specific Death," "Death from Other Causes," and "Death from Surgery." This model is shown in Figure 5. Each oval represents a health state. The arrows represent the possibility of transition from one state to another. The arrow that points back into the health state Alive indicates that the patient can remain in the health state Alive after a given cycle. Transitions between health states occur in a designated time period, known as the cycle period of the model. The cycle period for chronic diseases is typically 1 year, whereas the cycle period for acute diseases is often shorter—that is, months or even days. The probability that the patient will move from one health state to another in a given cycle is referred to as the transition probability. The life expectancy is the average length of time spent in all health states other than death.
The transition probabilities for the Markov model shown in
Figure 5 are as follows:
 |
 |
 |
 |
 |
Note that the health state at cycle number n is conditioned on the health state at cycle number n-1 and is independent of the health state before cycle number n-1. This property is the defining property of Markov models of this type, which are referred to as Markov chain models.
The transition probabilities for the example just described are given in Table 2, assuming a cycle period of 1 year. All these transition probabilities can be derived from the following three assumptions: if the patient has the disease, the probability of transitioning from Alive to Disease-Specific Death in 1 year's time is 0.15 if the patient does not undergo surgery and 0.03 if the patient undergoes successful surgery; the probability of transitioning from Alive to Death from Other Causes in 1 year's time is 0.05 if the patient does not experience surgery-related death; and surgery-related death is immediate.
Once the health states, allowed transitions between health states, and transition probabilities are identified, the life expectancy can be calculated using an algebraic solution, a cohort simulation, or a Monte Carlo simulation. All three approaches will be illustrated for estimating the life expectancy L1 in Figure 1A, where a 60-year-old patient presents with clinical symptoms, the decision is made to Do Nothing, and the patient actually has the disease (D+). In this case, p1 = 0.15, p2 = 0.05, p3 = 0, and p4 = 0.80, as shown in Table 2.
Algebraic Solution
If the transition probabilities are constant over time, then a closed-form
algebraic solution exists for estimating the life expectancy. In the simple
example above, the patient's life expectancy L1 is calculated as follows:
 |
In more complex Markov chain models with numerous transient, recurrent, and absorbing states, a matrix formalism may be necessary to evaluate the model using the closed-form, algebraic approach.
Cohort Simulation
If the transition probabilities are not constant over time, simulating the
outcomes of a cohort of patients is commonly implemented. This simulation
process is initiated by distributing a cohort among the health states. For the
above example, the entire cohort begins in the Alive state. At each cycle the
cohort is redistributed among the states, depending on the transition
probabilities. Markov cohort simulation for estimating the life expectancy L1
is illustrated in Table 3. The
initial cohort size is 10,000. At the start of the simulation, the 10,000
patients are in the Alive state. By the end of the first cycle,
p1 x 10,000 = 0.15 x 10,000 = 1,500 patients
enter Disease-Specific Death and p2 x 10,000 = 0.05
x 10,000 = 500 patients enter Death from Other Causes, leaving
10,000-1,500-500 = 8,000 patients in the Alive state for the start of the
second cycle. By the end of the second cycle, an additional
p1 x 8,000 = 0.15 x 8,000 = 1,200 patients
enter the Disease-Specific Death, and p2 x 8,000 =
0.05 x 8,000 = 400 patients enter Death from Other Causes, leaving
8,000-1,200-400 = 6,400 in the Alive state. The cumulative number of patients
in each of the three states for the first 40 cycles of the Markov process is
shown in the Table 3. Each row
totals 10,000 patients. Life expectancy is calculated as the average amount of
time a patient is in the Alive state. For this example, a 60-year-old patient
remains in the Alive state for 20 years on average, making his or her life
expectancy L1 = 60 + 20 = 80 years.
|
Monte Carlo Simulation
If complex dependencies exist in the state transition model, an intensive
computer simulation procedure called Monte Carlo simulation may be needed to
compute the life expectancy. In Monte Carlo simulation, patients traverse the
health states one at a time, with a random number generator (RNG) determining
what happens to an individual at each cycle of the process. An RNG is a
computer algorithm that produces sequences of numbers that on aggregate have a
specified probability distribution and individually possess the appearance of
randomness.
To estimate L1 in the above example via Monte Carlo simulation, an RNG samples a uniform distribution from 0 to 1. When the RNG produces a number in the range 0-0.05, the patient is assigned to Disease from Other Causes. This will happen 5% of the time, which corresponds to the transition probability from Alive to Disease from Other Causes. When the RNG produces a number greater than 0.05 but less than or equal to 0.20, the patient is assigned to Disease-Specific Death. This will happen 15% of the time, which corresponds to the transition probability from Alive to Disease-Specific Death. Finally, when the RNG produces a number greater than 0.2 but less than or equal to 1.0, the patient remains in the Alive state, which will happen 80% of the time. In this simple example, only one random number needs to be generated at every cycle. Once the patient enters a death-related state, the life history of that patient is terminated and a new run begins that traces the life history of the next patient. The process is repeated until a large number of runs (typically 10,000) are performed. There is no formula specifying the exact number of runs needed, but the number should increase with the complexity of the model to reduce simulation variability in the result.
Six sample runs of a Monte Carlo simulation are shown in Table 4. In each run, the patient is initiated in the Alive state at age 60 and ages 1 year in every cycle. Table 4 shows that in run 1, the patient dies of the disease after 5 years (at age 65); in run 2, the patient dies of other causes after 3 years (at age 63). The runs are repeated 10,000 times. The life expectancy is the average age at death. A valuable output of Monte Carlo simulation is a histogram of age at death, so that measures of variability in the life expectancy are easy to calculate [12, 13].
|
Markov models have much broader applicability than estimating life expectancy. They are used in a variety of fields to represent processes that evolve over time in a probabilistic manner. The article by Kuntz and Weinstein [14] is recommended further reading on Markov modeling in medical decision analysis.
Cost-Effectiveness Analysis
Cost-effectiveness analysis is a type of decision analysis in which both health and economic outcomes are considered simultaneously in making a decision [15, 16]. The decision analysis example described previously focused on maximizing life expectancy (LE). Although maximizing life expectancy is a reasonable value, it may not necessarily be the basis for a preferred decision. If the difference in life expectancy between an existing clinical protocol and a new clinical protocol is small but the difference in costs is large, it may be more prudent to follow the existing protocol and invest health care dollars in another clinical problem for which the incremental life expectancy is higher for the same health care expenditures.
In cost-effectiveness analysis, the expected value is reported as the marginal cost per year of life saved (MCYLS) [17]. When the decision tree is rolled back, the average cost is evaluated in parallel with the life expectancy. Dominant options are ranked in terms of incremental cost-effectiveness ratios.
The value of diagnostic testing is put to the greatest challenge in cost-effectiveness analysis. Often diagnostic testing increases both life expectancy and health care costs. The application of cost-effectiveness analysis to diagnostic testing is introduced in an article by Fryback [18] and discussed in more detail in an article by Singer and Applegate [19]. More general discussions on the role of cost-effectiveness analysis and recommendations for reporting results are found in other articles [20-22].
Summary
Decision analysis is a multifaceted concept. Underlying the decision analytic process is clarification of the decision and values for making a good decision, integration of data from multiple data sources, and mathematic modeling. The necessary steps for any decision analysis are summarized in Appendix 1.
|
The major strength of decision analysis is that the process offers an explicit and systematic approach to decision making based on uncertainty. The major weakness of decision analysis lies with the decision analyst who uses data to populate a model without understanding the biases in the data and therefore does not fully explore their impact on the decision [23, 24]. This problem is minimized when the decision analyst is fully knowledgeable of both clinical domain-specific and methodology-specific issues.
This article has focused the basic ideas of decision analysis toward the problem of evaluating a diagnostic imaging test on the basis of long-term patient outcomes when only the test's sensitivity and specificity are known. Markov models were introduced as means of linking intermediate to long-term outputs. Even when the inputs and structure of the decision analysis model may be incompletely supported by data, the decision analysis process itself can be valuable in identifying important areas of uncertainty and directing the investment of resources toward acquiring information needed to address the question of interest. Such analyses may be warranted before resources are committed to large-scale, costly clinical trials.
References
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  S. Mantha, M. F. Roizen, J. Foss, and J. E. Ellis Cardiac Troponin Surveillance After Abdominal Aortic Surgery Anesth. Analg., March 1, 2009; 108(3): 1047 - 1048. [Full Text] [PDF]
|
|
|
|
 |
|
 W. Hollingworth Radiology Cost and Outcomes Studies: Standard Practice and Emerging Methods Am. J. Roentgenol., October 1, 2005; 185(4): 833 - 839. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
