| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Original Research |
1 All authors: Department of Radiology, Monmouth Medical Center, 300 Second Ave., Long Branch, NJ 07740.
Received January 26, 2008;
accepted after revision April 10, 2008.
Address correspondence to A. J. Kwak
(andrew.kwak{at}gmail.com).
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. Two online surveys were distributed to diagnostic radiology programs approved by the Accreditation Council for Graduate Medical Education. Program directors were asked about the importance of the examination, and residents were asked about the importance and composition (proportion of repeated questions, known as recalls) of the examination and about preparation strategies.
RESULTS. Most of the program directors (122 of 178, 69%) responded. Most of the respondents believed that the examination correlated with their assessment of residents' knowledge, that passing was indicative of a qualified radiologist, and that the examination should not be eliminated. Although most of the program directors believed preparation increased residents' knowledge, they also believed that preparation detracted from reading assignments. Opinion was divided whether the examination should be continued in its present form. The majority of the 171 resident respondents spent most of their time studying recalls, which they believed made up 40–60% of the examination. One half of the residents believed that preparing helped in mastering the body of radiology knowledge, although most believed that core material made up only 20–60% of the examination. A minority of the residents considered passing the examination indicative of a qualified radiologist. Opinion was divided on continuing the examination in its present form.
CONCLUSION. Program directors and residents perceived value in the written clinical diagnosis board examination. The American Board of Radiology may need to reevaluate question content when restructuring the new examination.
Keywords: American Board of Radiology • education • program director opinion • resident opinion • written board examination
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Results of informal polling of residency graduates and program directors suggest that given the high proportion of recalls (questions repeated from previous administrations of the examination) on the yearly clinical diagnosis board examination, the objectivity of the examination can be questioned because passing the examination seems to depend on access to recalls. The goals of our study were to ascertain the perceived value among program directors and third- and fourth-year residents of studying for and passing the written clinical diagnosis board examination and to determine the changes, if any, that should be made. Although the recently proposed changes in the board examination address format and timing of administration, to our knowledge, question content has not been mentioned.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The survey addressed to program directors was composed of six mandatory questions with the answer choices completely agree, agree, neutral, disagree, completely disagree. Program directors were asked to assess how well the results of the written board examination correlated with program directors' perceptions of residents' fund of knowledge. The directors also were asked whether they believed preparing for the written board examination detracted from clinical duties and reading assignments that would otherwise be given. The final three questions attempted to gauge whether program directors believed the examination should be changed. An optional comments section was included.
The survey addressed to residents was composed of 10 questions and was separate from the survey addressed to program directors. The survey link was sent to Association of Program Directors in Radiology program directors, who were asked to forward the link to all residents who took the examination. In response to a request from one program, a separate cover letter with this link was sent to residents 1 week after the initial mailing. The resident survey polled those who had taken the 2007 written board examination on test content, that is, how much of the examination was composed of recalls, how much time was spent studying recalls, and how useful the test was for radiologists. Three questions regarding whether the examination should be changed were the same as those posed to program directors. This survey also queried residents about the size and geographic location of their programs. An optional comments section was included. Comments from both the program director and the resident surveys were categorized after the results were tabulated.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Three of the statements on the program director survey asked directors to rate the educational usefulness of the written clinical diagnosis board examination (Table 1). Sixty-six percent believed that the results correlated well with program directors' assessments of their residents' fund of knowledge. In addition, 66% believed that the time spent in preparation was useful in increasing residents' knowledge. However, 57% believed that this time came at the expense of reading assignments.
|
The second set of questions for program directors asked them to assess the board examination as a measure of competence. Table 2 shows that 53% of program directors believe that passing the examination serves as an indicator, at least in part, of a qualified radiologist. Sixty-four percent disagree that the written clinical diagnosis examination should be eliminated and that the only two portions should be the written physics and oral components of board certification. The views were split more evenly about whether the examination should be continued in its present form, although there was a slight preference favoring a change in examination structure.
|
The ABR did not release the number of examinees for the 2007 clinical diagnosis board examination. Using data obtained from the American Medical Association Fellowship and Residency Electronic Interactive Database [8] and the National Resident Matching Program [9] Websites, we estimated the number of fourth-year residents currently in training to be approximately 1,025. At the conclusion of the survey, a total of 171 responses were received from residents with no duplicates. Therefore, we estimated the response rate for residents to be approximately 17%. The demographics of the resident respondents (total number of residents in the program and geographic location) are summarized in Figures 1 and 2. Most of the resident respondents were in smaller programs (32%) and from the northeastern part of the United States (41%). Data stratified by region and program size did not appreciably change response patterns.
|
|
Results from the resident survey are shown in Tables 3 and 4 and Figures 1 and 2. A set of questions asked residents to assess their preparation for and the composition of the clinical diagnosis examination. The largest portion (37%) estimated that 40–60% of the examination was composed of core material relevant to radiology. The next highest estimate (30%) was slightly lower, 20–40%. When asked to estimate how much of the examination was composed of recalls, 108 of the residents (63%) answered that 40–60% of the examination was composed of previously asked questions. The next highest estimate from 40 residents (23%) was 20–40%. Most of the residents (71%) stated that they had spent 60–100% of their studying focused on learning recalls.
|
|
Table 4 shows residents' assessment of the examination as an indicator of radiologist competence. The largest portion (47%) seemed to believe that passing the written clinical diagnosis board examination does not necessarily qualify a radiologist as competent. Responses were more evenly split, however, when residents were asked whether the written clinical diagnosis portion should be eliminated, a slight majority disagreeing, and whether the examination should be continued in its current form, a slight majority agreeing.
One of the advantages of using an online survey generator is the ability to filter results according to answer choices. Answers were stratified according to size of program and geographic location to determine the existence of significant differences, taking these two independent variables into account. When results of the resident survey were filtered according to program size, the only apparent variation in response was on item 5, concerning whether the examination should be continued in its present form (Table 5). Of note, 28 of 55 residents (51%) in small programs (eight to 20 residents) tended to believe that the examination should be continued in its present form. Fourteen (25%) believed the examination should be changed, and the others (24%) were neutral. No difference was seen in residents' estimation of recalls when results were filtered according to program size. The clear majority of residents, regardless of program size, estimated that 40–60% of the examination was composed of direct recalls.
|
The final items on both surveys were optional free-text responses. Responses were collected and analyzed after the survey results had been tabulated. Sixty-one of 122 program directors (50%) and 60 of the 171 resident respondents (35%) contributed free-text responses.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
We attempted to determine factors that might have resulted in the pattern of response by program directors. The lack of images on the present examination was a common theme, one that has apparently already been acknowledged by the ABR in the proposed restructuring of the board examination. Another source of contention seems to be in the nature of the questions themselves. As one program director stated, "the high rate of repeat questions...detracts from the validity of the board examination." Another asked, "it apparently can't be passed without memorizing hundreds of past questions... how valid is that?" Of all of the themes found in the comments, the predominating sentiment was that there was a definite overreliance on previously asked questions. In addition, program directors believed that testing on "esoteric material" was "counter-productive" and that there must be flaws in the questions themselves when "even experts cannot answer many questions correctly."
The program director findings generally agree with the results derived from the resident survey. When asked to subjectively estimate how much of the examination was relevant to radiology, most residents (67%) answered that anywhere from 20–60% was core material. A clear majority (63%), however, believed that 40–60% of the examination was composed of recalls, and 71% estimated that they had spent 60–100% of their studying reviewing recalls as opposed to standard radiology texts. It appears that given the preponderance of recalls, most residents currently choose to focus most, if not all, of their studying on learning past examination questions.
It is no surprise then that in contrast to a clear minority (28%) of program directors, 47% of residents believed that passing the examination was not indicative, at least in part, of a highly qualified and trained radiologist. That is, almost one half of the resident respondents believed that passing the examination was not an indicator of competence, as opposed to 29% who believed that it was. This general response may be due to a prevalent belief that, as one respondent wrote, "many questions were truly direct recalls, with little or no variation from prior versions [which] could allow someone to simply memorize old test questions, and potentially pass." If that is the case, the respondent wrote, then "it does not truly test one's knowledge and is unfair to those who may have spent the time to truly learn the information." As another respondent stated, "the residents I know who... studied from books failed."
Given the comments, it might be expected that residents would be even more in favor of seeing a change in the examination structure. This attitude, however, was not clearly indicated in the study results. The views among residents about whether the examination should be composed of only a written physics and an oral section slightly favored (42%) inclusion of a written clinical diagnosis section, as opposed to 37% that did not. Moreover, many residents believed the examination should be kept as it is, 39% being in favor, 33% not in favor, and 28% neutral.
If there are flaws in the examination, 45% of residents believing that the examination is not an accurate indicator of a qualified radiologist, then the question to be asked is, Why there is less of a belief among residents that the examination should be changed? There may be a variety of explanations. Be cause 50% of residents believed that studying for the examination was helpful in solidifying and mastering the body of radiology know ledge, respondents may believe that the ends justify the means in that participation in intense study had the result of mastery of a solid body of knowledge. Second, some re sidents voiced concern about administration of the examination after the completion of residency. In those instances, despite the flaws of the examination, the alternative of admini stering some or all of the examination after residency may have affected their views. Third, retrospective review of the questions revealed ambiguity about the meaning of "the examination should be continued in its present form." Respondents might have interpreted "examination" to be the written clinical dia gnosis portion or the entire board examination.
Filtering the data according to program size and program location yielded only one major difference. As shown in Table 5, residents from smaller programs of eight to 20 residents tended to favor keeping the examination as it is. Those in larger programs showed less of a preference. This finding is surprising because many of the complaints aimed at the board examination concern the apparent disadvantage of smaller programs in terms of ability to amass recall questions. The exact factors taken into account are unclear, and evidence from this study is not sufficient for speculation on possible reasons.
With the written clinical diagnosis examination as it is, senior residents almost invariably enter into a "boards frenzy" to ensure a passing score. The ABR, according to Casarella [2], considers this sort of "hysterical, last minute studying" unnecessary and causes residents to "ignore their residency-based obligations." However, given the prevalent belief among residents that the questions are repetitive and esoteric, it is difficult to see how boards frenzy can be avoided. According to our study, most program directors confirmed that studying for the examination does take away from reading assignments that would otherwise be given. Nevertheless, most program directors also acknowledge a correlation between performance on the examination and resident knowledge, and most also see value in the amount of effort and time spent studying for the examination. It appears that there is an apparent consensus on the value of intense study and that the benefits, in general, outweigh the costs.
Perhaps the solution lies in establishment of a national curriculum for radiology residents with subsequent testing from that curriculum, as proposed by Goske and Reid [10]. With such a curriculum, residents would have a better-defined body of knowledge from which to expect questions and would be assured that the questions would be relevant to the practice of radiology. The effect of boards frenzy on senior residents would probably be lessened, and the intense study that elevates the resident's knowledge would be better focused.
Although our study was an attempt to shed light on current sentiment among radiology residents and program directors regarding the written clinical diagnosis portion of the board examination, it was affected by the inherent limitations of conducting a survey. The first was selection bias. Whenever non-compulsory surveys are offered, a certain bias may exist as to who responds. This bias seemed to apply more to radiology residents because, by our estimate, 69% of program directors responded, but only 17% of residents submitted surveys. Because the surveys were not sent to residents directly, it is conceivable that only residents from a relatively small number of programs participated. Although attempts were made to prevent accidental duplication of survey entries, there was the risk that one or more parties submitted multiple surveys to skew the results in a particular direction. There was the additional disadvantage of surveying residents through program directors. To our knowledge, no single database exists that contains the names and e-mail addresses of third- and fourth-year radiology residents. It is possible that program directors might have forwarded the surveys only to residents they believed would answer in a manner likely to skew the results in a particular direction. Given the nature of an anonymous online survey, we relied on the integrity of program directors.
Another criticism is the potential ambiguity of some of the survey questions. It is possible to interpret the statement "the examination should be continued in its present form" a variety of ways. Another argument can be leveled that many of the statements themselves were too general to shed light on specific problems. Although this may be true, we believe that the nature of the statements was better suited to the goals of our study. In light of the announcement issued by the ABR [4] regarding the restructuring of the board examination and residency training in general, it was not the purpose of the survey to attempt to generate a complaint list for what already appears to be a process that is being replaced. Rather, the findings may be of use for restructuring the examination, because the current board examination will be continued for at least 3 more years, even if the earliest implementation affects the incoming class of 2009. Furthermore, the recall issue should be carefully considered and perhaps formally addressed by the ABR, eliminating some of the secrecy surrounding the examination.
There are adverse effects of residents' studying primarily from recalls. As more than one program director commented, residents now seem to be studying primarily to pass a test. Given the perception that there is an emphasis on recalls, we attempted to contact a representative of the ABR via e-mail. We asked whether the ABR had considered the emphasis on recalls, especially given the fact that the American College of Radiology seems to be able to produce a new in-training examination every year. In accordance with the feedback received through this survey, we also asked whether the ABR had considered making the entirety of the question bank available to all residents equally to promote fairness and eliminate potential bias. A follow-up telephone call was made, but no reply had been received at the time of manuscript submission.
The issue of recalls is not limited to the examination administered by the ABR. In Canada, for example, the written radiology board examination administered by the Royal College of Physicians and Surgeons of Canada depends heavily on recall questions [11]. The American Board of Surgery uses the principle of recalls in its in-service examinations [12]. Other specialties also probably use the concept for various certification and in-training examinations. This trend illustrates a universal question regarding an appropriate test for determination of competency in medical specialties, a question that may require specific attention spanning all specialties.
By documenting the insights and perceptions of two key constituencies within academic radiology, we aimed to aid in the process of the restructuring the ABR is undertaking. We expect this restructuring will result in a clinically relevant examination representative of practical radiology, an examination that can assure medical practitioners and ultimately patients that the radiologist who passes it is highly qualified and competent.
|
|
Acknowledgments
We thank Tanzid Shamsand Keven Tuballes for their assistance in contacting
programs.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
