| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1
Beth Israel Deaconess Medical Center, 330 Brookline Ave., Boston, MA
02215.
2
Chiam Sheba Medical Center, Tel-Hashomer 52621, Israel.
3
University of Pennsylvania Medical Center, 3400 Spruce St., Philadelphia, PA
19104.
Received June 12, 2001;
accepted after revision August 3, 2001.
Partially supported by the Carl J. Shapiro Institute for Education and
Research at Harvard Medical School and Beth Israel Deaconess Medical
Center.
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. A sonographic simulator was used to teach and evaluate two consecutive classes of first-year radiology residents. A test consisting of 10 cases was given to each resident. Tests were scored with respect to image quality, measurement accuracy, and interpretation of results. Constructive feedback for improvement and additional training before taking call were provided, as needed. Surveys were given before and after the study to evaluate perceived knowledge and skills.
RESULTS. In the first year of the study, five of eight residents scanned appropriately, giving reasonable differential diagnoses, whereas three residents performed suboptimally and were given feedback, additional training, or both. The sonography curriculum was restructured based on initial resident performance. In the subsequent year, all eight residents performed satisfactorily on the test. Comparing the 2 years of the study, mean test scores increased from 3.5 to 4.0 (p > 0.05) for abdominal test questions and from 3.4 to 4.2 (p < 0.05) for early obstetric and gynecologic test questions. The residents' self-assessment of knowledge and scanning ability also significantly improved.
CONCLUSION. Sonographic simulation allows objective assessment and identification of weaknesses. These weaknesses can then be addressed before taking call, with resultant improved resident education and the presumed benefit of improved patient care.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
In today's busy sonography departments, rapid patient throughput is necessary. Clinical time pressures can negatively impact the educational experience. Having a novice resident scan a patient adds time to the examination, which can stress the schedule and may be seen by the patient as an inconvenience. Particularly during minimally invasive scanning, such as transvaginal sonography, the patient is understandably less likely to tolerate long examination times. In addition, because sonography training typically occurs during clinical practice, gaps may occur in residents' exposure to the range of patient problems and the diseases and abnormalities with which they must be familiar before taking call. It has been suggested that a trainee should be involved in more than 200 cases, representing the spectrum of anatomy and pathology pertinent to sonography, because residents involved in 200 or fewer cases performed poorly on a sonographic competency test similar to that of this study [1].
The potential of simulation-based medical education as an important tool for improving patient safety and the clinical competence of health professionals is increasingly recognized. As a result, simulators have been gaining popularity as a supplement to clinical training in various medical fields [4]. Apart from a simulated environment, a controlled setting in which to objectively assess and qualify resident scanning and interpretation abilities is difficult to achieve. Faculty are often hard pressed to assess objectively whether residents have sufficient knowledge and scanning ability to give appropriate patient care before taking call—a crucial consideration in the determination of clinical readiness. Without accurate evaluation, residents' discomfort or ineptitude may not be recognized until they are in an independent setting, such as on call, where patient care may be jeopardized.
In this study, we evaluate the effectiveness of a sonographic simulator for evaluating residents before taking call.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
A special test CD was made for this study. This CD contained 10 cases with questions on anatomy and pathology that allowed an assessment of the residents' interpretive skills, differential diagnoses, and imaging workup. Each case included relevant patient history and asked that two to four images with appropriate measurements be acquired and saved to floppy disks. The test cases included cholelithiasis without cholecystitis, dilated common duct with intrahepatic ductal dilatation, hydronephrosis, normal midline anatomy, normal uterus, small physiologic ovarian cyst, hyperstimulated ovaries, early obstetric sonography requiring measurement of crown—rump length, and two cases requiring differentiation of an early gestational sac from a missed abortion.
Two consecutive classes of eight first-year residents at Beth Israel Deaconess Medical Center's department of radiology were instructed in the use of the UltraSim and in clinical sonography during clinical rotations. In the first year of the study, 4 weeks of sonographic training was given before call, 2 weeks each at two hospital-based sonography departments. Before call in the second year of the study, residents were given 4 weeks of hospital-based sonography training similar to the training given the first group. This group was also given 2 weeks of training at an outpatient site, with emphasis on transvaginal scanning.
Residents in both classes were instructed to spend a total of 8-10 hr using the simulator and to print and submit their images for review. In practice, residents spent 0-10 hr on the simulator; the mean time spent was 4 hr.
Before beginning overnight call, residents were tested on the UltraSim by one of two attending radiologists or a senior resident, using the test CD and saving acquired images to a floppy disk for off-line assessment. In addition, standardized open-ended questions based on the scanning tasks regarding diagnosis and suggested treatment were graded on a 5-point scale: 1, completely incorrect; 2, shows some knowledge of the subject but gave an incorrect answer; 3, shows basic understanding of the subject and gave a reasonable answer; 4, almost completely correct; 5, completely correct. Constructive feedback for improvement was offered at the end of the examination, including commentary on technical aspects of sonography, criteria for normal and abnormal measurements, as well as on the resident's fund of knowledge. Supplemental reading and attention to particular aspects of the sonographic examination were recommended as deemed necessary by the examiner. A prospective assessment of passing or failing the examination was performed by the person administering the examination. A resident was thought to have failed the examination if he or she had performed poorly on 50% or more of the questions. The failing resident was then required to have additional sonographic training before beginning overnight call. Open-ended feedback regarding the fairness of the examination was obtained from each resident after the examination.
After all residents had completed the examination, the stored images were randomly coded. Images were compared for quality and measurement accuracy with images that were obtained by one of the authors and used as the gold standard. To assess resident expertise, the quality of each image was scored in a blinded review by two experienced radiologists using a 5-point scale: 1, the saved images completely missed the task requirements; 2, the saved images significantly missed the task requirements and did not show an understanding of the task (Fig. 1); 3, the saved image did not meet the task requirements but showed a basic understanding of the task (e.g., the image plane was incorrect but the region of interest was on the film); 4, the saved image met the task requirements, the image plane was correct or close to correct, and the image quality was acceptable (Fig. 2); 5, the saved image completely fulfilled task requirements and included good image quality (brightness, depth, and focal zone) (Fig. 3).
|
|
|
Measurements made by residents were compared with those obtained from the expert images and were graded on a 5-point scale: 1, greater than 40% difference between the resident-obtained and expert measurements; 2, within 40% of the expert measurement; 3, within 30%; 4, within 20% or 3 mm of the expert measurement (whichever was the larger); 5, within 10% or 2 mm (whichever was the larger). For each question, a combined overall score was given that was the average of the image quality, measurement, and differential diagnosis scores.
Surveys were given to residents and to attending radiologists who were giving instruction in sonography the year before residents used the UltraSim and in the second year of the study. These questionnaires addressed residents' self-assessment of their knowledge of sonography and of their scanning abilities. Surveys completed by attending radiologists evaluated the average first-year resident's knowledge and scanning ability when taking call. In addition, residents in both years of the study assessed the realism, content, difficulty, usefulness, and image quality of the simulation.
Chi-square and one- and two-tailed Mann-Whitney tests were used for statistical analysis.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
|
|
In year one, five (62.5%) of eight residents scanned appropriately on the UltraSim test, giving reasonable differential diagnoses and obtaining adequate images, with an overall mean score of 3.76, and answered more than half of the questions correctly. One resident performed suboptimally, receiving a mean score of 2.9, but answered six questions with a score of 3 or above. Two residents (25%) failed the test with mean scores of 2.3, answering five and six cases incorrectly, respectively, with scores below 3. These were the same two residents who were prospectively deemed to have failed the test and who had been given extra time on sonography rotations.
Of 80 questions (eight residents answering the same 10 questions each), 19 (24%) were missed in the first year, compared to one in 80 (1.2%) in the second year (p < 0.0001). Two areas of test questions were particularly problematic in the first year. The first were questions involving the porta hepatis: residents stated that lack of color Doppler sonography on the simulator made identification of the common duct difficult. The other problematic group involved transvaginal assessment of early gestations, for which residents stated that they had not had enough clinical experience.
Given the perceived lack of transvaginal scanning instruction in the first year of the study and the poor test results on the transvaginal cases, the sonography curriculum at our institution was restructured in the subsequent residency year. An additional 2 weeks of sonographic instruction with emphasis on transvaginal scanning was given at our outpatient site.
In year 2 of the study, eight (100%) of eight residents performed satisfactorily. Table 1 details the mean scores of the residents in the two years of the study. Significant improvement was seen in test scores overall, with most marked difference being in the obstetric and gynecologic portions of the test. Comparing the two years, mean scores in early obstetric and gynecologic test cases increased from 3.47 to 4.3 (p < 0.05), whereas mean scores in abdominal cases increased from 3.47 to 4.0 but without statistical significance. Time spent using the simulator did not appear to directly affect performance, although the small study size did not allow statistical analysis.
At the end of the 2 years of the study, resident self-assessment of knowledge and scanning ability significantly improved for early obstetric and gynecologic scanning (Tables 2 and 3). The attending radiologist assessment of resident knowledge and scanning, both during call and on clinical rotations, also improved, but many results were not statistically significant, likely because of the small sample size.
|
Regarding the simulator assessment, residents gave mean scores of 4 (on a 5-point scale from 1 = strongly disagree to 5 = strongly agree) with respect to the simulation-based test having an appropriate range of anatomy, pathology, and difficulty. However, mean scores were lower for adequacy of testing method (2.6), realism (2.8), and image quality (3.0).
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
In our study, two consecutive classes of first-year residents were trained in the usual fashion in clinical sonography and were required to use the UltraSim to supplement this training. Before taking overnight call, residents were tested using the UltraSim. Initially, three of eight residents showed weaknesses, which were addressed with additional instruction before those residents were allowed to take call.
Before this study, it was believed at our institution that the sonography curriculum adequately prepared residents to function at a level of competency appropriate for taking call. Our institution recently underwent a merger that resulted in three sonography sites; in two sites, faculty coverage and case material are heavily weighted to abdominal imaging rather than to obstetric and gynecologic sonography. First-year residents had rotations in these two sites for their first year of study.
In the first year of the study, it became apparent that residents subjectively and objectively were poorly prepared for transvaginal scanning. As a result, our sonography curriculum was restructured. Residents spent more time on sonography rotations (at least 6 weeks) before call, including 2 weeks at the third sonography site, which has high obstetric and gynecologic case volume. With these changes implemented, the subsequent class of residents scored better, and all were thought to be better prepared for overnight call compared with the prior class.
The improvement in performance evidenced by test scores was not necessarily a result of using the simulator for training. Other factors that may have led to improved performance are the restructured curriculum, increased attention to teaching transvaginal scanning after attending radiologists became aware of residents' weakness in that area, and increased resident awareness of the repercussions of poor performance on the test.
To our knowledge, little has been published regarding use of sonographic simulation in residency training. In a study from the Hospital of the University of Pennsylvania [5] it was shown that UltraSim performance improved when comparing first- and second-year residents. This finding indicated that the UltraSim could be used to evaluate scanning competency and, if appropriate tasks are chosen, can differentiate levels of experience. Sonographic simulation also has been used in training surgeons for interpretation of free fluid in the abdomen of trauma patients [2].
Although we found the use of the simulator helpful in evaluating residents, the limitations of simulation must be addressed. Using a sonographic simulator is not meant to and cannot replace scanning real patients. The data sets are limited, allowing the operator to scan only a small area of anatomy at any given time. At times, the reconstruction process resulted in artifacts during scanning, particularly when rotating between transverse and sagittal imaging planes. Lack of color Doppler sonography was also problematic, because residents use it to identify landmarks, particularly in the porta hepatis. Future improvement in the capabilities and data sets of the sonographic simulator may enable training programs to extend the training application and use of this modality even further.
We did not assess the performance of residents on call before and after use of the simulator and did not perform a comparison between the performance of the simulator and performance on human subjects. However, we do think that resident training in sonography has benefited from this project. The curriculum has been standardized, and the residents have a clear idea of the types of cases they will encounter in an on-call setting.
In conclusion, we have shown that sonographic simulation can be used to supplement clinical teaching and to evaluate residents. Not only do individual residents benefit from this approach, but also the adequacy of the curriculum can be assessed and modified. In our institution, such evaluation has resulted in a stronger training program that we expect will translate directly to improved patient care.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  A. J. Towbin, B. E. Paterson, and P. J. Chang Informatics in Radiology: Computer-based Simulator for Radiology: An Educational Tool RadioGraphics, January 1, 2008; 28(1): 309 - 316. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. M. Gaca, D. P. Frush, S. M. Hohenhaus, X. Luo, A. Ancarana, A. Pickles, and K. S. Frush Enhancing Pediatric Safety: Using Simulation to Assess Radiology Resident Preparedness for Anaphylaxis from Intravenous Contrast Media Radiology, October 1, 2007; 245(1): 236 - 244. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
