| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1 All authors: Department of Radiology, Box 3808, Rm. 2526, Blue Zone, South, Duke University Medical Center, Durham, NC 27710.
Received July 27, 1999;
accepted after revision October 1, 1999.
Supported by a grant from the Society of Radiologists in Ultrasound.
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. Sonographic competence tests were administered to 10 first-year diagnostic radiology residents after their involvement in increments of 50 cases, up to a total of 200 cases (four competency tests). Each competency test consisted of the resident's independently scanning and interpreting 10 clinically mandated studies that were scored in comparison with the examination performed by the sonographer and interpreted by an attending radiologist. Trainee studies were graded on the percentage of anatomic landmarks depicted, the number of reporting errors, the number of clinically significant reporting errors, and the percentage of cases receiving a passing score.
RESULTS. Although resident performance improved progressively with increasing experience for all parameters assessed, performance of the group was poor even after their involvement in 200 cases. At this testing level, the mean percentage of anatomic landmarks depicted successfully was 56.5%; the mean total reporting errors per case was 1.2; the mean clinically significant errors per case was 0.5; and the mean percentage of cases receiving a passing score was 16%. Impressive performance differences were observed among residents for all parameters assessed, and these differences were not explained by the number of months of radiology training the resident had taken before the sonography rotation.
CONCLUSION. Involvement in 200 or fewer cases during the training period is not sufficient for physicians to develop an acceptable level of competence in sonography.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The training guidelines established by the ACR and the AIUM for physicians who interpret diagnostic sonographic examinations require at least 3 months of sonographic training during the residency program and involvement in a minimum of 300-500 sonographic examinations during the training period [1,2]. These standards have been adopted by some accreditation programs, and there is a growing trend to link accreditation to reimbursement. Increasingly, the ACR and AIUM training requirements have been challenged by physician groups who claim that the standard is excessive and that competency can be achieved with less rigorous training and after involvement with substantially fewer cases or participation in a brief training course [3,4,5,6,7,8,9,10,11,12,13,14,15,16].
The goal of this study was to test physician competence in sonography after their involvement in prescribed numbers of studies in the hope of determining what case volume is needed for adequate training. In this report we describe and analyze trainee performance after exposure to increasing increments of cases up to a total of 200 examinations.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Each competency test consisted of 10 sonographic studies of patients referred for clinically indicated sonography. The resident independently acquired the basic image set for each study and then interpreted their findings for each of the 10 prescribed studies, without benefit of sonographer or sonologist assistance. Each patient scanned by the resident also underwent a complete sonographic examination performed in accordance with the standard practice in our laboratory, which consisted of images obtained by a trained sonographer and interpreted by an attending physician experienced in sonography. The attending physician reviewed the images generated by the sonographer and performed additional scanning when necessary. Case material was representative of the range of sonographic examinations performed in our laboratory (abdomen, superficial parts, obstetrics, gynecology, vascular). Test cases were assigned to the resident by the sonographer team leader on the basis of which patient was next to be scanned. If more than one patient was available when the resident was ready for the next test case, the supervising sonographer chose the patient whose requested study was less represented within the case material collected up to that time for that particular competency examination. The project was approved by the institutional review board, and patients were included in the testing process provided they gave informed consent for participation. If a patient declined to participate, the team leader selected the next available patient in accordance with the preceding criteria.
The resident was instructed to obtain a full set of sonographic images for each patient and to provide a list of findings and a summary interpretation of the examination. Images generated by the resident were assigned a unique history number to distinguish them from the sonographer or attending physician images that constituted the official examination. Residents could access only their own images when interpreting the study. The attending physician also completed a study sheet listing sonographic findings and issued a final interpretation using only the images produced by the sonographer and the attending physician team. The official report of the sonographic examination communicated to the referring physician was based exclusively on the images and interpretation generated by the sonographer and attending physician team.
The examination performed by the sonographer and interpreted by the attending radiologist was considered the standard for assessing the completeness and accuracy of the resident's study. The complete image sets obtained both by the trainee and by the sonographer and attending physician team were reviewed by a study investigator. To facilitate assessment of image sets, a checklist of specific images required by the operating protocols of our laboratory was generated for each type of sonographic study. Success in generating each of the required images in the relevant protocol was scored as follows: one point = successfully shown, one-half point = shown with qualifications (partial or questionable depiction of image), or no points = not shown. The completeness of the resident-generated image set was rated in comparison with the image set obtained by the sonographer and attending physician team by dividing the resident's point total for the case by the sonographer and attending physician's point total and multiplying by 100%. The resident scores for the 10 cases in each competency test were averaged together to give the percentage of anatomic landmarks successfully depicted by that resident for the particular competency test. A score was thereby generated for depiction of anatomic landmarks after each 50-case increment of experience for each resident and for the group as a whole.
The findings and interpretation of the resident were compared with those of the attending physician for each of the 10 cases in the competency test. Each discrepancy between the resident and attending interpretation was counted as a resident error. Errors were graded for severity as follows: minor error meant unlikely to have affected imaging workup, laboratory studies, medical, or surgical treatment; and clinically significant error meant likely to have required modification in medical or surgical treatment or additional imaging or laboratory studies to disprove or confirm. The total errors for each of the 10 cases in the competency test were averaged to generate an overall total error score at each 50-case increment of experience for each resident and for the group as a whole. Similar analysis was done for the clinically significant errors.
Total errors and clinically significant errors were further subdivided into false-positives (trainee reported a finding not seen by sonographer and attending physician team), false-negatives (trainee failed to identify a finding seen by sonographer and attending physician team), and interpretive errors (trainee accurately saw findings but incorrectly interpreted them).
The resident was assigned an overall grade of pass or fail for each case. Passing criteria were the acquisition of 80% or greater of the required images that had been successfully acquired by the sonographer and attending physician team, and no clinically significant errors in interpretation. The percentage of cases passed at each 50-case increment of experience was calculated for each resident and for the group as a whole.
Statistical analysis included analysis of variance using a mixed linear model to assess if significant differences existed in the percentage of anatomic landmarks shown for the competency tests performed at each testing increment. Differences in total and clinically significant errors with increasing case increments were assessed using a Poisson model with a generalized estimating equations adjustment for correlation within a single resident's outcomes. Differences in the percentage of cases passed with increasing case increments were assessed with a binomial model with generalized estimating equations adjustment. Analysis of variance with a mixed linear model was performed to determine if the month of residency (i.e., first, second, third month) of the sonography rotation correlated with the resident's performance on the competency tests at the various case increments.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
|
Resident performance after involvement in each of the predetermined case volumes is summarized in Table 2 and Figures 1,2,3,4. The percentages of anatomic landmarks depicted increased progressively with increasing experience. Despite this, after involvement in 200 clinical cases, the mean percentage of anatomic landmarks successfully shown by the group as a whole was only 56.5%. Total resident errors and significant errors displayed similar improvement with increasing experience: total errors per case averaged 2.0 at 50 cases and decreased to 1.2 at 200 cases. Corresponding values for clinically significant errors per case were 0.7 at 50 cases and 0.5 at 200 cases. The percentage of cases receiving a passing score with increasing experience also showed progressive improvement. However, even after involvement in 200 cases, the average percentage of cases receiving a passing score was only 16%. Statistical analysis confirmed a significant difference between resident performance at 50 cases compared with 200 cases for all parameters assessed as follows: percentage of anatomic landmarks depicted (p = 0.0001), total errors (p = 0.0001), significant errors (p = 0.01), and percentage of cases passed (p = 0.0002).
|
|
|
|
|
The percentage of total errors that were categorized as false-positive, false-negative, or interpretive error at each of the testing increments is shown in Table 3. Table 4 shows the subdivision of clinically significant errors into false-positive, false-negative, and interpretive errors. False-negatives were much more common than false-positives and interpretive errors at all of the testing increments for both total errors and clinically significant errors.
|
|
A wide range of errors was observed. Typical examples of false-positives included reporting the presence of deep vein thrombosis, a thickened endometrial stripe, or carotid artery stenosis when none was present. Examples of false-negatives included failure to identify gallstones, hydronephrosis, or liver masses. Interpretive errors included showing a groin pseudoaneurysm but incorrectly interpreting the lesion as an arteriovenous fistula, or misinterpretation of a uterine contraction as representing placenta previa.
Although all residents showed improved performance with involvement in increasing increments of sonographic cases, the difference in performance between individual residents was striking (Figs. 5,6,7,8). Variation in resident performance was evident at the time of the initial competency test, and differences between residents tended to persist with increasing experience: as a rule, residents who scored relatively high after involvement with 50 cases also tended to score relatively high after involvement in 200 cases. Similarly, several residents who performed relatively poorly initially did not achieve passing scores on any cases, even during the competency test administered after 200 cases. These differences in resident performance were not explained by the number of months of radiology training the resident had before taking the sonography rotation. No significant effect was seen from month of the sonography rotation for any of the parameters assessed as follows: percentage of anatomic landmarks depicted (p = 0.52), total errors (p = 0.83), significant errors (p = 0.28), or percentage of cases passed (p = 0.28).
|
|
|
|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Accreditation programs in sonography have more recently been adopted by both the ACR and the AIUM. These programs establish minimum requirements for physician competence, and in so doing address licensure, training, case reviews, continuing medical education, and ongoing involvement in studies after the training period. All the physicians performing sonography in a practice must meet these minimum requirements for the laboratory to be granted accreditation. Not surprisingly, accreditation has become an increasingly high-stakes issue as insurance carriers and the government begin linking reimbursement to accreditation. The standards set by the ACR and the AIUM have come under increasing scrutiny as various physician groups campaign for sonography privileges, reimbursement, and less stringent competency guidelines.
The case volumes required during the training period have been a particular point of contention. Some physician groups contend that competency can be achieved with less rigorous training, involvement in a substantially smaller number of examinations than required by the ACR and the AIUM, or participation in a brief training course [3,4,5,6,7,8,9,10,11,12,13,14,15,16]. For example, the Society for Academic Emergency Medicine (SAEM) has established a model curriculum for sonographic training that requires involvement in only 150 sonographic examinations with only 50% of these studies required to be on patients with clinical indications for sonography; the remainder can be performed on healthy models [3]. Emergency physicians have proposed credentialing criteria that are even lower than those advanced in the SAEM model curriculum and have published strategies designed to ensure that responsibility for emergency physician credentialing in sonography is assigned to the emergency department, where presumably training requirements would be less stringent than those in radiology [13,14].
The family practice literature likewise suggests that only 25-75 examinations are needed for competency to perform fetal biometry and fetal anatomic surveys and has further asserted there is no demonstrable learning curve for the acquisition of sonographic skills in fetal biometry and organ survey [6, 9, 10, 15, 16]. Studies published in the emergency medicine, family practice, and surgery literatures ostensibly support these assertions and further claim that particular types of sonographic examinations (e.g., sonography for ureteral colic, pelvic sonography, or trauma sonography) can be performed after exposure to low case volumes or participation in a brief training course [12, 17,18,19,20,21,22].
The performance benefits of training and experience have been convincingly documented for the interpretation of a range of imaging studies [23,24,25,26], but corresponding data are lacking for sonography. Indeed, to our knowledge no published studies exist that justify the substantially higher case volumes required during the training period by the ACR or the AIUM. By assessing physician competence in sonography after involvement in predetermined increments of cases, our study was designed to assess whether lowering the case volume training requirement to some lesser level is tenable.
Our data strongly suggest that involvement with case volumes of 200 or less during the training period is grossly insufficient for developing acceptable physician competence in sonography. Although what constitutes an acceptable level of competence is arguable, even the best scores attained on competency tests after 150 and 200 cases would have to be considered inadequate. Deficiencies were documented both in the ability to produce the basic images and in the ability to produce an accurate interpretation of the available images. These results were obtained without penalizing the trainee for failing to produce acceptable images in circumstances in which imaging was compromised by factors beyond the operator's control, such as patient obesity, bowel gas, or inability of a patient to cooperate fully with the examination. The completeness of the trainee's image set was always judged by comparing it with the image set actually obtained by the sonographer and attending physician team, rather than solely against checklists generated from practice protocols. Even when judged against this adjusted expectation, the image set produced by the residents included an average of only 56.5% of the required images after involvement in even 200 cases.
We attempted to set a standard for passing a case that was both clinically relevant and reasonable, rather than to expect the trainee to generate a perfect image set and an interpretation identical to that of the attending radiologist. With this in mind, cases were given a passing score if the resident produced 80% or more of the required images generated by the sonographer and attending physician and made no clinically significant errors in interpretation. Though one could argue that these criteria should be adjusted slightly higher or lower, such minor adjustments in the threshold for achieving a passing score would have had little overall effect on the numbers of cases passed. Indeed, even the resident with the best performance achieved a passing score on only 50% of the cases after involvement in 200 studies. With the welfare of patients at stake, this could hardly be considered an acceptable level of performance.
Despite the substandard sonographic skills of our trainees after 200 cases, the demonstrated benefit of training was encouraging. Progressive improvement in performance was noted as resident experience increased and residents were involved in an increasing number of cases during the training period. This training effect was seen for all parameters assessed (depiction of anatomic landmarks, total errors per case, clinically significant errors per case, and percentage of cases passed), and the differences in resident performance between 50 and 200 cases were statistically significant for all these parameters. Given this positive trend, we are hopeful that trainee performance will continue to improve with involvement in additional case increments.
Pronounced differences were observed in performance between individual residents. Although each resident exhibited a learning curve, as a general rule those residents who started out with lower scores continued to score relatively poorly even with increasing experience. We had anticipated differences in individual resident performance but were surprised by the magnitude of the variation between our residents, particularly because the residents at our institution are uniformly high-achieving individuals with long records of academic success before the residency. Likewise, though we expected that those residents assigned to the sonography rotation early in their residency would be at a disadvantage relative to those who had had collateral imaging training during other rotations in advance of the sonography rotation, the differences in performance between residents did not correlate with the month of the sonography rotation. Thus, what potential advantage was gained by exposure to other imaging rotations before the sonography rotation was dwarfed by the more dramatic difference in the inherent ability of trainees to learn sonographic skills.
The differences observed between our trainees underscore the operator dependence of sonography. There appear to be innate individual differences in ability to understand and assimilate the skills needed to successfully obtain and interpret sonograms. It is important that these differences be considered when establishing minimum standards for training. Indeed, to protect the public interest, training requirements should be set at a level that ensures most trainees—rather than just the most adept and fastest learners—can achieve an acceptable level of competence.
In summary, this study shows progressive improvement in sonographic skills with involvement in increasing numbers of cases during the training period. Despite this, involvement in 200 or fewer cases is not sufficient for physicians to develop an acceptable level of competence in sonography. In the upcoming years, we plan to retest our study group of trainees to determine if the 500-case level prescribed by the ACR and the AIUM is in fact the optimal threshold, but until this work and other studies are conducted, it would not seem prudent to substantially lower the case numbers required during the training period.
Acknowledgments
We thank Susan Murray for assistance with manuscript preparation and Rosa
Haithcock, Lesa Kurylo, and Sherri Pennell for data entry and case
management.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  J. Wise Everyone's a radiologist now BMJ, May 10, 2008; 336(7652): 1041 - 1043. [Full Text] [PDF]
|
|
|
|
 |
|
 K. Thoirs, M. A. Williams, and M. Phillips Systematic Review of Sonographic Measurements of the Ulnar Nerve at the Elbow Journal of Diagnostic Medical Sonography, September 1, 2007; 23(5): 255 - 262. [Abstract] [PDF]
|
|
|
|
 |
|
 C. L. Moore, S. Gregg, and M. Lambert Performance, Training, Quality Assurance, and Reimbursement of Emergency Physician-Performed Ultrasonography at Academic Medical Centers J. Ultrasound Med., April 1, 2004; 23(4): 459 - 466. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. Lee, A. N. Hodges, S. Williams, I. M. Vettraino, and B. McNie Fetal Ultrasound Training for Obstetrics and Gynecology Residents Obstet. Gynecol., February 1, 2004; 103(2): 333 - 338. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. A. Speed and P. W. P. Bearcroft Musculoskeletal sonography by rheumatologists: the challenges Rheumatology, March 1, 2002; 41(3): 241 - 242. [Full Text] [PDF]
|
|
|
|
 |
|
 W. L. Monsky, D. Levine, T. S. Mehta, R. A. Kane, A. Ziv, B. Kennedy, and H. Nisenbaum Using a Sonographic Simulator to Assess Residents Before Overnight Call Am. J. Roentgenol., January 1, 2002; 178(1): 35 - 39. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. J. Kasales, C. C. Coulson, D. Mauger, J. D. Chertoff, and A. Matthews Training in Obstetric Sonography for Radiology Residents and Fellows in the United States Am. J. Roentgenol., October 1, 2001; 177(4): 763 - 767. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. D. Maldjian, B. S. Hertzberg, and M. A. Kliewer Physician Training Requirements in Sonography Am. J. Roentgenol., April 1, 2001; 176(4): 1075 - 1076. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
