DOI:10.2214/AJR.06.5001
AJR 2006; 186:265
© American Roentgen Ray Society
Performance-Based Assessment of Radiology Faculty
Richard FitzGerald
The Royal Wolverhampton Hospitals NHS Trust Wolverhampton WV10 0QP,
United Kingdom
Donnelly and Strife [1] are
to be commended on the work in progress that they report on assessing
radiologists' performance. The peer-review database is a potent instrument
whereby all radiologists in a department can learn from mistakes and
discrepancies made by themselves and their colleagues. It also can serve as a
prompt whereby concern can be raised about the performance of an individual
radiologist in respect to a particular type of activity.
At first sight, it would seem entirely reasonable that anyone appearing
more than 2 SDs above the normalized mean should be notified and reported to
the radiologist in chief. Yet in a British hospital where 25 radiology
discrepancy meetings are held per year, only 81 cases involving error were
reviewed in a 12-month period during which 105,000 examinations were performed
[2]. The number of cases
entered into the Cincinnati Children's Hospital peer review database and the
individual workload figures of its radiologists are not disclosed
[1]. The number of potential
errors discussed at radiology discrepancy meetings typically is such that
there is not an adequate sample size to derive error rates of individual
radiologists. Studies from the development of the U.K. General Medical Council
Performance Procedures found that "generalizable results would be
obtained with not less than 3 judges all rating the same 60 reports from a
radiologist" and "any assessment of performance of technical
abilities in this field will need to use multiple assessors basing judgements
on an adequate sample of reports"
[3]. Radiology peer review
databases and discrepancy meetings are subject to many pitfalls: hindsight
bias, information bias, presentation or format bias, outcome bias, selection
bias, faulty data collection, confounders, and, very important, effect of
sample size. The difference between common cause and special cause variation
needs to be recognized [4].
When comparing individual radiologist performance with reported performance
in the literature, bear in mind that the quality of reporting in diagnostic
accuracy articles in radiology journals is "less than optimal"
[5]. Radiologic accuracy is
often much less than we would expect. Two experienced university
neuroradiologists, when evaluating lumbar disk herniation on MRI of patients
for whom they had adequate clinical information, reported that full agreement
is only 77% [6].
With the exception of breast imaging, Soffa et al.
[7] are quite right that
"radiologists do not currently have an objective benchmark for an
acceptable level of missed diagnoses." Their department has shown that a
blinded second interpretation quality assurance system can work in daily
practice [7]. It does not have
the problems of selection bias and recruitment of the ACR RADPEER system,
which has been described as a "less than perfect measuring
system"[8].
Setting a valid radiologic standard will take time. A radiologist whose
performance is below average may still be performing adequately. A reporting
discrepancy does not imply error
[9]. The clinical information
given to a reporting radiologist may be inadequate, incomplete, or incorrect.
Moreover, radiology reports are only part of the diagnostic jigsaw. Even if a
radiology report has the potential to adversely affect patient management, it
often does not. The genesis of radiologic error and discrepancy is frequently
systemic [10]. Although
monitoring systems such as the peer review database are important, they must
be used skillfully and sensitively. There needs to be equal emphasis on better
teamwork, more knowledge sharing, and the value of targeted instruction of
those radiologists whose performance is perceived to be less than adequate
[11].
References
- Donnelly LF, Strife JL. Performance-based assessment of radiology
faculty: a practical plan to promote improvement and meet JCAHO standards.
AJR 2005; 184:1398
-1401[Abstract/Free Full Text]
- Melvin C, Bodley R, Booth A, Meagher T, Record C, Savage P.
Managing errors in radiology: a working model. Clin
Radiol 2004; 59:841
-845[CrossRef][Medline]
- Jolly BC, Ayers B, MacDonald MM, et al. The reproducibility of
assessing radiological reporting: studies from the development of the General
Medical Council's Performance Procedures. Med Educ2001; 35[suppl 1]:36
-44
- Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and
clinical governance: Shewhart's forgotten lessons.
Lancet 2001; 357:463
-467[CrossRef][Medline]
- Smidt N, Rutjes AW, van der Windt DA, et al. Quality of reporting
of diagnostic accuracy studies. Radiology2005; 235:347
-353[Abstract/Free Full Text]
- van Rijn JC, Klemetso N, Reitsma JB, et al. Observer variation in
MRI evaluation of patients suspected of lumbar disk herniation.
AJR 2005; 184:299
-303; erratum in AJR 2005;
184:1027[Abstract/Free Full Text]
- Soffa DJ, Lewis RS, Sunshine JH, Bhargavan M. Disagreement in
interpretation: a method for the development of benchmarks for quality
assurance in imaging. J Am Coll Radiol2004; 1:212
-217[CrossRef][Medline]
- Borgstede JP, Lewis RS, Bhargavan M, Sunshine JH. RADPEER quality
assurance program: a multifacility study of interpretative disagreement rates.
J Am Coll Radiol 2004;1
: 59-65[CrossRef][Medline]
- Board of Faculty of Clinical Radiology, The Royal
College of Radiologists. To err is human: the case for review of reporting
discrepancies. London: The Royal College of Radiologists,2001
: 4-12
- FitzGerald R. Error in radiology. Clin
Radiol 2001; 56:938
-946[CrossRef][Medline]
- FitzGerald R. Radiological error: analysis, standard setting,
targeted instruction, and teamworking. Eur Radiol2005; 15:1760
-1767; DOI10.1007/s00330-005-2662-8[CrossRef][Medline]
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
Top
References
