AJR 2004; 183:135-140
© American Roentgen Ray Society
MRI of Fistula In Ano: Inter- and Intraobserver Agreement and Effects of Directed Education
Gordon N. Buchanan1,
Steve Halligan2,
Stuart Taylor2,
Andrew Williams1,
Richard Cohen1 and
Clive Bartram2
1 Department of Surgery, St. Marks Hospital, Intestinal Imaging, Middlesex,
England.
2 Intestinal Imaging Centre, St. Marks Hospital, Level 4V, Watford Rd., Harrow,
Middlesex, England HA1 3UJ.
Received October 21, 2003;
accepted after revision January 18, 2004.
Supported by Kodak, which funded the MRI via a grant from the Royal College
of Radiologists.
Address correspondence to S. Halligan
(s.halligan{at}imperial.ac.uk).
Abstract
OBJECTIVE. Preoperative MRI of fistula in ano is becoming more
common. This prospective study aimed to determine if a significant difference
occurred in interpretation between one expert and one novice observer and to
assess inter- and intraobserver agreement after both observers underwent a
period of directed education.
SUBJECTS AND METHODS. An outcome-derived reference standard was
defined in 100 patients with suspected fistula in ano via a combination of
preoperative MRI, surgical findings, and clinical outcome. The performances of
a single expert and a single novice interpreter were compared with this
reference standard both before and after a period of directed education, and
inter- and intraobserver agreement was determined.
RESULTS. Initially the expert correctly classified significantly
more fistulas than the novice (85% vs 63%, p = 0.024), but after
directed education there was no significant difference, with good agreement
for both the classification of the primary track (
= 0.71) and
identification of extensions (k = 0.61). Intraobserver agreement was very good
for the expert ( = 0.92) and novice ( = 0.88) for classification
of the primary track and good ( = 0.64 and 0.74, respectively) for
identification of extensions.
CONCLUSION. The diagnostic accuracy for fistula in ano
classification using MRI was significantly higher for one expert than for one
novice, though this was rectified by a short period of directed education.
Introduction
Fistula in ano is a common condition that is usually simple to treat
surgically. However, a significant proportion of patients are difficult to
treat either because the relationship between the fistula and anal sphincter
is unclear or because they suffer recurrence as a consequence of fistulas and
any secondary tracks ("extensions") left undetected and thus
untreated at surgery [1]. For
many years, examination under anesthesia (EUA) by an experienced colorectal
surgeon was considered the reference standard for the detection of fistulas
and any associated extensions, but it is now accepted that preoperative MRI
has a sensitivity surpassing EUA
[2,
3]. Furthermore, preoperative
MRI frequently alters the surgical approach
[4] and, most important,
MRI-guided surgery can significantly reduce postoperative recurrence in
complex cases by 75% [5].
For these reasons, MRI may become routine for assessment of complex or
recurrent fistulas [4,
5]. However, most MRI studies
have originated from specialist centers where the radiologists are familiar
with anorectal anatomy and the different types of fistula. Although good
reproducibility for MRI interpretation has been shown between experienced
observers [4], poor
reproducibility by inexperienced radiologists may negate the potential value
of imaging this condition [6].
To our knowledge, the level of agreement between experienced and inexperienced
observers has not been described. This prospective study assessed whether a
significant difference occurred in interpretation between an expert and a
novice observer and inter- and intraobserver agreement after a period of
directed education in MRI interpretation.
Subjects and Methods
One hundred consecutive patients (71 males, 29 females; median age, 42
years; age range, 17–65 years) with suspected fistula in ano were
recruited as part of an ongoing prospective trial assessing the value of
preoperative MRI [5,
7], for which local ethics
committee approval had been obtained. All patients provided written informed
consent.
All patients underwent preoperative 1.0-T MRI (Gyroscan T10-NT, Philips
Medical Systems) using a previously described protocol
[8]. Patients were scanned in
the supine position. The anal canal was identified using a midline sagittal
localizing scan from which axial and coronal STIR sequences were planned with
respect to the anal canal axis, using the following scanning parameters:
TR/TE, 1,500/15; field-of-view, 375 mm; matrix, 256 x 256; slice
thickness, 4 mm; interslice gap, 1 mm; number of excitations, 4. The body coil
or a phased array surface coil was used. No internal coils were used. MRI was
performed under the direct supervision of an experienced consultant
gastrointestinal radiologist. Each patient subsequently underwent EUA in the
operating room with the benefit of the MRI results, with surgery influenced by
this at the discretion of the operating surgeon. Patients were then followed
up as outpatients as per standard practice, their healing was documented, and
any sign of fistula recurrence was noted, including the findings of any
further EUA. Median duration of follow-up was 18 months (range, 10–46
months). Because no single investigation, including EUA, is known to be
definitive for precise fistula classification, a combination of preoperative
MRI, imaging-directed EUA, and postoperative outcome was used to determine an
outcome-derived reference standard fistula classification for each patient, a
procedure that has been previously validated and shown to be accurate
[9].
The principal investigator then divided the 100 cases into two groups of
50, matched for demographic features and fistula complexity
(Table 1). The first group of
50 patients was assessed independently by two observers blinded to the MRI
reports, surgical findings, and clinical outcome. The principal investigator
was not one of the observers but was a surgeon undertaking a postgraduate
thesis in MRI of fistula in ano. The expert observer was a consultant
gastrointestinal radiologist with experience of reporting approximately 1,400
MRI studies for fistula in ano. The novice observer was a fourth-year
radiology trainee with a subspecialty interest in gastrointestinal imaging who
had just commenced a fellowship in this field. The novice had no prior
training in reporting MRI for fistula in ano and had not experienced any
specific training in abdominopelvic MRI in general. The novice had prepared
for this study by reading peer-reviewed literature relevant to the topic.
The two observers recorded their findings on a fistula classification sheet
derived from the method of Parks et al.
[10], which has been validated
in previous studies of fistula in ano
[4,
5,
11]. If multiple fistulas were
present, these were recorded on the same sheet. Sepsis detected on MRI was
depicted as high signal using STIR sequences. Primary tracks were classified,
according to the method of Parks et al., as superficial where they course
medial to or below the internal sphincter, intersphincteric where they run
between internal and external sphincters, or transsphincteric where they cross
both internal and external sphincters to reach the ischiorectal fossa;
suprasphincteric tracks pass cranially from an anal canal internal opening
within the intersphincteric plane to loop over the puborectalis and pass
downward through the levator ani to the skin, and extrasphincteric tracks pass
directly from the rectum through levator ani to the skin lateral to the
external sphincter. In the absence of an internal opening, the track was
classified as a sinus rather than a fistula
[10]. The radial site of the
internal opening was defined according to clock position (6-o'clock
posterior), and its level was recorded as either rectal or anal. Any
associated secondary extensions or abscesses were defined by their anatomic
quadrant and location: ischiorectal, intersphincteric, or supralevator
[10]. Extensions in one
direction were termed abscesses; however, those extending to either side of
the primary track were termed "horseshoe extensions."
The principal investigator then determined the level of agreement between
the observers' assessments and the outcome-derived reference standard. Any
categoric discrepancy relating to the primary fistula track classification was
considered a disagreement. The radial site of the internal opening was
considered correct if recorded to within one quadrant on the clock face and
its level judged correct if the correct category for the enteric opening
(rectal or anal) had been indicated. Disagreement for the presence or absence
of any extension was noted, and further disagreement was noted if the anatomic
site or quadrant had been recorded incorrectly by the observer.
The principal investigator then undertook a detailed review with each
observer independently. This entailed comparing the completed observer
classification sheet and outcome-derived reference standard on a case-by-case
basis. The precise nature of any disagreement between the two assessments was
flagged by the principal investigator and verbally discussed with the observer
during re-review of the MRI hard-copy films on a viewing box. All studies were
reviewed, including those in which the observer's assessment was judged
correct; the whole process took approximately 3 hr for each observer. Each
observer was therefore educated on a case-by-case basis.
Each observer then independently reported the second group of 50 MR images
in an identical fashion to the first, again submitting findings to the
principal investigator when assessment was complete. To assess intraobserver
agreement, each observer performed a further assessment of the second group of
cases that was undertaken blinded to the previous assessment and that was
performed at least a month later, with the film order shuffled, in an attempt
to eliminate recall bias. Comparison with the outcome-derived reference
standard was again performed and agreement determined.
Any significant differences between the diagnostic accuracy of each
observer for the two groups of 50 cases were determined by comparing their
assessments with the outcome-derived reference standard. Primary tracks and
abscesses or horseshoe extensions were considered correctly identified when
described in the correct anatomic location according to the reference
standard. Internal openings were considered correctly identified when
described within the same quadrant (defined as within 2 hr on the clock face),
and at the same level (anal or rectal) as the reference standard. Differences
in categoric frequencies between the two observers were compared using
Fisher's exact test. Strength of agreement between the two observers for the
second group of 50 studies was determined using the kappa statistic
[12]. Analysis was performed
using Arcus Quickstat Biomedical (version 1.2, Research Solutions).
Statistical significance was assigned to a probability level of less than
0.05, and levels of agreement were defined according to Landis and Koch
[12].
Results
Of the 100 patients, four (4%) had no sepsis, six (6%) had sinuses, and 90
(90%) had 96 fistulas (Table
1).
Correct Classification
The experienced observer correctly classified significantly more fistula
tracks than the novice (85% vs 63%, p = 0.024;
Table 2) (Fig. 1) when the first 50
studies were assessed, but no significant difference existed between them for
the second 50 after the period of directed education (85% vs 72% and 83% vs
76% for each of the two second interpretations, respectively; p =
0.16 and 0.47, respectively; Table
2) (Fig. 2). No
significant difference existed between the expert and novice observer for
correct identification of abscesses, horseshoe extensions, or internal
openings when the first and second groups were compared
(Table 2).
View this table:
[in this window]
[in a new window]
|
TABLE 2 Comparison of Correct Fistula Classifications by the Expert and the
Novice Observers for the First 50 MRI Examinations and for the Second 50 After
Directed Education
|
|
View larger version (158K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 1. —Axial STIR MR image obtained at mid anal canal level in
41-year-old man in whom expert observer correctly identified intershpincteric
fistula in ano (arrow). Location was misclassified as
transsphincteric by novice before directed education.
|
|
View larger version (140K):
[in this window]
[in a new window]
[as a PowerPoint slide]
|
Fig. 2. —Axial STIR MR image obtained at mid anal canal level in
36-year-old woman in whom both expert and novice observers correctly
identified transsphincteric fistula in ano associated with ischiorectal
extension (arrow) after period of directed education.
|
|
Interobserver Agreement
After the period of directed education, good agreement ( = 0.71,
Table 3) was seen between the
expert and novice for classification of the primary track and also good
agreement ( = 0.61, Table
4) for the identification and localization of extensions
[12].
View this table:
[in this window]
[in a new window]
|
TABLE 3 Interobserver Agreement Between the Expert and the Novice for
Classification of the Primary Fistula Track After Directed Education
|
|
View this table:
[in this window]
[in a new window]
|
TABLE 4 Interobserver Agreement Between the Expert and the Novice for
Classification of the Identification and Localization of Extensions After
Directed Education
|
|
Intraobserver Agreement
After the period of directed education, very good intraobserver agreement
was seen for both the expert ( = 0.92,
Table 5) and the novice
( = 0.88, Table 5) for
classification of the primary track
[12]. Intraobserver agreement
for identification and anatomic localization of extensions was good for both
the expert ( = 0.64, Table
6) and the novice ( = 0.74,
Table 6)
[12].
View this table:
[in this window]
[in a new window]
|
TABLE 5 Intraobserver Agreement for Expert and Novice Observers for
Classification of the Primary Fistula Track After Directed Education
|
|
View this table:
[in this window]
[in a new window]
|
TABLE 6 Intraobserver Agreement for Expert and Novice Observers for
Identification and Localization of the Extensions After Directed
Education
|
|
Discussion
We have shown a significant difference in the correct classification of
fistula in ano with MRI between a single expert and a novice observer. This is
not surprising because it has long been recognized that experts perform better
than generalists on the presumption that their subspecialty experience
enhances their ability to interpret new cases on the basis of prior
experience, and they understand the clinical context more readily
[13,
14]. Supporting this premise,
an MRI study of 27 patients with fistula in ano found that only 42% of
fistulas were correctly classified by a radiologist new to the technique
[6]. However, the authors were
able to show that correct interpretation rose to 50% by the end of the study
and suggested that a learning curve was responsible
[6]. Nevertheless, this figure
is still well short of those achieved using experienced radiologists
[11,
15].
Surprisingly, we found no significant difference between the expert and the
novice observer when the identification and anatomic localization of
extensions were considered, even before the period of directed education was
undertaken. This implies that the major challenge for radiologists embarking
on preoperative MRI for fistula in ano lies with correct classification of the
primary track (i.e., inter-, trans-, supra-, or extrasphincteric) rather than
the identification of extensions. Beets-Tan et al.
[4] found good to very good
agreement between an experienced radiologist and a radiology resident,
although it is unclear whether the resident was also experienced in
interpretation of fistula MRI. As in our study, agreement was best for
detection of extensions rather than track classification
[4]. We were able to show no
significant difference (and good agreement) between our expert and novice
after directed education, suggesting that the technique may be rapidly
mastered given the right training environment
(Fig. 2). Both observers were
radiologists and intraobserver agreement was very good for classification of
the primary track, suggesting that the technique may be highly reproducible if
the same observer is used for any follow-up studies.
Accurate assessments of MRI for fistula in ano have been fraught with
difficulty because no easily available reference standard exists with which to
compare imaging results. Surgical EUA alone, once believed to be an acceptable
reference standard, has proven to be inadequate for vigorous studies. Indeed,
the first MRI study of cryptoglandular fistula in ano found that EUA was
incorrect in two of the 16 cases studied
[2], and a recent study of 71
patients with recurrent disease found that further disease in 16 patients
occurred at sites with apparently negative findings at EUA but suspicious at
MRI [5]. However, MRI is also
imperfect, especially with respect to the level of the internal opening
[5], and so a combination of
assessment techniques and clinical outcome after surgery is necessary to be
confident that the ultimate classification is correct
[5,
9]. We used such an
outcome-derived reference standard so that we could be confident that
incorrect reference classifications did not confound our assessments of
observer accuracy.
Our study does have some limitations, notably that only one expert and one
novice were used. Given this, each is effectively acting as a proxy for all
experts and novices respectively. Although we can be confident that our expert
really was so, based on substantial personal experience and previous
performance in blinded MRI comparisons
[5,
7,
8,
16], it could be argued that
our novice, because of a declared interest in the subspecialty of
gastrointestinal radiology, may have been unusually receptive to directed
training in this field. Nevertheless, he had not been responsible for the
reporting of MRI examinations for fistula in ano before the study yet had
achieved statistical parity with the expert by the end of it. It has been our
anecdotal experience with other novices that training is generally
straightforward and depends most on a thorough understanding of the disease
pathogenesis and the surgical issues
[17]; the imaging and
interpretation is generally straightforward once these are understood. It is
our belief that adequate performance could be achieved after only a few hours
of directed training with an expert as long as the novice understands the
pathogenesis and treatment of fistula in ano. Whether there are enough
radiologists presently to provide training is unknown, but increasing surgical
awareness of the technique will inevitably increase demand.
In summary, this study suggests that experts perform significantly better
than novices for preoperative classification of fistula in ano using MRI, but
that novices can achieve acceptable performance after a period of directed
education. After directed education, agreement between experts and novices is
good and intraobserver agreement is also acceptable.
References
- Seow C, Phillips RK. Insights gained from the management of
problematical anal fistulae at St. Mark's Hospital, 1984–88.
Br J Surg 1991;78:539
–541[Medline]
- Lunniss PJ, Armstrong P, Barker PG, Reznek RH, Phillips RK.
Magnetic resonance imaging of anal fistulae. Lancet1992; 340:394
–396[Medline]
- Beckingham IJ, Spencer JA, Ward J, Dyke GW, Adams C, Ambrose NS.
Prospective evaluation of dynamic contrast enhanced magnetic resonance imaging
in the evaluation of fistula in ano. Br J Surg1996; 83:1396
–1398[Medline]
- Beets-Tan RG, Beets GL, van der Hoop AG, et al. Preoperative MR
imaging of anal fistulas: does it really help the surgeon?
Radiology2001; 218:75
–84.[Abstract/Free Full Text]
- Buchanan G, Halligan S, Williams A, et al. Effect of MRI on
clinical outcome of recurrent fistula-in-ano. Lancet2002; 360:1661
–1662[Medline]
- Scholefield JH, Berry DP, Armitage NC, Wastie ML. Magnetic
resonance imaging in the management of fistula in ano. Int J
Colorectal Dis 1997;12:276
–279[Medline]
- Buchanan GN, Halligan S., Williams, AB, et al. Magnetic resonance
imaging for primary fistula-in-ano. Br J Surg2003; 90:877
–881[Medline]
- Halligan S, Bartram CI. MR imaging of fistula in ano: are endoanal
coils the gold standard? AJR1998; 171:407
–412[Abstract/Free Full Text]
- Schwartz DA, Wiersema MJ, Dudiak KM, et al. A comparison of
endoscopic ultrasound, magnetic resonance imaging, and exam under anesthesia
for evaluation of Crohn's perianal fistulas.
Gastroenterology 2001;121
:1064
–1072[Medline]
- Parks AG, Gordon PH, Hardcastle JD. A classification of
fistula-in-ano. Br J Surg 1976;63
: 1–12[Medline]
- Lunniss PJ, Barker PG, Sultan AH, et al. Magnetic resonance imaging
of fistula-in-ano. Dis Colon Rectum1994; 37:708
–718[Medline]
- Landis JR, Koch GG. The measurement of observer agreement for
categorical data. Biometrics1977; 33:159
–174[Medline]
- Robinson PJA. Radiology's Achilles' heel: error and variation in
the interpretation of the Röntgen image. Br J
Radiol 1997;70:1085
–1098[Abstract]
- Halligan S. Subspecialist radiology. Clin
Radiol 2002;57:982
–983[Medline]
- Spencer JA, Ward J, Beckingham IJ, Adams C, Ambrose NS. Dynamic
contrast-enhanced MR imaging of perianal fistulas. AJR1996; 167:735
–741[Abstract/Free Full Text]
- Taylor SA, Halligan S, Bartram CI. Pilonidal sinus disease: MR
imaging distinction from fistula in ano. Radiology2003; 226:662
–667[Abstract/Free Full Text]
- Halligan S. Imaging fistula-in-ano. Clin
Radiol 1998;53:85
–95[Medline]
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
Top
Abstract
Introduction
