| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Original Research |
1 Department of Radiology, Cincinnati Children's Hospital, 3333 Burnet Ave.,
M.L. 5031, Cincinnati, OH 45229-3039.
2 College of Business, University of Cincinnati, Cincinnati, OH.
Received September 7, 2007;
accepted after revision February 4, 2008.
Components of the workflow management system described here have been
licensed to a public company for commercialization in the radiology market
under the trade name RadStream; all data were collected and analyzed before
licensing.
Abstract
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
MATERIALS AND METHODS. The WMS prioritizes cases automatically on the basis of medical and operational acuity factors, automatically facilitates communication of critical radiology results, and provides permanent documentation of these results and communications. It runs in parallel with an integrated radiology information system (RIS)–PACS and speech-recognition system. Its effects on operations, staff stress and satisfaction, and patient satisfaction were studied.
RESULTS. Despite an increase in caseload volume after the launch of the WMS, case turnaround times, defined as the time between case availability on PACS and signing of the final radiology staff interpretation, decreased for all case types. Median case turnaround time decreased by 33 minutes (22%) for emergency department, 47 minutes (37%) for inpatient, and 22 minutes (38%) for outpatient radiology cases. All reductions were significant at a p value of < 0.05. Interruptions were reduced, consuming an estimated 28% less radiology staff time, after implementation. Patient perceptions of radiology service timeliness showed modest improvement after the WMS was implemented. Staff satisfaction showed no significant change.
CONCLUSION. There is room for improvement in radiology workflow even in departments with integrated RIS–PACS and speech-recognition systems. This study has shown that software tools that coordinate decentralized workflow and dynamically balance workloads can increase the efficiency and efficacy of radiologists. Operational benefits, such as reduced reading times, improvements in the timeliness of care (both actual and as perceived by patients), and reduced interruptions to radiologists, further reinforce the benefits of such a system. Secondary benefits, such as documenting communication about a case and facilitating review of results, can also promote more timely and effective care. Although use of the system did not result in a substantial improvement in staff perceptions, neither did it reduce their satisfaction, suggesting that these operational improvements were not achieved as a trade-off against the quality of the work environment.
Keywords: management • PACS • radiology information system • speech-recognition software • work efficiency • workflow management system
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Managing incoming and outgoing pages and telephone calls while reading new studies causes workflow interruptions that slow the interpretive process. Moreover, such workflow interruptions disrupt the mental focus of the radiologist and likely thereby increase the potential for errors [8]. It seems obvious that the radiologist distracted by multitasking is more likely to forget to report or further investigate important findings than is the radiologist working without interruption [9, 10].
Furthermore, the manual prioritization (or triage) of waiting cases can add significantly to the disruptions and delays that prevent a radiologist from making progress against the queue. Because different radiologists can have different mental heuristics for determining which case should be read next, this variation can negatively influence the quality of care by increasing the chance that the most acute cases will not be read first.
|
|
We designed, built, and launched a radiology workflow management system (WMS) to address these challenges. We then assessed the effects of the new software on various operational and perceptual metrics. This article reviews the methods used to test the effects of the WMS on patients and the institution and, based on those results, offers conclusions for practice.
For this study, we initially hypothesized that introducing the WMS into a multisite, academic radiology department with a fully deployed single-vendor commercially available RIS–PACS and speech-recognition system would, first, reduce case turnaround time; second, reduce radiologist workflow interruptions; third, improve patient satisfaction; and, fourth, improve radiology staff satisfaction and reduce job stress.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Software Development
The system was developed in a
Microsoft.net
environment using JavaScript (Sun Microsystems) and C++. Because we did not
have privileges to write to the database of our commercial, single-vendor
RIS–PACS and speech-recognition system, we built the WMS to operate as a
stand-alone system. Thus, it runs on the left-hand third monitor—the
same monitor that displays RIS information. The WMS runs rolling queries
against the RIS, importing patient, physician, and examination data
automatically, and performs continuous, near real-time updates of study status
changes, patient locations, ordering physician information, and availability
of radiology reports.
To enable the WMS to prioritize cases automatically, we developed a patent-pending algorithm that closely mimics the mental heuristics used by clinical radiologists (unpublished data). The algorithm takes into consideration a variety of factors, including the patient's medical acuity and psychologic state, the time the patient has been waiting, and so on, to continually prioritize cases. Radiologists are presented with a prioritized work list, but they are free to select any case from the work list at any time.
The WMS also facilitates and documents verbal critical result conveyances. It coordinates communication by "operators"—that is, medical assistants, file room personnel, physician extenders, and others who convey critical results by telephone—to referring clinicians. The WMS tracks whether the referring clinician requested a call as soon as results become available and, for those cases, automatically routes the final signed dictations to the operators' work screens along with full patient demographic information, contact information for all clinicians involved with that patient's care, and the location and contact information for the interpreting radiologist. The operator verbally conveys the result to the referring clinician or his or her staff and permanently logs this conveyance for future audit, if necessary. If the operator is unable to reach the referring physician immediately, the case moves to a hold queue; when the referring clinician returns the page or telephone call, any available operator can pick up the case from the hold queue and complete the report conveyance. Whichever operator completes the report conveyance then documents this in the communication log, which is permanently saved in the WMS's database.
Pilot testing of the WMS was performed with a limited launch using studies ordered by the emergency department. Based on those initial results, functionality and implementation of the system were then expanded to drive radiology workflow department-wide. The new system was used for approximately 12 weeks to allow users to become familiar with its functions and the revised (paperless) workflow. Operational metrics and survey measurements of radiology staff stress and satisfaction, as well as patient satisfaction, were collected, as described later in this article.
Measurement and Analysis
Before initiating the actual study, psychometric instruments were refined
and validated in a pilot study. Then, to establish a baseline before WMS
launch for our metrics, we collected process time stamps for each step of the
radiology study acquisition and interpretation process and, through
observation, we measured the duration and type of workflow interruptions
experienced by radiologists. Radiology staff job satisfaction and stress were
measured on online surveys, and patient satisfaction was assessed on paper
surveys distributed by technologists and front desk staff.
After the baseline data were collected, staff members were trained to use the WMS, process changes to correspond to new information flow were implemented, and the system was launched. After launch, additional user support and training were provided. After a previously established ramp-up period of 12 weeks (to complete department-wide training and ensure staff were comfortable with the system) after roll-out, a second round of survey and operational data collection exactly replicated the baseline data collection process. A timeline of these events is shown in Figure 1. In the following sections, we review data collection and analysis methods in detail and discuss our hypotheses mentioned earlier.
Measurement of process flow—To measure the effects the WMS had on clinical operations, we first identified four important process points that exist across all patient types as shown in Figure 2. Time-stamp data for these four radiology process points were routinely recorded by the RIS and extracted for the two 4-week periods (before and after WMS implementation) shown in Figure 1. We collected data across three general study types: stat (emergency department and urgent care centers) studies, inpatient studies, and outpatient studies.
Assessing the difference between process times before and after implementation involved calculating the difference between consecutive process point time stamps to get activity durations. For example, the duration of the reading activity was calculated as the difference between the first time stamp (end of procedure) and the second time stamp (initial dictation completed).
A small number of outlier cases exhibited durations between time stamps exceeding 72 hours (these outliers were typically tied to special causes, exceptions, or patient handling outside the normal workflow). These outlier cases represented fewer than 1% of all cases included in the data set and were removed (listwise deletion) from further analysis. This same threshold and treatment were applied for both pre- and postimplementation measurements.
Given that these time durations were not normally distributed even after truncating the distributions at 72 hours, we relied on nonparametric tests. For comparing mean process durations, we used the Wilcoxon's rank sum (Mann-Whitney U) test. For comparing median process durations, the Kruskal-Wallis one-way analysis of variance by ranks test was used.
Measurement of workflow interruptions—To measure the effect of the WMS on clinical radiologist workflow interruptions, trained observers were stationed in reading rooms where two radiologists interpreted urgent cases from 5:00 pm to 9:00 pm on weeknights. For each measurement stage (pre- and postimplementation), interruption data were recorded during 20 hours of direct observation time (40 hours total over both stages). As mentioned earlier, all data were collected using the same methods in the periods before and after implementation.
Observers who were not hospital employees were trained to record, in real time, the start time, duration, and nature of each interruption. Interruptions included walk-ins (physicians and technologists with questions), telephone calls from out side physicians and their staff, and other medical care–related interruptions. Self-induced work stoppages, such as attending to scheduled, supplemental duties or personal issues, were monitored but were not included in calculations. For comparing interruption attributes (mean durations and interarrivals) before and after implementation, we used the one-sided (interarrivals increasing and durations decreasing) Wilcoxon's rank sum test.
These objective interruption data were complemented by subjective measures of perceived interruptions. Included on radiology staff surveys (detailed more completely later) was a single-item measure (7-point Likert format) assessing staff members' perceptions of the frequency with which their work was interrupted. Mean differences in this metric between pre- and postimplementation states were evaluated using separate Student's t tests for physicians and technologists.
Measurement of patient and parent perceptions—Single-item measures were developed to assess patient and parent satisfaction with excellence of medical treatment, timeliness of service, and likelihood of returning. Paper surveys with instructions were distributed by technologists to a parent (or an accompanying adult) or to the patient at the end of each patient's procedure. Surveys contained 19 questions in total covering a variety of issues and came with prestamped, preaddressed envelopes to facilitate their return.
Patient and parent satisfaction and timeliness of service were assessed using 7-point Likert scales. Likelihood of returning was assessed using a 5-point Likert scale. Pilot testing suggested that these items exhibited sufficient reliability (vis-à-vis other previously validated items) and were likely to be easily understood by respondents. Mean differences between pre- and post implementation states for each of these three metrics were evaluated using Student's t tests.
Because the implementation environment was a pediatric facility, it is possible that most survey respondents were parents, guardians, or other adults who accompanied a patient. In no case, however, did both a patient and an accompanying adult complete different surveys for the same visit—only one survey was given to each patient. Because inpatients and emergency department patients were largely unable to complete surveys or their experience with radiology was not distinct from other services provided at the hospital, the patient and parent surveys reported here were collected solely for outpatient radiology cases. No individually identifiable patient data were collected.
Measurement of radiology staff perceptions— To measure staff and patient perceptions of the work environment, a combination of previously validated and newly developed measurement instruments was used. Staff members were invited via institutional e-mail to participate in Web-based surveys. Two e-mail reminders were sent over a period of 2 weeks. All radiologists who perform clinical work and technologists who use the RIS–PACS as their primary information system were invited to participate in the survey. Staff surveys included 43 questions, 33 of which constituted the following six constructs.
Staff perceptions of the technical (information system) environment were measured across three constructs: overall information system quality, ease of use, and usefulness. All were measured using multiitem scales. The overall information system quality scale initially consisted of 12 items from the end-user computing satisfaction instrument [11]. Results from the pilot study supported reducing this scale to a more parsimonious eight items while maintaining acceptable reliability and validity. The ease-of-use scale and the usefulness scale each initially consisted of four items from the Technology Acceptance Model instrument [12]. Pilot results suggested that all four items should be retained for each scale. All items were assessed on 7-point Likert scales.
Staff perceptions of the work environment were assessed across three constructs: psychologic stress, empowerment, and job satisfaction. The psychologic stress scale initially consisted of seven items based on the Job Content Questionnaire [13]. Pilot results indicated that three items sufficiently captured this construct while exhibiting satisfactory reliability and validity. The empowerment scale initially consisted of three items also from the Job Content Questionnaire [13]. Pilot study data indicated that keeping all three items was necessary to maintain adequate reliability and validity. Finally, the job satisfaction scale initially consisted of four items newly developed for this study. Pilot results indicated that one of the proposed items did not exhibit adequate convergent validity, so it was dropped; the remaining three items constituted an adequately reliable and valid measurement scale for this construct. Complete scale construction and item wording are available on request from the authors. A 7-point Likert scale was used for all items.
Unweighted scale means were calculated separately for radiologists and technologists. Differences between pre- and postimplementation states were then evaluated using Student's t tests for each scale mean.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
|
Process Flow
Over 14 consecutive days before implementation of the WMS, complete and
usable time-stamp data were acquired for 6,093 cases (1,057 emergency
department patients; 1,479 inpatients; 3,557 outpatients). During the 15-day
measurement period after implementation, data were collected for 7,493 cases
(1,528 emergency department patients; 1,313 inpatients; 4,652 outpatients).
Results of the comparison are shown in Tables
2 (means) and
3 (medians). Medians were
primarily relied on for interpretation because of the data's distributional
deviations from normality.
|
|
Implementation of the WMS led to a decrease in median outpatient report reading times from 0.95 to 0.59 hour (a 38% improvement) and a decrease in median inpatient report reading times from 1.71 to 0.99 hour (a 42% improvement). Emergency department median report reading times did not significantly change, remaining approximately 0.38 hour. Median inpatient report sign off times decreased from 0.41 to 0.35 hour (a 15% improvement), and median emergency department report sign off times decreased from 2.10 to 1.54 hours (a 27% improvement). Median outpatient report sign off times were unchanged at 0.0 hours and, for all report types, median transcription times were unchanged at 0.0 hours due to widespread use of speech recognition. These results support our first hypothesis—that is, implementation of the WMS would reduce case turnaround time.
Workflow Interruption
Based on the objective (third-party) interruption data, mean interarrival
times significantly increased (i.e., interruptions occurred less frequently)
from 11.8 to 15.1 minutes (p < 0.10) after the WMS was
implemented. There was no statistically significant change in mean
interruption durations (2.4 minutes before implementation and 1.6 minutes
afterward, p > 0.10). If a typical interruption lasts 
2
minutes, this reduction in the arrival rate would reduce each radiologist's
interruptions by roughly 90 minutes per 40 hours of work or nearly 2 full
weeks per year.
Staff perceptions of workflow interruptions showed no significant changes at p = 0.05. On a 7-point scale (with smaller numbers indicating less frequent interruptions), mean values were 5.42 and 5.14 before and after implementation, respectively, for radiologists and 4.48 and 4.89 for technologists. These results offer modest support for our second hypothesis—that is, implementation of the WMS would reduce radiologist workflow interruptions.
Patient and Parent Perceptions
Patient perceptions of timeliness of service significantly increased
(p < 0.05), whereas patient and parent satisfaction and likelihood
of returning metrics showed no significant change
(Table 4). These results
support our third hypothesis—that is, implementation of the WMS would
improve patient satisfaction—only for the timeliness of service
metric.
|
Radiology Staff Perceptions
Radiologists' and technologists' perceptions of overall information system
quality, ease of use, and usefulness showed no significant (p = 0.05)
changes between pre- and postimplementation; radiologists' and technologists'
perceptions of psychologic stress, empowerment, and job satisfaction were
similarly unchanged (Table 5).
These results offer no support for our fourth hypothesis—that is,
implementation of the WMS would improve radiology staff satisfaction and
reduce job stress.
|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Process Flow
The reductions in the time durations involved in reading cases and signing
off cases are likely attributable to three primary effects of the WMS on
workflow. First, effort and time are saved by removing the manual task of
sorting, prioritizing, and distributing waiting requests that were involved
before moving to the paperless workflow that the WMS enabled. Before
implementation, radiologists were constantly sifting through various piles of
work requests and requisitions to make momentary triage decisions. The WMS
automated that task and provides radiologists with presorted on-screen work
lists accessible from any workstation. While this automation does not
constrain the radiologist from selecting any waiting patient in his or her
service, it does offer an efficient, effective presort to an otherwise chaotic
and constantly changing "pile" of waiting cases.
Second, radiologists were interrupted less frequently after the WMS was in place. Because results were transmitted more quickly and completely under the new WMS-facilitated workflow, referring clinicians received results more quickly and generated fewer expediting requests to the radiologists on service. This reduction in interruptions allows better concentration and results in faster readings. These faster readings then further reinforce the ability of the WMS (and operators) to convey results more quickly, which sets up a self-reinforcing positive feedback loop.
This phenomenon is the crux of why speech recognition alone is insufficient to reduce radiologist workflow interruptions. Although speech recognition enables radiologists to get reports into an electronic system rapidly, if those reports do not quickly get to the clinicians who ordered the studies, those clinicians are still likely to interrupt the radiologists to ask for study results. The mere presence of reports in an electronic system is not sufficient to decrease interruptions to radiologist workflow because the ultimate customer of those reports—the referring physician—has not yet been satisfied. This ability to close the radiology communication loop—getting a report back to the clinician who requested it—is a key to the success of this WMS.
The third improvement was the improvement in the department's ability to distribute, track, and verify radiologists' attention to waiting cases. By removing the added work of determining who is reading which case at any given time and eliminating the possibility that two different radiologists may separately spend time reading the same case, the WMS helps avoid valueless effort and focuses radiologist workflow on value-adding activities. For example, because radiologists can read from more than one service work list at a time, cross-coverage is facilitated and geographically separate satellite imaging centers can be serviced with no appreciable loss in efficiency. The WMS is able to reprioritize active physicians' work lists to include all covered services, brokering communications and accounting for different geographic locations and statuses of all patients in the system. Note that no changes were made to service assignments or reading policies to support the WMS; the WMS was designed to support existing workflows (through automation, documentation, and so on) rather than to alter them significantly.
Emergency department study reading times were not significantly affected by implementation of the WMS. This lack of change was perhaps due to the fact that these cases were already read fairly quickly (< 23 minutes) before implementation, offering minimal room for improvement.
Workflow Interruption
Introduction of the WMS decreased the frequency, but not the duration, of
interruptions to radiologists' workflow. This seems appropriate given that the
WMS should reduce the need for clinicians to call the radiology department
asking for results or requesting expedited readings (thus increasing the time
between interruptions), but the type of interruptions (and therefore, how long
they should require to be resolved) was not likely to be affected. Perceived
interruptions showed no significant change; the low statistical power of our
collected data may be partly responsible for that null finding. Because the
negative impact of interruptions on both the efficiency and the quality of
radiologist work is likely to be significant, this specific aspect of workflow
management certainly deserves further research.
Patient Perceptions
Outpatient perceptions of the radiology experience showed a significant
improvement regarding timeliness, but no significant change in perceived
satisfaction and likelihood of returning. We are heartened to see that the
significant reduction in case reading times, and improved communication
brokering (as discussed earlier) was perceived by patients and parents as an
improvement in the timeliness of radiology services at this facility. A lack
of change in the satisfaction metric is possibly due to the already high
quality of medical care provided at this institution, thus leaving little
opportunity for a back-office workflow improvement to affect it. Many patients
may feel they have little choice where to go for radiology services due to
referrers' specificity, so this metric may not be directly affected by
workflow improvements to any appreciable extent.
Staff Perceptions
Staff perceptions of departmental information systems and overall work
environment exhibited no statistically significant changes. When the WMS was
launched, department radiologists were generally unsatisfied with the quality
of the information systems in the department, in large part because we had
recently made several major changes over the prior several months, including
combining our RIS and PACS and then launching speech recognition with a 100%
usage requirement. This large number of changes gave us concern that the WMS,
a new and untried system, would be poorly perceived. Add to that the fact that
case loads had increased with no compensating changes in staffing levels, our
findings of no significant degradation in staff perceptions were actually
reassuring.
Critical Results Communication and Documentation
As implemented, the WMS helps to ensure that critical results are
appropriately conveyed and conveyances are permanently documented. It provides
feedback to radiologists so they know when results have been conveyed, to
whom, and by whom. The WMS provides a communication log separate from the RIS
that is permanently saved and accessible to users. On occasion, to address
questions in particular cases, audits of the log produce documentation of all
key critical results conveyance events.
Our approach is to use human operators to close the communication loop with referring clinicians. Humans are involved to ensure that inaccuracies in clinician contact information databases do not delay timely conveyance of critical results. The human interface also makes it easy for referring clinicians to reach interpreting radiologists if they have outstanding questions—they simply ask to be connected. If a radiologist wishes to discuss a case with a clinician, the WMS will notify an operator who then locates the clinician, connects the two caregivers, and documents that they have spoken in the communication log. Because the radiologist who generated the report is then in direct contact with the referring physician, we do not require that the radiologist produce additional documentation as to what was said on the telephone; it is implied that the content of the report was discussed. We believe it is important to maintain personal contact with our referring base, and this approach helps improve the quality of service we deliver.
Using a nonphysician human operator to read the final, signed radiologist's report has proved adequate. Receiving clinicians can request repetition and operators can fax reports to clinicians at their locations. The clinician acknowledges understanding and receipt of results during the conversation with the operator. This approach is in keeping with the Joint Commission's 2008 National Patient Safety Goals regarding critical results notification [10].
In conclusion, despite our deployment of an integrated RIS–PACS and speech-recognition system, critical workflow problems persisted. In an effort to streamline workflow and increase productivity and responsiveness and, as an extension, quality of care, we developed the WMS described here. The system as designed automatically triages cases, improves distribution of work, facilitates communication of results, and provides an audit trail of report conveyances.
Although many such efforts are validated a posteriori, such as through anecdote, we desired to test rigorously the actual impact of the WMS across a variety of operational and perceptual metrics. By measuring the production environment before and after implementation, we assessed its clinical impact and found that it decreased radiologist workflow interruptions and significantly decreased departmental turnaround times for all reports. However, these improvements did not come at the expense of the radiology work environment or patient satisfaction; rather, patients' perceptions of radiology service timeliness improved significantly after deployment.
In summary, this internally developed WMS significantly improves our ability to provide high-quality care to both our patients and our referral sources by allowing radiologists to read more cases, more quickly, and with fewer interruptions. Moreover, it supports hospital and Joint Commission objectives by facilitating communication of critical results and documentation of critical result conveyance more comprehensively than our previous system.
Acknowledgments
The authors acknowledge the contributions of Hong Yang, Laurie Perry, and
Neil Johnson in this work.
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
