Medical Physics and Informatics
April 25, 2013

The Future of the Radiology Information System


OBJECTIVE. Today in the hospital setting, several functions of the radiology information system (RIS), including order entry, patient registration, report repository, and the physician directory, have moved to enterprise electronic medical records. Some observers might conclude that the RIS is going away. In this article, we contend that because of the maturity of the RIS market compared with other areas of the health care enterprise, radiology has a unique opportunity to innovate.
CONCLUSION. While most of the hospital enterprise spends the next several years going through the digital transformation converting from paper to a digital format, radiology can leap ahead in its use of analytics and information technology. This article presents a summary of new RIS functions still maturing and open to innovation in the RIS market.
The future is here, it's just not evenly distributed yet.
—William Gibson
Radiology departments were among the first clinical departments in health care to implement electronic systems as part of their clinical workflow, with the first such systems to assist radiology reporting processes appearing as early as the mid-1960s [1]. The early systems were information islands used to manage the operations of radiology independently of the hospital. This included managing the patient identification database and the ordering physician database, as well as tracking the patient through the steps of acquiring the images and tracking report interpretation. The advancements in radiology informatics to date, including integration of PACS and the radiology information system (RIS) into department workflow, have done much to increase departmental efficiency. Multiple groups have highlighted the decreased number of steps within the standard workflow when using these systems compared with traditional film-and-paper–based systems; this has led to departmental efficiency (higher volume of studies being performed and interpreted), improved customer service (via faster report turnaround time and ready availability of images for clinician review), and decreased costs. The fundamental advantages of these systems lie in their ability to keep massive amounts of data (of all types, including images, demographic and clinical information, and billing and scheduling) readily accessible and to streamline work-flow via elimination of previously required steps, more efficient workflow management, and facilitation of rapid communication.


The first RISs were developed in the 1960s [1] and focused mainly on improving departmental and radiologist efficiency in two core problem areas—report coding as well as delivery of reports.
In the early to mid-1970s, RISs continued to evolve and become more reliable as new server technologies became available, incorporating more robust programming and database applications, with such tools as the Massachusetts General Hospital Utility Multi-Programming System. This new technology served well in departments‧ efforts to automate other functions, including initiatives to implement structured reporting methods to improve reporting efficiency, film jacket tracking, and improved report delivery via remote printing technology to distributed areas for better results communication.
In 1980, a group of university and private hospitals formed the Radiology Information Systems Consortium (RISC) to develop requirements for an improved RIS and to create a request for proposal for commercial entities to build this system. This request for proposal was eventually won and developed by Digital Equipment Corporation. RISC continued to guide the development of additional features of the RIS (the product now called DECrad) through the mid-1980s, working with Digital Equipment Corporation and helping to foster the DECrad users group, a national group of users formed with the purpose of enhancing clinical workflow, as well as systems and film management. The RISC later became the Society of Imaging Informatics in Medicine.
Beginning in the late 1990s and continuing into the early 2000s, as PACS technology became available and stable, radiology departments were again transformed, this time moving away from physical film and incorporating the new workflow of digital imaging for diagnostic interpretation. Digital dictation also became available in the early 2000s, allowing speech-to-text functionality, again improving diagnostic report turnaround.
The struggle to achieve improved efficiencies in radiology through the use of improved information system technology is a journey that has lasted over 40 years and will continue to evolve in radiology.

Areas of Innovation

We conducted a literature review and grouped advanced RIS functionality into eight categories. Advanced functionality was classified as not commonly available through commercial RIS vendors. This functionality was predominantly developed at academic medical center settings by in-house informatics groups. The categories chosen to classify the functionality are electronic medical record (EMR) aggregation, order entry decision support, advanced workflow, clinical decision support, digital dashboards, data mining, customer service, and, finally, surveillance and outcomes. We acknowledge that this categorization scheme is arbitrary. There is overlap between categories, and the nomenclature is vague. In our opinion, this is precisely because the technology is still evolving and will define itself as it matures. The available literature on radiology informatics (and, more broadly, health informatics) provides no formal conventions for classifying types of health information technology. Our categorization and nomenclature derives from common conventions in the current literature that we think are sufficiently descriptive.

Electronic Medical Record Aggregation

As radiologists and other clinicians become more dependent on information systems, it has become clear that improved access to the full EMR to ascertain pertinent clinical information can affect diagnosis and potentially improve patient care. Modern informatics systems can provide a number of benefits by linking disparate hospital information systems containing unique sources of data for a given patient, such as PACS for diagnostic images, RIS for examination scheduling and diagnostic reporting, and general hospital information systems for other clinical data. Unfortunately, radiologists are often provided images for interpretation without supporting clinical data other than that provided by the requesting provider on the examination request. These data, including patient history, are often brief, unstructured, and possibly inaccurate. Although this is not a new problem, increasing interest in patient-centered care, individualized medicine, and quality improvement should prompt calls for change. The incorporation of demographic and clinical data are vital for optimal modality and protocol selection, examination interpretation, and recommendations for further patient management.
The expansion of EMR utilization has allowed electronic storage of clinical data [15], yet utilization of this information in radiology has been limited. Radiologists often must use separate workstations, software, and login credentials to access the EMR. Practically, this often means moving to a different computer, performing an additional or separate login, and manually retrieving, reviewing, and analyzing patient data. To exacerbate this issue, a radiologist would need to navigate to multiple locations with an EMR, often entailing several minutes and several clicks, just to see whether there is pertinent information available.
Despite the negative effect on workflow, studies have shown that radiologists use the EMR frequently, with one study (in which the EMR was located on a remote workstation) showing EMR usage in as many as 73% of examination interpretations with certain modalities, accounting for 21% of the diagnostic effort (defined as time spent on image analysis) [6]. Our institution, like several other academic medical centers, needed to develop an in-house program, integrated into the worklist and dictation systems, that would automatically provide a summary of relevant EMR data on opening an examination. Prior imaging reports, clinical notes (including operative narratives, history and physicals, and discharge summaries), and laboratory and pathology data were provided in a simple user interface on one of the workstation's side monitors. The system was also integrated with the PACS, allowing retrieval of images for viewing (even from separate patients) without exiting the current examination.
An evolving challenge is presenting the vast and growing amount of data available on the EMR in an efficient way. Added functionalities, such as EMR indexing, structured searches, and automated searches, could further streamline workflow. Zalis and Harris [7] designed a programmable search system for the EMR that allowed time-of-service queries of patients‧ medical records. The system stored structured complex queries, filtered the EMR dataset to more specified subsets, and transmitted the output of the search to a readable form, such as a web browser or other software. Benefits of the system were highlighted by a sample project that included a query built for use before interventional procedures, ultimately providing the same search satisfaction and accuracy as a manual EMR search while reducing search time by a factor of 8. Other uses include automated EMR search for contraindications to certain procedures and automatic queries at the time of order entry to identify possibly unnecessary duplicate examinations [7]. Future work should continue to optimize the integration of clinical and radiographic information. For example, efforts are under way to embed images from RIS and other clinical imaging systems into the EMR, which should improve the accuracy and efficiency of patient management while increasing clinician and patient satisfaction via collaboration and information sharing [8].

Clinical Decision Support at the Time of Ordering and Interpretation

Integrated RIS-based decision support tools are available that may improve patient management and optimize resource utilization at several points along the workflow chain, including at order entry, during image interpretation, and when recommending further patient management. There is increased pressure to limit utilization of imaging to evidence-based applications, in an effort to rein in costs of health care in the United States [9]. At the same time, the effective use of medical imaging can save costs beyond radiology; for example, the appropriate use of CT in the emergency department has been shown to be cost effective by decreasing the rate of unnecessary operations [10]. Nevertheless, there are wide variations in clinician ordering behavior [11]. A lack of agreement on what is appropriate, difficulty in changing practice patterns, and increased types and complexity of imaging tests contribute to the problem.
“Just-in-time” interventions—that is, interventions delivered at the point of care—have been shown to significantly affect the safety and quality of patient management beyond radiology by reducing serious medical errors [12]. Computerized order entry systems with embedded imaging decision support have been developed in response and have strong data showing clinical acceptance and decreased utilization of low-yield examinations [9, 13, 14]. Most systems use web-based software programs that provide a choice of tests from structured menus linked to predefined indications (with varying levels of detail). Appropriateness criteria, such as that provided by the American College of Radiology [15], or direct evidence is provided at the time of order entry. The utility of the requested examination can be ranked and alternatives provided [9, 11, 13, 14]. In addition, some systems have integrated data collection functionality to track ordering behavior, providing another vehicle for intervention [11, 14].
Decision support can also improve quality at the time of image interpretation. Just-in-time learning tools are available, including web-based image search applications designed specifically for radiologists (e.g., Yottalook [16] and the American Roentgen Ray Society's Goldminer [17]), diagnostic decision support systems in web portal formats (e.g., STATdx [18]), and biomedical literature databases (e.g., PubMed [19]). In addition, the evidence in currently available literature can be codified into computer models for diagnostic and recommendation guidance. For example, decision support applications are available for mammography wherein image features and clinical data are inputted as variables in a computer model, and then a Bayesian network is used to provide post-test probabilities for various diagnoses to help guide further management [2022]. As with all nonessential informatics tools, seamless integration into the workflow is vital for usability and clinical acceptance.

Advanced Workflow

Peer Review

Peer review, in which redundant interpretations are rendered for a given study, is a commonly used method for performance assessment in radiology [23]. Integrating peer review into departmental workflow may be accomplished in a variety of ways; however, authors have emphasized the importance of simplicity and streamlined integration in order for any peer review system to be accepted and effective [24]. Modern informatics solutions can help accomplish these goals using several tools. The American College of Radiology's peer review system RADPEER [25], in which interpretations are scored on a scale from 1 to 4, is the leading method for peer review in the United States [26]. This program can be integrated into the RIS or PACS user interface for seamless use during clinical workflow. Alternatively, RIS-integrated software can be developed that allows users to tag prior imaging if problems are identified, or randomly selected cases can be reviewed on standalone software in which discrepancies are identified, classified, communicated, and further analyzed [27]. Kruskal et al. [28] found that a secure online peer review system promoted reporting of problematic cases and allowed the identification of performance trends.
In addition to routine performance assessment, peer review could also serve as a vital component for improved radiology resident training. Although this has traditionally been accomplished in real time with joint review of examinations by attending physicians and residents, this is not always feasible, especially with increasing volumes and spatial and temporal separations of attending physician and resident interpretations. When attending physicians remotely edit dictations, trainees are forced to manually access the RIS, search for reports, and compare finalized versions using only their memory of the preliminary interpretations. This is a considerable time burden, and, indeed, studies have shown that a significant number of residents forgo this process [29]. One proposed solution is software that can query the RIS; extract preliminary reports, finalized reports, and report metadata; store this information in an indexed database; and present a comparison view with changes highlighted through a web-based application. One training program that implemented this system found significant increases in the number of trainees who routinely reviewed their reports, from 46.2% before implementation to 80.8% afterward [29]. Such software solutions also afford the ability to store, track, and analyze a variety of other metrics obtainable from reports, such as relative value units, trends in dictation style, and so forth. Our institution has recently introduced a similar system for rapid review of trainee reports, which is integrated into a web-based portal that contains several of the other informatics applications reviewed in this article.

Critical Findings and Automated Reporting

Effective communication of imaging results is one of the key components of quality in radiology [30], with recent studies suggesting that communication problems are a significant source of overall radiology errors [31]. As a result, there is an increasing medicolegal emphasis on accurate and timely reporting of critical findings [32], and several regulatory bodies have highlighted the need for direct reporting of critical findings with subsequent documentation of the communication [33, 34]. Although direct synchronous communication with ordering physicians allows two-way interaction, immediate feedback, and confirmation of message receipt, it is also associated with a number of problems, not least of which is a significant interruption in workflow. A variety of solutions have been developed to streamline this process [3540]. The ideal solution will be specific to the nuances of a given institution, but, in general, should be integrated into existing systems, deliver messages at the point of care (i.e., just in time), provide options for the delivery method according to the specific situation, and provide automatic documentation in the patient record. Automated asynchronous message delivery in the form of e-mail or text-page can provide an extremely efficient solution; however, this cannot supplant direct communication in critical cases. One group used a workflow management system to automatically facilitate direct reporting and documentation of critical findings by nonradiologist personnel within the department, whereas another created an automated alert system for emergency department physicians that required acknowledgment of receipt for documentation purposes [37]. Although the communication of critical findings will likely remain a tangible component of routine workflow, ongoing and future informatics efforts should offer solutions to minimize interruptions.

Technologist Feedback

Technologists have always been and remain an integral part of any radiology department; they are as important as any other individual within the workflow chain for optimal efficiency and quality. Although the rise of digital radiology has improved workflow for technologists [41, 42], it has also led to a trend in decreased quality control [43], despite research showing the importance of continued quality analysis [44] and the release of quality-control guidelines by the American Association of Physicists in Medicine [43, 45]. There are tools built into most modern imaging acquisition or transmission machines that collect valuable quality metrics (e.g., exposure data, repeat examinations, fluoroscopy time, and sonographic, thermal, and mechanical indexes), yet informatics tools to aggregate, analyze, and present these data are sparse [46]. Furthermore, one of the most important quality-control measures, feedback from the interpreting radiologists, is increasingly limited in the new digital environment. Increased workflow demands and geographically remote image acquisition and interpretation have limited direct radiologist-technologist interactions [47].
Several informatics tools have been examined to address these problems, often borrowing from the principles presented elsewhere in this review. Nagy et al. [48] incorporated technologist-specific metrics into their departmental dashboard, such as overall image quality, number of quality control issues submitted by radiologists, and examination repeat rate. These data were used as a part of a systemwide approach to quality improvement according to the principles of business analytics (discussed later in this article).
Another group developed software specifically aimed at improving the training, coaching, and management of technologists [46]. Their system pulled data from the CT reader for analysis of departmentwide data and individual technologist performance (e.g., number of examinations performed, repeat rate, and common reasons for repeat examinations). Data were presented in a digital x-ray dashboard for fast review. This group was not only able to identify stark differences in individual performance (e.g., 80% of examinations were performed by 21% of the technologists), they also found that they could significantly reduce repeat examination rate simply by making specific systemic changes in workflow (e.g., positioning for scoliosis examinations) [46].
To improve direct communication, Nagy et al. [48] developed a web-based tool that enabled radiologists to quickly and efficiently document and provide feedback to technologists in specific cases, which resulted in the identification of systemic problems for root-cause analysis, improved technologist receptiveness to problems (with the ability to coach those technologists with high numbers of issues), and significantly higher submissions of quality-control issues (compared with prior paper-based methods). In addition, radiology turnaround times were improved after implementation [47].

Digital Dashboards

The complexity of radiology operations presents numerous challenges to departmental efficiency and quality. The overall theme of this review is to emphasize that a massive amount of available data is not being fully used, largely because of the multiplicity of data sources and a lack of convenient data procurement, analysis, and display that can lead to actionable intervention. The concept of a “digital dashboard” has been used in a variety of other complex systems to overcome these problems [49], and applications in radiology are becoming increasingly common.
The overall goal of a radiology dashboard, like dashboards in other systems, is to quickly present real-time data that can be used to facilitate operational corrections. This borrows from established principles in business intelligence and analytics, in which data analysis tools provide the means to make smaller evidence-based adjustments more frequently, providing increased transparency, fewer and smaller systemwide interruptions, and, theoretically, increased efficiency and quality [48]. The available applications are vast, including technologist feedback [46, 47], radiologist workflow management [5052], and overall departmental quality metrics [48, 49, 53]. The reader is referred to the reference material for specific details. There are certain design considerations, however, that should be applied in most circumstances. First, an efficient dashboard must be able to aggregate and store data from multiple systems; this is accomplished via extraction from the source systems and storage in a centralized data warehouse. Data should be indexed so that they are accessible across multiple platforms. Analytics are then performed on these data and displayed in web-based graphical user interfaces [48]. Morgan et al. [49] nicely summarize several factors that can increase the success of a dashboard: interface optimization, where there is concise presentation of only the relevant data at the appropriate time; context sensitivity, which should account for the variability in needs between different users; user customization, to improve user satisfaction and account for interindividual preferences; and workflow integration, because nonintegrated systems have been shown to have a poor impact on end behaviors [52, 54]. Perhaps most important, there must be a system in place to drive change on the basis of the information acquired, whether through real-time alert systems [50], individual self-assessment [51], or organized departmental meetings [48].
There are multiple published examples showing clinical acceptance of dashboard implementation and the identification of unexpected data, which can drive highly specified and efficient quality improvement programs. Improvement in multiple quality metrics following dashboard implementation have also been identified, including report turnaround time, outpatient waiting time, status order turnaround time, quality-control resolution time, and repeat examination rate [4648, 50, 52]. We anticipate that the use of digital dashboards will expand in radiology as it has in other fields, especially as demands for efficiency and quality continue to grow while our informatics infrastructure becomes increasingly complex.

Data Mining

Data mining is a broad field of computer science that can have multiple meanings in the context of radiology informatics. Generally speaking, the goal is to extract data from a source and transform that data into a form suitable for further use. Many of the concepts mentioned in this review contain some form of data mining, such as the use of digital dashboards, as already discussed. The concept has been applied to individual radiologist performance [5558], the optimization of radiology reports [59], ionizing radiation dose audits [60], technologist quality control [46, 47], and overall departmental quality metrics [48, 5052]. In this section, we will focus on retrieval of the end product: radiology text reports. Advances in digital acquisition and storage have resulted in huge archives of data. Like other large data sources, such as the World Wide Web, efficient methods of data extraction become increasingly important as data accumulate. Several tools have been developed that should improve the ability of the radiology community to use our own product for enhancement of real-time performance and teaching and research initiatives.
The overall design of most described report mining tools involves extraction of data from a source (generally, the RIS, but other data sources, such as the hospital information systems and PACS, could also be integrated) and storage in a separate database. The most efficient systems often take advantage of indexing, for example, by building a relational database with linked tables containing different subsets of information [61]. The database serves as the back end, while a query client, usually a simple graphical user interface in a web browser, serves as the front end. Boolean queries (AND, OR, NOT, and so forth) enhance the efficiency of searches. Output can be in the form of hyperlinks to reports matching the search query, with various options for data display (e.g., ordered by relevance or date) [61, 62].
A wide variety of freeware and commercial software is available to build such systems [61], which can be tailored to specific applications or institutional needs. Although the potential utility of these tools is exciting, the inherent information and capabilities contained within them necessitates strict HIPAA compliance and robust database security [61, 62]. Ongoing work in natural language processing (i.e., computer extraction of meaningful data from nonsystematic text reports), structured reporting, and radiology ontology (e.g., the RadLex lexicon) should improve the capabilities and expand the utilization of report mining software [6266].

Customer Service

Health care is part of the service sector; as such, customer relationships are a vital component of quality and success. Radiology, in particular, is service oriented, because the field has two major “customers”—patients and referring physicians. The recent digital revolution has provoked fears that the specialty is becoming increasingly marginalized and commoditized, and optimal service may be more important now than ever before. Informatics initiatives can improve customer relationships in several ways.
Easy scheduling, thorough information on how to prepare for an examination and what to expect, prompt access to results, and patient autonomy are factors that may improve radiology-patient relationships. Much of this can be accomplished via increasingly available web products providing data security, easy navigability, context-appropriate information, and direct communication with providers (e.g., online consultations) [38]. Growing emphasis on patient-centered care is also driving web-based communications solutions [37]; furthermore, these initiatives are supported by evidence that increased patient involvement in medical decision making positively impacts outcomes [6769].
The ubiquitous utilization of medical imaging in routine clinical practice necessitates that the needs and requests of referring physicians are also routinely considered. RIS-based personnel tracking systems, integrated into the hospital intranet and available to clinicians, have been shown to increase radiologist accessibility and save clinicians time when they have a question or need to find a specific radiologist [70]. Automated reporting systems can track message delivery and receipt [29] while decreasing results turnaround time and reliance on fixed computer terminals [3739]. Computerized order entry systems have been developed with add-ons designed specifically for referring providers, including risk-management tools (e.g., electronic notifications or reminders) and outcomes tracking [11]. Ongoing work on structured reporting may improve physician satisfaction with clarity, brevity, and clinical correlations, which are the three metrics most valued by both patients and clinicians [71]. Finally, informatics initiatives to increase image access, distribution [11], and integration with the EMR [8] are under way to address demands for more comprehensive information management.

Surveillance and Outcomes

Radiologists gain clinical experience with every case; however, feedback is necessary to optimize the learning and self-improvement process. This feedback often comes in the form of subsequent clinical data, such as operative reports, pathology reports, laboratory data, clinical visits, and follow-up imaging. Many systems are used to keep track of interesting or difficult cases, from notes scribbled on napkins to more sophisticated electronic-based tracking (e.g., an e-mail to oneself). Manual methods, electronic or not, introduce considerable inefficiency for those radiologists with dedicated follow-up habits. Today's digital environment provides an ideal medium for robust informatics solutions. At our institution, a teaching file integrated into the PACS provides a convenient and commonly used method for case tracking. A single click opens a web browser containing the selected image with automatically populated information. A brief description is added, and the case is saved in a database that is accessible at any later date via a web browser. Sharing and sorting of cases is also available. Alkasab et al. [72] described a similar program; however, their system possessed added functionality to further automate the acquisition of subsequent clinical data. The program, named “RaceTrack,” automatically searches the patient record and populates relevant clinical information (e.g., operative notes and pathology reports) into the database entry. In addition, there is a direct link from the web browser to the patient's EMR entry and to a web-based display of the study images [72]. Radiologists should seek and encourage future initiatives in this area, such as automated patient surveillance with electronic notification of predefined outcomes. In an increasingly competitive market, efficient feedback promoting lifelong skills improvement can provide one method to distinguish individual value.


For over 40 years, radiology departments have been early adopters of technology, incorporating the latest innovations from fields beyond health care to improve their clinical practices. Beginning with early server and database improvements and evolving through newer technologies and workflows, such as remote image distribution and teleradiology, radiology departments have been under constant pressure to both automate and improve their practices through the innovative use of information technology. Radiology continues to be one of the most technology-heavy clinical endeavors, potentially serving as a key proving ground for information technology specialists looking to improve quality, efficiency, and patient care through improved access to relevant clinical data and innovative software tools. Radiologists themselves may spend more time than any other physician specialty directly interacting with computer systems as they provide patient care, and they are well positioned to lead in the transformation of medicine through electronic health records that is currently under way. We think that the practice of radiology will continue to evolve with innovative technologies and that there are several promising opportunity areas for those looking to improve patient care both within radiology and throughout the health care system.


Ash JS, Bates DW. Factors and forces affecting EHR system adoption: report of a 2004 ACMI discussion. J Am Med Inform Assoc 2005; 12:8–12
Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003; 348:2526–2534
Berner ES, Detmer DE, Simborg D. Will the wave finally break? A brief view of the adoption of electronic medical records in the United States. J Am Med Inform Assoc 2005; 12:3–7
Jha AK, DesRoches CM, Campbell EG, et al. Use of electronic health records in U.S. hospitals. N Engl J Med 2009; 360:1628–1638
Middleton B, Hammond WE, Brennan PF, Cooper GF. Accelerating U.S. EHR adoption: how to get there from here: recommendations based on the 2004 ACMI retreat. J Am Med Inform Assoc 2005; 12:13–19
Lin A, Harris M, Zalis M. Initial observations of electronic medical record usage during CT and MRI interpretation: frequency of use and impact on workflow. AJR 2010; 195:188–193
Zalis M, Harris M. Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol 2010; 7:625–633
Ratib O, Swiernik M, McCoy JM. From PACS to integrated EMR. Comput Med Imaging Graph 2003; 27:207–215
Blackmore CC, Mecklenburg RS, Kaplan GS. Effectiveness of clinical decision support in controlling inappropriate imaging. J Am Coll Radiol 2011; 8:19–25
Rao PM, Rhea JT, Novelline RA, Mostafavi AA, McCabe CJ. Effect of computed tomography of the appendix on treatment of patients and use of hospital resources. N Engl J Med 1998; 338:141–146
Khorasani R. Computerized physician order entry and decision support: improving the quality of care. RadioGraphics 2001; 21:1015–1018
Khajouei R, Jaspers MW. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010; 49:3–19
Ip IK, Schneider LI, Hanson R, et al. Adoption and meaningful use of computerized physician order entry with an integrated clinical decision support system for radiology: ten-year analysis in an urban teaching hospital. J Am Coll Radiol 2012; 9:129–136
Rosenthal DI, Weilburg JB, Schultz T, et al. Radiology order entry with decision support: initial clinical experience. J Am Coll Radiol 2006; 3:799–806
American College of Radiology. ACR appropriateness criteria. American College of Radiology website. Updated June 2012. Accessed July 7, 2012
Yottalook. Yottalook website. Accessed July 7, 2012
American Roentgen Ray Society. ARRS Gold-miner. American Roentgen Ray Society website. Accessed July 7, 2012
Amirsys. STATdx. STATdx website. Accessed January 8, 2013
National Center for Biotechnology Information, U.S. National Library of Medicine. PubMed. NCBI website. Accessed January 8, 2013
Burnside E, Rubin D, Shachter R. A Bayesian network for mammography. In: Proceedings of the American Medical Informatics Association Annual Symposium. Los Angeles, CA: AMIA, 2000:106–110
Burnside ES, Rubin DL, Fine JP, Shachter RD, Sisney GA, Leung WK. Bayesian network to predict breast cancer risk of mammographic micro-calcifications and reduce number of benign biopsy results: initial experience. Radiology 2006; 240:666–673
Burnside ES, Rubin DL, Shachter RD. Using a Bayesian network to predict the probability and type of breast cancer represented by microcalcifications on mammography. Stud Health Technol Inform 2004; 107:13–17
Donnelly LF. Performance-based assessment of radiology practitioners: promoting improvement in accordance with the 2007 Joint Commission standards. J Am Coll Radiol 2007; 4:699–703
Mahgerefteh S, Kruskal JB, Yam CS, Blachar A, Sosna J. Peer review in diagnostic radiology: current state and a vision for the future. RadioGraphics 2009; 29:1221–1231
Jackson VP, Cushing T, Abujudeh HH, et al. RADPEER scoring white paper. J Am Coll Radiol 2009; 6:21–25
Weinreb JC. In: ACR annual conference on quality and safety. Scottsdale, AZ: American College of Radiology, 2012
Prevedello L, Khorasani R. Enhancing quality assurance and quality control programs: IT tools can help. J Am Coll Radiol 2009; 6:888–889
Kruskal JB, Yam CS, Sosna J, Hallett DT, Milliman YJ, Kressel HY. Implementation of online radiology quality assurance reporting system for performance improvement: initial evaluation. Radiology 2006; 241:518–527
Sharpe RE, Jr, Surrey D, Gorniak RJ, Nazarian L, Rao VM, Flanders AE. Radiology Report Comparator: a novel method to augment resident education. J Digit Imaging 2012; 25:330–336
Rubin DL. Informatics in radiology: measuring and improving quality in radiology: meeting the challenge with informatics. RadioGraphics 2011; 31:1511–1527
Renfrew DL, Franken EA Jr, Berbaum KS, Weigelt FH, Abu-Yousef MM. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992; 183:145–150
Altman DJ, Gunderman RB. Outsourcing: a primer for radiologists. J Am Coll Radiol 2008; 5:893–899
Institute of Medicine. Improving the 21st century health care system. In: Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press, 2001:39–60
The Joint Commission. Improve the effectiveness of communication among caregivers. In: The Joint Commission 2013 national patient safety goals. The Joint Commission website. Published 2013. Accessed January 24, 2013
Eisenberg RL, Yamada K, Yam CS, Spirn PW, Kruskal JB. Electronic messaging system for communicating important, but nonemergent, abnormal imaging results. Radiology 2010; 257:724–731
Juluru K, Eng J. Internet-based radiology order-entry, reporting, and workflow management system for coordinating urgent study requests during off-hours. AJR 2005; 184:1017–1020
Halsted MJ, Froehle CM. Design, implementation, and assessment of a radiology workflow management system. AJR 2008; 191:321–327
Johnson AJ, Hawkins H, Applegate KE. Web-based results distribution: new channels of communication from radiologists to patients. J Am Coll Radiol 2005; 2:168–173
Mates J, Branstetter BF, Morgan MB, Lionetti DM, Chang PJ. “Wet Reads” in the age of PACS: technical and workflow considerations for a preliminary report system. J Digit Imaging 2007; 20:296–306
Rubin DL. Informatics methods to enable patient-centered radiology. Acad Radiol 2009; 16:524–534
Reiner B, Siegel E, Carrino JA, McElveny C. SCAR Radiologic Technologist Survey: analysis of technologist workforce and staffing. J Digit Imaging 2002; 15:121–131
Reiner BI, Siegel EL, Carrino JA, Goldburgh MM. SCAR Radiologic Technologist Survey: analysis of the impact of digital technologies on productivity. J Digit Imaging 2002; 15:132–140
Honea R, Elissa Blado M, Ma Y. Is reject analysis necessary after converting to computed radiography? J Digit Imaging 2002; 15(suppl 1):41–52
Hammerstrom K, Aldrich J, Alves L, Ho A. Recognition and prevention of computed radiography image artifacts. J Digit Imaging 2006; 19:226–239
American Association of Physicists in Medicine. Quality control in diagnostic radiology: report of task group #12—Diagnostic X-ray-Imaging Committee. Madison, WI: Medical Physics Publishing, 2002
Minnigh TR, Gallet J. Maintaining quality control using a radiological digital X-ray dashboard. J Digit Imaging 2009; 22:84–88
Nagy PG, Pierce B, Otto M, Safdar NM. Quality control management and communication between radiologists and technologists. J Am Coll Radiol 2008; 5:759–765
Nagy PG, Warnock MJ, Daly M, Toland C, Meenan CD, Mezrich RS. Informatics in radiology: automated Web-based graphical dashboard for radiology operational business intelligence. Radio-Graphics 2009; 29:1897–1906
Morgan MB, Branstetter BF 4th, Mates J, Chang PJ. Flying blind: using a digital dashboard to navigate a complex PACS environment. J Digit Imaging 2006; 19:69–75
Awan OA, van Wagenberg F, Daly M, Safdar N, Nagy P. Tracking delays in report availability caused by incorrect exam status with web-based issue tracking: a quality initiative. J Digit Imaging 2011; 24:300–307
Chen R, Mongkolwat P, Channin DS. RadMonitor: radiology operations data mining in real time. J Digit Imaging 2008; 21:257–268
Morgan MB, Branstetter BF 4th, Lionetti DM, Richardson JS, Chang PJ. The radiology digital dashboard: effects on report turnaround time. J Digit Imaging 2008; 21:50–58
Seltzer SE, Kelly P, Deibel GM, Ros P. Radiology quality and performance metrics on the Web: a management information and communications tool. Acad Radiol 2000; 7:981–985
Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996; 156:1551–1556
Reiner B. New strategies for medical data mining. Part 1. Dynamic and performance-based reimbursement. J Am Coll Radiol 2010; 7:975–979
Reiner B. New strategies for medical data mining. Part 3. Automated workflow analysis and optimization. J Am Coll Radiol 2011; 8:132–138
Reiner B. New strategies for medical data mining. Part 2. The customizable productivity pacer. J Am Coll Radiol 2011; 8:33–38
Reiner BI, Siegel EL. The clinical imperative of medical imaging informatics. J Digit Imaging 2009; 22:345–347
Reiner B. Uncovering and improving upon the inherent deficiencies of radiology reporting through data mining. J Digit Imaging 2010; 23:109–118
Wilde R, Charnock P, McDonald S, Moores BM. Qualifying the use of RIS data for patient dose by comparison with DICOM header data. Radiat Prot Dosimetry 2011; 147:329–332
Desjardins B, Hamilton RC. A practical approach for inexpensive searches of radiology report databases. Acad Radiol 2007; 14:749–756
Erinjeri JP, Picus D, Prior FW, Rubin DA, Koppel P. Development of a Google-based search engine for data mining radiology reports. J Digit Imaging 2009; 22:348–356
Langlotz CP. RadLex: a new method for indexing online educational materials. RadioGraphics 2006; 26:1595–1597
Lacson R, Khorasani R. Practical examples of natural language processing in radiology. J Am Coll Radiol 2011; 8:872–874
Rubin DL. Creating and curating a terminology for radiology: ontology modeling and analysis. J Digit Imaging 2008; 21:355–362
Safdar NM, Siegel E, Erickson BJ, Nagy P. Enabling comparative effectiveness research with informatics: show me the data! Acad Radiol 2011; 18:1072–1076
Greenfield S, Kaplan S, Ware JE Jr. Expanding patient involvement in care: effects on patient outcomes. Ann Intern Med 1985; 102:520–528
Wagner EH, Glasgow RE, Davis C, et al. Quality improvement in chronic illness care: a collaborative approach. Jt Comm J Qual Improv 2001; 27:63–80
Wagner EH, Grothaus LC, Sandhu N, et al. Chronic care clinics for diabetes in primary care: a system-wide randomized trial. Diabetes Care 2001; 24:695–700
Rumball-Smith A, MacDonald S. Development and utilisation of a real-time display of logged in radiology information system users. J Digit Imaging 2011; 24:295–299
Lafortune M, Breton G, Baudouin JL. The radiological report: what is useful for the referring physician? Can Assoc Radiol J 1988; 39:140–143
Alkasab TK, Harris MA, Zalis ME, Dreyer KJ, Rosenthal DI. A case tracking system with electronic medical record integration to automate outcome tracking for radiologists. J Digit Imaging 2010; 23:658–665


Unique customized medical search engine service from ARRS! ARRS GoldMiner® is a keyword- and concept-driven search engine that provides instant access to radiologic images published in peer-reviewed journals. For more information, visit

Information & Authors


Published In

American Journal of Roentgenology
Pages: 1064 - 1070
PubMed: 23617491


Submitted: November 17, 2012
Accepted: November 20, 2012


  1. clinical informatics
  2. imaging informatics
  3. informatics
  4. innovation
  5. radiology informatics



John W. Nance Jr
Department of Radiology, Johns Hopkins University School of Medicine, Baltimore, MD.
Christopher Meenan
Department of Radiology, University of Maryland School of Medicine, 22 S Greene St, Baltimore, MD 21201.
Paul G. Nagy
Department of Radiology, Johns Hopkins University School of Medicine, Baltimore, MD.


Address correspondence to P. G. Nagy ([email protected] or [email protected]).

Metrics & Citations



Export Citations

To download the citation to this article, select your reference manager software.

Articles citing this article







Copy the content Link

Share on social media