Performance	Choose Top of pageABSTRACTPerformance <<TimeWork FlowCostConclusionAcknowledgmentReferences

The most important condition that AI must satisfy is to improve the performance of mammography. Without this achievement, there is no utility in the technology for interpretation purposes. The sensitivity of currently available CAD is acceptable. The major drawback, however, is the high rate of false-positive marks, also known as flags. False-positive flags can distract the interpreting radiologist with too much noise and lead to unnecessary workups and biopsies. Increases in recall rates and false-positive examination results of screening may be partially attributed to the false-positive flags of current CAD technology. One report in the literature [2] describes significantly decreased specificity of radiologists with CAD, along with an overall increase in false-positive recalls. If AI earns a place in breast imaging, its best opportunity is to decrease false-positive flags compared with the number associated with currently available CAD programs. There are limited research data, but so far AI programs analyzing mammograms have exhibited performance equal to that of radiologists in assigning breast density [3]. In the recent Dialogue for Reverse Engineering Assessments (DREAM) challenge that ended in May 2017, a competition of AI algorithms to detect breast cancer in a test set of mammograms, the winning algorithm achieved an ROC AUC of 0.87 in the final test, which indicates good generalization ability.

Time	Choose Top of pageABSTRACTPerformanceTime <<Work FlowCostConclusionAcknowledgmentReferences

A second desirable feature for making AI successful within diagnostic radiology is that it must not unnecessarily increase interpretation time. Ideally, AI CAD would take no more time for the radiologist to evaluate than currently available CAD, which is estimated to increase interpretation time approximately 20%. If AI CAD contains fewer flags to review, interpretation time will also be decreased. The previously described concept of a quick negative examination may become a reality if the negative predictive value of AI CAD can approach 100% [4]. A quick negative examination is one which an AI algorithm identifies a negative finding with 100% certainty, generates an appropriate report, and closes the case without a radiologist's ever looking at the examination. If even a small percentage of cases can be identified as negative with 100% certainty, a substantial amount of valuable physician time can be directed at more complicated cases and other activities requiring human intervention.

In general, for successful clinical implementation of AI CAD, the AI CAD algorithm must not delay the delivery of images to radiologists. This is particularly relevant, for example, for radiology studies performed in the emergency department, where quick and timely intervention is often paramount. In contrast, the speed of imaging analysis of AI algorithms will likely not be a rate-determining issue in breast imaging because most screening examinations are interpreted off-line and not in real time, facilitating the successful clinical implementation of AI CAD in breast cancer screening.

Work Flow	Choose Top of pageABSTRACTPerformanceTimeWork Flow <<CostConclusionAcknowledgmentReferences

A third feature that must be present is seamless work flow integration. The AI effort should start before image acquisition by combing the medical record to identify patients in need of appropriate imaging. The ordering process and scheduling may then be initiated automatically on behalf of the referring physician. The radiologist's review of the AI image analysis should be completely integrated into the postacquisition process. There should not be separate workstations or monitors. An ideal format would be similar to current CAD, in which AI information can be toggled on and off in an image overlay manner. There will likely be many more vendors of AI software than hardware given their lower barriers to market entry. This means that AI algorithms must be compatible and fully integrated with multiple manufacturers' hardware and software systems and be functional across varying platforms.

A potential application of AI to improve both work flow and patient care is the concept of rereading, whereby the algorithm begins to review archived recent prior examinations not subjected to initial AI CAD evaluation. Any cases flagged would be automatically moved into a queue to be reviewed by a radiologist [4]. If even one cancer per thousand cases were to be detected, this process would positively affect the cost-benefit ratio.

Cost	Choose Top of pageABSTRACTPerformanceTimeWork FlowCost <<ConclusionAcknowledgmentReferences

The cost to provide AI interpretive assistance must not be so high that it tilts the value equation against its use. Currently, CAD is included as part of a bundled charge for both screening and diagnostic mammograms rather than an individual separate charge. Introducing a new code for billing is unlikely. Increasing speed and availability of high-performance computers along with newly minted ranks of computer scientists should lower barriers to entry into AI development. Over time, this should lower prices. In addition, with the open access initiatives of groups such as the IBM DREAM challenge, the top-performing software will be released to everyone free of charge and open to upgrades from anyone. This would obviate expensive licensing fees and purchase agreements. Given that false-positive mammograms are estimated to cost the health care system approximately $4 billion each year, a decrease in false-positive CAD flags has strong potential to lower the overall cost of breast imaging [5].

Conclusion	Choose Top of pageABSTRACTPerformanceTimeWork FlowCostConclusion <<AcknowledgmentReferences

If developers can achieve the conditions we outline, AI will have an attractive cost-benefit ratio in women's imaging. The most likely starting point for AI in women's imaging will be mammography. The intent of this article is to frame radiology's approach to the benefit side of the cost-benefit ratio. We anticipate a flood of data on the impact of AI in the realm of image interpretation in the near future. Specifically, data related to sensitivity, positive predictive value, turnaround time, and prices are needed to conduct a valid cost-benefit analysis. Without many validated commercially available products and reimbursement figures, the process for calculating the cost side of the ratio will have to occur over time after data are collected from clinical practice and rigorously evaluated. As those data become available, we anticipate the emergence of a number of models to quantify the cost-benefit ratio of AI. The framework for such a model may be similar to that developed by Lee et al. [6] for investigating the cost-effectiveness of tomosyn-thesis for screening dense breast tissue. Our intent is that this article sets in motion the process of collectively crowd-sourcing both ideas for the use of AI in general and specific projects for collecting the data needed.