Clinical Perspective
Cardiopulmonary Imaging
May 23, 2019

Tackling the Radiological Society of North America Pneumonia Detection Challenge

Abstract

OBJECTIVE. We provide overviews of deep learning approaches used by two top-placing teams for the 2018 Radiological Society of North America (RSNA) Pneumonia Detection Challenge.
CONCLUSION. Practical applications of deep learning techniques, as well as insights into the annotation of the data, were keys to success in accurately detecting pneumonia on chest radiographs for the competition.
Deep learning, a class of machine learning techniques using multilayered neural networks [1], has attracted significant attention in radiology for its applicability to solving a variety of clinical imaging problems. However, clinical deep learning articles typically feature a limited set of techniques and neural network architectures as a proof-of-concept solution for a given problem. As a result, the optimal approach for a particular task may be left as an open question or a topic for future research.
In this context, machine learning competitions offer an approach toward clarifying the state of the art for solving a particular problem. Competitions allow direct quantitative performance comparisons among a variety of problem-solving approaches. The rise in popularity of deep learning in computer vision can be attributed to the 2012 ImageNet Large Scale Visual Recognition Challenge [2], a competition involving classification of thousands of color photographs into image categories. The winning solution helped popularize the use of deep convolutional neural networks for state-of-the-art image classification [3].
In 2018, the Radiological Society of North America (RSNA) organized an international machine learning challenge about detecting and localizing pneumonia in chest radio-graphs [4]. The challenge used images from a publicly available chest x-ray dataset from the National Institutes of Health [5] with annotations made by radiologists [6]. The challenge was hosted on a platform provided by Kaggle, Inc., which also provided $30,000 in prize money for the competition.
More than 1400 teams participated in the preliminary stage of the challenge and 346 teams in the final stage. This article describes the approaches of the first-place and third-place teams, whose members have medical backgrounds. The narratives illustrate the iterative process of incremental performance improvement typical of machine learning optimization. A glossary of technical terms is provided in Table 1.
TABLE 1: Glossary of Technical Terms
TermExplanation
Bounding boxA rectangle on an image specifying the location of an object (e.g., pneumonia). In the competition it is described by four numbers: x-coordinate, y-coordinate, width, and height.
ClassificationThe task of assigning to an image a label from a fixed set of categories (e.g., lung opacity, no lung opacity).
Convolutional neural networkA type of hierarchic multilayered neural network well suited for processing images. Filters encoding spatial patterns are applied (convolved) across the image to detect image features; successive layers learn increasingly complex features.
Deformable convolutional networkA modification of convolutional neural networks where the typical square receptive fields of the convolutional filters can be warped by learned pixel offsets [9]. This may improve detection of objects of different scales or shapes.
DetectionSee object detection.
EpochOne complete pass through the training set during training of a machine learning model.
FoldsSubsets of a dataset of approximately equal size.
Ground truthThe labels used to assess the performance of a machine learning system. In the competition, the ground truth consists of pneumonia bounding boxes derived from human expert annotations.
HyperparameterA configuration setting that is not learned during training, often tuned to optimize the learning process or model performance. Examples include learning rate and batch size (i.e., the number of images processed before each parameter update).
ImageNetA large annotated image database of more than 14 million images used to train and test computer vision systems [2, 13, 22]. Models pretrained on ImageNet images are often used for other computer vision tasks via transfer learning.
Intersection over Union (IoU)For object detection, a measure of how closely a predicted bounding box matches a ground truth bounding box. The area in common is divided by the total area encompassed by both boxes.
MetadataData that describe other data. For instance, chest radiograph metadata include the patient's age, sex, and positioning (anteroposterior or posteroanterior).
Nonmaximum suppressionPostprocessing technique used to limit one bounding box per detected object. Among a group of bounding boxes overlapping by a prespecified amount, only the highest scoring bounding box is retained.
Object detectionThe task of identifying the location of an object in an image, most often using a bounding box.
OverfittingThe process whereby a model learns to fit training data so well that it generalizes poorly to new data (e.g., in a separate test set).
Relation networkNeural network containing modules that reason about relations among objects. Relation networks have been proposed to improve object detection [16].
Test setA dataset completely separated from the training set and validation set that is used to evaluate final model performance. In research, the test set is used for evaluation purposes when all learning is completely finished. In competitions, the labels of the test set are withheld by the organizers and used to evaluate performance of participants' models.
Training setLabeled data used by a machine learning model to learn the parameter values that best predict the labels from the data. In a neural network, the learned parameters are the weights of connections between nodes.
Transfer learningThe process by which a model pretrained on a given task can be fine-tuned to perform well on a related task. This strategy is particularly advantageous when the volume of data available for the original task is much larger than for the new task.
Validation setLabeled data separate from the training set that are used to estimate how well a model has been trained to select the best performing model and training approach.

The Challenge

The full details of the RSNA Pneumonia Detection Challenge are provided on the Kaggle competition website [7]. The competition was a two-stage challenge that began with the release of a training set of 25,684 radiographs and a test set of 1000 radiographs; all radiographs were released in an anonymized DICOM format at 1024 × 1024 pixels resolution and 8-bit depth. The training images were labeled by nonthoracic radiologists from RSNA, whereas the test images were labeled by a panel of subspecialty-trained radiologists from the Society of Thoracic Radiology. The labels of the test set were withheld. Within the training set, 5659 were labeled positive for “lung opacity,” 8525 images were labeled “normal,” and 11,500 images were labeled as being neither normal nor having a lung opacity (“no lung opacity/not normal”). Each site of pneumonia was provided as a bounding box represented by a set of four numbers to describe the localization. A positive image could include from one to four different bounding boxes.
During stage 1 of the competition from August 27, 2018, to October 24, 2018, teams were allowed up to five bounding box prediction submissions per day for the 1000 test images. Each prediction set was scored using an average precision metric calculated at several thresholds of intersection over union (IoU) for the predicted bounding boxes (Figs. 1 and 2). The IoU metric measures both classification (correctly identifying the presence of pneumonia in a chest radiograph) and detection (precisely localizing the opacity). Teams were ranked on a public leaderboard according to each team's highest scoring submission. To continue to stage 2 of the competition, teams were required to upload their best model before the end of stage 1.
Fig. 1 —Graphic shows intersection over union (IoU) as measure of overlap between predicted and ground truth bounding boxes. See Figure 2.
Fig. 2A —Example of Radiological Society of North America scoring metric for detecting pneumonia on single image.
A, Chest radiograph of 55-year-old woman shows two ground truth bounding boxes (solid boxes) and two predicted bounding boxes (dashed boxes). IoU = intersection over union.
Fig. 2B —Example of Radiological Society of North America scoring metric for detecting pneumonia on single image.
B, Image shows calculation of average precision for single image in A based on illustrated bounding boxes. Average precisions for all images in test set are averaged to obtain test set score. TP(t), FP(t), and FN(t) represent number of true-positives, false-positives, and false-negatives, respectively, for threshold t. IoU = intersection over union, TP = true-positive, FP = false-positive, FN = false-negative.
In stage 2 of the competition, a new test set of 3000 images was released, and teams used their uploaded models to submit bounding box predictions for the new test set. The final rankings in the competition (Table 2) were based exclusively on the test set scores in stage 2.
TABLE 2: Final Leaderboard Ranking
RankTeam NameRSNA Metric Score
1Ian Pan and Alexandre Cadrin0.25475
2Dmytro Poplavskiy [ods.ai]0.24781
3Phillip Cheng0.23908
416bit.ai / layer60.23901
5JRS_HP0.23807
6PFNeumonia0.23478
7DEEPRADIOLOGY0.23089
8Mu Song0.22801
9DASA-FIDI-IARA0.22497
10Dancing Bears0.22456
Mean of 346 teams 0.14124

Note—Adapted with permission from [26] (RSNA Pneumonia Detection Challenge: private leaderboard. Kaggle website. www.kaggle.com/c/rsna-pneumonia-detection-challenge/leaderboard. Accessed March 2019). Teams ranked 9 and 10 differ from the listing in [26], because the original ninth place winning team declined their prize. RSNA = Radiological Society of North America.

First-Place Team: Ian Pan and Alexandre Cadrin-Chênevert

Forming the Team

Both members of this team participated in previous machine learning challenges applied to medical imaging including the inaugural 2017 RSNA Pediatric Bone Age Machine Learning Challenge [8]. To incorporate different algorithmic approaches, the team was formed during the competition after 4 weeks of individual and stand-alone work. This sequential strategy favors independent development of complementary solutions before team formation.

Framing the Problem

We first focused on how we wanted to frame the problem from a high-level perspective. We settled on a two-step classification-detection approach. This involved training two types of models: classifiers that would label each image as positive or negative for opacity and object detectors that would predict bounding boxes around the opacity. To allow precise spatial localization of lung opacities, detector models are frequently inferior in raw image classification compared with image classifiers. Consequently, we thought that combining these two approaches would be synergistic and would improve the competition metric, which incorporated both classification and detection performance.

Understanding the Data

Understanding the dataset is, in our view, the most important part of a data science competition. Initial visualization of the data allows a better representation of the specific characteristics of a dataset. In this competition, images were provided along with metadata including patient age and sex and radiographic view. We experimented with integrating this metadata but found no benefit. One of the keys to our success was understanding the annotation process. Although the images in the training set were labeled by a single radiologist, images in the two test sets were labeled by three radiologists; the final bounding box label was created by taking the intersection of the radiologists' bounding boxes [6]. We decreased each edge of the bounding boxes produced by our model by 12.5%, which resulted in significant improvement on the test set. Furthermore, the two test sets were labeled by subspecialty-trained thoracic radiologists, whereas the training set was labeled by a general radiologist. We hypothesized that the thoracic radiologists had higher sensitivity in detecting opacities; as a result, we used a lower threshold than indicated on the training set to label images and boxes as opacities, which also conferred significant performance improvement.

Model Architectures

All model architectures were based on deep convolutional neural networks, which are the main components to produce state-of-the-art results in various computer vision and image analysis tasks. Application of successive learnable image filters (i.e., convolution kernels) allows the progressive transformation of spatial to semantic information. Each model architecture has a capacity to learn complex representations of the data to solve a specific task. Deeper networks with multiple hierarchic layers usually have a higher representational capacity. Deformable convolutions [9] are a specialized type of convolution offering even higher representational capacity by learning a different position, or offset, for each element of the convolutional kernel (Fig. 3). By training different model architectures, we were able to obtain— at almost negligible computational cost—multiple model readings similar to multiple human readings of the same examination.
Fig. 3A —Illustrations show standard and deformable convolutional filters.
A, Standard convolutional filters (A) are square, whereas deformable convolutional filters (B) can be offset by variable distance to better capture natural image features. Optimal offset is parameter that is learned by network during training and can differ among individual filters. B is adapted with permission. (© 2017 IEEE. Adapted, with permission, from Dai J, Qi H, Xiong Y, et al. Deformable convolutional networks. 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017:764–773)
Fig. 3B —Illustrations show standard and deformable convolutional filters.
B, Standard convolutional filters (A) are square, whereas deformable convolutional filters (B) can be offset by variable distance to better capture natural image features. Optimal offset is parameter that is learned by network during training and can differ among individual filters. B is adapted with permission. (© 2017 IEEE. Adapted, with permission, from Dai J, Qi H, Xiong Y, et al. Deformable convolutional networks. 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017:764–773)

Selecting and Training Models

Our approach emphasized diversifying the model portfolio. For classification, we used an ensemble of InceptionResNetV2 [10], Xception [11], and DenseNet169 [12] architectures, which were pretrained on images from the ImageNet challenge [2, 13]. We also trained models using different outcomes. One set of models was trained to predict two classes: opacity versus no opacity. Another set was trained to predict three classes: lung opacity, no lung opacity/not normal, normal. To add additional heterogeneity, we also trained models on re-sized square images at resolutions of 256, 320, 384, 448, and 512 pixels. For object detection, we used an ensemble of RetinaNet [14], deformable region-based fully convolutional network (R-FCN) [9, 15], and deformable relation network [16] architectures. We trained on re-sized square images at resolutions from 256, 288, 320, 352, 384, 416, 448, 480, and 512 pixels. We found that training on smaller images did not decrease performance but resulted in faster model training, thus allowing us to run more experiments. We divided the data into 10 folds stratified by metadata and class, and each model was trained on a different fold.
For the classification networks and RetinaNet, we stopped training when model performance on the validation fold plateaued. For the two deformable architectures (R-FCN, deformable relation network), we used the implementation's default hyperparameters. Deformable architectures were trained only on images with opacity bounding boxes; images without labeled bounding boxes were ignored. For the RetinaNet architecture, we adopted two separate training processes. The first process involved training on only positive images. The second process involved training on concatenated images, where one positive image was attached adjacent to a negative image (Fig. 4). This allowed the model to better learn how to reduce false-positive findings. Because most of our object detection models were trained only on positive images, we relied heavily on the classification networks to reduce false-positives. Ultimately, this resulted in 60 separate models: 10 for classification and 50 for object detection.
Fig. 4 —Concatenated images used by first-place team as input to RetinaNet object detection model. This improved single-architecture performance by forcing model to distinguish between true pneumonialike lung opacities and false-positives within same image. Images show ground truth bounding boxes (solid boxes) for lung opacities in positive training example.

Combining Models for Final Predictions

Our final predictions integrated output from the classifiers and the object detectors. First, the classification networks each made a prediction on an image. These predictions were averaged into a single classification score. Object detectors were run independently on each image as well and produced bounding boxes with an associated prediction score. To combine all of the predictions, we developed a complex ensembling process that was a product of many experiments and fine-tuning (Fig. 5). Our code for this challenge is publicly available online [17].
Fig. 5 —Diagram shows ensembling method used by first-place team. In total, 50 object detection models across three different architectures and 10 different image sizes were used. Additional 10 classification models were incorporated to exclude box predictions from images with low predicted probabilities of lung opacities. DR-FCN = deformable region-based fully convolutional network, DRelNet = deformable relation networks, ConcatRetina = RetinaNet trained on concatenated images. Final image shows two predicted bounding boxes (solid boxes) for lung opacities.

Third-Place Team: Phillip M. Cheng

The RSNA Pneumonia Detection Challenge was my first machine learning competition, and I spent several days at the beginning learning about image object detection. The most useful introductory resources were lecture videos by Andrew Ng [18] and Jeremy Howard [19]. These works led me to the article by Lin et al. [14] in which they describe RetinaNet, a neural network architecture for object detection. The single-stage architecture incorporates a convolutional neural network backbone, the output of which is used in subnetworks that perform object classification and bounding box localization. I used the Keras-RetinaNet implementation [20] that was pretrained on images from the ImageNet competition [2, 13].
After viewing training images at various resolutions, I decided that the high resolution of the original images was not necessary for pneumonia bounding box prediction and would slow down training. I used the images at a low 224 × 224 resolution, which was much more efficient on my computing hardware, consisting of a workstation with a 12-GB graphics card (Titan Xp, Nvidia). I divided the 25,684 training images into a large training set (95%) and a small validation set (5%). I had originally used a much larger percentage of images in the validation set, but I obtained significantly better training results when I shifted more images into the training set. To further increase the size of the training set, I augmented the training images with random rotations, translations, scaling, horizontal flipping, and addition or subtraction of random constants (Fig. 6).
Fig. 6 —Third-place solution: flow diagram shows use of training data for training and validating RetinaNet model.
I trained each model up to 25 epochs, saving a model snapshot after each epoch. I used the trained models to predict pneumonia bounding boxes with confidence scores for the validation set images. To limit the number of predicted bounding boxes, I kept only boxes that exceeded a score threshold set to provide the best validation set performance. I also eliminated any overlapping bounding boxes, because I believed that the ground truth bounding boxes specified by the radiologist annotators were unlikely to be overlapping.
One question that occupied my time was when to stop training a model. Training beyond 25 epochs resulted in worsened validation set performance, indicating that the models were overfitting the training data. To define a stopping point more systematically, I calculated after each epoch the maximum Youden J index (sensitivity + specificity − 1) that I could achieve on the validation set based on adjusting the score threshold. I found that the models that performed best on the test set were the ones with the highest Youden index on the validation set, with the score threshold slightly lowered to give a sensitivity close to 90%. I suspect that the benefit of lowered score thresholds was because of the higher prevalence of pneumonia in the test set relative to the training set.
The most important observation I made about the test data was that reducing all bounding boxes by a fixed percentage (17% in each dimension) improved my test set leaderboard score substantially. I had first observed this when I had a larger validation set and I manually reviewed the predicted bounding boxes for the internal validation set images, and they seemed too large. Interestingly, when I switched to the smaller validation set, shrinking the bounding boxes reduced my internal validation set score but still improved my leaderboard score. I then read a forum post describing how the test set cases had been annotated by multiple readers, with the ground truth determined by the intersections of the annotators' bounding boxes. This suggested that bounding boxes of the test set images would be systematically smaller in the test set compared with the training set. I therefore based my modeling decisions more on the stage 1 test set leaderboard scores than on my validation set scores.
My final best model was an ensemble of two RetinaNet models, each with a different convolutional neural network backbone. I took weighted means of the coordinates of overlapping bounding boxes from both trained neural networks, using the scores for the boxes from each network as the weights. For bounding boxes that did not overlap between the two neural networks, I used a separate higher threshold value to decide whether the solitary box should be retained (Fig. 7). In retrospect I wish I had spent more time experimenting with larger ensembles. My code is available online [21].
Fig. 7 —Third-place solution: flow diagram shows neural network ensemble and postprocessing for creation of final output bounding boxes. ResNet-50 and ResNet-101 are neural network backbones [27].

Discussion

Although the winning solutions in the 2018 RSNA Pneumonia Detection Challenge were diverse, several common features were seen, as exemplified by the two solutions described in this article. The top solutions in the competition all used deep convolutional neural networks, which have become pervasive in computer vision tasks such as image classification, detection, and segmentation. Both solutions in this article leveraged transfer learning, whereby networks pretrained on the large ImageNet dataset [13, 22] and were fine-tuned using the smaller training set of chest x-ray images. The prior training allowed the networks to start with a baseline “knowledge” of basic image features, facilitating subsequent training on the radiographic images. Both teams used model ensembles to improve performance, with the first-place team distinguishing itself in the competition by the number of creative model combinations in their solution.
Significantly, the top teams made similar discoveries regarding differences between the training and test sets. Both teams found that a lower threshold for pneumonia detection improved test set scores. In addition, systematic reductions in the sizes of bounding boxes improved performance on the test sets. These differences between the training set and test sets appear to reflect differences in the annotation process for these datasets, with a consensus of expert radiologists used as ground truth in the test sets.
Finally, success in the competition was driven mainly by iterative empirical evidence rather than by purposeful designs regarding how pneumonia could best be detected. Even though all the competitors in this article had medical training, they used relatively little radiologic knowledge about imaging appearances of pneumonia to reach their final solutions. Neither team was able to effectively use image metadata in its solution, suggesting that the metadata did not provide additional distinguishing information beyond the image data alone for classifying x-ray images for the presence of pneumonia.
Despite the competition success of the teams in this article, the magnitude of the gap between the top solutions and human expert-level performance remains an open question. In particular, we do not know how well the pneumonia bounding boxes drawn by an expert radiologist would overlap with consensus boxes determined by other experts. Accurate measurement of single radiologist detection performance on the test set in this competition would require redefining the test set ground truth bounding boxes to eliminate the bias arising from the reduced bounding box sizes in the consensus annotations. Understanding expert level pneumonia detection performance will therefore involve additional analysis and research. In addition, the diagnostic performance of a deep learning model tested on a specific dataset does not necessarily transfer to other datasets from different institutions or imaging equipment [2325]. Broader systematic validation of model performance is therefore necessary before large-scale clinical use.

Conclusion

Effective applications of modern deep learning techniques, as well as insights into the labeling process of the image data, were critical for success in the RSNA Pneumonia Detection Challenge.

Acknowledgments

We thank the National Institutes for Health Clinical Center for providing the chest x-ray images used in the competition [5]; the original dataset is available at nihcc.app.box.com/v/ChestXray-NIHCC. We also gratefully acknowledge the considerable work of radiologists and staff from the Radiological Society of North America, the Society of Thoracic Radiology, and Kaggle, Inc., in annotating the images and organizing the competition.

References

1.
Chartrand G, Cheng PM, Vorontsov E, et al. Deep learning: a primer for radiologists. RadioGraphics 2017; 37:2113–2131
2.
Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis 2015; 115:211–252
3.
Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ, eds. Advances in neural information processing systems 25 (NIPS 2012). San Diego, CA: Neural Information Processing Systems Foundation, 2012
4.
Prevedello LM, Halabi SS, Shih G, et al. Challenges related to artificial intelligence research in medical imaging and the importance of image analysis competitions. Radiol Artific Intell 2019; 1:e180031
5.
Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE Proceedings: 30th IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE, 2017:3462–3471
6.
Shih G, Wu CC, Halabi SS, et al. Augmenting the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia. Radiol Artif Intell 2019; 1:e180041
7.
RSNA Pneumonia Detection Challenge: overview. Kaggle website. www.kaggle.com/c/rsna-pneumonia-detection-challenge. Accessed July 17, 2019
8.
Halabi SS, Prevedello LM, Kalpathy-Cramer J, et al. The RSNA pediatric bone age machine learning challenge. Radiology 2019; 290:498–503
9.
Dai J, Qi H, Xiong Y, et al. Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017:764–773
10.
Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, Inception-ResNet and the impact of residual connections on learning. arXiv website. arxiv.org/abs/1602.07261. Published February 23, 2016. Accessed May 18, 2018
11.
Chollet F. Xception: deep learning with depthwise separable convolutions. arXiv website. arxiv.org/abs/1610.02357. Published October 7, 2016. Accessed May 18, 2018
12.
Huang G, Liu Z, van der Maaten L, Weinberger KQ. Densely connected convolutional networks. arXiv website. arxiv.org/abs/1608.06993. Published August 24, 2016. Accessed May 18, 2018
13.
ImageNet website. ImageNet. www.image-net.org/. Accessed February 20, 2019
14.
Lin TY, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision. Los Alamitos, CA: IEEE, 2017:2999–3007
15.
Dai J, Li Y, He K, Sun J. R-FCN: object detection via region-based fully convolutional networks. arXiv website. arxiv.org/abs/1605.06409v2. Published May 20, 2016. Accessed February 22, 2019
16.
Hu H, Gu J, Zhang Z, Dai J, Wei Y. Relation networks for object detection. arXiv website. arxiv.org/abs/1711.11575v2. Published November 30, 2017. Accessed February 22, 2019
17.
Code for 1st place solution in Kaggle RSNA Pneumonia Detection Challenge. GitHub website. github.com/i-pan/kaggle-rsna18. Accessed July 17, 2019
18.
Ng A. Convolutional neural networks. Coursera website. www.coursera.org/learn/convolutional-neural-networks. Accessed December 27, 2018
19.
Howard J. Cutting edge deep learning for coders: part 2. Onwards fast ai website. course18.fast.ai/part2.html. Accessed January 26, 2019
20.
Gaiser H. Keras implementation of RetinaNet object detection: keras-retinanet. github.com/fizyr/keras-retinanet. GitHub website. Accessed January 19, 2019
21.
3rd Place solution for RSNA Pneumonia Detection Challenge. GitHub website. github.com/pm-cheng/rsna-pneumonia. Accessed March 2019
22.
Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE, 2009:248–255
23.
Pan I, Agarwal S, Merck D. Generalizable inter-institutional classification of abnormal chest radiographs using efficient convolutional neural networks. J Digit Imaging 2019 Mar 5 [Epub ahead of print]
24.
Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med 2018; 15:e1002683
25.
Chang K, Balachandar N, Lam C, et al. Distributed deep learning networks among institutions for medical imaging. J Am Med Inform Assoc 2018; 25:945–954
26.
RSNA Pneumonia Detection Challenge: private leaderboard. Kaggle website. www.kaggle.com/c/rsna-pneumonia-detection-challenge/leaderboard. Accessed July 17, 2019
27.
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. arxiv.org/abs/1512.03385. arXiv website. Published December 10, 2015. Accessed April 14, 2019

Information & Authors

Information

Published In

American Journal of Roentgenology
Pages: 568 - 574
PubMed: 31120793

History

Submitted: March 27, 2019
Accepted: March 28, 2019
Version of record online: May 23, 2019

Keywords

  1. artificial intelligence
  2. convolutional neural network
  3. deep learning
  4. detection
  5. pneumonia

Authors

Affiliations

Ian Pan
Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI.
Department of Diagnostic Imaging, The Warren Alpert Medical School of Brown University, Providence, RI.
Alexandre Cadrin-Chênevert
Department of Medical Imaging, CISSS Lanaudière, Saint-Charles-Borromée, QC, Canada.
Department of Radiology, Université Laval, Quebec City, QC, Canada.
Phillip M. Cheng
Department of Radiology, Keck School of Medicine of the University of Southern California, 1441 Eastlake Ave, Ste 2315B, Los Angeles, CA 90033.

Notes

Address correspondence to P. M. Cheng ([email protected]).

Metrics & Citations

Metrics

Citations

Export Citations

To download the citation to this article, select your reference manager software.

Articles citing this article

View Options

View options

PDF

View PDF

PDF Download

Download PDF

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share on social media