IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 35, NO.

5, MAY 2016 1153

Guest Editorial
Deep Learning in Medical Imaging: Overview and
Future Promise of an Exciting New Technique
I. INTRODUCTION markedly after the development of the backpropagation algo-
rithm. However, performance was still insufficient. Other clas-

D EEP learning is a growing trend in general data analysis

and has been termed one of the 10 breakthrough technolo-
gies of 2013 [1]. Deep learning is an improvement of artificial
sifiers were subsequently developed, including decision trees,
boosting and support vector machines. Each of these has been
applied to medical image analysis, especially for detection of
neural networks, consisting of more layers that permit higher abnormalities, but also in related fields such as segmentation.
levels of abstraction and improved predictions from data [2]. Despite such developments, relatively high false positive rates
To date, it is emerging as the leading machine-learning tool in for detection were the norm.
the general imaging and computer vision domains. CNNs were applied to medical image processing as long
In particular, convolutional neural networks (CNNs) have ago as 1996 in work of Sahiner et al. [4]. In this work, ROIs
proven to be powerful tools for a broad range of computer containing either biopsy-proven masses or normal tissues were
vision tasks. Deep CNNs automatically learn mid-level and extracted from mammograms. The CNN consisted of an input
high-level abstractions obtained from raw data (e.g., images). layer, two hidden layers and an output layer and used backprop-
Recent results indicate that the generic descriptors extracted agation. Training times were described as “computationally
from CNNs are extremely effective in object recognition and intensive” in this pre-GPU era, but no times were given. In
localization in natural images. Medical image analysis groups 1993, CNNs were applied to lung nodule detection [5]. A CNN
across the world are quickly entering the field and applying was used to detect microcalcifications on mammography in
CNNs and other deep learning methodologies to a wide variety 1995 [6].
of applications. Promising results are emerging. The typical CNN architecture for image processing consists
In medical imaging, the accurate diagnosis and/or assessment of a series of layers of convolution filters, interspersed with a
of a disease depends on both image acquisition and image in- series of data reduction or pooling layers. The convolution fil-
terpretation. Image acquisition has improved substantially over ters are applied to small patches of the input image. Like the
recent years, with devices acquiring data at faster rates and in- low-level vision processing in the human brain, the convolution
creased resolution. The image interpretation process, however, filters detect increasingly more relevant image features, for ex-
has only recently begun to benefit from computer technology. ample lines or circles that may represent straight edges (such
Most interpretations of medical images are performed by physi- as for organ detection) or circles (such as for round objects like
cians; however, image interpretation by humans is limited due to colonic polyps), and then higher order features like local and
its subjectivity, large variations across interpreters, and fatigue. global shape and texture. The output of the CNN is typically
Many diagnostic tasks require an initial search process to de- one or more probabilities or class labels. The convolution filters
tect abnormalities, and to quantify measurements and changes are learned from training data. This is desirable because it re-
over time. Computerized tools, specifically image analysis and duces the necessity of the time-consuming hand-crafting of fea-
machine learning, are the key enablers to improve diagnosis, tures that would otherwise be required to pre-process the images
by facilitating identification of the findings that require treat- with application-specific filters or by calculating computable
ment and to support the expert’s workflow. Among these tools, features. There are other network architecture variants, such
deep learning is rapidly proving to be the state-of-the-art foun- as a deep recurrent neural network known as long short-term
dation, leading to improved accuracy. It has also opened up memory [7].
new frontiers in data analysis with rates of progress not before CNNs are highly parallelizable algorithms. Much of the
experienced. practicality of using CNNs today lies in the vast acceleration
(approximately 40 times) enabled by graphics processing unit
A. Historical Perspective on Networks (GPU) computer chips compared to CPU processing alone.
An early paper describing the value of the GPU for training
Neural networks and the basic ideas behind deep learning CNNs and other machine learning techniques was published in
have been around for decades [3]. They have typically had just 2006 [8]. In medical image processing, GPUs were introduced
a few layers. The performance of neural networks improved first for segmentation, reconstruction and registration, and then
much later for machine learning [9], [10]. Interestingly, while
Eklund et al. [10] discuss convolutions extensively in their
Digital Object Identifier 10.1109/TMI.2016.2553401 2013 paper, convolutional neural networks and deep learning

0278-0062 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1154 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 35, NO. 5, MAY 2016

are not mentioned at all. This highlights how rapidly the revo- learning. In the standard approach to CAD [11] candidate le-
lution of deep learning has refocused medical image processing sions are detected, either by supervised methods or by classical
research. image processing techniques such as filtering and mathemat-
ical morphology. Candidate lesions are often segmented, and
B. Networks Today described by an often large set of hand-crafted features. A
Deep neural networks have recently gained considerable classifier is used to map the feature vectors to the probability
commercial interest due to the development of new variants that the candidate is an actual lesion. The straightforward way
of CNNs and the advent of efficient parallel solvers optimized to employ deep learning instead of hand-crafted features is to
for modern GPUs. The main power of a CNN lies in its deep train a CNN operating on a patch of image data centered on the
architecture, which allows for extracting a set of discriminating candidate lesion. Several articles in this issue use this approach.
features at multiple levels of abstraction. Training a deep CNN Setio et al. [12] combine three previously developed candidate
from scratch (or full training) is a challenge. First, CNNs re- detectors for pulmonary nodules in 3D chest CT scans and ex-
quire a large amount of labeled training data, a requirement that tract 2D patches centered on these candidates under nine dif-
may be difficult to meet in the medical domain where expert an- ferent orientations. A combination of different CNNs is used
notation is expensive and the diseases (e.g., lesions) are scarce. to classify each candidate. A slight improvement compared to
Second, training a deep CNN requires large computational and a previously published classical CAD system for the same task
memory resources, without which the training process would be is reported.
extremely time-consuming. Third, training a deep CNN is often Roth et al. [13] applied CNNs to improve three existing CAD
complicated by overfitting and convergence issues, which often systems for the detection of colonic polyps on CT colonog-
require repetitive adjustments in the architecture or learning raphy, sclerotic spine metastases on body CT and enlarged
parameters of the network to ensure that all layers are learning lymph nodes on body CT. They also used previously developed
with comparable speed. Given these difficulties, several new candidate detectors and 2D patches in three orthogonal direc-
learning schemes, termed “transfer learning” and “fine-tuning”, tions, and up to 100 randomly rotated views. The randomly
are shown to provide solutions and are increasingly gaining rotated “2.5D” views are a method of decompositional image
popularity. These will be discussed further in Section II-C. representation from the original 3D data. The CNN predictions
on these 2.5D views are later aggregated to attain additional
C. Networks in the Medical Domain gains in accuracy. The sensitivity for lesion detection im-
Deep learning methods are most effective when applied to proved 13 – 34% for all three CAD systems with the use of
large training sets, but in the medical domain large data sets are CNNs, indicating that the approach was both generalizable and
not always available. We are therefore faced with major chal- scaleable. Improvements of this magnitude have been nearly
lenges including (a) Can deep networks be used effectively for impossible to achieve using non-deep learning classifiers such
medical tasks? (b) Is transfer learning from general imagery to as committees of support vector machines.
the medical domain relevant? (c) Can we rely on learned fea- Dou et al. [14] detected cerebral microbleeds from suscep-
tures alone or may we combine them with handcrafted features tibility weighted MRI scans. They use 3D CNNs and also re-
for the task? This special issue of IEEE-Transactions on Med- placed the candidate detection stage with a CNN, proposing a
ical Imaging (IEEE-TMI) on deep learning in medical imaging two stage approach. They report improved results with their 3D
focuses on progress in this new era of machine learning and CNN compared to various classical and 2D CNN approaches
its role in the medical imaging domain. This issue presents re- from the literature that the authors re-implemented and trained
cent achievements of CNNs and other deep learning applica- and tested on the same data set.
tions to medical tasks. It contains 18 papers by various inves- Sirinukunwattana et al. [15] detected and classified nuclei
tigators from around the world, selected from 50 submissions, in histopathological images. They use a CNN that takes small
an unusually high number for IEEE special issues, and this was patches as input and instead of just predicting if the central pixel
achieved in a period between publication of the call for papers of the patch is a cell nucleus, they model the output as a high
and the submission deadline that was shorter than usual. The pa- peak in the vicinity of the center of each nucleus and flat else-
pers focus on a variety of classical tasks, from detection to cat- where. This spatially constrained CNN, in combination with a
egorization (e.g., lesion detection, image segmentation, shape fusion of overlapping patches in the test stage, produces better
modeling, image registration), as well as opening up new and results than previously proposed techniques for this task also
novel application domains. Also included are several works that based on CNNs and on classical feature-based approaches.
focus on the exploration of the networks and provide insight Anthimopoulos et al. [16] focus on detecting patterns of inter-
stitial lung diseases from 2D patches of chest CT scans. They are
on the architectures to be chosen for various tasks, parameters,
one of three groups in this issue (Shin et al. [17] and van Tulder
training sets and more.
et al. [18]) using a public data set from [19]. They train a CNN
II. OVERVIEW OF TOPICS AND PAPERS IN THE JOURNAL to classify patches of 32 32 pixels into one of 7 classes and
report higher accuracy than three previously published methods
A. Lesion Detection that employ hand-crafted features.
Computer-aided detection (CAD) is a well established area In several other articles in this issue, lesion detection is also
of medical image analysis that is highly amenable to deep the topic of interest, but the focus of these papers are broader or
DEEP LEARNING IN MEDICAL IMAGING: OVERVIEW AND FUTURE PROMISE OF AN EXCITING NEW TECHNIQUE 1155

zoom in on particular methodological issues. These papers will applications, including detection, segmentation and labeling.
be briefly discussed below. Still, several works focus on unsupervised schemes which are
mostly shown to be useful in image encoding, efficient image
B. Segmentation and Shape Modeling representation schemes and as a preprocessing step for further
supervised schemes. Unsupervised representation learning
For a large dataset of 2891 cardiac ultrasound examinations, methods such as Restricted Boltzmann Machines (RBM) may
Ghesu et al. [20] combine deep learning and marginal space outperform standard filter banks because they learn a feature
learning for object detection and segmentation. The combina- description directly from the training data. The RBM is trained
tion of “efficient exploration of large parameter spaces” and a with a generative learning objective; this enables the network
method to enforce sparsity in the deep networks, increased com- to learn representations from unlabeled data, but does not
putational efficiency, and led to a 13.5% reduction in mean seg- necessarily produce features that are optimal for classification.
mentation error compared to a reference method published by Van Tulder et al., [18] conducted an investigation to com-
the same group. bine the advantages of both generative and a discriminative
Three studies focused on segmentation of brain structures learning objectives in a convolutional classification restricted
or brain lesions. The problem of multiple sclerosis brain le- Boltzmann machine, which learns filters that are good both for
sion segmentation on MRI was addressed by Brosch et al. [21]. describing the training data and for classification. It is shown
The authors developed a 3D deep convolutional encoder net- that a combination of learning objectives outperforms purely
work that combined interconnected convolutional and decon- discriminative or generative learning.
volutional pathways. The convolutional pathway learned higher 3) Training Data Considerations: CNNs enable learning
level features, and the deconvolutional pathway predicted the data-driven, highly representative, layered hierarchical image
voxel level segmentation. They applied their network to two features. These features have been demonstrated to be a very
publicly available datasets and one clinical trial data set. They strong and robust representation in many application domains,
compared their method with 5 publicly available methods. The as presented in this issue. In order to provide such a rich repre-
method was reported to perform “comparably to the best state- sentation and successful classification, sufficient training data
of-the-art methods”. are needed. The amount of data required is a key question to be
Brain tumor segmentation on MRI was investigated in explored. Related questions include the following: How can we
Pereira et al. [22]. The authors used small kernels, a deeper use the training data we have most efficiently? What can we do
architecture, intensity normalization and data augmentation. in cases in which data are not available? And finally, are there
Different CNN architectures were used for low and high grade alternative methods for acquiring and medically annotating
tumors. The method separately segmented the enhancing part data?
and the core of the tumor. They attained top results on a 2013 Several of these questions are addressed by papers in this
public challenge dataset, and second place in an on-site 2015 issue. Van Grinsven et al. [24] attempt to improve and speed-up
challenge. the CNN training for medical image analysis tasks by dynami-
For brain structure segmentation, CNNs performed well cally selecting misclassified negative samples during training.
on five different datasets for patients in different age groups, The CNN training process is a sequential process requiring
spanning from pre-term infants to older adults in a study by many iterations (or epochs) to optimize the network parame-
Moeskops et al. [23]. A multi-scale approach was used to ters. In every epoch, a subset of samples is randomly selected
achieve robustness. The method attained good results on eight from the training data and is presented to the network to update
its parameters through back-propagation, minimizing a cost
tissue classes, with Dice similarity coefficients averaging 0.82
function. Classification tasks in the medical domain are often
to 0.91 for the five datasets.
a normal vs pathology discrimination task. In such a scenario,
the normal class is extremely over-represented; moreover, the
C. Network Exploration
majority of normal training samples are highly correlated due
1) Data Dimensionality Issues- 2D vs 3D: In most works to the repetitive pattern of normal tissues in each image. Only
to date we see analysis performed in 2D. It is often questioned a small fraction of these samples are informative. Treating
if the transition to 3D will be a key to major step forward in uniformly this data during the learning process leads to many
performance. Several variations exist in the data augmentation training iterations wasted on non-informative samples, making
process, including 2.5D. For example, in Roth et al., [13] axial, the CNN training process unnecessarily time-consuming. An
coronal and sagittal images were taken centered on a voxel in a approach to identify informative normal samples, as shown in
colonic polyp or lymph node candidate and fed into the cuda- the work, increased the efficiency of the CNN learning process
convnet CNN, which incorporates three channels normally used and reduced the training time.
to represent the red, green and blue color channels of a nat- 4) Transfer Learning and Fine Tuning: Obtaining datasets
ural light image. Explicitly 3D CNNs were used in this issue in the medical imaging domain that are as comprehensively
by Brosch et al.[21] and Dou et al. [14]. annotated as ImageNet remains a challenge. When sufficient
2) Learning Methodology- Unsupervised vs Supervised: data are not available, there are several ways to proceed: 1)
When we look at the literature of networks, it is evident that the Transfer learning: CNN models (supervised) pre-trained from
majority of works focus on the supervised CNNs in order to natural image dataset or from a different medical domain are
achieve categorization. Such networks are important for many used for a new medical task at hand. In one scheme, a pre-trained
1156 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 35, NO. 5, MAY 2016

CNN is applied to an input image and then the outputs are ex- classifier, and two different tasks are addressed: i) breast den-
tracted from layers of the network. The extracted outputs are sity segmentation, and ii) scoring of mammographic texture,
considered features and are used to train a separate pattern clas- with state-of-the-art results achieved. To control the model
sifier. For instance, in Bar et al. [25], [26] pre-trained CNNs capacity a sparsity regularizer is introduced that incorporates
were used as a feature generator for chest pathology identifica- both lifetime and population sparsity. The convolutional layers
tion. In Ginneken et al. [27] integration of CNN-based features in the unsupervised parts are trained as autoencoders; In the
with handcrafted features enabled improved performance in a supervised part the (pre-trained) weights and bias terms are
nodule detection system. 2) Fine Tuning: When a medium sized fine-tuned using softmax regression.
dataset does exist for the task at hand, one suggested scheme is Yan et al. [33] design a multi-stage deep learning framework
to use a pre-trained CNN as initialization of the network, fol- for image classification and apply it on body part recognition.
lowing which further supervised training is conducted, of sev- In the pre-train stage, a convolutional neural network (CNN)
eral (or all) the network layers, using the new data for the task is learned using multi-instance learning to extract the most dis-
at hand. criminative and non-informative local patches from the training
Transfer learning and fine tuning are key components in the slices. In the boosting stage, the pre-trained CNN is further
use of deep CNNs in medical imaging applications. Two works boosted by these local patches for image classification. A hall-
that explore these issues are Shin et al. [17] and Tajbakhsh et mark of the method was that it automatically discovered the dis-
al. [28]. The experiments conducted in these works consistently criminative and non-informative local patches through multi-in-
show that using pre-trained CNNs with fine-tuning achieved
stance deep learning. Thus, no manual annotation was required.
the strongest results, regardless of specific applications domain
Regression networks are not very common in the medical
(Tajbakhsh et al.), and for all network architectures (Shin et al.).
imaging domain. In Miao et al. [34], a CNN regression ap-
In Tajbakhsh et al., [28] further analysis shows that deep fine-
proach is presented, for real-time 2-D/3-D registration. Three
tuning led to improved performance over shallow fine-tuning,
algorithmic strategies are proposed to simplify the underlying
and the importance of using fine-tuning increases with reduced
mapping to be regressed, and to design a CNN regression model
size training sets. In Shin et al., [17] the GoogLeNet architec-
with strong non-linear modeling. Results show that the DL
ture led to state-of-the-art detection of mediastinal lymph nodes
compared to other less deep architectures. method is more accurate and robust than two state-of-the-art
5) Ground Truth – From the Experts and the Non-Experts: accelerated intensity-based 2-D/3-D registration methods.
The lack of publicly available ground-truth data, and the diffi- We have yet to explore the domains to which NNs can be ap-
culty in collecting such data per medical task, both cost-wise as plied, and the applications and dimensionality of tasks to which
well as time-wise, is a prohibitively limiting factor in the med- they will provide a substantial contribution. In an exploratory
ical domain. Though crowdsourcing has enabled annotation of work, Golkov et al. [35] provide an initial proof-of-concept,
large scale databases for real world images, its application for applying DL to reduce diffusion MRI data processing to a
biomedical purposes requires a deeper understanding and hence, single optimized step. They show that this modification en-
more precise definition of the actual annotation task Nguyen et ables one to obtain scalar measures from advanced models
al. [29], McKenna et al. [30]. The fact that expert tasks are being at twelve-fold reduced scan time and to detect abnormalities
outsourced to non-expert users may lead to noisy annotations without using diffusion models. The relationship between the
introducing disagreement between users. Many issues arise in diffusion-weighted signal and microstructural tissue properties
combining the knowledge of medical experts with non-profes- is non-trivial. Golkov et al. [35] demonstrate that with the use
sionals, such as how to combine the information sources, how to of a DNN such relationships may in fact be revealed: DWIs
assess and incorporate the inputs weighted by their prior-proved are directly used as inputs rather than using scalar measures
accuracy in performance and more. These issues are addressed obtained from model fitting. The work shows microstructure
in Albarqouni et al. [31] who present a network that combines prediction on a voxel-by-voxel basis as well as automated
an aggregation layer that is integrated into the CNN to enable model-free segmentation from DWI values, into healthy tissues
learning inputs from the crowds as part of the network learning and MS lesions. Diffusion kurtosis is shown to be measured
process. Results shown give valuable insights into the function- from only 12 data points and neurite orientation dispersion
ality of deep CNN learning from crowd annotations. The most and density measures from only 8 data points. This may allow
astonishing fact about crowdsourcing studies in the medical do- for fast and robust protocols facilitating clinical routine and
main, however, is the conclusion that a crowd of nonprofes- demonstrates how classical data processing can be streamlined
sional, inexperienced users can in fact perform as well as the by means of deep learning.
medical experts. This has also been observed by Nguyen et al.
[29] and McKenna et al. [30] for radiology images. III. DISCUSSION: KEY ISSUES AND MAIN PROMISES
In the majority of works presented, use of a deep network is
D. Novel Applications and Unique Use Cases
shown to improve over the state-of-the-art. As these improve-
Unsupervised feature learning for mammography risk ments seem to be consistent across a large variety of domains,
scoring is presented in Kallenberg et al. [32]. In this work, and as is usually the case, development of a deep learning solu-
a method is shown that learns a feature hierarchy from un- tion is found to be relatively straight-forward, we can view this
labeled data. The learned features are then input to a simple as a major step forward in the medical computing field. Still, a
DEEP LEARNING IN MEDICAL IMAGING: OVERVIEW AND FUTURE PROMISE OF AN EXCITING NEW TECHNIQUE 1157

major question remains as to how and when we can reach a sub- data sets only: challenges provide a precise definition of a task
stantial leap in performance – equivalent to the 10% increase to be solved and define one or more evaluation metrics that
in large scale category recognition of 2012. Are we asking the provide a fair and standardized comparison between proposed
right questions and investigating the right tasks? Are we using algorithmic. Without such a standardization, it is often difficult
strong enough input representations (e.g., 2D vs 3D)? Do we to compare different approaches to the same problem even if
need to work on getting real Big Data for each medical task, they use the same data set. This is illustrated in this issue where
or will transfer learning be sufficient? These issues and more three studies (Anthimopoulos et al. [16], Shin et al. [17], and
are partly addressed in the papers of this issue (Section II), and van Tulder et al. [18]) use the same data set of chest CT scans
mostly remain as the challenges to be answered in the future. with annotations of interstitial lung disease patterns [19], but
In the literature we can find both unsupervised as well as su- they all report results in a different form.
pervised learning via deep networks. It seems that the majority One study in this issue (Setio et al. [12]) has resulted in a
of works are in fact using supervised learning. The question challenge for pulmonary nodule detection (http://luna16.grand-
arises if in the medical domain, where data numbers are a crit- challenge.org/), organized in conjunction with the IEEE ISBI
ical factor, the formalism should combine the benefits of both conference, using the publicly available LIDC/IDRI data set,
the unsupervised and the supervised. It is likely that to leverage and thus the system described in this issue can be compared
really big data for which hand annotations are unavailable or directly to alternative approaches.
intractable to obtain, the field will need to move more towards Last year we have seen the first large challenges devoted
semi-supervised and unsupervised learning. to medical image analysis organized on major platforms tra-
The literature contains many network architectures. The vari- ditionally focusing on other machine learning applications.
ability is large, as can be seen across the works in the current Kaggle organized a competition on detection and staging
journal. Possibilities include selecting a known architecture, de- of diabetic retinopathy from color fundus images for which
vising a task-specific architecture, fusing across architectures 661 teams submitted results, $100,000 prize money was
and more. One interesting question going forward is if the very awarded, and around 80,000 images were made available
deep residual networks that used 152 layers and performed best (https://www.kaggle.com/c/diabetic-retinopathy-detection).
in the ILSVRC 2015 classification task [36] also obtain good This data was used in one study in this special issue (van
results on medical tasks. Grinsven et al. [24]). Recently, a second medical image
One key aspect of deep learning is that it can benefit from analysis competition was completed using MRI to measure
large amounts of training data. Breakthrough results in com- cardiac volumes and derive ejection fractions in which 192
puter vision were obtained on the ILSVRC challanges based teams participated and $200,000 prize money was available
on the ImageNET data set (http://www.image-net.org/). This (https://www.kaggle.com/c/second-annual-data-science-bowl).
data set is very large compared to most of the training and test In both these competitions, the top contenders all used convo-
data sets used in the papers in this special issue (millions versus lutional neural networks. We expect both trends to continue:
100’s or 1000’s). Our community could likely benefit strongly if challenges will use larger data sets and deep learning will be a
similarly large publicly available medical image data sets were dominant technology among the best performing solutions. In
constructed. this context, the upcoming series of worldwide competitions
There are several reasons why this is challenging. First, it to increase the accuracy of various imaging based cancer
is difficult to obtain funding for the construction of data sets. screenings (coding4cancer.org) could be of interest for readers
Second, scarce and expensive medical expertise is needed for of this special issue.
high quality annotation of medical imaging data. Third, privacy On-line platforms, such as the ones used for competitions,
issues make it more difficult to share medical data than natural serve multiple purposes. They lead to new collaborations and
images. Fourth, the breadth of applications in medical imaging joint solutions, could also be the most effective way to obtain
requires that many different data sets need to be collected. large amounts of data annotations via crowd-sourcing, which
Despite these potential hurdles, we see rapid progress in the was shown by the study from Albarqouni et al. [31] in this issue.
area of data collection and data sharing. Many public data sets Finally, we are grateful for the guidance from the current Ed-
have been released and studies today routinely use them in itor-in-Chief, the help from the TMI office, and most impor-
experimental validation. Examples include VISCERAL and tantly the tremendous efforts by the authors and the reviewers.
The Cancer Imaging Archive (http://www.visceral.eu/ and This Special issue provides a snapshot in time of a very fast
http://www.cancerimagingarchive.net/). Roth et al. [13] and moving field of medical image processing. We hope you will
Shin et al. [17] analyze a dataset of enlarged lymph nodes on enjoy it and look forward to your future contributions to this
CT scans that they have made public on The Cancer Imaging dynamic field.
Archive [37]. The same group has made a pancreas dataset
available online [38]. ACKNOWLEDGMENT
Since 2007, it has become customary to organize challenge
workshops at medical imaging conferences such as MICCAI, We would like to thank all the submissions to the journal. We
ISBI, and SPIE Medical Imaging. This has led to a large amount value greatly the support from our large set of expert reviewers,
of data sets and ongoing benchmark studies, documented at the who helped select the papers and shape this issue.
website http://www.grand-challenge.org/. Using these public H. Greenspan is funded in part by INTEL Collaborative Re-
benchmark data sets has a distinct advantage over using public search Institute for Computational Intelligence (ICRI-CI). This
1158 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 35, NO. 5, MAY 2016

work was funded in part by the Intramural Research Program of [16] M. Anthimopoulos, S. Christodoulidis, A. Christe, and S.
the National Institutes of Health, Clinical Center. Mougiakakou, “Lung pattern classification for interstitial lung
diseases using a deep convolutional neural network,” IEEE Trans.
Disclaimer: The opinions expressed herein are those of the Med. Imag., vol. 35, no. 5, pp. 1207–1216, May 2016.
authors and do not necessarily represent those of the National [17] H.-C. Shin et al., “Deep convolutional neural networks for computer-
aided detection: CNN architectures, dataset characteristics and transfer
Institutes of Health. Financial Disclosures: R. M. Summers re- learning,” IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1285–1298, May
ceives patent royalties from iCAD Medical. 2016.
[18] G. van Tulder and M. de Bruijne, “Combining generative and discrim-
HAYIT GREENSPAN, Guest Editor inative representation learning in convolutional restricted Boltzmann
Biomedical Image Computing Lab machines,” IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1262–1272,
May 2016.
Department of Biomedical Engineering [19] A. Depeursinge et al., Comput. Med. Imag. Graph., vol. 36, no. 3, pp.
Faculty of Engineering 227–238, 2012.
Tel-Aviv University [20] F. Ghesu et al., “Marginal space deep learning: Efficient architecture
for volumetric image parsing,” IEEE Trans. Med. Imag., vol. 35, no. 5,
Tel-Aviv, 69978 Israel pp. 1217–1228, May 2016.
[21] T. Brosch et al., “Deep 3D convolutional encoder networks with
BRAM VAN GINNEKEN, Guest Editor shortcuts for multiscale feature integration applied to multiple scle-
rosis lesion segmentation,” IEEE Trans. Med. Imag., vol. 35, no. 5,
Diagnostic Image Analysis Group pp. 1229–1239, May 2016.
Radboud University Medical Center [22] S. Pereira, A. Pinto, V. Alves, and C. Silva, “Brain tumor segmenta-
Nijmegen, 6525 GA The Netherlands tion using convolutional neural networks in MRI images,” IEEE Trans.
Med. Imag., vol. 35, no. 5, pp. 1240–1251, May 2016.
[23] P. Moeskops et al., “Automatic segmentation of MR brain images with
RONALD M. SUMMERS, Guest Editor a convolutional neural network,” IEEE Trans. Med. Imag., vol. 35, no.
Imaging Biomarkers and Computer-Aided Diagnosis Lab 5, pp. 1252–1261, May 2016.
[24] M. van Grinsven, B. van Ginneken, C. Hoyng, T. Theelen, and C.
Radiology and Imaging Sciences Sánchez, “Fast convolutional neural network training using selective
National Institutes of Health Clinical Center data sampling: Application to hemorrhage detection in color fundus
Bethesda, MD 20892 USA images,” IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1273–1284, May
2016.
[25] Y. Bar, I. Diamant, L. Wolf, and H. Greenspan, “Deep learning with
REFERENCES non-medical training used for chest pathology identification,” Proc.
[1] MIT Technol. Rev., , 2013 [Online]. Available: https://www.technolo- SPIE Med. Imag. Computer-Aided Diagnosis, vol. 9414, 2015.
gyreview.com/s/513696/deep-learning [26] Y. Bar, I. Diamant, L. Wolf, S. Lieberman, E. Konen, and H.
[2] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. Greenspan, “Chest pathology detection using deep learning with
521, no. 7553, pp. 436–444, 2015. non-medical training,” in Proc. IEEE 12th Int. Symp. Biomed. Imag.,
[3] J. Schmidhuber, “Deep learning in neural networks: An overview,” 2015, pp. 294–297.
Neural Netw., vol. 61, pp. 85–117, 2015. [27] B. van Ginneken, A. A. Setio, C. Jacobs, and F. Ciompi, “Off-the-shelf
[4] B. Sahiner et al., “Classification of mass and normal breast tissue: A con- convolutional neural network features for pulmonary nodule detec-
volution neural network classifier with spatial domain and texture im- tion in computed tomography scans,” in Proc. IEEE 12th Int. Symp.
ages,” IEEE Trans. Med. Imag., vol. 15, no. 5, pp. 598–610, Oct. 1996. Biomed. Imag., 2015, pp. 286–289.
[5] S. C. B. Lo, J. S. J. Lin, M. T. Freedman, and S. K. Mun, “Computer- [28] N. Tajbakhsh et al., “Convolutional neural networks for medical image
assisted diagnosis of lung nodule detection using artificial convolution analysis: Full training or fine tuning?,” IEEE Trans. Med. Imag., vol.
neural-network,” Proc. SPIE Med. Imag., Image Process., vol. 1898, 35, no. 5, pp. 1299–1312, May 2016.
pp. 859–869, 1993. [29] T. B. Nguyen et al., “Distributed human intelligence for colonic polyp
[6] H.-P. Chan, S.-C. Lo, B. Sahiner, K. L. Lam, and M. A. Helvie, “Com- classification in computer-aided detection for CT colonography,” Ra-
puter-aided detection of mammographic microcalcifications: Pattern diology, vol. 262, no. 3, pp. 824–833, 2012.
recognition with an artificial neural network,” Med. Phys., vol. 22, no. [30] M. T. McKenna et al., “Strategies for improved interpretation of
10, pp. 1555–67, 1995. computer-aided detections for CT colonography utilizing distributed
[7] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural human intelligence,” Med. Image Anal., no. 6, pp. 1280–1292, 2012.
Comput., vol. 9, no. 8, pp. 1735–1780, 1997. [31] S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, and N.
[8] G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learning algorithm for Navab, “Agg-Net: Deep learning from crowds for mitosis detection in
deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, 2006. breast cancer histology images,” IEEE Trans. Med. Imag., vol. 35, no.
[9] D. Castano-Diez, D. Moser, A. Schoenegger, S. Pruggnaller, and A. S. 5, pp. 1313–1321, May 2016.
Frangakis, “Performance evaluation of image processing algorithms on [32] M. Kallenberg et al., “Unsupervised deep learning applied to breast
the GPU,” J. Struct. Biol., vol. 164, no. 1, pp. 153–160, 2008. density segmentation and mammographic risk scoring,” IEEE Trans.
[10] A. Eklund, P. Dufort, D. Forsberg, and S. M. LaConte, “Medical image Med. Imag., vol. 35, no. 5, pp. 1322–1331, May 2016.
processing on the GPU–Past, present and future,” Med. Image Anal., [33] Z. Yan et al., “Multi-instance deep learning: Discover discriminative
vol. 17, no. 8, pp. 1073–94, 2013. local anatomies for bodypart recognition,” IEEE Trans. Med. Imag.,
[11] B. van Ginneken, C. M. Schaefer-Prokop, and M. Prokop, “Computer- vol. 35, no. 5, pp. 1332–1343, May 2016.
aided diagnosis: How to move from the laboratory to the clinic,” Ra- [34] S. Miao, Z. J. Wang, and R. Liao, “A CNN regression approach for
diol., vol. 261, no. 3, pp. 719–732, 2011. real-time 2D/3D registration,” IEEE Trans. Med. Imag., vol. 35, no. 5,
[12] A. Setio et al., “Pulmonary nodule detection in CT images using multi- pp. 1352–1363, May 2016.
view convolutional networks,” IEEE Trans. Med. Imag., vol. 35, no. 5, [35] V. Golkov et al., “q-Space deep learning: Twelve-fold shorter and
pp. 1160–1169, May 2016. model free diffusion MRI scans,” IEEE Trans. Med. Imag., vol. 35,
[13] H. Roth et al., “Improving computer-aided detection using convolu- no. 5, pp. 1344–1351, May 2016.
tional neural networks and random view aggregation,” IEEE Trans. [36] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image
Med. Imag., vol. 35, no. 5, pp. 1170–1181, May 2016. recognition ArXiv, 2015 [Online]. Available: arXiv:1512.03385, to be
[14] Q. Dou et al., “Automatic detection of cerebral microbleeds from MR published
images via 3D convolutional neural networks,” IEEE Trans. Med. [37] H. R. Roth et al., “A new 2.5 d representation for lymph node detection
Imag., vol. 35, no. 5, pp. 1182–1195, May 2016. in CT,” Cancer Imag. Arch., 2015 [Online]. Available: http://dx.doi.
[15] K. Sirinukunwattana et al., “Locality sensitive deep learning for de- org/10.7937/K9/TCIA.2015.AQIIDCNM
tection and classification of nuclei in routine colon cancer histology [38] H. R. Roth et al., “Data from pancreas-CT,” Cancer Imag. Arch.,
images,” IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1196–1206, May 2016 [Online]. Available: http://dx.doi.org/10.7937/K9/TCIA.2016.
2016. tNB1kqBU
DEEP LEARNING IN MEDICAL IMAGING: OVERVIEW AND FUTURE PROMISE OF AN EXCITING NEW TECHNIQUE 1159

Hayit Greenspan received the B.S. and M.S. degrees in electrical engineering from the Technion-
Israel Institute of Technology, Haifa, Israel, and the Ph.D. degree in electrical engineering from the
California Institute of Technology, Pasadena, CA, USA.
She was a Postdoc with the Computer Science Division at University of California Berkeley fol-
lowing which she joined Tel-Aviv University and founded the Biomedical Image Computing lab,
which she heads. She is a Tenured Professor at the Biomedical Engineering Department in the Fac-
ulty of Engineering, Tel-Aviv University. She is affiliated with the International Computer Science
Institute (ICSI), Berkeley, CA, USA. From 2008 until 2010, she was a visiting Professor at Stan-
ford University, Department of Radiology, Faculty of Medicine. She has been conducting research
in medical image analysis for the past 20 years, with a special focus on image modeling and anal-
ysis, deep learning, and content-based image retrieval. Among her current research projects: MRI
resolution augmentation, brain MRI research (structural and DTI), CT and X-ray image analysis
and medical imagery indexing and retrieval. She has over 100 publications in leading international
journals and conference proceedings. She is a coauthor on several patents. She has received grants from several Israeli and U.S.
agencies. Currently her Lab is funded for Deep Learning in Medical Imaging by the INTEL Collaborative Research Institute for
Computational Intelligence (ICRI-CI).
Dr. Greenspan has received several awards. She is a member of several journal and conference program committees, including
SPIE medical imaging, IEEE ISBI and MICCAI. She is an Associate Editor for the IEEE TRANSACTIONS ON MEDICAL IMAGING.

Bram van Ginneken studied physics at the Eindhoven University of Technology, Eindhoven, The
Netherlands, and at Utrecht University, Utrecht, The Netherlands. He received the Ph.D. degree
in computer-aided diagnosis in chest radiography from the Image Sciences Institute, Utrecht, The
Netherlands, in March 2001.
He is Professor of Functional Image Analysis at Radboud University Medical Center Nijmegen.
Since 2010, he is co-Chair of the Diagnostic Image Analysis Group within the Department of
Radiology and Nuclear Medicine, together with N. Karssemeijer. He also works for Fraunhofer
MEVIS, Bremen, Germany, and is one of the founders of Thirona, a company that provides quan-
titative analysis of chest CT scans. From 2001 through 2009, he led the Computer-Aided Diagnosis
group at the Image Sciences Institute, where he still has an Associated Faculty position. He has
(co-)authored over 150 publications in international journals. He is a member of the Editorial Board
of Medical Image Analysis. He has pioneered the concept of challenges in medical image analysis,
where multiple teams apply their algorithm to solve a particular medical image analysis task to the
same test data set, so that a fair comparison between different approaches can be made.
Dr. van Ginneken is an Associate Editor for the IEEE TRANSACTIONS ON MEDICAL IMAGING.

Ronald M. Summers received the B.A. degree in physics and the M.D. and Ph.D. degrees in
medicine/anatomy and cell biology from the University of Pennsylvania, Philadelphia, PA, USA.
He completed a medical internship at the Presbyterian-University of Pennsylvania Hospital,
Philadelphia, PA, USA, a radiology residency at the University of Michigan, Ann Arbor, MI,
USA, and an MRI fellowship at Duke University, Durham, NC, USA.
In 1994, he joined the Diagnostic Radiology Department at the NIH Clinical Center, Bethesda,
MD, USA, where he is now a tenured Senior Investigator and Staff Radiologist. He is currently
Chief of the Clinical Image Processing Service and directs the Imaging Biomarkers and Com-
puter-Aided Diagnosis (CAD) Laboratory. His research interests include deep learning, virtual
colonoscopy, CAD and development of large radiologic image databases. His clinical areas of spe-
cialty are thoracic and abdominal radiology and body cross-sectional imaging. He is a member of
the editorial boards of the journals Radiology, Journal of Medical Imaging, and Academic Radi-
ology. He has co-authored over 400 journal, review and conference proceedings articles and is a
coinventor on 14 patents.
Dr. Summers received the Presidential Early Career Award for Scientists and Engineers, presented by Dr. Neal Lane, President
Clinton’s science advisor, in 2000. In 2012, he received the NIH Director’s Award, presented by NIH Director Dr. Francis Collins.
He is a program committee member of the Computer-aided Diagnosis section of the annual SPIE Medical Imaging conference.