Mobile Networks and Applications

https://doi.org/10.1007/s11036-018-1113-0

Cognitive IoT-Cloud Integration for Smart Healthcare: Case Study

for Epileptic Seizure Detection and Monitoring
1
Musaed Alhussein & Ghulam Muhammad 1 & M. Shamim Hossain 2 & Syed Umar Amin 1

# Springer Science+Business Media, LLC, part of Springer Nature 2018

Abstract
We propose a cognitive Internet of Things (IoT)–cloud-based smart healthcare framework, which communicates with smart
devices, sensors, and other stakeholders in the healthcare environment; makes an intelligent decision based on a patient’s state;
and provides timely, low-cost, and accessible healthcare services. As a case study, an EEG seizure detection method using deep
learning is also proposed to access the feasibility of the cognitive IoT–cloud smart healthcare framework. In the proposed
method, we use smart EEG sensors (apart from general healthcare smart sensors) to record and transmit EEG signals from
epileptic patients. Thereafter, the cognitive framework makes a real-time decision on future activities and whether to send the data
to the deep learning module. The proposed system uses the patient’s movements, gestures, and facial expressions to determine the
patient’s state. Signal processing and seizure detection take place in the cloud, while signals are classified as seizure or non-
seizure with a probability score. The results are transmitted to medical practitioners or other stakeholders who can monitor the
patients and, in critical cases, make the appropriate decisions to help the patient. Experimental results show that the proposed
model achieves an accuracy and sensitivity of 99.2 and 93.5%, respectively.

Keywords IoT-cloud . Smart healthcare . Seizure detection . EEG . Deep learning

1 Introduction paradigm of smart cities. Furthermore, the integration of IoT

and cloud technologies has provided a seamless and ubiqui-
The Internet of Things (IoT), which can be considered an tous framework for smart healthcare monitoring. At present,
interconnected network of intelligent sensor devices, often residents in smart cities have access to smart sensor devices
has limited storage and low processing power capability. and advanced mobile technologies. In an environment such as
IoT, together with cloud computing, which has a large storage that of a smart city, finding specialized doctors, healthcare
and sufficient processing power capability, has made essential centers, and hospitals nearby is difficult. The movement of
services, such as smart healthcare [1, 2], possible in a smart patients in critical conditions is also quite difficult. Hence,
city environment. However, monitoring and communicating we need to create a smart healthcare monitoring framework
remotely with patients are necessary in such environments. In by integrating the resources available at our disposal to im-
addition, the need to provide low-cost, high-quality, and prove the quality and accessibility of healthcare services. In
patient-centric smart healthcare to patients has emerged. such a smart healthcare monitoring framework, we can trans-
The advancements in the field of IoT [3] and cloud tech- mit and process medical-related multimedia signals from
nologies [4] has resulted in a tremendous demand for real- smart sensors and mobile devices to provide timely assistance
time, intelligent, and remote healthcare services under the and quality healthcare services to patients. However, such
healthcare data and signals are often naturally large and chal-
lenging to handle because of their complexity.
* M. Shamim Hossain The healthcare industry has emerged as one of the major
[email protected] industries with tremendous demands. Apart from providing
patients with critical and crucial services, this industry is also
1
Department of Computer Engineering, College of Computer and generating large revenues for the government and the private
Information Sciences, King Saud University, Riyadh, Saudi Arabia sector. The smart healthcare industry has recently witnessed a
2
Department of Software Engineering, College of Computer and competition among various healthcare providers in providing
Information Sciences, King Saud University, Riyadh, Saudi Arabia mature and sophisticated services and devices with high
 Mobile Netw Appl

accuracy and dependability and low cost [5]. Therefore, the framework. Previous studies [10, 11] showed that when cloud
IoT–cloud integration within healthcare has recently been the technologies, IoT, and smart sensors are integrated, real-time
focus of considerable research initiative. Several types of IoT smart healthcare services can be provided in a smart city.
devices built for healthcare include smart wearable devices, Another study focused on integrating edge computing with
such as blood pressure devices, portable insulin syringe, stress cognitive technology for smart healthcare [12]. Therefore,
monitoring devices, weight tracking and general fitness de- such a smart healthcare monitoring framework should be able
vices, hearing aids, and EEG and ECG monitors. to effectively process multimedia signals and sensor data and
Although healthcare data, such as EEG, are complex in in real time to provide quality healthcare services.
nature, we have had numerous technological advances in the Epilepsy has been called a neurological disorder that can
fields of big data analytics and cloud computing to manage the occassionally cause jerking in several parts of the human
complexity of such data and provide the processing power and body, loss of consciousness, and in severe cases, convulsions
storage capability needed by such information. However, in the entire body. This disorder can affect a patient’s quality
many interconnected IoT devices and sensors and extensive of life, cause social and economic problems in people of all
multimedia, healthcare, and communication data make it dif- ages, and lead to premature death in extreme cases [13].
ficult to build a smart healthcare framework that can cater to Approximately 50 million people are affected by epileptic
the needs of all stakeholders in a smart city environment. seizure globally [13]. However, only approximately 70% of
However, the concept of smart IoT–cloud integration is patients can be treated with medication, while approximately
impossible without human brain-like intelligence. With big 8% of patients who do not respond to medication require
data and its real-time processing coming into the picture, the surgical intervention. If epilepsy is detected at an early stage,
research community faces multiple challenges to develop a epileptic seizures can be suppressed using antiepileptic drugs
smart and intelligent IoT–cloud framework, which would be or by electrical stimulation [14].
able to make its own decisions. Consequently, the cognitive However, the main problem is that epileptic patients need
computing framework was introduced and proposed to turn immediate and quality care. Any delay in treatment or
IoT into a brain-powered cognitive IoT (CIoT) [6], which reaching specialized medical centers or hospitals can be cata-
would possess a high level of intelligence. When we consider strophic for patients who are affected by seizures. Therefore, a
complex and big data, such as healthcare data, for a smart city smart healthcare monitoring system is essential for epileptic
paradigm, the CIoT becomes considerably important. patients and could generally solve this problem. Medical prac-
Ecidently, having a cognitive IoT–cloud smart healthcare titioners can monitor and advise patients regularly. In serious
framework would require that the IoT devices inside a pa- cases, other smart city services, such as smart ambulances and
tient’s body (i.e., attached or around him/her) cooperate to mobile clinics, can be used to provide urgent medical assis-
sense his/her body signals, movements, voice, disease bio- tance to patients.
markers, or monitored signals, such as EEG and ECG; and This study proposes an automatic and advanced method for
deduce the state of the patient. The cognitive healthcare frame- EEG-based seizure detection and monitoring to be used as a
work is sufficiently intelligent to make the corresponding de- component of the proposed cognitive healthcare IoT (CHIoT)
cisions to make the patient comfortable and decide the future framework. In the proposed seizure detection method, we use
course of activities by involving different stakeholders of the scalp EEG, which is recorded by smart EEG sensors, as the
smart city. Similar to the assumption that making life in a input signal. Our system also uses other healthcare smart sen-
smart city comfortable without intelligence is unachievable, sors to record the psychological and physiological signals and
an IoT framework without cognitive capability is only a waste transmit them to the cloud via the Internet. Apart from EEG,
of resources. Therefore, considerable research effort has been these signals include patients’ movements, gestures, and facial
exerted [5] to develop such a healthcare framework, which expressions to determine their state. Thereafter, the cognitive
caters to the needs of the patients, medical practitioners, hos- system makes a real-time decision on the activities and the
pitals, and all other service providers. medical attention and services to be provided to patients based
A previous study [1] highlighted the challenges of provid- on their state and whether to send the data to the deep learning
ing smart healthcare services using smart sensors and cloud module. If the cognitive system believes that a patient is hav-
connectivity in a smart city environment. Temperature, traffic, ing a seizure, then this system makes the decision to inform
humidity, and other related parameters should be controlled other stakeholders and sends the EEG data to the deep learn-
through smart sensors to provide effective smart healthcare ing module. Signal processing and seizure detection occur in
services [6]. In [7], the researcher presented a patient status the cloud and signals are classified as seizure or non-seizure
monitoring framework, in which medical data are accessed with a probability score. The results are transmitted to medical
through voice and facial expression, to address such complex practitioners or other stakeholders who can monitor the pa-
medical-related data. In [8, 9], the authors proposed an emo- tients and, in critical cases, make the appropriate decisions to
tion recognition system that can be utilized in a cloud assist the patient. EEG signals have a low signal-to-noise ratio
Mobile Netw Appl

and are sensitive to external and internal noises. Eye blinking given that the domain is new and still developing, improve-
and muscle movement artifacts can cause numerous problems ments in framework design and enhancement of the end ser-
in extracting good features. Therefore, we use a novel deep vices, intelligent behavior, and smart processing of multimod-
learning model composed of a deep convolutional neural net- al health data are needed to fuel further exploration and inter-
work (CNN) and stacked autoencoder. The deep CNN is used est in this field.
to extract features, while the stacked autoencoder is used to All objects in CHIoT, such as sensors and devices, are
improve the subject-specific features and accuracy. The major interconnected and cooperate to understand the physical and
contributions of the proposed CHIoT framework are as fol- social environments, store and process the learned information
lows. (1) The cognitive IoT–cloud technology is integrated and the extracted knowledge, and learn to adapt themselves.
with the smart healthcare monitoring framework. (2) To the The framework has intelligent decision-making capability that
best of our knowledge, this study is the first to use the smart requires minimum human intervention.
CHIoT framework for epileptic seizure detection and moni- Many cognitive IoT frameworks for various applications
toring. (3) A novel integration of deep CNN and stacked [3, 12, 15–17] are presented in the literature. In [18], the au-
autoencoders, which is better than the state-of-the-art model thors propose a cognitive framework to assist in smart city
for seizure detection, is proposed in this research. development and make it considerably sustainable. In [19], a
The remainder of this paper is organized as follows. three-layer cognitive ring is proposed to achieve a good per-
Section 2 discusses several related studies on CHIoT, smart formance and high intelligence and merges human cognition
healthcare, and EEG seizure detection. Section 3 describes the with the system design. In [20], a cognitive system, which
proposed smart healthcare monitoring framework. Section 4 could model human knowledge and process relative informa-
presents the experiments, results, and discussion. Lastly, tion, was proposed. In [18], the authors developed a cognitive
Section 5 concludes this research. system that has the open question answering capability and
makes use of text data and natural language processing. In
[21], big data are analyzed to build a cognitive system.
2 Related work Many researchers have also applied cognitive computing to
different applications in the healthcare domain. A few re-
In this section, we discuss a few of the state-of-the-art methods searchers have used it for physiological [5, 22, 23] and psy-
for smart healthcare and the recent methods for seizure chological applications [15]. In [10], an emotion-aware cog-
detection. nitive system is proposed based on cloud computing. In [19],
an emotion-aware cognitive system is also proposed, although
2.1 Cognitive healthcare-IoT it uses human facial expression recognition. In [22], another
emotion-aware cognitive system is proposed but uses voice
CHIoT is an advancing field that is creating a plethora of and facial expression recognition.
applications to improve healthcare and revolutionizing the
entire concept of connected healthcare devices. Moreover, 2.2 Smart healthcare
CHIoT has a tremendous scope in providing smart services,
such as remote monitoring of patients; tracking, detection, and Smart healthcare is attracting considerable interest from gov-
generation of alerts; medical equipment operation and control; ernment organizations, private companies, and researchers
and smart pill disbursal. CHIoT enables us to deal with med- from different fields because of its social and economic ben-
ical emergencies in an improved manner and provide rapid efits. Accordingly, numerous studies [18], models [24], and
response anywhere. CHIoT connects with smart sensors, services [19, 25] related to smart healthcare have emerged
which are inside, on the surface, or around a patient’s body; because of the integration of IoT–cloud technologies. In such
and monitors and interprets multimodal health data, including a type of smart healthcare model, medical practitioners, in-
the patient’s physiological and psychological signals, through cluding healthcare staff members, can analyze healthcare data
these smart sensors. Research initiatives have integrated in real time. In another study [5], the author discussed a smart
CHIoT with 5G technology to make it considerably phenom- healthcare framework, in which smart city residents can use
enal in the smart healthcare perspective [15]. Several of the smart devices to find a route to healthcare centers. In another
consumer-based cognitive systems have integrated smart plat- study [19], the author discussed an interconnected smart
forms, such as Microsoft Kinect, to fuse smart cognitive be- healthcare framework involving processing of electronic
havior in healthcare IoT frameworks. The Kinect platform is health records. In [21], the researchers proposed a smart glu-
based on gesture and activity recognition to understand human cose monitoring system for diabetic patients, which involved
behavior cognition. In one of such research initiatives, the daily activities and locations. In [25], the author proposed a
authors [15] proposed a speech emotion recognition-based robot-controlled automatic ambulance to treat patients with
5G cognitive system for providing health services. However, cardiac arrhythmia requiring immediate care. Other studies
 Mobile Netw Appl

have explored medical forgery in smart healthcare [26]. The vision. Therefore, considerable enhancement is needed in
effective use of communication technology, data analytics, deep learning methods applied to EEG analysis and seizure
smart sensors, and cloud resources is crucial for the smart detection.
healthcare environment [27]. Deep CNN is a variant of deep networks that learn local
Our proposed framework for smart healthcare monitoring and spatial features in data using convolutions. Deep CNN
caters to the various needs and challenges discussed in the often has multiple successive convolutional layers [30]. In
literature and provides a solution for deploying an automatic the initial layers, these networks extract features that are spa-
EEG-based seizure detection system. In particular, we use tial and of low level using the raw input. Thereafter, the net-
smart sensing, IoT-aware, and cloud technologies. In the sub- works in the deeper layers progressively extract considerably
sequent section, our seizure detection and classification global, high-level features. CNN has been successfully used
methods are discussed. for such inputs as images, which are 2D. By contrast, the EEG
signal has different properties and comprises dynamic time
2.3 EEG seizure detection series data from electrode recordings on the three-
dimensional scalp surface. The EEG signal has the disadvan-
This section reviews state-of-the-art seizure detection tage of having a low signal-to-noise ratio. That is, this signal is
methods. affected by noise from artifacts that lack task-related informa-
The scalp EEG technique is commonly used for epileptic tion. Therefore, the existing CNN architectures should be
seizure detection, prediction, classification, diagnosis, and on- modified and adapted to the EEG data. We use a novel deep
set detection. The EEG recordings are often analyzed by learning model composed of a deep CNN and stacked
trained neurologists who scan hours of EEG recordings for autoencoder.
characteristic patterns of seizures. This activity can be time- First, we use the deep CNN model to extract features from
consuming, tedious, and expensive even when done for a sin- the raw input data. Thereafter, we use these features as input to
gle patient. Numerous researchers have developed automated the stacked autoencoder. During the supervised training phase
techniques for seizure detection and prediction to address of the deep CNN model, the network parameters are
these issues [28–31]. Automated techniques also face many pretrained for the EEG seizure data set. These features are
problems because of the inherent nature of the EEG signal. eventually used by the stacked autoencoder to learn general
The EEG seizure patterns are different from patient to patient. subject-specific features. In this manner, we are able to
An epileptic seizure pattern in one patient may appear to be a achieve good specificity and accuracy for the cross-patient
normal EEG in another patient. Hence, building a generic dataset.
automated method that functions correctly for every epileptic
patient is relatively a difficult task [29] because of the overlap
between seizure and non-seizure patterns. 3 Proposed cognitive IoT-cloud framework
In several methods [30], EEG signals are transformed into and seizure detection technique
images. In [31], the authors transformed EEG signals into
images by projecting the patient electrodes into 2D. These This section presents the proposed IoT–cloud-based smart
studies employed hand-crafted features for automated seizure healthcare monitoring framework and discusses the EEG-
detection and classification. These techniques are specific be- based seizure detection and classification method.
cause epileptic seizures are nonstationary and the EEG pattern
varies significantly in epileptic patients [32]. 3.1 Cognitive IoT-cloud smart healthcare scenario
Deep learning techniques have recently achieved promis-
ing results [33] in computer vision and speech recognition, The use of smart healthcare systems in an interconnected
thereby proving that automatically extracted features work smart city environment enables residents and medical practi-
better than manual extracted features. Deep learning methods tioners to use smart sensor devices and cloud and cognitive
have also been applied to detect epileptic seizures. Deep neu- IoT aware technologies to access their electronic health re-
ral networks trained with dropout are used in [34] for patient- cords. Through smart wearable technology and communica-
specific epileptic seizure detection. Deep belief networks are tion, patients can regularly update their health-related data.
used in [35] to detect seizures in multichannel and high- The cognitive system can analyze these data in real time and
resolution EEG data. Deep CNN and stacked autoencoders perform the best action to provide services to patients. These
are used in [36] for the analysis and classification of the uploaded data can also be remotely viewed and analyzed by
EEG data. medical staff members who can also provide assistance and
The majority of the deep learning methods discussed pre- advice to the concerned patients. This type of patient-
viously exhibit a good performance but are incomparable to dependent healthcare monitoring is essential for smart
the deep learning models in other fields, such as computer healthcare to achieve its major objectives, such as low
Mobile Netw Appl

healthcare cost, high efficiency, easy accessibility, error-free 5G, or WiFi, to transmit data to the cloud. Once all health-
diagnosis, reduced hospital and medical staff member visits, related data and signals, including EEG, reach the cloud, user
and enhanced quality of life. Accordingly, we present a sce- authenticity is verified by the system and the data are proc-
nario for our proposed smart healthcare monitoring frame- essed by the seizure detection system. Figure 1 shows that the
work for a smart city environment. IoT sensors may include different types of healthcare-related
Residents should use smart city infrastructures and register sensor devices, such as wristbands, smartwatch, wearable sen-
with a smart healthcare service provider to enable the cogni- sors, and headgears; and can measure diverse health-related
tive system and the medical practitioner and staff members to data, such as heart rate, blood pressure, respiratory rate, body
communicate with the service provider and obtain access to a temperature, body movement, ECG, and EEG. These smart
patient’s health record in a remote manner. The person’s loca- sensors’ devices can be worn by patients or may be embedded
tion information is also available in the system and can be in smart city environments, such as smart homes, medical
used in case of an emergency. The smart healthcare service centers, offices, or automobiles. These smart sensor devices
provider has a communication link with all specialized med- can also communicate with each other using short-range com-
ical practitioners, including those involved in epilepsy. Many munication. The LAN interface layer is built upon smart com-
healthcare smart sensors can be used to record the real-time munication protocols for short- to medium-range communica-
psychological and physiological data from patients. These tion and interconnection between devices. These protocols
signals include a patient’s movements, gestures, and facial include Zigbee, Bluetooth, Z-wave, and LoWPAN.
expressions. The cognitive system continuously calculates The hosting layer comprises heterogeneous smart devices,
the patient’s state in real time. The epileptic patient also has such as smartphone, tablets, personal digital assistant, laptops,
to wear a smart EEG sensor, which comes in the form of a or workstations, which collect data to be processed locally.
comfortable and lightweight skull cap. Through sensor elec- These devices have processing capabilities to detect general
trodes, the EEG sensor continuously records the concerned health abnormalities through dedicated applications or pro-
patient’s EEG. When the cognitive system detects that the grams. Health problems can include abnormal blood pressure,
patient is having seizures, the recorded data are sent to the heart rate, or body temperature. These data are eventually sent
cloud for further processing in real time. From the cloud, the to the cloud for further processing through the WAN interface.
data are sent to the healthcare professionals for further analy- The WAN interface is used for long-range communications
sis. The medical staff members can further investigate and using WiFi, 4G, or 5G technologies. Once the data reaches
advise the patient accordingly. The signals are processed con- the healthcare service providers, they can analyze the patients’
tinuously based on the severity of the seizure detection and health records to make any necessary immediate decision.
classification result. The cognitive system sends a warning to Thereafter, the data (e.g., EEG signals) are further processed
the medical staff members and related stakeholders through using the proposed seizure detection and classification system.
the smart healthcare provider. In case of emergencies, a smart After final processing, the result is returned to the medical
ambulance can rush through traffic to provide urgent care to a practitioners for a detailed analysis. In non-emergency cases,
patient. In such cases, ambulances are assisted by a smart the visiting medical centers require time and involve high cost.
traffic light system, which ensures that these vehicles move Hence, we can save money, time, and hospital space by using
easily through traffic. Therefore, the smart city healthcare such a smart healthcare framework.
framework can provide remote, real-time, and critical The cloud comprises the cloud manager, data center, fea-
healthcare services to residents and patients while they remain ture extraction server, detection server, and classification serv-
in their respective locations. er. The cloud manager first authenticates whether a resident is
registered with a smart healthcare provider. The cloud manag-
3.2 System architecture er is also responsible for verifying the identity of all stake-
holders in the smart healthcare system, such as doctors, med-
In our proposed smart healthcare monitoring framework (see ical staff members, hospital representatives, and patients. The
Fig. 1), different types of health-related multimedia and EEG cloud manager also controls the data flow to and from the
signals are obtained using smart IoT sensors. We have a local various servers and manages communication, storage, and
area network (LAN) layer, which comprises short-range smart other resources. The cloud manager sends the data to the cog-
communication devices. This LAN layer acts as an interface nitive engine, which uses multimodal data including EEG,
to transmit the obtained signals, such as EEG, from IoT de- psychological and physiological data and determines whether
vices’ layer to a hosting layer consisting of heterogeneous the patient needs emergency care. The sensor signals also
smart devices, such as a smartphone or laptop. Thereafter, include patient’s movements, gestures, facial expressions to
these smart devices transmit the received data through the know about the patient’s state. The cognitive system then
wide area network (WAN) layer to the cloud. The WAN in- makes a real-time decision based on patient’s state, about the
terface uses smart communication technologies, such as 4G, activities and the medical attention, services to be provided to
 Mobile Netw Appl

Fig. 1 Cognitive IoT-cloud smart healthcare framework

the patient and whether to send data to the deep learning mod- and a resolution of 16-bit is used. Out of the total 686 EEG
ule. If cognitive system believes that the patient is having a recordings, only 198 contain one or more seizures. In most of
seizure, then it makes the decision to inform other stake- the EEG recordings, the seizure activity lasts about 25 s.
holders and the EEG data is sent to a deep learning module.
Deep learning techniques are used to extract features in the
feature extraction server. After the signals are preprocessed, 3.3.2 Input representation
signal processing techniques are used to extract features from
the feature extraction server. The detection server detects and There are many techniques used for the EEG input represen-
classifies seizure data and sends the detection results to the tation in the deep learning models. Since deep learning models
cloud manager. The data, features, results, and other model like convolution neural networks need 2D inputs, therefore, a
parameters are eventually stored in the data center. lot of researchers have converted EEG recordings into images
Healthcare professionals make the final decision on the type and topo-maps [30, 31]. Some deep learning models used
of service to be provided to epileptic patients using the differ- electrode voltage to transform the EEG recordings into topo-
ent types of signal from various smart sensors and the EEG graphical images organized in a time series [31]. However,
classification results. These healthcare service details are there is evidence that EEG signal is correlated over time series
shared with all smart city stakeholders using smart communi- data, [38], therefore in this study we used raw EEG data as
cation that can access and analyze the patient health reports for input, represented as a two-dimensional array in which the in
further care of the residents. width we have the time steps (samples) and in the height there
are all the electrodes. The input is represented as a two-
dimensional array in which the width has all time steps and
3.3 Seizure detection and classification the height has all electrodes.

3.3.1 Dataset
3.3.3 Deep CNN
We used the CHB-MIT dataset for our study, collected at the
Children’s Hospital Boston, which is the largest freely avail- Deep CNN learns local and spatial features in such signals by
able dataset for EEG epileptic seizure data [37]. It has 686 using convolutions and nonlinearity and have the ability to
multiple channel scalp EEG recordings from paediatric pa- represent higher-level features as a combination of lower level
tients who were affected by intractable seizures. The data is features. The pooling layers help the network to represent
recorded from 23 epilepsy patients, which includes 5 males intermediate feature maps in a concise way by retaining the
and 18 females in the age group of 10 to 22 years. The 10–20 most important information using downsampling. Hence deep
international EEG electrode montage system is used to record CNN models are best for end-to-end learning, which use raw
the dataset. There are 969 h of scalp EEG recordings contain- data to extract features automatically. CNN takes raw data as
ing 173 epileptic seizures. The sampling rate used is 256 Hz input and learns spatial features in the initial layers, and
Mobile Netw Appl

Fig. 2 Using sliding window to crop the signal and Deep CNN Architecture

progressively in the deeper layers, and it is able to learn tem- dimension. For activations function we used the Exponential
poral features. linear units as shown below in eq. (3):
Our deep CNN model is based on the architecture [33],
which has convolution and max-pooling blocks followed by
yijk ¼ xijk for xijk > 0 and yijk ¼ exijk −1 for xijk ≤ 0 ð3Þ
fully connected layers. We changed the first convolution layer
to manage the multi-channel EEG input, as suggested in [39].
We tried with a different number of convolution and max-
pooling layers, with different types of training, normalization
exijk
strategies, and activation functions. Since our input is 2D, yijk ¼ ð4Þ
having time and electrodes dimension so we make CNN first ∑D
t¼1 e
xijt

convolve overtime followed by convolution over electrodes.

Convolution layers are followed by the max-pooling layer. The eq. (4) above shows softmax operation or the normal-
Then we have another normal convolution-max-pooling ized exponential function, which is used to represent a prob-
block. The next convolution layer is followed by fully con- ability distribution of D dimensional vector.
nected softmax classifier as shown in Fig. 2. In CNN we have Batch normalization is applied to intermediate outputs of
a convolutional layer which is composed of filters or kernels. layers to set them to unit variance and zero mean so that we
Each of these kernels is slid over the input signal and in this can apply training examples in batches. It is also applied to the
way, the kernel is convolved with the input signal. We also output of convolutional layers before we apply nonlinearity on
have a parameter called stride, which decides the amount of it. Dropout sets randomly some inputs to zero, in each iteration
convolution of the kernel with the input signal. The output for of the training phase. It helps to achieve generalization and
three-dimensional convolution, which is a set of kernel or prevents overfitting. Dropout is used with a 0.5 probability at
feature maps, can be shown with eq. (1) below. the beginning.

3.3.4 Stacked autoencoders

yi0 j0 k 0 ¼ ∑ijk wijkk 0 xiþi0 ; jþ j0 ;k ð1Þ
An autoencoder is usually a three-layer neural network with
x = (x1, …xk) is the input for the network layer where (i, j) an input layer, one hidden layer and an output layer [40], with
are the dimension of the weight filter, (i’, j’) are the location of the output layer having the same number of neurons as the
the input on the 2D map where the weight filter is convolved, input layer in order to reconstruct its own inputs. Therefore,
w = (w1, …wn) is the learned weight, k is the number of chan- autoencoders are unsupervised learning methods. An
nels and k’ is the number of filters. The max-pooling operation autoencoder is trained so that the input x is mapped to the
can be shown with following eq. (2): hidden layer, this stage is called the encoding stage, then the
output of hidden layer z is mapped to the output layer, to
n 0 0
o reconstruct the input, this stage is called the decoding stage.
yijk ¼ max yi0 j0 k : i ≤ i < i þ p; j < j < j þ p ð2Þ
These steps are shown in the following equations.
Where p is called the padding operator,(i, j) are pooling z ¼ σðWx þ bÞ
location in the 2D map, (i’, j’) are the pooling window
 Mobile Netw Appl

Fig. 3 Deep Learning model

using CNN and stacked
autoencoder

3.3.5 Proposed model architecture

0 0
 0 0

x ¼σ W xþb
Even when recording EEG in controlled environments there
are always some differences among subjects and among re-
Where W and W′ are weight matrix, b and b′ are bias vec- cording sessions. Therefore it is difficult to identify common
tors. The functions σ and σ′ are element wise activation func- features for cross-subject recordings. In the case of EEG, since
tions. The weights are said to be tied if we set W = [W′]T. Then the signal to noise ratio is poor, it is sensitive to external and
training the autoencoders helps to minimize reconstruction internal noise. Eye blinking and muscle movement artifacts
error, E(x, x′). can cause a lot of problems in extracting good features.
Therefore we use a novel deep learning model composed of
h  0 i deep CNN followed by stacked autoencoder.
argminW;b E x; x In our model, during the pre-training stage, CNN is
employed to train all the weights and biases in the network
After the autoencoder is trained and we obtain the learned simultaneously. In this supervised learning strategy, the deep
hidden layer features, these features are used for classification CNN maps each of the time steps in the input data to one of the
tasks. We can also use these features as input to the next layer given classes (output labels) and the error is used to train the
in stacked autoencoder. A stacked autoencoder consists of network parameters. Hence the important features are learned,
multiple layers of autoencoders. Stacking autoencoder is the noise artifacts in the EEG dataset are ignored, and it does not
greedy layer-wise training technique for pre-training a deep over-fit to noise in the data.
neural network. It trains each autoencoder layer in turn. The We used cropped training technique that uses sliding win-
encoding and decoding step in autoencoder is run for each dows within the input. By using this technique we achieve a
layer of the stacked model while the parameters of all other large number of training examples. We used this technique as
layers are frozen. After training step is complete then a super- the CHB-MIT database has only 173 seizure events, hence
vised fine-tuning step is also performed in which all the net- especially for seizure class we have limited training data,
work parameters are trained simultaneously using and hence there are chances that our deep CNN model could
backpropagation algorithm [40], to improve results. In this overfit. It not only increases stabilize the dataset but also in-
study, we use stacked autoencoder with 2 hidden layers. creases classification accuracy. Figure 2 illustrates the use of
the sliding window in the proposed architecture.
Figure 3 shows the overall architecture of the deep learning
model. Since the sampling rate for the dataset is 256 Hz there-
Table 1 Structure of CNN fore in a 2-s window we have about 500 samples for all chan-
nels, which are presented to the first convolution layer. The
Layers Type

1 Convolution (10 × 1, 20 filters) Table 2 Structure of Stacked Autoencoder

2 Convolution (20 × 23, 20 filters)
Layers Type
3 Max-pooling (2 × 1, stride 2)
4 Convolution (10 × 20, 40 filters) 1 Input layer (1000 neurons)
5 Max-pooling (2 × 1, stride 2) 2 Autoencoder (500)
6 Convolution (10 × 40, 80 filters) 3 Autoencoder(200)
7 Fully connected(2 classes) 4 Softmax, Fully connected(2 classes)
Mobile Netw Appl

Fig. 4 Sensitivity: Deep CNN

model comparison with Deep 120
CNN + Stacked Autoencoder 100
80
60
40
20
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Sensivity (%) Deep CNN

Sensivity (%) Deep CNN + Stacked Autoencoder

first convolution layer uses 20 filters of size 10 × 1, which are general subject-specific features. These learned features from
slid over every sample, therefore this convolution is over time CNN are used individually for each patient but in cross trial
for each electrode at a time. The second convolution is over all manner. This is achieved by having the stacked autoencoder to
electrodes or channels having 20 filters of size 20 × 23 as there reconstruct a different trial of the same patient. In this way, we
are 23 channels. Then within this block, we have max pooling also increase the training set as there are many trials available
layer having a filter of size 2 × 1 with stride 2. After this, we for each patient. After training the stacked autoencoders using
again apply a convolution and max-pooling block having 40 trials from the same patient, we again trained them with trials
filters of size 10 × 20 and max-pooling filter of the same size from different patients but belonging to the same class. In this
of 2 × 1 with stride2. In the last block, we apply convolution way were able to achieve good sensitivity and at the same time
without max-pooling with 80 filters of size 10 × 40 and finally good accuracy for the cross-patient dataset.
a fully connected layer with softmax function to give the out-
put probability for the two classes.
The structure of CNN and stacked autoencoder are shown 4 Results
in Table 1 and Table 2, respectively. In our model the training
takes place one patient at a time, therefore the softmax classi- We used PyTorch deep learning framework to build deep
fier gives us patient-specific probability distribution over out- CNN-stacked autoencoder model and used GTX 960 GPU
put classes, which is 2 here. card on Intel Pentium i7 machine. We tested the performance
We first use deep CNN model as explained earlier for the of the model using sliding window strategy to increase the
extracting features from the raw input data, then these features dataset size. Since we took 2 s window as an input to our
are used as input to the stacked autoencoder. During the su- model for testing the output label was the mean of the labels
pervised training phase of deep CNN model, the network pa- for all the events in a single trial.
rameters are pre-trained for the EEG seizure dataset. Then There are improvements in the result when using
these features are used by the stacked autoencoder to learn autoencoders with CNN as compared to only CNN method.

Fig. 5 Accuracy: Deep CNN

model comparison with Deep 105
CNN + Stacked Autoencoder
100

95

90

85

80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Accuracy (%) Deep CNN

Accuracy (%) Deep CNN + Stacked Autoencoder
 Mobile Netw Appl

Table 3 The performance of some methods for seizure detection

Study Problem Design choices Accuracy Sensitivity

[37] Patient Specific Seizure Detection and Prediction SVM 96% –

[31] Cross patient Epileptic Seizure detection CNN + RNN 95% 85%
[42] General seizure detection across different subjects without prior knowledge k-NN 93% 88%
[43] Patient Specific Seizure Detection stacked autoencoders – 100%
[44] Context-learning based EEG analysis for seizure detection SVM 88.8% –
[45] DWT to improve accuracy and to reduce computational cost of seizure detection DWT 92.30% –
Our Model Patient Specific and Cross Patient Seizure Detection and Classification CNN + stacked autoencoders 99.20% 93.5%

It can be seen that the average values for sensitivity are close 5 Conclusion
to 93% and average recognition accuracy is 99.2%. For some
patients, the sensitivity was less as is the case for patient 5 and This study proposes a cognitive smart healthcare monitoring
6. It shows that the EEG is highly variable across patients. framework based on the integration of IoT and the cloud. An
When we applied cropped training strategy on the CHB- epileptic seizure detection and classification system inside the
MIT dataset, the deep CNN-stacked autoencoder model gave framework is built using deep CNN and stacked autoencoders.
us an overall accuracy of 99.5%, which is better than the state The proposed system also uses other healthcare smart sensors
of the art on the same dataset. The sensitivity of the system to record the psychological and physiological signals and
was comparable with state of the art, which means this deep transmit them to the cloud via the Internet. These signals in-
CNN- stacked autoencoder architecture can be used as a clude patients’ movements, gestures, and facial expressions to
patient-specific as well as cross-patient seizure detector. The determine their state. Thereafter, the cognitive system makes a
sensitivities and the accuracies are shown in Figs. 4 and 5, real-time decision on the activities and the medical attention
respectively. and services to be provided to patients based on their state and
Our result was better than the state of the art when com- whether to send the data to the deep learning module. If the
pared with patient-specific approaches as well as cross- cognitive system believes that a patient is having a seizure,
patient techniques. Researchers in [37] built a patient- then it makes the decision to inform other stakeholders and
specific detector for detection of epilepsy seizures and re- sends the EEG data to the deep learning module. Signal pro-
ported an accuracy of 96%, which is less than our model’s cessing and seizure detection occur in the cloud and signals
accuracy. Our model also got better accuracy and sensitivity are classified as seizure or non-seizure with a probability
than REVEAL algorithm discussed in [41], which had good score. The results are transmitted to medical practitioners or
results for cross-patient seizure detection. One of the stud- other stakeholders who can monitor the patients. In critical
ies [31] used recurrent and convolution deep learning meth- cases, these medical practitioners make the appropriate deci-
od for seizure detection reported a sensitivity of 85% for sions to assist patients.
cross-patient detection, is also less than our model’s sensi- This study also evaluated deep CNN and stacked
tivity of 90%. In [42], they used supervised k-NN classifier autoencoders for EEG epileptic seizure detection and classifi-
for seizure detection across subjects and reported a classi- cation. Deep CNN is able to extract a robust and wide range of
fication accuracy of 93% and sensitivity of 88%, which is features from EEG, while stacked autoencoders facilitate the
also less than what we report. In [43], the authors used removal of noise artifacts from the signal. Deep CNN and
stacked autoencoders with logistic classifiers for patient- stacked autoencoders also increase the overall accuracy of
specific seizure detection on the CHB-MIT dataset, to the system and assist in building a generic cross-patient
achieve 100% sensitivity. In [44], authors proposed classifier.
context-learning for seizure detection by extracting hidden Experiments showed that the proposed system is accu-
inherent features within EEG fragments and the temporal rate and sensitive for cross-patient seizure detection.
features from EEG contexts. They used SVM to achieve an However, we should address several issues before such
accuracy of 88.8%. Discrete Wavelet Transform (DWT), framework can be operative in a secure manner. These is-
was applied to selected frequency bands, in [45] to attain sues include interoperability, availability, scalability, and
an accuracy of 92.30% on the CHB-MIT dataset. Table 3 security. We attempted to solve the issues of interoperabil-
compares accuracies and sensitivities of the proposed ity in the proposed framework. Security should be provided
CNN-stacked autoencoder model with the discussed state at the end of the healthcare service provider. For future
of the art models, showing our method outperforms them in research direction, we can create a cognitive framework to
accuracy and is comparable in terms of sensitivity. handle big data of EEG in the cloud.
Mobile Netw Appl

Acknowledgements This work is supported by the Deanship of 22. Muhammad G, Alsulaiman M, Amin SU, Ghoneim A, Alhamid
Scientific Research at King Saud University, Riyadh, Saudi Arabia MF (2017) A facial-expression monitoring system for improved
through the Research Group Project No: RG-1436-016 healthcare in smart cities. IEEE Access 5(1):10871–10881
23. Chen M, Song J, Lai C, Hu B (2016) Smart clothing: connecting
human with clouds and big data for sustainable health monitoring.
Mob Netw Appl 21(5):825–845
References 24. Vlacheas P et al (2013) Enabling smart cities through a cognitive
management framework for the internet of things. IEEE Commun
1. Yin Y, Zeng Y, Chen X, Fan Y (2016) The internet of things in Mag 51(6):102–111
healthcare: an overview. Journal of Industrial Information 25. Piniewski B et al (2010) Empowering healthcare patients with
Integration 1:3–13 smart technology. Computer 43(7):27–34
2. Chen M, Zhang Y, Qiu M, Guizani N, Hao Y (2018) SPHA: smart 26. Ghoneim A, Muhammad G, Amin SU, Gupta B (2018) Medical
personal health advisor based on deep analytics. IEEE image forgery detection for smart healthcare. IEEE Commun Mag
Communications 56(3):164–169 56(4):33–37
3. Hu, L et al (2016) Internet of Things Cloud: Architecture and 27. Muhammad G (2015) Automatic speech recognition using inter-
Implementation. IEEE Commun Mag 54(12-Supp):32–39 laced derivative pattern for cloud based healthcare system. Cluster
4. Chen, M et al (2018) Edge-CoCaCo: Toward Joint Optimization of Computing (Springer) 18(2):795–802
Computation, Caching, and Communication on Edge Cloud. IEEE 28. Hills M (2014). Seizure detection using FFT, Temporal and Spectral
Wireless Commun 25(3):21–27 Correlation Coefficients, Eigenvalues and Random Forest.
Technical Report. Github
5. Muhammad G, Rahman SKMM, Alelaiwi A, Alamri A (2017)
29. Park Y, Luo L, Parhi KK, Netoff T (2011) Seizure prediction with
Smart health solution integrating IoT and cloud: a case study of
spectral power of EEG using cost-sensitive support vector ma-
voice pathology monitoring. IEEE Commun Mag 55(1):69–73
chines. Epilepsia 52:1761–1770. https://doi.org/10.1111/j.1528-
6. Chen M, Herrera F, Hwang K (2018) Cognitive computing: archi-
1167.2011.03138.x
tecture, technologies and intelligent applications. IEEE Access 6:
30. Parvez MZ, Paul M (2015) Epileptic seizure detection by exploiting
19774–19783
temporal correlation of electroencephalogram signals. IET Signal
7. Hossain, MS (2016) Patient State Recognition System for Processing 9(6):467–475. https://doi.org/10.1049/iet-spr.2013.
Healthcare Using Speech and Facial Expressions. J Medical 0288
Systems 40(12):272:1–272:8 31. Thodoroff P, Pineau J, Lim A (2016). Learning robust features
8. Hossain MS, Muhammad G (2016) Audio-visual emotion recogni- using deep learning for automatic seizure detection, Proceedings
tion using multi-directional regression and Ridgelet transform. of the 1st Machine Learning for Healthcare Conference. PMLR
Journal on Multimodal User Interfaces 10(4):325–333 56:178–190
9. Hossain MS et al (2015) Audio–visual emotion-aware cloud gam- 32. Panayiotopoulos, CP (2010) A Clinical Guide To Epileptic
ing framework. IEEE Transactions on Circuits and Systems for Syndromes And Their Treatment. Springer-Verlag, London, pp 654
Video Technology 25(12):2105–2118 33. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classifica-
10. Zanella A (2014) Internet of things for smart cities. IEEE Internet tion with deep convolutional neural networks. In: Pereira F, Burges
Things J 1(1):22–32 CJC, Bottou L, Weinberger KQ (ed) Advances in neural informa-
11. Hossain MS, Muhammad G (2016) Cloud-assisted industrial inter- tion processing systems 25, pages 1097–1105. Curran Associates,
net of things (IIoT)-enabled framework for health monitoring. Inc.. URL http://papers.nips.cc/paper/ 4824-imagenet-
Comput Netw 101:192–202 classification-with-deep-convolutional-neural-networks.pdf
12. Chen M, Li W, Hao Y, Qian Y, Humar I (2018) Edge cognitive 34. Pramod S, Page A, Mohsenin T, Oates T (2014) Detecting epileptic
computing based smart healthcare system. Futur Gener Comput seizures from EEG data using neural networks. arXiv preprint arXiv
Syst. https://doi.org/10.1016/j.future.2018.03.054 1412:6502
13. World Health Organization (2017) Available: http://www.who.int/ 35. Turner JT, Page A, Mohsenin T, Oates T (2014) Deep belief net-
mediacentre/factsheets/fs999/en/ works used on high resolution multichannel electroencephalogra-
14. Litt B, Echauz J (2002) Prediction of epileptic seizures. Lancet phy data for seizure detection. In 2014 AAAI Spring Symposium
Neurol 1:22–30. https://doi.org/10.1016/S1474-4422(02)00003-0 Series
15. Chen M, Yang J, Hao Y, Mao S, Hwang K (2017) A 5G cognitive 36. Li D, Wang G, Song T, Jin Q (2016) Improving convolutional
system for healthcare. Big Data Cogn Comput 1(1):2 neural network using accelerated proximal gradient method for an
epilepsy diagnosis. In Control (CONTROL), 2016 UKACC 11th
16. Qian Y, Chen M et al (2018) Secure enforcement in cognitive in-
International Conference on. IEEE, 1–6
ternet of vehicles. IEEE Internet of Things Journal 5(2):1242–1250
37. Shoeb A (2009) Application of Machine Learning to Epileptic
17. Hossain MS, Muhammad G, Al-Qurishi M (2018) Verifying the Seizure Onset Detection and Treatment. Ph.D. thesis,
images authenticity in cognitive internet of things (CIoT)-oriented Massachusetts Institute of Technology
cyber physical system. MONET 23(2):239–250 38. Canolty RT, Edwards E, Dalal SS, Soltani M, Nagarajan SS, Kirsch
18. Zhang M et al (2012) Cognitive internet of things: concepts and HE, Berger MS, Barbaro NM, Knight RT (2006) High gamma
application example. Int J Comput Sci Issues 9(3):151–158 power is phase-locked to theta oscillations in human neocortex.
19. Hossain MS, Muhammad G (2017) Emotion-aware connected Science 313:1626–1628
healthcare big data towards 5G. IEEE Internet of Things Journal 39. Schirrmeister RT, Springenberg JT, Fiederer LDJ, Glasstetter M,
5(4):2399-2406 (2018) Eggensperger K, Tangermann M, Hutter F, Burgard W, Ball T
20. Hossain MS, Muhammad G, Alamri A (2017) Smart healthcare (2017) Deep learning with convolutional neural networks for
monitoring: a voice pathology detection paradigm for smart cities. EEG decoding and visualization. Hum Brain Mapp 38:5391–
Multimedia Systems. https://doi.org/10.1007/s00530-017-0561-x 5420. https://doi.org/10.1002/hbm.23730
21. Samani H, Zhu R (2016) Robotic automated external defibrillator 40. Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy
ambulance for emergency medical service in smart cities. IEEE layer-wise training of deep networks. Adv Neural Inf Process Syst
Access 4:268–283 19:153
 Mobile Netw Appl

41. Wilson S, Scheuer M, Emerson R, Gabor A (2004) Seizure detec- and Biology Society, Chicago, IL, pp. 4184–4187. https://doi.org/
tion: evaluation of the Reveal algorithm. Clin Neurophysiol 10: 10.1109/EMBC.2014.6944546
2280–2291 44. Xun G, Jia X, Zhang A (2015) Context-learning based electroen-
42. Fergus P, Hussain A, Hignett D, Al-Jumeily D, Abdel-Aziz K, cephalogram analysis for epileptic seizure detection, 2015 IEEE
Hamdan H (2016) A machine learning system for automated International Conference on Bioinformatics and Biomedicine
whole-brain seizure detection. Applied Computing and (BIBM), Washington, DC, 2015, pp. 325–330. https://doi.org/10.
Informatics 12(1):70–89, ISSN 2210-8327. https://doi.org/10. 1109/BIBM.2015.7359702
1016/j.aci.2015.01.001 45. Chen D, Wan S, Xiang J, Bao FS (2017) A high-performance sei-
43. Supartak A, Li L, Guo Y (2014) Feature extraction with stacked zure detection algorithm based on discrete wavelet transform
autoencoders for epileptic seizure detection, 2014 36th Annual (DWT) and EEG. PLoS One 12(3):e0173138. https://doi.org/10.
International Conference of the IEEE Engineering in Medicine 1371/journal.pone.0173138