Eg sensors (nb Article LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes Sakorn Mekruksavanich !® and Anuchit Jitpattanakul 2“ Gates Upaates Jiptanal, A LSTM Networks Using Sarthe Dat for Sensor Based Human Activity Recognition in Set Homes Ses 202121, 1696. tp oon? 103990/s01051636 Academie Editors: Giowann Pau ae Tsun You Recsved: 5 February 2021 ‘Accepted: 22 February 221 Pabst 26 Febrry 2021 Publisher's Note: MDP stays neutral swith rondo jaro cam in published maps and ineittiona ft This attic isan open acess atic iste under the terms and Atibation (CC BY) lense tps reatvecommonsorg/eenses/by/ 40), Department of Computer Engineering, School of Taformation and Communication Technology, University of Phayao, Phayao 56000, Thailand; sakorn.medup.acth Intelligent and Nonlinear Dynamic Innovations Research Center, Department of Mathematics, Faculty of Applied Science, King Mongkut's University of Technology North Bangkok, Bangkok 10800, Thailand > Coesespondence: [email protected] acth Abstracts Human Activity Recognition (HAR) employing inertial motion data has gained consid ‘erable momentum in recent years, both in research and industrial applications. From the abstract perspective, this has been driven by an acceleration in the building of intelligent and smart environ- ‘ments and systems that cover al aspects of human life inclucling healthcare, sports, manufacturing, ‘commerce ete. Such environments and systems necessitate and subsume activity recognition, aimed. at recognizing the actions, characteristics, and goals of one or more individuals from a temporal series of observations streamed from one oF more sensors. Due to the reliance of conventional Machine Learning (ML) techniques on handcrafted features in the extraction pro suggests that deep-learning approaches are more applicable to automated feature extraction from, raw sensor data, In this work, the generic HAR framework for smartphone sensor data is proposed, based on Long Short-Term Memory (LSTM) networks for time-series domains. Four baseline LSTM networks are comparatively studlied to analyze the impact of using different kinds of smartphone sensor data, In addition, a hybrid LSTM network called 4-layer CNN-LSTM is proposed to improve recognition performance. The HAR method is evaluated on a public smartphone-based dataset of UCLHAR through various combinations of sample generation processes (OW and NOW) and. validation protocols (10-fold and LOSO cross validation). Moreover, Bayesian optimization tech- ‘niques are used in this study since they are advantageous for tuning the hyperparameters of each, LSTM network. The experimental results indicate that the proposed 4-layer CNN-LSTM network performs well in activity recognition, enhancing the average accuracy by up to 2.24% compared to prior state-of-the-art approaches. ss, current research, Keywords: HAR; LSTM; deep learning; time-series data; smartphone sensor, feature extraction, 1. Introduction In the present day, with many countries experiencing aging populations, more elders tend to live alone and are often unable to receive care from family members. It is a recognized fact that elders are subject to falls and accidents when carrying out the activities of daily life. To help single elders live safely and happily, through the Internet of Things (IoT), smart home equipment has been developed to identify the daily activites of elders. In fact, activity recognition isa vital objective in a smart home situation [1]. The ML society is intrigued by Human Activity Recognition (HAR) [2] due to its availability in real-world applications such as fall detection for elderly healthcare monitoring, exercise tracking in sport science [3,4], surveillance systems [5-7], and preventing office work syndrome [8]. Currently, HAR is becoming a challenging research topic due to the accessibility of sensors in wearable devices (e.g., smartphone, smartwatch, ete.) which are cost-effective and consume less power, including live cascading of time-series data [9] ‘Sonsors 2024, 21,1636. hutps://dok.org/10.3390/<21051636 -ntps//wwwmdpi.com/journal/sensorsSensors 2021, 21,1636 20125 Recent research, involving both dynamic and static HAR, uses sensor data collected from wearable devices to better understand the relationship between health and behavioral biometric information [10,11]. The HAR methods can be categorized into two categories according to data sources: visual-based and sensor-based [12]. With visual-based HAR, video or image data are recorded and processed using computer vision techniques [13]. The authors [14] proposes a new approach for identifying sport-related events with video data based on fusion and multiple features. This work achieves high recognition rates in the video-based HAR. Whereas sensor-based HAR works on time-series data captured froma wide range of sensors embedded in wearable devices [15,16]. In [17], the authors researches a context-aware HAR system and notices that an accelerometer is mostly adequate to detect simple activities including walking, sitting, and standing. As adding gyroscope data, the sstem recognition performance will be increased for employing more complex activities such as drinking, eating, and smoking. However, there are some works to explore other sensors. Fu et al. [18] indicate to improve a HAR framework by using a sensor data of air pressure system along with inertial measurement unit (IMU). This HAR model shows at least 1.78% higher recognition performance than others that is not appliable to sensor data. In the last decade, there isa generational shift in HAR study from device-bound strategies to device-free approaches. Cui et al, [19] introduce a WiFi-based HAR framework by using channel state data to recognize common activities, However, WiFi-based HAR is capable of detecting basic behaviors only, such as running and standing. The cause is that CSI cannot have enough knowledge to understand dynamic events [20]. Sensor-based HAR is becoming more commonly used in smart devices since, with the advancement of pervasive computer and sensor automation, smartphones and their privacy are well protected. Therefore, smartphone sensor-based HAR is the focus of this study. As a wearable device, modern smartphones are becoming increasingly popular. Furnished with an assortment of implanted sensors such as accelerometers, gyroscopes, Bluetooth, and ambient sensors, smartphones also allow researchers to study the activities of daily life. Sensor-based HAR on a device can be considered as an ML model, built to constantly track the actions of the user, despite being connected to a person’s body. Traditional methods have made major strides through the implementation of state-of-the-art deep-learning techniques, including decision tree, naive Bayes, support vector machine (21, and artificial neural networks [22]. Nevertheless, these traditional ML methods may eventually focus on heuristic, handcrafted feature extraction, which is typically constrained by human domain expertise. However, the efficiency of traditional ML methods is constrained in terms of sorting accuracy and other measurements. Here, Deep Learning (DL) methods are employed to moderate the previously men- tioned limitations. Using multiple hidden layers instead of manual extraction through human domain knowledge allows raw sensor data features to be learned spontaneously The mining of appropriate in-depth, high-level features for dealing with complex issues such as HAR is facilitated by the deep architecture of these approaches. These DL ap- proaches are now being used to construct a resilient smartphone-based HAR [23,24] ‘The Convolutional Neural Network (CNN) is a potential DL approach which has achieved favorable results in speech recognition, image classification, and text analysis [25] When applied to time-series classification-related HAR, the CNN has superiority over other conventional ML approaches, cue to its local dependency and scale invariance [26]. Studies ‘on one-dimensional CNNs have shown that these DL models are more effective in solving the HAR problem with performance metrics than conventional ML models [27]. Due to the temporal dependency of sensor time-series data, LSTM networks are introduced to tackle the issue. The LSTM network can identify relationships in the temporal knowledge dimension without combining the time steps as in the CNN network [25] Ullah et al. [29] proposed a Stacked LSTM network, trained along with accelerometer and gyroscope data, inspired by the emerging DL techniques for sensor-based HAR. ‘These researchers found that recognition efficiency could be enhanced using the Stacked ISTM network to repeatedly extract temporal features. Zhang et al. [30] proposed aSensors 2021, 21,1636 30125 Stacked HAR model based on an LSTM network. The findings revealed that with no extra difficulty in training, the Stacked LSTM network could enhance recognition accuracy. Better recognition performance was achieved by combining the CNN network with the LSTM network, based on the study by Mutegeki et al. [51] who used the robustness of CNN network feature extraction while taking advantage of the LSTM model for the classification of time series. To provide promising results in recognition performance, Ondoiez and Roggen [32] combined the convolutional layer with LSTM layers. In order to capture diverse data during training, Hammerla et al. [33] compared different deep neural networks in HAR, including CNN and LSTM, and significantly improved the performance and robustness of recognition. However, existing practices have their own weaknesses and involve various sample generation methods and validation protocols, making them unsuitable for comparison. To better understand LSTM-based networks for solving HAR problems, this research aims to study LSTM-based HAR using smartphone sensor data. Five LSTM networks were comparatively researched to evaluate the impact of using different kinds of smartphone sensor data from a public dataset called UCI-HAR. Moreover, Bayesian optimization is utilized to manipulate LSTM hyperparameters. Therefore, the primary contributions of this research are as follows: + A ayer CNN-LSTM is proposed: a hybrid DL network consisting of CNN lay- ers and an LSTM layer with the ability to automatically learn spatial features and temporal representation. * The various experimental results demonstrate that the proposed DL network is suit- able for HAR through smartphone sensor data * The proposed framework can improve the recognition operation and outperform other baseline DL networks. The remainder of the paper is structured as follows. Section 2 provides details of the preliminary concept and background theory used in this study. Section 3 presents the proposed HAR framework for obtaining smartphone sensor data. Section 4 shows the experimental conditions and results. The derived results are then discussed in Section 5. Finally, Section 6 presents the conclusion. 2. Theoretical Background 2.1. HAR from Sensor Data Generally, HAR systems aim to (1) determine (both online and offline) the ongoing actions/activities of a person, a group of persons, or even a crowd, based on sensory observation data; (2) determine personal characteristics such as the identity of people in a given space, gender, age, etc.; (3) knowledge of the context within which the observed activities are taking place [34]. Therefore, general human activities can be determined as a set of actions performed by a person over a certain period according to a given protocol It is assumed that a person performs some types of activities by applying a predefined activity set A [26] A= {ay 03,05,...,0m} a where m represents the number of activity classes. Then, a data sequence from sensors reading (s) gathers the activity data 5 = {di dads,...sdn} @ where d; represents the data reading from sensor at time f of a number of sensor data n, while 1 > m. ‘The HAR work is to construct the recognition function F for predicting the activity sequence based on the data reading s of a sensor. F(s) = (Ayes. rAighea, € A, @)Sensors 2021, 21,1636 40125 while a sequence of actual activity is mean as, F(s) = {aj 03,03, }ap € A. “ ‘Commonly, a HAR system is developed in five fundamental steps: Data collection, ‘Segmentation, Feature extraction, Model training, and Classification, as shown in Figure a. (a Sensorbased HAR using Machine Learn Approaches lection —P Segmentation —b Deep Feature Enraston & Model Training —P Classification Te waning (©) Sensor-based HAR using Deep earning Approaches Figure 1. Sensor-based HAR approaches using (a) ML technique and (b) the DL technique. ‘A detailed explanation of each process is designated in the following. The first step in the HAR process is to continuously capture sensor data from a wearable device while the participant is performing predefined activities, The raw sensor data obtained from wearable devices must be commonly pre-processed to remove unwelcome noise. Since the sensor data is represented in time-series format, it must be divided into segments of equal length with a defined window size and a proportion of overlap. Feature extraction is deemed to be the most important step because it defines the operation of the recognition model, and either conventional ML algorithms or DL techniques may be used in this step. Using conventional ML in the time and frequency domain, experts can carefully extract heuristic or handcrafted functions. Numerous time-domain characteristics are available, such as correlation, max, min, mean, standard derivation, etc. A range of frequency-domain characteristics is also available, such as energy, entropy, time between peaks, etc. However, in both domains, handcrafted features have certain drawbacks since they are based on knowledge of the domain and human condition, Such expertise could assist with a specific issue in a unique setting, but cannot be extended to include distinct parameters with the same problem. Moreover, human experience is specifically employed to derive handmade characteristics [35], such as statistical evidence, but refuses to differentiate between events With identical patterns such as standing and sitting behaviors. Some studies have used the methodological approach in ML to construct HAR on smartphones [36-38] DL can help to avoid the drawbacks encountered with role extraction in traditional ML [39]. Figure Ib explains how DL with multiple forms of networks can be applied to HAR In the DL approach, the feature extraction and model training operations are concur-Sensors 2021, 21,1636 50125 rent. Whereas in the traditional ML approach, the functions can be learned dynamically through the network rather than being individually assembled. 2.2, LSTM Networks Nowadays, LSTM networks [40] are giving an impressive performance across diverse temporal schemes, The LSTM is one of the expancting Recurrent Neural Networks (RNNS). Afterward, their remarkable architecture which actions the dividing gradient problems, LSTM is satisfying at dealing with time-series classification issues. The input set is defined as X = x0, %1, X27 -++) tt Xep1. oe], the output set as Y= [yor Vi Yor ver Yo Mesto vel and the hidden layers as H = (lta ft tay oss fits hey «ele Then, U, W, and V represent the weight metrics of each layer. U represents the values of the weight metrics from the input layer to the hidden layer, W represents the values of the ‘weight metrics from the hidden layer to another hiclclen layer, and V represents the values of the weight metrics from the hidden layer to the output layer. The computing mechanism of the LSTM network can hereinafter be summarized. The input data is processed and transformed to the hidden layer using a matrix transformation, accompanied by the hidden layer’s data in the last step. The output data of the hidden layer then passes through an activation function to become the concluding value in the output layer. These detailed processes are illustrated in Figure 2 @ ®@ @ @ =) + Figure 2. The unfold architecture of one-layer standard LSTM, Results of hidden layers and output layers can be formally determined as tanh(Ux; +H) while n= uu ) tanh(Ux; + Wh, +04) while i=1,2,3,.. ye = tanh( Vij +69) while i=0,1,.. 6 where X = bt, 81,2) .00/%e p41 is the input set In the DL technique, RNNs can predict the current time output based on prior informa- tion, However, Bengio et al. [41] inform that RNN networks can recognize the data for only a moment, owing to the dissolving gradient issue. While the deep network backpropaga- tion technique is applied, gradients will be dissolved if permitting gradients to flow deeply are not taken, Hochreiter and Schmidhuber [12] introduced a new neuron into the RNN family, named LSTM, to tackle the problem of long-term dependency. In comparison to the input combination and processing used in RNNs, LSTMs gain the appropriate architecture for recognizing data as an input gate for longer. A forget gate compares the inner memory with new data to overwrite it, This process allows gradients to flow efficiently through time. The input gate, forget gate, output gate, and a memory cell of LSTM (defined asi, f 0 and C, respectively) are arranged to manipulate the data which should be distemembered, recognized, and restored as described in Figure 3. The gating technique is chosen to carrySensors 2021, 21,1636 6.0825 the required data. This approach consists of both an activated function (sigmoid function) and an element-wise multiplication function. The output value should be within [0, 1] to enable the multiplication to proceed, and subsequently to allow data to flow (or not). Satisfactory operation can be achieved by assigning the related initialize gates a value of 1 (or close to 1), so training at the beginning is not diminished. Individual criterion in the LSTM neuron at point ¢ can be described as follows, Figure 3. The structure of an LSTM neuron The forget gate fi is accountable for gathering prior data, Then, the recurrent input Jy and current input x; are multiplied by their weights for being input of a sigmoid function: o(x) = (1-+e°*) 1. The output fr is a number within [0, 1] which is multiplied with cell state (cy). If value output f; is 1, the LSTM will hold this new data, otherwise, if f= 0 the LSTM will disremember absolutely this data fu = o(Upre + Wl a + bp) 0 After that, the input gate i+ consists of a sigmoid function. It presents output if is dlefines the ongoing vale to restore the LSTM cell: ir = (Ui + Why +b) 6 ‘The state gate g; builds different values of a vector that are consolidated with a status restore Sr = tanh(Ugxy + Weliy_1 + bg) e) The output gate o; defines data from the cell state that should appear instantly and associates with previous data: oy = o(Uoxs + Woltt-1 +b) (10) “The update state cy consists of disappeared data what is to be forgotten =f Voi Pi Qs a) The hidden output Jt, of LSTM composes of short and long terms with: Inn = 0, @ tann(cr) (2) ‘The computational mechanism of LSTM cell is detailed in Equations (7}-(12). Firstly, there is a requirement to disremember previous data that relates to the forget gate. The next process is to define new suitable data to hold in memory with an input gate. The old.Sensors 2021, 21,1636 70125 cell state, c)_1, to the latest cell state, cy are to be restored as possible. The last step, proper data are determined to be output of the layer above with an output gate. Recent research studies have proposed a variety of LSTM networks to tackle the problem of time-series classification in HAR, One such network has a simple LSTM con- figuration called Vanilla LSTM to differentiate it from deeper LSTMS and the suite of more elaborate configurations. This original LSTM architecture was defined by [12], and will give good results on most small sequence prediction problems. The Vanilla LSTM is defined as: the input layer which is fully connected to the hidden layer and output layer of LSTM. The Vanilla LSTM is proposed to overcome the HAR problem [45]. Later, different architecture-based LSTM networks were proposed to solve the HAR problem such as deep LSTM networks called stacked-LSTMs [29], hybrid LSTM networks called CNN-LSTM; combining the CNN with the LSTM [31], mixed LSTM networks called ConvLSTM [4], and bidirectional LSTM networks called Bidir-LSTM [28,45] Proposed Methodology The proposed LSTM-based HAR framework in this study enables the sensor data captured from the smartphone sensor to classify the activity performed by the smartphone user. Figure 4 illustrates the overall methodology used in this study to achieve the research goal. To enhance the recognition efficiency of LSTM-based DL networks, the proposed TSTM-based HAR is presented. Raw sensor data is split into two main subsets during the first stage: raw training data and test data. In the second stage of model training and hyperparameter tuning, the raw training data is further split into 75% for training and 25% for validating the trained model. Five LSTM-based models are tested by the validation data, and the Bayesian optimization approach then tunes the hyperparameters of the trained models. Finally, the hyperparameter-tuned models will be checked against the test results and their recognition performance compared. Figure 4. The proposed! framework of LSTM-based HAR, 3.1. UCLHAR Smartphone Dataset ‘The recommended system in this work uses UCT human behavior recognition through a mobile dataset [36] to monitor community activities. Activity data gained from 30 par- ticipants of varying ages, races, heights, and weights (aged between 18 and 48 years) was included in the UCI-HAR dataset. While holding a Samsung Galaxy $-I smartphone (Su- won, Korea) at waist level, the subjects carried out everyday tasks. Each person conducted six tasks (i.e,, walking, walking upstairs, walking downstairs, sitting, standing, and lying down), The combined tri-axial values of the smartphone accelerometer and gyroscope were used to record sensor data, while the six preset tasks were performed by each of theSensors 2021, 21,1636 80825 participants. At a steady rate of 50 Hz, tri-axial values of linear acceleration and angular velocity data were obtained. A detailed description of the UCI-HAR dataset is provided in ‘Table 1. Figures 5 and 6 show the accelerometer and gyroscope data samples, respectively. Table 1, Description of UCHHAR dataset Activity Abbreviates Description No. of Samples Walking Wa Participant walks horizontal forward in a direct position 172 Walking (Upstairs) Wo Participant walks upstairs 154 Walking (Downstairs) Wd Participant walks downstairs 1406 Sitting si Participant sits on a chair W777 Standing st Participant stands inactive 1906 Laying la Participant sleeps or lies down 1944 Walking asx —y —2 Waking Upstairs ans:—x —y —2 Walking Downstarsais:—x — 5 a § «| ie ‘Acceleration [mis] ee ee Figure 5. Accelerometer data from UCI-HAR dataset, = Walking Walking Upstairs axis: —x — y — gy : pileup) Sytnmncays g 3 ! 3 B i ee tg ee Cee eee eee eee Figure 6. Gyroscope data from UCI-HAR dataset. An intermediate filter was applied for sound quality pre-processing of the sensor data in the UCL-HAR dataset. A third-order Butterworth low-pass filter with a cutoff frequency of 20 Hz is sufficient for capturing human body motion since 99% of its energy is contained below 15 Hz [16]. The sensor information was then sampled in 2.56 s fixed-width sliding windows with a 50% overlap between them as shown in Figure 7. The four reasons for choosing this window size and overlapping proportion [36] are as follows: (1) The rate of walking of an average person is between 90 and 130 steps/min [47], ie., a minimum of 1.5 steps/s. (2) For each window study, at least one complete walking period (two steps) is desired. (3) This approach can also help people with a slower cadence, such asSensors 2021, 21,1636 9.0825 the elderly and those with disabilities. A minimum speed equal to 50% of the average human cadence was assumed by the researchers [36]. (4) Signals were also mapped via the Fast Fourier Transform (FFT) in the frequency domain, optimized for two-vector control (2.56 s x 50 Hz = 128 cycles). The available dataset contains 10,299 samples, split into two classes (ie., two sets of training and testing). The former has 7352 samples (71.39%), while the latter has the remaining 2947 samples (28.61%). The dataset is imbalanced, as shown in Figure 8. Since the use of accuracy only is insufficient for analysis and fair comparison, we additionally apply the F1-score to compare the performance of LSTM-based networks in this work i Actual Activities clas #2 ee Step sag Sensor eannel #1 wanisdata | Sensor data Sensor channel #2 sans data Sensor channel #3 pans data ‘Sting 1] Window Label = class Label = class #2 Figure 7. Data segmentation process by a sliding window. peeaeceec asta ye on Figure 8. Activity label distribution of UCI-HAR dataset.Sensors 2021, 21,1636 10.025 To evaluate the DL models, the dataset is first standardized. After employing the nor- malization approach, the dataset shows zero mean and unit variance. Figure 9 displays the histogram activity data from the tri-axial values of both the accelerometer and gyroscope. axisi—x —y —z Ps Lop ES! & oo woe a B25 Walking Upstairs 4 oo ate a pos Walking Downstairs £00 f - ww Siting i, aN i | Standing ° 8 oh ov Density : Siting i A ee Standing Lm ne Density % Laying : a 1 0 1 (b) Histograms of the gyroscope data by activity. Figure 9. Histograms visualization of data from (a) accelerometer (b) gyroscope.Sensors 2021, 27,1636 110825 3.2, LSTM Architectures ‘The following LSTM network architectures are used in this work: Vanilla LSTM net- work, 2-Stacked LSTM network, and 3-Stacked LSTM network, as illustrated in Figures 10-12, respectively. The original LSTM model (or Vanilla LSTM network) comprises an individual hidden layer of LSTM, followed by a common feedforward output layer. The Stacked LSTM networks are upgraded versions of the original model with multiple hidden LSTM layers. Each layer of the Stacked LSTM network contains multiple memory cells. A. Stacked LSTM structure can be technologically defined as an LSTM model, consisting of, multiple LSTM layers to take advantage of the temporal feature extraction obtained from each LSTM layer. O 2—O LJ O lopst STM Fully Connecied Layer Output Layer Layer ‘layer Raw Data Figure 10, Vanilla [STM network architecture. Input Lsrattayers Fly Connected Layer i i ft O.60 Figure 11. 2 Stacked LSTM network architecture. q a Dropout | 3 | repent | | Dro : g : A | beet smi Layer FulyConreced yee QU Ope i te Figure 12, 3-Stacked LSTM network architecture. ‘The CNN-LSTM architecture employs CNN layers in the feature extraction process of input data incorporated with LSTMs to support sequence forecasting, as shown in Figure 13. The CNN-LSTMS are built to solve forecasting problems in visual time series and applications to achieve textual descriptions from image sequences, This architecture is appropriate for issues involving a temporal input structure or requiring output generationSensors 2021, 21,1636 120625 with a temporal structure. In this work, an LSTM network called 4-layer CNN-LSTM is proposed to improve recognition performance. The architecture of the proposed 4-layer CNN-LSTM is illustrated in Figure 14. Raw Data rt comotsimltayes {SM rtycammattayr OME Figure 13 ONNLSTM actor art | K vane i—> tn. = py ving Figure 14, The proposed architecture of 4layer CNN-LSTM network ‘The network architecture of the proposed 4-layer CNN-LSTM network is shown in Figure 1, The tri-axial accelerometer and tri-axial gyroscope data segments were used as network inputs. To extract feature maps from the input layer, four one-dimensional convolutional layers were used for the activation feature in ReLU. Then, to summarize the feature maps provided by the convolution layers and reduce the computational costs, a max-pooling layer is also added to the proposed network. Their dimensions also need to be reduced after reducing the size of the function maps to allow the LSTM network to operate. For this reason, the flattened layer converts each function map’s matrix representation into a vector. In addition, several dropouts are inserted on top of the pooling layer to decrease the risk of overfitting, The output of the pooling layer is processed by an LSTM layer after the dropout function is applied. This models the temporal dynamics to trigger the feature maps. A fully connected layer, followed by a SoftMax layer to return identification, is the final layer. Hyperparameters such as filter number, kernel size, pool size, and dropout ratio were determined by Bayesian optimization, as shown in Figure 3.Sensors 2021, 21,1636 130825 3.3, Tuning Hyperparameter by Bayesian Optimization Hyperparameters are essential for DL approaches since they directly manipulate the actions of training algorithms and have a crucial effect on the performance of DL models. Bayesian optimization is a practical approach for solving the function problems prevalent in computing for finding the extrema. This approach is suitably employed for solving a related-function problem where the expression has no closed form. Bayesian optimization can also be applied to related-function problems such as extravagant computing, hard derivative evaluation, or a non-convex function. In this work, the optimization goal is to discover the maximum value for an unknown function f at the sampling point: xt = angmax f(x) (13) where A represents the search space of x. Given evidence data E are derived from Bayes’ theorem of Bayesian optimization. Then, the posterior probability PUM) of a model M is comparable to the possibility P(E | M) of over-serving F given model M multiplied by the prior probability of P/M): P(EIM)P(M) PCE) ‘The Bayesian optimization technique has recently become popular as a suitable ap- proach for tuning deep network hyperparameters by reason of their capability in admin- istering multi-parametric issues with valuable objective functions when the first-order derivative is not applicable. P(MIE) = ayy 3.4, Sampling Generation Process and Validation Protocol ‘The first process in sensor-based HAR is to create raw sensor data for the samples. This procedure involves separating it into small windows of the same size, called temporal windows. The raw sensor data is then interpreted as time-series data. Then, as the data samples are separated into training data, the temporal windows from the signals are used to lean a model and test the data to validate the leamed model. There are many strategies for using temporal windows to obtain data segments. The overlapping temporal window (QW), whereby a fixed-size window is applied to the in- put data sequence to provide data for training and test samples using certain validation protocol, is the most generally used window in sensor-based HAR studies (e.g., 10-fold cross validation). However, this technique is highly biased since there is a 50% overlap between subsequent sliding windows, Another method called the non-overlapping tem- poral window (NOW) can prevent this bias. As opposed to the OW technique, the NOW has the disadvantage of only a limited number of samples since the temporal windows no longer overlap. Evaluating the prediction metrics of the trained models is a critical stage of the process. Cross Validation (CV) [48] is used as the standard technique, whereby the data is separated into training and test data. Various approaches, such as leave-one-out, leave-p-out, and k-fold cross validation, can be used to separate the data for training and testing [49,50]. ‘The objective of this process is to evaluate the ability of the learning algorithm to generalize new data [50]. In sensor-based HAR, the purpose is to generalize a model for a different subject. The cross-validation protocol should also be subject-specific, meaning that the training and testing data contain records of various subjects. Cross validation of this protocol is called Leave-One-Subject-Out (LOSO). In order to create a model, the LOSO employs samples of all subjects but leaves one out to shape the training data. Then, using the samples of the excluded subject, the trained model is examined [51]. In this work, four combinations of sample generation processes and validation proto- cols evaluate the output recognition of LSTM networks, as shown in Table 2Sensors 2021, 21,1636 140825 ‘Table 2. Combination between sample generation processes and validation process used in this study Validation Protocol Sample Generation Process_|§ "S08 T0080 10-FoldCV _Leave-One-Subject-Out CV Overlapping-Windows owcy OWLS Non Overlapping-Windows NOCV NOLS 3.5, Performance Evaluation Metrics ‘The LSTM network classifiers employed in HAR can be measured using a few perfor- mance assessment indices. Five assessment metrics: accuracy, precision, recall, Fl-score, and AUC are chosen for performance evaluation. ‘The accuracy metric represents the corrected ratio of forecasting samples to total samples. Its suitable for assessing the classification performance in the case of balanced data. On the other hand, the Fl-score represents the weighted average of both precision and recall. Therefore, this metric can be properly applied in the event the data is imbalanced. Six metrics can be mathematically defined as the following expressions. 1. The accuracy is the evaluation ratio metric to all true assessment results of summarize the total grouping achievement for all types: Truep + Truey AcoUTRCY = Tren + Falsep + True + Falsen (a5) 2. The precision represents the ratio of positive samples classified correctly to total positive samples: Truep Precision = Trep + Falsep (16) ‘The recall represents the ratio of positive samples classified correctly to total positive samples: Truep vee Truep + False a7) The Fl-score represents the union of the recall value and the precision value in a separated value: Recall x Pre Recall + Precision Fl-score = 2 x (18) 5. Receiver Operating Characteristics (ROC) Curve, known as precision-recall rate, presents an approach for determining the true positive rate (TPR) against the false positive rate (FPR): Truep ae Truep + Falsep (a9) The false positive rate refers to the proportion of positive data points which are inappropriately claimed to be negative in comparison to all negative data points. Falsep : FPR = Trey + False ted where Truep means the number of true positives, Truey means the number of true negatives, Falsep means the number of false positives, and Falsey means the number of false negatives Not only Area Under the Curve (AUC) indicates the overall efficiency of classifiers, but it also describes the likelihood that positive cases selected will be ranked higher than negative cases [34]Sensors 2021, 21,1636 150625 4, Experiments and Results In the experimental section, the process is described and the results used to evaluate the LSTM networks for HAR, 4.1, Experiments ‘To compare the performance of each LSTM network and address the HAR issue, the experiments are varied. For the first variation, a basic LSTM network called Vanilla LSTM network is used, composed of only one LSTM hidden layer with dropout and one dense layer. For the second variation, one more LSTM hidden layer is added. This kind of configuration is called 2-Stacked LSTM. In the third variation, as well as adding an LSTM hidden layer, a Stacked LSTM is included. The fourth variation is the CNN- LSTM; an LSTM network combined with a convolution layer to create an LSTM layer. In [process of the hyperparameter optimization, the fifth LSTM-based networks were tuned by Bayesian optimization algorithm. The accuracy of each model after optimization is shown in Figure 15. After hyperparameter tuning, the 4-layer CNN-LSTM achieves an accuracy of 93.51% that outperforms other models after 140 iterations. The hyperparameters for the Vanilla LSTM network, 2-Stacked LSTM network, 3-Stacked LSTM network, CNN-LSTM network, and the proposed 4-layer CNN-LSTM network are summarized in Tables 3-7, respectively. The ROC of five LSTM models determining the ability of each classifier model is shown in Figure 16. E096 Poo (o ¥ oss a 7 1 20 40 oO 80 100 120 140 = Perera ree St Figure 15, The accuracy of each model after optimization process Table 3. The summarized hyperparameters of Vanilla LSTM network found by SigOpt. Phase Hyperparameter Value LSTM-neuron, on Structure Dropout 0.28385 Dense 784 Loss Function CCross-entropy Optimizer RMSprop. Training Batch Size ot Learning Rate 1x 19-3507 Number of Epoches 100Sensors 2021, 21,1636 160625 ‘Table 4, The summarized hyperparameters of 2 Stacked LSTM network found by SigOpt. Phase Hyperparameter Value LSTM-neuron: 63 Dropout: 0.46892 Structure LSTM-neuron 39 Dropouts 0.06469 Dense 181 Loss Function ‘Cross-entropy Optimizer RMSprop Training Batch Size 64 Learning Rate 1x 10-3288 Number of Epoches 191 ‘Table 5, The summarized hyperparameters of 3-Stacked LSTM Network found by SigOpt Phase Hyperparameter Value TSTM-neurony 74 Dropout: 0.08753 LSTM-neurong 4B Structure Dropout, 032057 LSTM-neurons 36 Dropouts 0.30374 Dense 338 Loss Function ‘Cross-entropy Optimizer RMSprop Training Batch Size 64 Learning Rate 1x 19-26H01 Number of Epoches 50 Table 6. The summarized hyperparameters of CNN-LSTM network found by SigOpt Phase ‘Hyperparameter Value Kemel-Size 3 Convolution, Stride 1 Filters 39 KernelSize 3 : Convolution: Stride i Structure a es Dropout; 0.02205 Maxpooling 2 Dense 83 Dropout, 0.27907 Dense 10 Toss Function Cross-entropy Optimizer ‘Adam Training Batch Size ot Learning Rate 1 x 19-2671 Number of Epoches 100Sensors 2021, 21,1636 170825 ‘Table 7. The summarized hyperparameters for 4-layer CNN-LSTM network found by SigOpt Stage ‘Hyperparameter Value Kemel-Size 3 Convolution, Stride 1 Filters 507 Kernel-Size 3 Convolution Stride 1 Filters 1 Kemel-Size 3 eee Convolutions Stride 1 Filters 468 Kernel-Size 3 Convolution, Stride 1 Filters 509 Dropout, 0.00952 Maxpooling 2 LsTM-neuron 127 Dropout, 0.27907 Dense ‘Loss Function Optimizer Training Batch Size Number of Epoches : a of | -=— a a a a i i r me : 2 i A. i i i a, Figure 16. Receiver operating, characteristic curves of five LSTM models. 4.2. Results The results derived from the conducted experiments are presented in this section, The experimental hardware and software configurations are as follows. The LSTM-based HAR ‘models are implemented using Python’s Scikit-Learn, TensorFlow [52] Core v2.0.0-rc0, and Keras v2.4.0 with each test consisting of different LSTM models. All experiments were executed using Tesla K80 GPU (NVIDIA, USA) on the Google Colab platform. Furthermore, the hyperparameters of LSTM networks were optimized using the Bayesian hyperparameter optimization on the SigOpt platform [53]. This is a scalable and regulated platform accompanied by an API to facilitate the generation of well-performing models. It also allows the process to proceed in parallel for faster evaluation The LSTM networks trained on the UCF-HAR dataset were evaluated using two different sample generation processes and two validation protocols, 10-fold and LOSO cross validation. Several experiments were conducted to evaluate the recognition performance of the LSTM networks with a set of evaluation indices: accuracy, precision, recall, Fl-score,Sensors 2021, 21,1636 180625 and AUC. Table 8 shows the accuracy derived from the various LSTM networks trained on the UCI-HAR dataset. The accuracy, precision, recall, Fl-score, and AUC of different LSTM networks are shown in Tables § and 9 using two different sample generation procedures (OW and NOW, respectively) and evaluated by the 10-fold protocol of cross validation. As can be observed from both tables: (1) The accuracy in all five recognition models was greater than 95%, with FI scores greater than 99%, This means that for the six regular operations, the proposed HAR system could accurately identify behavior. (2) In all five examples, the precision, recall, and Fl-scores outperformed the accuracy level. The accuracy metrics not only take Truep and Falsep into calculation, but also Truey and Falsey, by comparing Equations (15)-(18). Lower precision means that the Truew is substantially less than Falsew and Falsep, In other words, the most positive samples can be found in all LSTM networks. (3) All the evaluation metrics of the five OW process data-trained LSTM networks were greater than those of the NOW process. This shows that the amount of data affected the recognition efficiency of the five LSTM networks, but the difference was not statistically significant, (4) With a high accuracy of 99.39% from the OW process and 98.76% from the NOW process, the proposed 4-layer CNN-LSTM network outperformed other LSTM-based networks. Table 8. Performance metrics of five LSTM networks used in the experiment by OW sample generation and 10-fold cross validation protocol Evaluation Metrics (hata) Network ‘Accuracy Precision Recal Fi-Score ‘AUC Vanilla LSTM network 965% (E0408) SOSA CLD TOON) EY (LOAIHH) WAM (LOATH) 9979 (20.36%) 2estacked LSTM network 7.3D% (LOGS) 99.93% (DISD) — 943% (LO.6OR) 9.68% (402%) 994639 (40.135%) Stacked LSTM network 659% (LO47S%) — 99.66% (NARS) — ATT (LOADE) — 9ATI% (EO2SR%) 99.73% (40.2460) CNNASTM network SBAV% [EO751%) 99.90% (L0205%) 9935 (LOGO) 99.72% (-EO313%) 99.78% (40.088) layer CNN-LSTM network 99.39% (LO2489,) 99.93% (0.205%) 997% (407489) OORT CLUSTAM) 99.87% -£0080%) ‘Table 9, Performance met validation protocol ‘of five LSTM networks used in the experiment by NOW sample generation and 10-fold cross Network Evaluation Metrics (Esta) Accuracy Precision Recall FrScore Vanilla [STM network 2rstacked LSTM network Stacked LSTM network CNN-ASTM network ‘Hlayer CNN-LSTM network 34.93% ([=0908%) 96.23% (1.060%) 4.85% (£0800%) 98.95% (£24557) 99.73% (0.115%) 97.48% (£0607) 99.39% (1 99.51% (£0911%) 99.63% (£0231%) 98.76% (£0:500%) 99.96 (EOATIN) TB (LOOM) VAT EAN) 99.65% (402840) ‘9.86% (E0.186%) 99.58% (EO537%) 99.26% (=0823%%) : The performance metrics (accuracy, precision, recall, Fl-score, and AUC) of various LSTM-based networks using two separate sample generation processes (OW and NOW) are shown in Tables 10 and 11, evaluated by the leave-one-subject-out cross-validation protocol. As can be observed from the experimental results in Table 10: (1) The accuracy ofall five LSTM-based models was greater than 92% and Fl score greater than 75%. The highest average precision was 96.70% and the highest Fl-score 81.05%, indicating that the proposed 4-layer CNN-ISTM was achieved. (2) In all five situations, the precision, recall, and Fl-score outperformed the accuracy level. The precision metrics not only take Truep and Falsep into calculation, but also Truey and Falsey, by comparing the metric Equations (15)-(18). Higher accuracy indicates that Tren was substantially greater than Falsey and Falsep. This means that most negative samples could be recognized by all LSTM networks. It can be observed from the experimental results in Table 1 that the proposed 4-layer CNN-LSTM outperformed other LSTM-based networks with the highest accuracy of 93.94%. The confusion matrix of the proposed 4-layer CNN-LSTM networks is illustrated in Table 12.Sensors 2021, 21,1636 190625 ‘Table 10, Performance metrics of five LSTM networks used in the experiment by QW sample generation and LOSO crass validation protocol ‘Evaluation Mewies (Esta) Network Accuracy Precision Recall FLScore ‘auc Vanilla LSIM network SREP (ESS) WILLA) “TSH EAT) AH ESSSO) —_ 9B EIA) 2estacked LSTM nctwork 92.52% (ER77I%) BOSS EIGT%) — TTA EERE) ANOS (45.2598) 97.65% (24. 318%) Sstacked LSTM network 93.75% (£68387) OOS (EB.00D%) 792% (424%) TH [4B4D3%) 9818 (23314) CNN-SIM network SLID (76%) — NUL (EL990%) THU (SLERDAY) TTT (EIASIT) 97 8N(H4236) Payer CNNAISTM network —9670%(23516%) — BOSULCES61) BLS ESA) LOS (ERAID) 9G (=LB27™%) Table 11. Performance metsies of five LSTM networks used in the experiment by NOW sample generation and LOSO cross validation protocol valuation Metis (Esta) Network Accuracy Precision Recall Fi-score AUC Vania TSTM newwork D6I% (ERS) — OREN TTI OTST (FIVRISW) SBA (HIZMY) ORI CERAII 2stacked LSTM network SRB" (ZK2IN%) VHP ERIN) AH CETLIBID) — SART(ERTIM) 97.0% (HAT) Stacked LSTM network —$252%(£7972%) 97 381(47499%) — 9BSWY (29761) — SHA (ERIM) 98.19% (£3598) CNNASTM network S413% (A7954%) STEM CATASI) — RBH (LO26HN) 9.6% (L792%) ROD (44.324) AyerCNN-ISTM network 9394 (81P) — 96SKGTSITA) THAT ARI) ORAM CHIOITIN) 9762 EARS) Table 12. Confusion Matrix forthe proposed! -layer CNN-LSTM, Walking —Walkingupsteins _Walkingpownstairs Sitting Standing Laying Recall Walking 12 0 0 0 0 0 1.00) Walkinguipstars 0 134 0 0 0 0 1.00 Walking panstans 0 0 141 0 0 0 1.00 Sitting 0 0 0 178 0 0 1.00 Standing, 0 0 0 3 188 0 0984 Laying 0 0 0 0 0 41.00 Precision 1.000 1.000 1.000 09831000 —_~1.000 4.3, Comparative Results Tables 8-11 show the comparative results between the experimental LSTM hybrid networks and the LSTM baseline network. Since the UCLHAR dataset is imbalanced, the accuracy is inadequate for valid comparison to be evaluated, so the Fl-score is also used! to compare the recognition performance of these LSTM networks, ‘Considering to experimental results on average of 10-fold cross validation by OW sam- ple generation, the derived results demonstrates that hybrid LSTMs has better recognition performance than conventional LSTMs with higher accuracy and Fl-score upto 2.13% and 0.11%, respectively. Additional with NOW sample generation, They are also better than conventional LSTMs with 2.77% andl 0.24% of accuracy and Fl-score, respectively. At the same time with hybrid LSTMs, the proposed 4-layer CNN-LSTM has more performance of activity recognition than CNN-LSTM with 0.90% and 0.11% of accuracy and FI- OW sample generation, and 1.32% and 0.44% of accuracy and Fl-score for NOW ‘generation, respectively. In LOSO cross validation with results on average, hybrid LSTMs have more recog- nition performance than conventional LSTMs wi th 2.45% of accuracy for OW sample generation. Moreover with NOW sample generation, hybrid LSTMs still have more recog nition performance than conventional LSTMs with 1.98% and 0.745% of accuracy and Fl-score, respectively. Considering to hybrid LSTMs, the proposed 4-layer CNN-LSTMSensors 2021, 27,1636 290625 hhas more performance of activity recognition than CNN-LSTM with 2.58% and 3.28% of, accuracy and Fl-score in OW sample generation, respectively. Moreover, the effectiveness of the proposed 4-layer CNN-LSTM with the Fl-score of each activity class was compared. The Fl-score results of the different LSTM-based DL. models trained from smartphone sensor data are shown in Figure 17. The accuracy and loss details of the training process for four models (Vanilla LSTM network, 2-Stacked [STM network, 3-Stacked LSTM network, and CNN-LSTM network) and the proposed 4-layer CNN-LSTM are illustrated in Figures 18 and 19, respectively. Waking Mabing Laying pears Downstairs Vana LSTM a 2Stacked LSIM tw 3Slached LSTM WCNNALSTM mtLayer CNNASTIM g ee ee Figure 17. Bar chart showing Fl-score of the different LSTM networks on the UCI-HAR dataset (a) Vanilla LSTM. (b) 2-Stacked LSTM (@) CNN-LSTM Figure 18. Accuracy and loss examples of training process of Vanilla LSTM, 2-stacked LSTM, 3- stacked LSTM, and CNN-LSTM,Sensors 2021, 21,1636 210825 10 ‘Accuracy and Loss 02 Aecaracy ot training cata recuracy of validation data == toss of raining data oo = = ° 07S 1) aS aS0 AS —<200 ‘Talning Epoch Figure 19. Accuracy and loss examples of training process of the proposed 4-layer CNN-LSTM. 44, Comparative Analysis An accuracy comparison between the proposed model and other LSTM networks is, shown in Table 15. The proposed 4-layer CNN-LSTM network outperformed the other previous works. This is because the spatial feature extraction performed by the 4-layer CNN improved the overall accuracy by up to 2.24% compared to the most recent work [54] ‘Table 13. Performance comparison results Ref. “Architecture ‘Accuracy Year Lee [25] 1D-CNN 92.71% 2017 Hernandez [45] Bidir-LSTM 92.67% 2n19 Mutegeki [31] CNN-LSTM 92.13% 2020 Ni [4] SDAE 97.15% 2020 ‘The proposed approach ‘Mayer CNN-LSTM. 99.39% 7 5. Discussion of Results ‘The results derived from the experiments presented in Section 4 are the main contribu- tions of this research, and the following discussion is provided to ensure the consequences are clear. LSTM-based DL networks determined in this study could be employed to precisely categorize activities, specifically typical human daily living activities. The categorized pro- cess used tri-axial data on both accelerometers and gyroscopes embedded in smartphones. The categorized accuracy and other advanced metrics were considered to evaluate the sensor-based HAR of five LSTM-based DL architectures. A publicly available dataset of. previous studies [25,31,36,45,54] was applied to compare the generality of DL algorithms With the 10-fold cross-validation technique. ‘The UCL-HAR raw smartphone sensor data were evaluated with the Vanilla LSTM network, 2-Stacked LSTM network, 3-Stacked LSTM network, CNN-ISTM network, and the proposed 4-layer convolutional LSTM network (4-layer CNN-LSTM network). First, activity classification based on sensors with raw tri-axial data from both the accelerometer and gyroscope has been demonstrated. Figure 18a illustrates the process of learning raw sensor data with the Vanilla LSTM network, The loss rate decreased gradually and the accuracy rate increased slowly without any appearance of dilemma. This indicates thatSensors 2021, 21,1636 20625 the network leams appropriately without overfitting problems. The final result shows an average accuracy of 96.54% with the testing set. Figure 1Sb shows the training result of the 2-Stacked LSTM network. With the training set, both loss and accuracy were thoroughly trained and provide a decent performance. However, during the testing set process, some bouncing occurred. Specifically, when the epoch was 23, bouncing in both loss and accuracy could obviously be observed. Fortunately, the final average accuracy result is still better than that for the Vanilla LSTM network. During the testing set process, the loss rate was 0.1394 and the average accuracy 97.32%. The training result of the 3-Stacked LSTM network in Figure 8c shows a satisfactory learning process, but the testing set was significantly unstable. There are many bouncing spots in the middle and final epochs. While the epoch was repeated, the loss rate appeared to fluctuate. The process appears in worse shape when considering it as a graph. Nonetheless, the results in the actual testing set did not change for the worse with a loss rate of 0.189 and an accuracy rate of 96.59%. In Figure 18d, the training process of the CNN-LSTM network is shown to generate unstable parts in the validation set. On the other hand, the accuracy is generally higher than when training with the baseline [STM model, including the 2-Stacked LSTM network and 3-Stacked LSTM network, which achieved 98.49% in the testing set, Figure 9 shows the learning, process of the proposed 4-layer CNN-LSTM. In the testing set, with the epoch at 50, the loss rate decreased significantly and the accuracy increased significantly. Both the loss rate and accuracy quickly stabilized. Finally, the accuracy was 99.39% in the testing set. The derived classification results in this study reveal that hybrid DL architectures can improve the prediction performance of the baseline LSTM. When associating two DL. architectures (CNN and LSTM), the hybrid architecture could be the dominant reason for the increased scores, since the CNN-LSTM network produced higher average accuracy and Fl-score than the baseline [STM networks. Therefore, it can be inferred that the hybrid model delivers the advantages of both CNN and LSTM in terms of extracting regional features within short time steps and a temporal structure across a sequence. A limitation of this study is that the algorithms for deep learning, are trained and tested using laboratory data. Previous studies have shown that the performance of learning algorithms could not accurately reflect performance in everyday life under laboratory conditions [55]. Another constraint is that when looking at real-world situations, this study does not discuss the issue of transitional behaviors (Sit-to-Standing, Sit-to-Lay, etc), which isa challenge goal. However, the proposed HAR architecture can be applicable to many realistic applications in smart homes with high performance deep learning networks in- cluding optimal human movement in sports, healthcare monitoring and safety surveillance for elderly people, and baby and child care 6. Conclusions and Future Works In this study, the LSTM-based framework explores the LSTM network, providing, high performance in addressing the HAR problem. Four LSTM networks were selected to study their recognition performance using different smartphone sensors, ie, tri-axial accelerometer and tri-axial gyroscope. These LSTM networks were evaluated using a publicly available dataset called UCI-HAR by considering predictive accuracy and other performance metrics such as precision, recall, Fl-score, and AUC. The experimental results show that the 4-layer CNN-LSTM network proposed in this study outperforms the other baseline LSTM networks with a high accuracy rate of 99.39%. Moreover, the proposed LSTM network was compared to previous works. The 4-layer CNN-LSTM network could improve the accuracy by up to 2.24%, ‘The advantage of this model is that the CNN layers perform direct mapping in the spatial representation of raw sensor data for feature extraction. The LSTM layers take full advantage of the temporal dependency to significantly improve the extraction features of HAR Future work would involve the further development of LSTM models using various hyperparameters, including regularization, learning rate, batch size, and others. Fur- thermore, the proposed model could be applied to more complicated activities to tackleSensors 2021, 21,1636 230825 other DL challenges and HAR by evaluating it on other public activity datasets, such as. OPPORTUNITY and PAMAP2. Author Contributions: Conceptualization and model analysis, SM. resource angi data curation, A.J: methodology and validation, S.M. data visualization and graphic improvement, AJ. discussion and final editing, SM; writing-review and editing, SM. funding acquisition, AJ. and SM. All authors hhave wad and agreed to the published version of the manuscript. Funding: This research was funded by University of Phayao with Grant No, FF61-UoE008, Thailand ‘Science Research and Innovation Fund, and King Mongkut's University of Technology North Bangkok ‘with Contract No. KMUTNB-BasicR-6433-2. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable Acknowledgments: The authors thank the SigOpt team for the provided optimization services Conflicts of Interest: The authors declare no conflict of interest References 1. Shih, CS; Chou, JJ Lin, KJ. WusKong; Secure Run-Time environment and data-driven loT applications for Smart Cities and Smart Buildings. J. Interiet Ser. Inf. Secur. 2018, 8, 1-17. 2. Jobanputra, C-; Bavishi, J; Doshi, N. Human Activity Recognition: A Survey. Proceia Comput. Sci. 2019, 155, 698-708. doi:10.1016/,procs.2019.08.100. [CrossRef] 3. Qi Jz Yang, P; Hanneghan, M; Tang, S; Zhou, B. A Hybrid Hierarchical Framework for Gym Physical Activity Recognition and Measurement Using Wearable Sensors. [EEE Internet Things [.2019, 6, 1384-1393. oi:10.1109/JIOT.2018.2846359. [Crossief] 4. Mekruksavanich, S; Jitpattanakul, A. Exercise Activity Recognition with Surface Electromyography Sensor using Machine Learning Approach. In Proceedings of the 2020 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecormunications Engineering (ECTI DAMT and NCON), Pattaya, Thailand, 11-14 March 2020; pp. 75-78 5. Atapour, C.; Agrafiotis, I; Creese, S. Mocteling Advanced Persistent Threats to enhance anomaly detection technique Mob. Netw, Ubiguitous Comput. Dependable Appl. 2018, 9, 71-102. 6 Park, M; Seo, J: Han, J Oh, H.; Lee, K. Situational Awareness Framework for Threat Intelligence Measurement of Android. Malware. J. Wirel. Mob, Netw. Ubiquitous Comput. Dependable Appl. 2018, 9, 25-38. Kotenko,L; Saenko, I; Branitskiy, A. Applying Big Data Processing and Machine Learning Methods for Mobile Internet of Things Security Monitoring, |. Internet Sere Inf. Secu: 2018, 8, 54-63, 8, Mekruksavanich, S; Hnoohom, N,;Jitpattanakul, A, Smartwatch-based sitting detection with human activity recognition for office workers syndrome. In Proceedings of the 2018 International ECTI Northern Section Conference on Electrical, Electronics, (Computer and Telecommunications Engineering (ECTI-NCON), Chiang Rai, Thailand, 25-28 February 2018; pp. 160-164. 9, Casale, P; Pujol, O.; Radeva, P. Human Activity Recognition from Accelerometer Data Using a Wearable Device. In Pattern Recognition and Image Analysis; Vitra, J, Sanches, |M., Herndndez, M., Eds.; Springer: Berlin/Heidelberg, Germany, 2011; pp. 289-296, 10. Mekruksavanich, S. Jitpattanakul, A; Youplao, P; Yupapin, P. Enhanced Hand-Oriented Activity Recogni ‘Smartwatch Sensor Data Using LSTMs. Symmetry 2020, 12,1570. [CrossRef] IL, Mekruksavanich, S; Jitpattanakul, A. Biometric User Identification Based on Human Activity Recognition Using Wearable Sensors: An Experiment Using Deep Learning Models. Electronics 2021, 10, 308. doi:10.3390 /electronies10030308, [CrossRef] 12. Minh Dang, L; Min, Ky Wang, H.; Jalil Piran, M,; Hee Lee, Cs Moon, H. Sensor-based and vision-based human act recognition: A comprehensive survey. Pattern Recagnit. 2020, 108, 107561. [CrossRef] 13. Zhang, S; Wei, Z.; Nie, J; Huang, L; Wang, S; Li, Z. A Review on Human Activity Recognition Using Vision-Based Method. J. Hoalthe, Eng. 2017, 2017, 1-31. do:10.1155/2017 3090343, [Crosse] [PubMed] 14. Afza, F; Khan, MA. Sharif, Mz Kadry, $ Manogaran, G.; Saba, T Ashraf, L; Damasevigius, R, A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection. Image Vis. Comput. 2021, 106, 104090, doi:10.1016/}.imavis.2020,104090, [Crossitet] De-La-Hoz-Franco, E.; Ariza-Colpas, P; Quero, JM. Espinilla, M. Sensor-Based Datasets for Human Activity Recognition—A, Systematic Review of Literature. IEEE Access 2018, 6, 59192-59210, doi:10.1109/ ACCESS.2018.2873502, [CrossRef] 16. Hnoohom, N.; Mekruksavanich, S; Jitpattanakul, A. Human Activity Recognition Using Triaxial Acceleration Data from Smartphone and Ensemble Learning. In Proceedings of the 2017 13th International Conference on Signal-Image Technology Internet-Based Systems (SITIS), Jaipur, India, 4-7 December 2017; pp. 408-412, J. Wirt n Based on.Sensors 2021, 21,1636 240825 17, Agac,S Shoaib, M; Ince, .D. Contextaware and dynamically adaptable activ omsmoking. Compt. Elect Eng. 2021, 9, 106949, [CrossRef] 18, Fu, Z; He, X; Wang, E; Huo, |; Huang, Jz Wa, D. Personalized Human Activity Recognition Based on Integrated Wearable Sensor and Transfer Learning, Sensors 2021, 21,885. dois 0.39%0/s21030885, [Cros 19, Cui, W;Li,B Zhang, L, Chen, Z. Devicefee single-user activity recognition using diversified deep ensemble learning. Appl Sef Comput, 2021, 102, 107066. do:101016 /jasoe.2020.107066, [CeossRef] 20, Hussain, Z; Sheng, M.;Zhang, WE. Different Approaches for Human Activity Recognition: A Survey. CoRR 2019, abs:1996.05074 21. Sargano, AB AngelovP; Habib, . Hunan Action Recognition from Multiple Views Based on View-lnvariant Feature Descriptor Using Support Veetor Machines. App. So. 2016, 6, 309, doi:10.390 appi00309. [Cross] 22, Ramasamy Ramamurthy, S; Roy N. Recent trends in machine leaning for human activity recognition—A survey. WIREs Data Min. Know, Dist. 208, 8, 1254 [CrossRef] 23, Almastukh, By Muhtadi, J; Artoli, A. A robust convolutional neural network for online smartphonebased human activity recognition, [Intel Fussy Syst. 2018, 35,112, doil03253/]IFS-169699 [CrossRef] 24, Ignatov, A. Real-time human activity recognition from accelerometer data using Convolutional Neural Networks. Appl. Soft Comput. 2018, 62, 915-822. do:10.1016/jasoc.201709.027 [CrossRef] 25, Lee, SM, Yoon, SM.;Cho, H. Human activity recognition from accelerometer data using Convolutional Neural Network. In Proceedings of the 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), Jeju, Korea, 13-16 February 2017; pp. 131-134. 26, Wang | Chen, Y; Hoo, S; Peng, Xz Lisha, H. Deep Leaming fr Sensor-based Activity Recogn Lett. 2017. [CrossRef] 27. Baldominos, A; Cervantes, A; Saez, Vlas, P. A Comparison of Machine Leaming and Deep Learning Techniques for Activity Recognition using Mobile Devices. Sensors 2019, 19,521, [CrossRef] 28, Zhao, ¥; Yang, R; Chevalier, G; Xu, X; Zhang, Z. Deep Residual Bid LSTM for Human Activity Recognition Using Wearable Sensors. Math, Pro. Eng. 2018, 2018, 1-13, doit0.1155/2018/7316954, [Crosske] 28. Ullah, M; Ullah, H.; Khan, SD; Cheikh, FA. Stacked Lstm Network for Haman Activity Recognition Using Smartphone Daa. In Proceedings ofthe 2019 Sth European Workshop on Visual Information Processing (EUVIP), Roma, Italy, 28-31 October 2019; pp. 175-180. 30, Zhang, Pj Zhang, Z:;Chao, H.C. A Stacked Human Activity Recognition Model Based on Parallel Recurrent Network and Time Series Evidence Theory. Sensors 2020, 20,4016. [CrossRef] 31, Mtegeki, R; Han, DS. A CNN-LSTM Approach to Human Activity Recognition, In Proceedings of the 2020 Intemational Conference on Artificial Intelligence in Information and Communication (ICAMC), Fukuoka, Japan, 19-21 February 2020; pp. 62-366 32, Oronez, Fly Roggen, D. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Act Recognition. Sensors 2016, 16,115, doi:10.3390/ s16010115. [CrossRef] 38, Hammerla, N.¥ Halloran, . Plate, T. Deep, Concoutional and Recurrent Moves or Huan Activity Recogiton Using Wearables AAAI Pres: Menlo Park, CA, USA, 2016; pp. 1533-1540, Balling, A. Blanke, U; Schiele, B. A Tutorial on Human Activity Recognition Using Body-Worn Inertial Sensors. ACM Comput, Sur 2014, 46. Crosske] Bao, L, Intille, SS. Activity Recognition from User-Annotated Acceleration Data. In Presive Computing Ferscha,A., Matter, F, Eds, Springer: Berlin/Heidelberg, Germany, 2004 pp. 1-17, Anguita,D,;Ghio, A Oneto, L, Parra, X; Reyes-Ontz |. Energy eicient smartphone-haset activity recognition using fixed-point arithmetic. J. Unioes. Comput. Sc. 2013, 19, 1295-1318 37, Kwvapisz, LR; Weiss, GM, Moore, S.A. Activity Recognition Using Cell Phone Accelerometers. SIGKDD Explor. News. 2011, 12, 74-82, [CrossRef] 38. Hu, L; Chen, ¥; Wang, &; Wang, J; Shen, J; fang, X; Shen, Z. Less Annotation on Personalized Activity Recognition Using Context Data, In Proceedings ofthe 2016 Intemational IEEE Conferences on Ubiquitous Intelligence Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Cloud and Big Data Computing, Intemet of People, and Smart World Congress (UIC/ ATC /SsalCom/CBDCom/loP/SmartWorld), Toulouse, France, 18-2 July 2016; pp. 327-352 38. Nan, ¥; Lovell, NH; Redmond, SJ; Wang, K; Delbaere, K; van Schouten, KS. Deep Learning for Activity Recognition in Older People Usinga Pocket-Worn Smariphone Sensors 2020, 20,7195, [Crosskei] 40, _Krizhevsky, A Sutskever,; Hinton, G, ImageNet Classification wth Deep Convolutional Neural Networks. Neural If. Process Syst. 2012, 25. [CrossRef] 41. _Bengio, ¥; Simard P; Frascon,P: Leaming long-term dependencies with gradient descent i difficult. IEEE Trans, Neural Netu: 1994, 5, 157-166. [CrossRef] [PubMed] 42, Hochriter,S; Schmichuber, J. Long Short-Term Memory. Neus! Comput. 1997, 9, 1735-1780. [CrossRef] 43. Chen, ¥; Zhong, K. Zhang, J; Sun, Qz Zhao, X. LSTM Networks for Mobile Haman Activity Recognition. In Proceatngs ofthe 2016 international Conference on Arial Intligence: Technolgies and Applications; Atlantis Press: Amsterdam, The Netherlands, 2016; pp. 50-53. [CrossRef] recognition with smart watches: A case study fon: A Survey. Pattern Recognit.Sensors 2021, 21,1636 250825, 4A. Singh, SP; Lay-Ekuakille, A. Gangwar, D. Sharma, MK. Gupta, S. Deep ConvLSTM with selfattention for human activity decoding using wearables. arXiv 2020, arXiv:2005.00698, 45, Hemandez, F; Suarez, LF; Villamizar, |; Altuve, M, Human Activity Recognition on Smartphones Using a Bidirectional LSTM. Network. In Proceedings of the 2019 XXII Symposium on Image, Signal Processing and Artificial Vision STSIVA), Bucaramanga, Colombia, 24-26 April 2019; pp. 1-5, 46, Karantonis, DM, Narayanan, M.R.; Mathie, M.; Lovell, NEL; Celler, B.G, Implementation of a eal-time human movement classifier using a triaxial accelerometer for ambulatory monitoring. IEEE Trans. if Tein. Biomed. 2006, 10, 156-167. [Ceoss8e¥] [PubMed] BenAbselkader, C.; Cutler, R; Davis, L. Stride and cadence asa biometric in automatic person identification and verification. In Proceedings of the Fifth IEEE Intemational Conference on Automatic Face Gesture Recognition, Washington, DC, USA, 21-21 May 2002; pp. 372-377. [CrossRef] 48. Hastie, ; Tibshirani, R, Friedman, J. The Elements of Statistica! Learning: Data Mining, byerence, and Prediction; Springer Series in Statistics; Springer: Berlin/ Heidelberg, Germany, 2009, 49. Arlot,S; Celisse, A. A survey of cross-validation proceclures for model selection. Stat. Sur, 2010, 4, 40-79. [Crossief] 50, Saeb,$, Loni, L. Jayaraman, A Mohs, D,; Kording, K. The need to approximate the use-case in clinical machine leamiing, GigaScience 2017, 6. [CrossKe!] 51. Wang, $: Zhou, G.; Ma, ¥ Hu, L. Chen, Z.; Chen, Ys Zhao, H.; Jung, W, Eating detection and chews counting through sensing ‘mastication muscle contraction. Smart Health 2018, 9-10, 179-191. [CrossRef] 52. Abadi, Ms Barham, P; Chen, J; Chen, Z.; Davis, A; Dean, J; Devin, M.; Ghemawat, 8; Irving, G.; Isard, M.et al. TensorFlow: A System for Large-Scale Machine Learning, In Proceclings of the 12th USENIX Conference on Operating Systems Design ard Inplementation; USENIX Association: Berkeley, CA, USA, 2016; pp. 265-288, 53, _Dewancker, I; McCourt, M; Clark, S; Hayes, P; Johnson, A. Ke, G. A Stratified Analysis of Bayesian Optimization Methods. ari 2016, arXiv:1608.09441 54. Ni,Qs Fan, Z~; Zhang, L.; Nugent, C.D.;Cleland, I; Zhang, Ys Zhou, N. Leveraging Wearable Sensors for Human Daily Activity Recognition with Stacked Denoising Autoencoders. Sensors 2020, 20, 5114. {Crossef] 55, Gyllensten, LC; Bonomi, A.. Identifying Types of Physical Activity with a Single Accelerometer: Evaluating Laboratory-trained Algorithms in Daily Life. IEEE Trans. Biomed. Eng. 2011, 58, 2656-2603. [Crossiet]