Received: 30 March 2021 Revised: 7 August 2021 Accepted: 27 September 2021

DOI: 10.4218/etrij.2021-0097

ORIGINAL ARTICLE

Comparing automated and non-automated machine

learning for autism spectrum disorders classification using
facial images

Basma Ramdan Gamal Elshoky1,2 | Eman M. G. Younis3 |

Abdelmgeid Amin Ali2 | Osman Ali Sadek Ibrahim2

1
Information Technology Section, Korean
Egyptian Faculty for Industry and Energy Abstract
Technology, Beni Suef Technological Autism spectrum disorder (ASD) is a developmental disorder associated with
University, Beni Suef, Egypt
2
cognitive and neurobehavioral disorders. It affects the person’s behavior and
Computer Science Department, Faculty
of Science, Minia University, Minia, Egypt
performance. Autism affects verbal and non-verbal communication in social
3
Department of Information Systems, interactions. Early screening and diagnosis of ASD are essential and helpful
Faculty of Computers and Information, for early educational planning and treatment, the provision of family support,
Minia University, Minia, Egypt
and for providing appropriate medical support for the child on time. Thus,
Correspondence developing automated methods for diagnosing ASD is becoming an essential
Basma Elshoky, Information Technology need. Herein, we investigate using various machine learning methods to build
Section, Korean Egyptian Faculty for
predictive models for diagnosing ASD in children using facial images. To
Industry and Energy Technology, Beni
Suef Technological University, Beni Suef, achieve this, we used an autistic children dataset containing 2936 facial images
Egypt. of children with autism and typical children. In application, we used classical
Email: [email protected]
machine learning methods, such as support vector machine and random for-
est. In addition to using deep-learning methods, we used a state-of-the-art
method, that is, automated machine learning (AutoML). We compared the
results obtained from the existing techniques. Consequently, we obtained that
AutoML achieved the highest performance of approximately 96% accuracy via
the Hyperpot and tree-based pipeline optimization tool optimization. Further-
more, AutoML methods enabled us to easily find the best parameter settings
without any human efforts for feature engineering.

KEYWORDS
artificial intelligence, autism spectrum detection, deep learning, facial images, machine
learning

1 | INTRODUCTION support, and for providing an appropriate medical care to

the child on time [1]. Autism affects verbal and non-
Early diagnosis of autism spectrum disorder (ASD) is verbal communication and social interaction for children.
essential and helpful for the early design of special Body expressions can provide information about the
educational planning and treatment, provision of family affective state of a person [2]. The body expressions of the

This is an Open Access article distributed under the term of Korea Open Government License (KOGL) Type 4: Source Indication + Commercial Use Prohibition +
Change Prohibition (http://www.kogl.or.kr/info/licenseTypeEn.do).
1225-6463/$ © 2022 ETRI

ETRI Journal. 2022;44(4):613–623. wileyonlinelibrary.com/journal/etrij 613

 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
614 ELSHOKY ET AL.

subject with autism play a vital role in reflecting his or automatically for the ML classification model. The contri-
her behavior and feelings. They also have the same face butions of this study are as follows:
pattern. The research [3] studied facial asymmetry in
autistic children compared with typical children and chil- • using facial images to identify ASD using classical ML
dren who have siblings with autism. They found that methods;
autistic children had greater facial asymmetry than other • using DL convolutional neural network (CNN) and
children. The intensity of autistic symptoms also appears other well-known DL architectures with transfer learn-
in facial asymmetry. ing for ASD detection;
Recent developments in artificial intelligence • optimizing the performance of ML using AutoML tech-
(AI) have heightened the need for applying it in medical niques, which saves time and effort in determining the
diagnosis systems. Machine learning (ML) is a subfield of best parameters and selecting the best ML algorithm
AI [4], which learns patterns from data. Over the past automatically for the ML classification model; and
century, ML techniques have been widely used to analyze • comparing the results obtained from previous
and classify medical images for the prediction of diseases. approaches.
These techniques help physicians in the screening and
diagnosis of diseases. The remainder of the paper is organized as follows.
Classical ML algorithms use historical labeled Related works are presented in Section 2. In Section 3,
training data to create predictive models for classifying we explain the methods and techniques. Section 4
new instances of data [5]. ML algorithms can be classi- presents the results of various methods and their
fied into supervised, semi-supervised, and unsupervised comparison. Section 5 provides a discussion of the results.
learning. Supervised learning algorithms learn and Finally, Section 6 presents the conclusions and
predict the output from the input labeled data. In the future work.
classification task, the data are labeled with class or
category labels, such as has a disease and does not have
a disease. Every ML method has some parameters that 2 | RELATED WORK
need to be adjusted to improve performance. However,
deep-learning (DL) algorithms take the input data 2.1 | Analyzing body expressions for ASD
without performing feature engineering and produce an detection
internal representation of the data to generate the pre-
dictive models. Rehg (2011) created a multimodal dyadic behavior
In automated machine learning (AutoML) methods, (MMDB) dataset for children with autism. It contains
hyperparameters are adjusted automatically to produce audio and video data of controlled behavioral interac-
models that achieve the best performance, besides choos- tions of children between the ages of 15 and 30 months
ing the ML method that achieved the best accuracy on through social games with a trained adult examiner [10].
the dataset [6]. The AutoML method achieves more suc- He captured and analyzed the controlled behavioral
cess in prediction tasks and solves classification problems interactions of children with autism through images
in studies [7–9]. The study [9] used the AutoML tool, collected from videos that show hand, head, and eye
tree-based pipeline optimization tool (TPOT), with movements. In 2013, some studies [11–13] analyzed and
Kalman Filter to predict the number of deaths due to diagnosed autism using body expressions of the subject
COVID-19 that achieved accuracy 99% of R2 score. The with autism. Rehg and others employed the MMDB
study [8] used the TPOT approach to obtain ML models dataset to investigate the activity recognition by analyz-
that predict brain age accurately. The paper [7] investi- ing children’s behavior [13]. The results enabled the
gated the performance of AutoML (auto-sklearn, H2O, detection of discrete behaviors (gaze shifts, smiling, and
and TPOT) in text classification tasks using 13 English play gestures). Rajagopalan and others created a self-
textual datasets. stimulatory behavior dataset for analyzing stimming
Previous studies on ML for ASD have only been behaviors in children with autism [11]. It will help clini-
conducted using classical and DL methods. In this paper, cians to diagnose autism. The self-stimulatory refers to
we compare these models with state-of-the-art AutoML the movement of body parts frequently. The dataset
methods for ASD using facial images. The main novelty contains 75 videos collected from the YouTube website
is to use the AutoML techniques to optimize the perfor- for stimming behavior (arms flapping, headbanging, and
mance of ML for autism classification using facial spinning). They were collected in an uncontrolled
images. These will save time and effort in determining natured setting during regular daily activities. The videos
the best parameters and selecting the best ML algorithm are annotated with a set of representative attributes of
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ELSHOKY ET AL. 615

the behavior, such as body parts (hand, head, face, and training on a few images took a long time. Beary and
eye) attributes. After feature extractions, training, and others used DL MobileNet and two dense layers to diag-
testing the model, the result achieved 47.1% accuracy. nose autism using autistic children facial images dataset
Rajagopalan developed a model for the diagnosis of on Kaggle Version 1 [22]. The dataset contains 3014
autism by analyzing the body behavior of children with images split into 90% and 10% for training and testing,
autism [12]. The dataset was videos for children con- respectively. The training model was completed after
taining a sequence of repetitive actions and typical 15 epochs, and each epoch required 10 min to run. The
behaviors. The highest accuracy achieved was 77.6%. test model achieved an accuracy of 94.64%, indicating
The study [14] proposed a computational framework that autism can be identified using facial images. The
to analyze facial expression in ASD and Tourette’s disor- images in this version are low quality and duplicated.
der (TD) children. The production of facial expressions The study [23] used autistic children’s facial images
will be helpful to understand the basic emotions of an dataset to detect autism using facial images. It applied the
ASD child. In [15], the facial expression in children with VGG19 to train the model. The training model achieved
ASD was analyzed by studying the behavior of face, an accuracy of 96%. Finding the classification model to
vision, and hand movement. The study suggested that identify ASD from facial images will help the clinical
the facial expressions of these children have a typical diagnosis of ASD. To the best of knowledge, many ML
behavior. The study [16] used an eye-tracking image techniques have been applied for ASD detection, but they
dataset to train an ML classifier to identify children did not use AutoML. Therefore, we compare the use of
affected by autism. The paper [17] presented a review of traditional ML, DL, and AutoML for ASD detection.
using computer vision to analyze faces face in health and
medical applications. It studied the main features:
mouth, skin, eyes, muscles, and shape. 3 | MATERIAL A ND METHODS

3.1 | Dataset description

2.2 | Diagnosis using facial images
The autistic children dataset Version 9 [24] found on
In 2016, the study [18] used an ML method to identify Kaggle consists of 2936 jpg images of 224
224 pixels
children with autism by analyzing face images. The with two class labels. These classes are autistic and non-
dataset contains face images for tracking eye movement autistic children. Images are collected online from
when the child is viewing human faces. The support websites and Facebook pages associated with autism.
vector machine (SVM) model achieved an accuracy of Images in this version are better than those in previous
88.51% using grayscale images (width: 500
height: versions, and there is no duplication. It is also for youn-
700 pixels). ger children. Images are facial images for girls and boys.
Patnam and others (2017) used a DL RCNN algorithm The dataset is split into train, valid, and test folders. Each
to build a system recognition meltdown in autistic kids folder is split into autistic and non-autistic. Figure 1
[19]. For recognized behavior, they took images from shows a sample of images for both cases. This dataset
videos covering ears and face as two instances. They was selected because it is free to access; that is, it does
achieved an accuracy of 92%. not require approval. Besides, it is easier to process
Rudovic and others (2018) automated estimation of images instead of videos.
autism using DL residual network (ResNet) algorithm
with five fully connected (FC) layers and 20 epochs on
face images (256
256 pixels) of children with autism
[20]. They used the robot-assisted to record 30 videos
with different cultural backgrounds (Asia vs. Europe).
Then, they selected face images from these videos. The
automated task achieved an accuracy of 43% using only
face images.
Tamilarasi and Shanmugam (2020) applied CNN
ResNet-50 (50 layers) architecture on images dataset to
identify children with autism [21]. The dataset contains
39 image faces (224
224 pixels) split into 19 non-autism
and 20 autism images. The classification achieved an F I G U R E 1 Image sample, (A) child without autism and
accuracy of 89.2% in 1.5 min. However, the model (B) child with autism
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
616 ELSHOKY ET AL.

3.2 | Computing platform PyTorch, which can run on TPU, GPUs, or CPU. Tradi-
tional ML has limitations that require tools for feature
We used Python programming language Version 3.7 with engineering. However, DL has an important property,
GPU in the Google Colab platform. The Google Col- the ability to extract features from images automatically
aboratory service is an online Jupyter Notebook in Python within training data. This property saves work and time.
that runs on Google’s servers [25]. Colab servers provide DL has many architectures, and each architecture is suit-
powerful hardware options, such as GPUs and TPUs, for able for specific applications.
up to 12 h at a time, completely free. Moreover, it is easy to
access it anywhere with an internet connection. We used
Scikit-learn and open-cv Python packages. Scikit-learn is 3.5 | Convolution neural network
an open-source package of ML algorithms written in
Python that provides a range of libraries and algorithms CNN architecture is widely used for image classification
(supervised and unsupervised learning) [26]. The com- applications. There are three types of layers in the CNN
puter vision library (open-cv) is an open-source library convolution layer: pooling, dropout, and FC. DL phases
used to process and prepare images and videos [27]. are as follows:

1. Prepare the dataset of images.

3.3 | Machine learning techniques 2. Build the CNN model.
3. Split the data into train test.
ML is a field of study concerned with algorithms that 4. Train the model.
learn from examples for solving a problem. The popular 5. Test the model.
supervised ML algorithms are classification and regres- 6. Evaluate the model.
sion trees (CART), logistic regression (LR), linear 7. Change parameters to improve performance (hyper-
discriminant analysis (LDA), boosting, SVM, gradient parameter optimization). Figure 3 shows the CNN
boosting machine (GBM), ada-boost (AB), random forest architecture that has been applied. The architecture
(RF), naive Bayes (NB), k-neighbors (KNN), and extra consists of the following:
trees (ET). Figure 2 shows the ML architecture that has
been applied. We use the computer vision technique (a) Input layer: This is called the visible layer; it takes
open-cv to extract features from images. RGB images are a color image with 224-pixel height and width.
resized to 90
90 pixels and converted to grayscale (b) Four blocks of hidden layers (convolution/
images coded as an array with the label autistic (1) or pooling): These layers are used to extract features
non-autistic (0). Train test and cross-validation tech- to create a feature map.
niques are used to split data for train, validation, and test (c) FC layers (flatten/dense): They combine features
set. Then, apply several ML algorithms. and make the prediction for the output layer. The
performance of the CNN accuracy can be
improved using the transfer learning. Transfer
3.4 | Deep learning learning provides a pre-trained model that saves
training time. There are many DNN architectures
DL is a multilayer neural network called a deep neural used for image classification problems, such as
network (DNN). Keras is an open-source DL API written VGG, ResNet, MobileNet, DenseNet, Xception,
in Python by Francois Chollet in 2015 [28]. It uses DL EfficientNet, and Inception, that can be off the
mathematical libraries like TensorFlow, Theano, and

F I G U R E 2 Machine learning architecture. ASD, autism

spectrum disorder FIGURE 3 Convolutional neural network architecture
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ELSHOKY ET AL. 617

shelf used for transfer learning. It means using best configuration is re-trained on the entire dataset to
pre-trained models in new applications. produce the classifier that handles subsequent predict
calls.

3.6 | Automated machine learning 3.8 | Pipeline optimization

Most ML algorithms must be configured before training Pipeline optimization is one or more tasks that help
the model. It is not easy to find an ML model with an automate the ML process. TPOT is an open-source
optimized parameter that achieves the highest accuracy. Python package built on the scikit-learn in the
There are many tools and libraries that help in optimiza- Computational Genetics Laboratory at the University of
tion, for example, AutoML. There are many popular Pennsylvania by Dr. Randal Olson and Dr. Jason
AutoML tools, such as AutoSklearn, AutoKeras, H. Moore. It attempts to find pipelines that maximize
AutoWeka, H2O, TPOT, Hyperopt-Sklearn, and the classification accuracy for a given dataset by selecting
AutoPytorch. These tools have an important advantage in the appropriate models and optimizing their parameters
that they extract features automatically from the dataset [31]. The TPOT implements four main types of pipeline
without using a feature extraction algorithm. Addition- operators as follows:
ally, it enhances the ability of non-expert users of ML
techniques to obtain the optimum model. Data are split 1. Preprocessors: It is a standard scaling operator that
into 70% and 30% for training and testing of the model. generates interacting features via polynomial combi-
nations of numerical features (PolynomialFeatures).
2. Decomposition: It implements a RandomizedPCA, a
3.7 | Hyperparameter optimization variant of PCA that uses randomized singular value
decomposition.
Hyperparameter optimization was performed using 3. Feature selection: It implements a recursive feature
Hyperopt-sklearn. Hyperopt-sklearn is an open-source elimination strategy that selects the top k features and
Python package built on scikit-learn based on the search removes features that do not meet a minimum vari-
space approach. It uses Hyperopt [29] to describe a search ance threshold.
space over possible configurations, preprocessing, and 4. Models: It implements individual and ensemble tree-
classification modules. based models (e.g., decision tree classifier, RF, and
Hyperopt has three components: (1) A search domain GBM), non-probabilistic, and probabilistic linear
is specified via random variables and can include models (e.g., SVM and LR), and KNN.
operators and functions. (2) An objective function that
maps random variables to a scalar-valued score that the These pipelines are automatically generated and opti-
optimization algorithm will try to minimize. (3) Hype- mized using an evolutionary computation technique
ropt’s fmin: It is an optimization algorithm. The fmin called genetic programming as implemented in the
function carries out the optimization process and stores Python package DEAP [32]. The optimization process
the search result in a database (e.g., Python list). Then, it works as follows:
returns the best performing configuration to the caller.
It provides six possible classifiers and preprocessing 1. Each time a dataset passed through a modeling
modules [30]. The possible classifiers are RF, KNN, operator.
support vector clustering (SVC), ET, stochastic 2. The resulting classifications are stored such that the
gradient descent (SGD), and multinomial NB. The most recent classifier to process the data overrides any
possible preprocessing modules are none (0), N (1) previous predictions.
(normalization), min–max scaler (NMS), and principal 3. The earlier classifier’s predictions are stored as a new
component analysis (PCA) (2). feature.
The estimator class consists of fit and predict 4. The final predictions are used to evaluate the classifi-
methods. The fit method executes optimization. Then, cation performance of the pipeline.
the predict method applies the best model to test data.
During optimization, each evaluation training on a Figure 4 shows the AutoML architecture. TPOT,
large fraction of the training set estimates test set accu- Hyperpot, and AutoKeras required extracting features
racy on a validation set and returns the validation set from images and splitting data to train and test the
score to the optimizer. At the end of the search, the models.
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
618 ELSHOKY ET AL.

4 | R E SUL T S parameter learning rate of 0.001 and a momentum of 0.9.

Dropout and augmentation did not improve the model
4.1 | Results of classical machine accuracy. Figure 6 shows the accuracy versus computa-
learning tional time of DL models.

Table 1 presents the performance of traditional ML algo-

rithms in terms of accuracy and computational time.
Figure 5 shows the accuracy versus computational time
of classical ML models. The highest accuracy achieved by
ML was 72.64% with ET classifier using train test split
technique with 4.25 s. Applying cross-validation could
not improve the classification accuracy and take
more time.

4.2 | Results of deep learning F I G U R E 5 Accuracy versus computational time of classical

machine learning models. AB, ada-boost; CART, classification and
Table 2 presents the performance of DL models in terms regression trees; ET, extra trees; GBM, gradient boosting machine;
of accuracy. The patch size is “64,” used for all DL KNN, k-neighbors; LDA, linear discriminant analysis; LR, logistic
models. Then, CNN applied five convolution blocks with regression; NB, naive Bayes; RF, random forest; SVM, support
vector machine
100 epochs for the model. The SGD optimizer has a

TABLE 2 Performance of deep-learning model

Model No. of epoch Test accuracy

CNN 100 84
VGG16 10 89
VGG19 10 88
ResNet50 20 86
ResNet101 30 85.33
ResNet152 30 87

Note: Bold, highlight for max value.

F I G U R E 4 AutoML architecture. AutoML, automated Abbreviations: CNN, convolutional neural network; ResNet, residual
machine learning; TPOT, tree-based pipeline optimization tool network.

TABLE 1 Accuracy and computational time of machine learning

Algorithm Accuracy Precision F1 score Time in second

LR 55.60 54.20 56.74 3.23
LDA 57.32 56.03 59.89 17.85
CART 60.38 59.06 59.89 15.50
NB 69.42 67.82 69.13 0.31
KNN 62.42 67.82 70.40 37.72
SVM 71.28 78.60 70.40 64.03
AB 67.08 69.35 67.08 69.52
GBM 70.82 73.44 70.64 393.60
RF 70.94 69.47 69.47 12.00
ET 72.64 76.76 69.47 4.25

Abbreviations: AB, ada-boost; CART, classification and regression trees; ET, extra trees; GBM, gradient boosting machine; KNN, k-neighbors; LDA, linear
discriminant analysis; LR, logistic regression; NB, naive Bayes; RF, random forest; SVM, support vector machine.
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ELSHOKY ET AL. 619

T A B L E 3 Tree-based pipeline optimization tool automated

machine learning results for autistic children dataset
Generation 1—Current best internal CV score:
0.9435143314936798
Generation 2—Current best internal CV score:
0.9435143314936798
Generation 3—Current best internal CV score:
0.9435143314936798
Generation 4—Current best internal CV score:
0.9435143314936798
F I G U R E 6 Accuracy versus computational time of deep-
Generation 5—Current best internal CV score:
learning models. CNN, convolutional neural network; ResNet,
0.9454643641326925
residual network
Best pipeline: ExtraTreesClassifier(SelectFwe(input_matrix,
alpha = 0.011), bootstrap = False, criterion = entropy,
max_features = 0.25, min_samples_leaf = 1,
min_samples_split = 19, n_estimators = 100)

T A B L E 4 Hyperpot automated machine learning #cross-

validation results for autistic children dataset for the extra trees
classifier

Time for each

No. of pipelines pipeline in second Accuracy
- - 96.4
100 45 96.1
100 10 96.5
10 2 96.0
50 5 96.5
Average 96.2

Note: Bold, highlight for max value.

parameters: first layer (128 + activation = rule,

kernel_initializer = “he_uniform”) and second layer (1
+ activation = sigmoid). Parameters are batch_size = 64,
loss = “binary_crossentropy,” mean = [123.68, 116.779,
103.939], featurewise_center = true, and optimizer =
SGD. We applied different optimizers; the best optimizer
is SGD, with a learning rate of 0.001 and a momentum of
0.9. Figure 7 shows the VGG16 model accuracy and loss
for 10 epochs.
F I G U R E 7 Performance of training and validation VGG16
(A) accuracy and (B) loss

4.3 | Results of AutoML

The result of different transfer learning architectures Table 3 presents the results of the TPOT AutoML applied
shows that ResNet and VGG16 achieved the best to the autistic children dataset. We train TPOT with dif-
performance, and they improved the CNN accuracy. The ferent population and generation sizes. The best process
highest accuracy achieved was 89% with the VGG16 runs for 5 generations and 60 population sizes. The ET
model by applying two dense layers with specific classifier is an output of the pipeline that processes as
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
620 ELSHOKY ET AL.

F I G U R E 8 Summary performance of Keras slim residual F I G U R E 1 0 Performance of machine learning techniques.

neural network AutoML, automated machine learning

T A B L E 5 Comparison of the proposed model performance

with other papers

Using same Using other

dataset datasets
Proposed model [22] [21] [20]
96.0% 94.4% 89.2% 43.0%

F I G U R E 9 Part of the activation maps for features of Keras

5 | DISCUSSION
slim residual neural network classifier
Using traditional ML algorithms, the best model was ET
with an accuracy of 72.64% with the least time of 4.25 s.
However, using VGG16, a DL neural network architec-
the best ML model with specific parameters. The test ture applied with transfer learning achieved an accuracy
evaluates the accuracy of 96.6%. The TPOT process with of 89% compared with 84% using the CNN model with
GPU took 4 h and 45 min to obtain this result. We 5 layers and 100 epochs. However, using the AutoML
trained the HyperoptEstimator class with different pipe- technique to search among possible classifiers and
lines and times. The ET classifier is repeated more than preprocessing modules, we obtained an optimized model
once in the output, so we choose it. that achieved approximately 96% accuracy on the test set.
Table 4 presents the results of the Hyperparameter Figure 10 compares the accuracy of traditional ML
AutoML applied to the autistic children dataset. The final methods, DL, and AutoML. When we trained the dataset
Hyperpot performance of 96.2% is the average of five using traditional ML and DL, the test accuracy of DL was
results achieved using different times and pipelines. The 16% higher than the traditional ML. However, AutoML
results are for four classifiers: Keras slim residual neural improved the accuracy by about 24% more than using
network, average blender, Elastic-Net, and baseline classical ML methods and 7% more than the best DL
image classifier. Keras slim residual neural network clas- model. DL with transfer learning VGG16 achieved an
sifier is the best classifier compared with other classifier accuracy of 89% more than the CNN DL algorithm with
performance. an accuracy of 84%. Moreover, AutoML achieved the
Figure 8 presents the summary performance of highest accuracy of 96%.
Keras slim residual neural network classifier using Therefore, we can obtain the highest performance of
training schedule (one layer: 64 units). It measures the the ML model with optimization and hyperparameters
performance using a confusion matrix that explains the using AutoML, which saves time and effort required for
actual and predated results. It also presents a number feature engineering.
of the statistical results: ROC curve, prediction of distri- Table 5 compares our proposed model performance
bution, and cumulative charts. Figure 9 shows the with other models in [20–22] using the facial image
description of a part of the activation maps for features dataset. Our proposed model achieves higher accuracy
that the Keras slim residual neural network used in than [22] in the same dataset. It also achieves higher
the validation set. accuracy than [20,21] using another facial image dataset.
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ELSHOKY ET AL. 621

Our approach increases the accuracy of the ASD model RE FER EN CES
that identifies autism using facial images. The proposed 1. A. Cox, K. Klein, T. Charman, G. Baird, S. Baron-Cohen, J.
AutoML model outperforms other ML models in the Swettenham, A. Drew, and S. Wheelwright, Autism spectrum
literature for the different ASD image datasets. Addition- disorders at 20 and 42 months of age: Stability of clinical and
ally, the AutoML model achieves higher accuracy than ADI-R diagnosis, J. Child Psychol. Psychiatry 40 (1999), no. 5,
719–732.
models in the literature using the same dataset.
2. A. Kleinsmith and N. Bianchi-Berthouze, Affective body expres-
sion perception and recognition: A survey, IEEE Trans. Affect.
Comput. 4 (2012), no. 1, 15–33.
6 | C ON C L U S I ON S 3. M. Boutrus, S. Z. Gilani, G. A. Alvares, M. T. Maybery, D. W.
Tan, A. Mian, and A. J. O. Whitehouse, Increased facial
This paper investigates the hypothesis of creating a model asymmetry in autism spectrum conditions is associated with
that identifies autism from facial images using various symptom presentation, Autism. Research 12 (2019), no. 12,
1774–1783.
classical, DL, and AutoML ML techniques. The result
4. S. Russell and P. Norvig, Artificial intelligence: A modern
shows that using AutoML methods can generate an opti-
approach, 3rd ed., Prentice Hall Press, USA, 2009.
mal model for ASD image classification with high accu- 5. C. M. Bishop, Pattern recognition and machine learning,
racy. AutoML is a state-of-the-art method for ML. Our Springer, 2006.
aim is to find the best optimized ML to identify autism 6. M. Feurer and F. Hutter, Hyperparameter optimization,
using autistic children dataset “facial images.” Tradi- Automated machine learning, Springer, Cham, 2019,
tional ML models require more time and process to find pp. 3–33.
7. M. Blohm, M. Hanussek, and M. Kintz, Leveraging automated
the highest performance parameters.
machine learning for text classification: Evaluation of AutoML
DL cannot make the model more optimized. Thou-
tools and comparison with human performance, 2020. arXiv
sands of possible pipelines can be evaluated using the preprint arXiv:2012.03575.
AutoML method to find the best model. 8. J. Dafflon, W. H. L. Pinaya, F. Turkheimer, J. H. Cole, R.
This paper is the first that used AutoML in ASD clas- Leech, M. A. Harris, S. R. Cox, H. C. Whalley, A. M. McIntosh,
sification. The result supports the hypothesis that we can and P. J. Hellyer, An automated machine learning approach to
create an optimized model using the AutoML method, predict brain age from cortical anatomical measures, Hum.
which identifies autism in children from facial images. Brain Mapp. 41 (2020), no. 13, 3555–3566.
9. T. Han, F. N. B. Gois, R. Oliveira, L. R. Prates, and M. M. De
The AutoML method requires time to produce the best
Almeida Porto, Modeling the progression of COVID-19 deaths
classifier; however, it outperforms the traditional ML using Kalman Filter and AutoML, Soft. Comput. 1 (2021),
algorithm and DL models. Although AutoML achieved 1–16.
the best accuracy, it still suffers from the same problem 10. J. M. Rehg, Behavior imaging: Using computer vision to study
of DL “Black-Box” models, which is model interpretabil- autism, in Proc. IAPR Conf. Machine. Vision Appicat. (Nara,
ity. The produced models are not understandable by Japan) June 2011, pp. 14–21.
humans. 11. S. Rajagopalan, A. Dhall, and R. Goecke, Self-stimulatory
behaviours in the wild for autism diagnosis, in Proc. IEEE Int.
In future studies, we will combine image data with
Conf. Comput. Vision Workshops (Sydney, Australia), Dec.
other modalities of data, such as behavioral and genetic 2013, pp. 755–761.
data, to predict ASD in children using AutoML. We will 12. S. S. Rajagopalan, Computational behaviour modelling for
also improve the interpretability of the AutoML methods. autism diagnosis, in Proc. ACM Int. Conf. Multimodal Interac-
tion (Sydney Australia), Dec. 2013, pp. 361–364.
CONFLICT OF INTEREST 13. J. Rehg, G. Abowd, A. Rozga, M. Romero, M. Clements, S.
The authors declare that there are no conflicts of interest. Sclaroff, I. Essa, O. Ousley, Y. Li, and C. Kim, Decoding
children’s social behavior, in Proc. IEEE Conf. Comput. Vision
Pattern Recogn. (Portland, OR, USA), June 2013,
ORCID pp. 3414–3421.
Basma Ramdan Gamal Elshoky https://orcid.org/0000- 14. M. Leo, P. Carcagnì, C. Distante, P. L. Mazzeo, P. Spagnolo, A.
0003-2247-1498 Levante, S. Petrocchi, and F. Lecciso, Computational analysis
Eman M. G. Younis https://orcid.org/0000-0003-2778- of deep visual data for quantifying facial expression production,
4231 Appl. Sci. 9 (2019), no. 21, 4542.
15. M. D. Samad, N. Diawara, J. L. Bobzien, J. W. Harrington,
Abdelmgeid Amin Ali https://orcid.org/0000-0002-
M. A. Witherow, and K. M. Iftekharuddin, A feasibility study of
6291-4516 autism behavioral markers in spontaneous facial, visual, and
Osman Ali Sadek Ibrahim https://orcid.org/0000-0001- hand movement response data, IEEE Trans. Neural Syst.
9254-3093 Rehabil. Eng. 26 (2017), no. 2, 353–361.
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
622 ELSHOKY ET AL.

16. P. Mazumdar, G. Arru, and F. Battisti, Early detection of AUTHOR BIOGRAPHIES

children with autism spectrum disorder based on visual exploration
of images, Signal Process. Image Commun. 94 (2021), 116184.
Basma Elshoky received her B.Sc.
17. M. Leo, P. Carcagnì, P. L. Mazzeo, P. Spagnolo, D. Cazzato,
and M.Sc. degrees in computer sci-
and C. Distante, Analysis of facial information for healthcare
applications: A survey on computer vision-based approaches, ence from the Faculty of Science,
Information 11 (2020), no. 3. https://doi.org/10.3390/ Minia University, Minia, Egypt, in
info11030128 2016 and 2021, respectively. From
18. W. Liu, M. Li, and L. Yi, Identifying children with autism 2016 to 2019, she worked as a free-
spectrum disorder based on their face processing abnormality: A lancer. Since 2019, she has been with
machine learning framework, Autism Res. 9 (2016), no. 8, the Information Technology Section, Korean Egyptian
888–898.
Faculty for Industry and Energy Technology, Beni
19. V. S. P. Patnam, F. T. George, K. George, and A. Verma, Deep
learning based recognition of meltdown in autistic kids, in Proc.
Suef Technological University, Beni Suef, Egypt,
IEEE Int. Conf. Healthcare Inform. (Park City, UT, USA), where she is now a lecture assistant. Her main
Aug. 2017, pp. 391–396. research interest is machine learning.
20. O. Rudovic, Y. Utsumi, J. Lee, J. Hernandez, E. C. Ferrer, B.
Eman M. G. Younis is currently
Schuller, and R. W. Picard, Culturenet: A deep learning
approach for engagement intensity estimation from face images
working as an Associate Professor at
of children with autism, in Proc. IEEE/RSJ Int. Conf. Intell. Minia University, Faculty of Com-
Robots Syst. (Madrid, Spain), Oct. 2018, pp. 339–346. puters and Information, Information
21. F. C. Tamilarasi, and J. Shanmugam, Convolutional neural net- Systems Department. She obtained
work based autism classification, in Proc. Int. Conf. Commun. her B.Sc. degree from Zagazig Uni-
Electron. Syst. (Coimbatore, India), June 2020, pp. 1208–1212. versity, Egypt, in 2002. She obtained
22. M. Beary, A. Hadsell, R. Messersmith, and M.-P. Hosseini,
her M.Sc. degree from Menoufia University, Egypt, in
Diagnosis of autism in children using facial analysis and deep
2007. She obtained her Ph.D. degree from Cardiff Uni-
learning, 2020. arXiv preprint arXiv:2008.02890.
23. S. Jahanara and S. Padmanabhan, Detecting autism from facial versity, UK, in 2014. She joined NTU, the Computing
image, 2021. Department, as a researcher.
24. Gerry PIOSENKA, Autistic children data set j Kaggle, 2020.
Abdelmgeid Amin Ali is a Profes-
25. E. Bisong, Google Colaboratory, Building machine learning and
deep learning models on Google Cloud Platform: A comprehen-
sor of Computer Science—Dean of
sive guide for beginners, Apress, Berkeley, CA, 2019, pp. 59–64. Faculty of Computers and Informa-
26. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. tion, Minia University, Egypt.
Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V.
Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M.
Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine
learning in Python, J. Mach. Learn. Res. 12 (2011), 2825–2830.
27. G. Bradski, The OpenCV library, Dr. Dobb’s J Softw Tools (2000). Osman Ali Sadek Ibrahim is an
28. Francois. Chollet, and other, Keras, 2015. https://github.com/ Assistant Professor at the Depart-
fchollet/keras ment of Computer Science, Faculty
29. J. Bergstra, D. Yamins, and D. D. Cox, Hyperopt: A python of Science, Minia University, Egypt.
library for optimizing the hyperparameters of machine learning His research interests include data
algorithms, in Proc. Python Sci. Conf. (Austin, TX, USA), 2013,
mining, information retrieval, evolu-
pp. 13–19.
30. B. Komer, J. Bergstra, and C. Eliasmith, Hyperopt-sklearn:
tionary algorithms, and machine
Automatic hyperparameter configuration for scikit-learn, in learning techniques. He obtained his B.Sc. degree
Proc. Python Sci. conf., 2014, pp. 32–37. from the Department of Computer Science, Faculty of
31. R. S. Olson, N. Bartley, R. J. Urbanowicz, and J. H. Moore, Science, Minia University, Egypt, in 2002. He also
Evaluation of a tree-based pipeline optimization tool for auto- obtained his M.Sc. degree by research in Computer
mating data science, in Proc. Genetic Evolutionary Computat. Science titled: “Using Genetic Algorithms to improve
Conf. (Denver, CO, USA), July 2016, pp. 485–492. Information Retrieval” in 2007 from the Department
32. F.-A. Fortin, F.-M. De Rainville, M.-A. G. Gardner, M.
of Computer Science, Minia University, Egypt. The
Parizeau, and C. Gagné, DEAP: Evolutionary algorithms made
easy, J. Mach. Learn. Res. 13 (2012), no. 1, 2171–2175.
main subject of his M.Sc. research is about using a
 22337326, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2021-0097 by Egyptian National Sti. Network (Enstinet), Wiley Online Library on [28/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ELSHOKY ET AL. 623

genetic algorithm to improve query learning prob- How to cite this article: B. R. G. Elshoky, E. M.
lems. Recently, he obtained his Ph.D. degree from the G. Younis, A. A. Ali, and O. A. S. Ibrahim,
University of Nottingham, Faculty of Science, School Comparing automated and non-automated machine
of Computer Science. His Ph.D. thesis is titled: “Evo- learning for autism spectrum disorders classification
lutionary Algorithms and Machine Learning Tech- using facial images, ETRI Journal 44 (2022),
niques for Information Retrieval.” 613–623. https://doi.org/10.4218/etrij.2021-0097