TYPE Original Research

PUBLISHED 21 September 2022

DOI 10.3389/fpsyt.2022.993077

A machine learning-based
OPEN ACCESS diagnostic model for children
EDITED BY
Antonio Narzisi,
Stella Maris Foundation (IRCCS), Italy
with autism spectrum disorders
REVIEWED BY
Daniel Holzinger,
complicated with intellectual
Hospitaller Brothers of Saint John
of God Linz, Austria
Muhammad Awais Bin Altaf,
disability
Lahore University of Management
Sciences, Pakistan
Chao Song1 , Zhong-Quan Jiang2 , Li-Fei Hu1 , Wen-Hao Li1 ,
*CORRESPONDENCE
Zhi-Wei Zhu Xiao-Lin Liu1 , Yan-Yan Wang1 , Wen-Yuan Jin1 and
[email protected] Zhi-Wei Zhu1*
SPECIALTY SECTION 1
Department of Developmental and Behavioral Pediatrics, The Children’s Hospital, Zhejiang
This article was submitted to
University School of Medicine, National Clinical Research Centre for Child Health, Hangzhou,
Autism,
China, 2 School of Public Health, Lanzhou University, Lanzhou, China
a section of the journal
Frontiers in Psychiatry

RECEIVED 13July 2022

ACCEPTED 01 September 2022
PUBLISHED 21 September 2022 Background: Early detection of children with autism spectrum disorder (ASD)
CITATION and comorbid intellectual disability (ID) can help in individualized intervention.
Song C, Jiang Z-Q, Hu L-F, Li W-H,
Appropriate assessment and diagnostic tools are lacking in primary care. This
Liu X-L, Wang Y-Y, Jin W-Y and
Zhu Z-W (2022) A machine study aims to explore the applicability of machine learning (ML) methods in
learning-based diagnostic model diagnosing ASD comorbid ID compared with traditional regression models.
for children with autism spectrum
disorders complicated with intellectual Method: From January 2017 to December 2021, 241 children with ASD, with
disability.
Front. Psychiatry 13:993077. an average age of 6.41 ± 1.96, diagnosed in the Developmental Behavior
doi: 10.3389/fpsyt.2022.993077 Department of the Children’s Hospital Affiliated with the Medical College
COPYRIGHT of Zhejiang University were included in the analysis. This study trained the
© 2022 Song, Jiang, Hu, Li, Liu, Wang,
traditional diagnostic models of Logistic regression (LR), Support Vector
Jin and Zhu. This is an open-access
article distributed under the terms of Machine (SVM), and two ensemble learning algorithms [Random Forest
the Creative Commons Attribution (RF) and XGBoost]. Socio-demographic and behavioral observation data
License (CC BY). The use, distribution
or reproduction in other forums is were used to distinguish whether autistic children had combined ID. The
permitted, provided the original hyperparameters adjustment uses grid search and 10-fold validation. The
author(s) and the copyright owner(s)
are credited and that the original Boruta method is used to select variables. The model’s performance was
publication in this journal is cited, in evaluated using discrimination, calibration, and decision curve analysis (DCA).
accordance with accepted academic
practice. No use, distribution or Result: Among 241 autistic children, 98 (40.66%) were ASD comorbid ID. The
reproduction is permitted which does
not comply with these terms.
four diagnostic models can better distinguish whether autistic children are
complicated with ID, and the accuracy of SVM is the highest (0.836); SVM
and XGBoost have better accuracy (0.800, 0.838); LR has the best sensitivity
(0.939), followed by SVM (0.952). Regarding specificity, SVM, RF, and XGBoost
performed significantly higher than LR (0.355). The AUC of ML (SVM, 0.835
[95% CI: 0.747–0.944]; RF, 0.829 [95% CI: 0.738–0.920]; XGBoost, 0.845 [95%
CI: 0.734–0.937]) is not different from traditional LR (0.858 [95% CI: 0.770–
0.944]). Only SVM observed a good calibration degree. Regarding DCA, LR,
and SVM have higher benefits in a wider threshold range.

Frontiers in Psychiatry 01 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

Conclusion: Compared to the traditional regression model, ML model based

on socio-demographic and behavioral observation data, especially SVM, has
a better ability to distinguish whether autistic children are combined with ID.

KEYWORDS

artificial intelligence, machine learning, child, autism spectrum disorder, intellectual

disability, diagnostic model

Introduction problems. Our clinical experience also suggests that the goal
of intervention is to promote integration into mainstream
Autism spectrum disorder (ASD) is a neurodevelopmental society for most children with normal abilities, while working
disorder characterized by social disorder and restricted, toward self-care for most children with low abilities. Overall,
repetitive, stereotyped behavior (1). The worldwide prevalence its judgment of an autistic child’s IQ informs the planning of
of ASD has increased year by year. According to the monitoring individualized interventions.
data released by the United States in 2021, the prevalence of ASD Wechsler Intelligence Scale for Children-IV (WISC-IV) is
is higher than 2.27%, that is, one in 44 children has ASD. ASD the most commonly used tool for evaluating intelligence levels
has become one of the fastest-growing diseases in children and a (20), and it is also considered to be suitable for children
public health problem threatening children’s health (2–4), which on the ASD (21). However, the use and scoring of WISC-
not only affects the life quality of children but also increases the IV need to be authorized by relevant parties, with high
economic burden on society and families due to the high cost of acquisition costs, many standardized test items, and specific
the intervention (5). training for evaluators, which hinder the primary medical
The proportion of ASD combined with intellectual disability workers from evaluating the intelligence level of children with
(ID) is about 33–35% (2, 6), with more comorbidities, such as ASD. Furthermore, medical and health resources distribution is
epilepsy and self-injurious behavior (7, 8), and more medical uneven in developing countries like China (22), with millions
expenses (9), but they may be less effective in social skills of children with ASD (23). Therefore, it is unrealistic to
training through behavioral interventions (10). Thus, the basic carry out comprehensive intelligence assessments like WISC-
intelligence quotient (IQ) level of children with ASD affects the IV. Therefore, it is particularly urgent to develop a simple and
intervention effect (11). Since the dose-response relationship effective diagnostic model for primary pediatric medical workers
between weekly intervention duration (dose) and IQ scores to identify ID in autistic children. Using behavioral observation
(response) was confirmed (12), increasing the intensity of the results to evaluate the intelligence of children with ASD can
intervention for ASD combined with ID can be considered. simplify the diagnostic process of autistic children with ID (24),
It is also critical to identify children with autism of average and help promote the application at the grass-roots level.
intelligence, as their adaptive functioning lags behind their IQ In the research of disease diagnosis, regression analysis is a
(13–15). Early identification and intervention for children with commonly used diagnostic method (25–29), which is simple and
autism of average intelligence can improve social and vocational interpretable, such as Cox regression and Logistics regression
outcomes in this population (16). In terms of intervention (LR). However, traditional regression methods mainly deal with
content, training in social adjustment should be as important low-level relations, making it difficult to analyze high-level non-
as social intervention for children with autism who have an linear relations (28, 30–32). The correlation between influencing
IQ greater than 70, while training in cognitive skills is also factors and outcomes is often non-linear in epidemiological
important for children with autism with comorbid ID. Not data. Concurrently, linear regression models are used to fit the
only that, but the intelligence level of children with ASD is non-linear relation, and the results are often biased. Machine
also related to their emotions. Anxiety is the most common learning (ML) is a set of computational methods that can
emotional problem in children with ASD, but in most cases discover complex non-linear relations between inputs and
it is difficult to distinguish the symptoms of anxiety from outputs, which has been widely used in disease diagnosis and
those of ASD (17). Although anxiety and depression also health research (27, 30, 33, 34). Support Vector Machine (SVM)
frequently occur in children with ASD who have normal IQ is a class of ML learners that performs exceptionally well
(17, 18), identifying emotional problems such as anxiety in on small sample datasets (35). Ensemble learning is a widely
children with ASD combined with ID can be more difficult used method with excellent performance (36). The Random
(19). Therefore, intellectual assessment of children with ASD Forest (RF) of Bagging ensemble idea and XGBoost of Boosting
may help in the early detection of their emotion-related ensemble idea are two of the most representative models.

Frontiers in Psychiatry 02 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

The existing predictive diagnosis of ASD or ASD ASD combined with ID. Socio-demographic information and
comorbidities (e.g., attention deficit hyperactivity disorder) behavioral observation data were used as variables in this study,
usually includes four aspects, disease prevention or risk and the findings were the result of the WISC-IV assessment
factor identification, disease diagnosis, disease efficacy (see the Appendix). Socio-demographic information includes
prediction, and disease prognosis prediction. Most of these gender, age at the time of behavioral observation, parent’s
diagnostic models use complex diagnostic-related data, such education. The behavioral observation items were formulated
as expensive head MRI, EEG (37–39), and blood biochemical concerning Autism Diagnostic Observation Schedule (ADOS).
indicators. These data were used to build diagnostic models to First, the language ability was assessed and divided into three
diagnose ASD-related disorders and to determine their type types: pre-verbal and single words, phrase speech, as well
or severity. Currently, much of the past research on diagnostic as fluent speech. Eleven aspects of behavioral observations
models has focused on diagnostic imaging, neglecting the include whether ASD children have stereotyped use of words
importance of demographic and behavioral observational or phrases, pointing/gestures, unusual eye contact, facial
data (40). Meanwhile, fewer studies have focused on early expressions toward others, the quality of actively expressing
diagnosis and screening of ASD combined with ID (41, 42). social intentions, unusual sensory interest in-game materials
Furthermore, in the context of healthcare resource shortages or people, complex mannerisms, unusual, repeated interest
and COVID-19 pandemic, the application of simple and or stereotyped behavior, overactivity, negative behaviors, and
effective diagnostic tools geared toward most primary care anxiety (Table 1). The scoring criteria are detailed in the
physicians can greatly reduce the burden on the healthcare table below. The intelligence level of children with ASD
system. Thus, early diagnosis can identify at-risk populations was assessed using WISC-IV, the fourth edition, suitable for
and initiate personalized interventions while seeking further children and adolescents aged 6–16 years (20). This study
help from higher levels of care. This is the most cost-effective was approved by the Ethics Committee of the Children’s
approach. Fortunately, with the increased availability of Hospital, Zhejiang University School of Medicine (No. 2022-
data from cross-sectional pediatric studies in China, both IRB-014).
demographic and behavioral observations are available. Using
these data, diagnostic models can be constructed to help
identify autistic children combined with ID at the early Feature selection with Boruta
diagnostic stage.
This article intends to combine the behavioral observations The choice of variables has a decisive impact on the
and socio-demographic data of children with ASD, apply final model performance. Therefore, this study used the
ML methods to the diagnosis of ID in autistic children, Boruta method to process the variable list, a feature
and optimize the diagnostic model through feature selection. selection method established using a random forest
Simultaneously, SVM, RF, and XGBoost models are compared classifier (43). It determined the importance of variables
with the traditional LR model. Finally, comprehensive use by comparing the correlation between real and shaded
of discrimination, calibration, and decision curve analysis features. Traditional feature selection algorithms often use
(DCA) to evaluate and screen the optimal diagnostic model filtering, so it is easy to discard some relevant features to
provides a new perspective for early diagnosis of autistic minimize errors. However, Boruta is a wrapper method
children with ID. that can find all feature sets through a fully correlated

TABLE 1 Behavioral observation scoring criteria.

Materials and methods
Behavioral observation variables 0 point 1 point 2 points
Study design and population Stereotyped speech Scarcely Partial Many
Pointing/gestures Many Partial Scarcely
This study retrospectively collected ASD data from January
Unusual eye contact Scarcely Partial Many
2017 to December 2021 in the department of Developmental
Facial expression Many Partial Scarcely
and Behavioral Pediatrics, the Children’s Hospital, Zhejiang
Social quality Good Average Bad
University School of Medicine. These children with ASD
Unusual sensory interest Scarcely Partial Many
met the diagnostic criteria for ASD in the Diagnostic and
Complex mannerisms Scarcely Partial Many
Statistical Manual of Mental Disorders, 5th Edition (DSM-
Repetitive stereotyped behaviors Scarcely Partial Many
5). They were also assessed by the WISC-IV and the
Overactivity Scarcely Partial Many
Adaptive Behavior Assessment System-II (ABAS-II) for adaptive
Negative behaviors Scarcely Partial Many
behavior. Those children who scored below 70 on the WISC-
Anxiety Scarcely Partial Many
IV and had an adaptive disorder were diagnosed with

Frontiers in Psychiatry 03 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

TABLE 2 The socio-demographic information and behavioral observations of autistic children.

Variable Total Whether or not with ID P-value

Yes No

241 98 143 –
Gender
Male 202 (83.82%) 75 (76.53%) 127 (88.81%) 0.013
Female 39 (16.18%) 23 (23.47%) 16 (11.19%)
Age 6.41 ± 1.96 6.01 ± 1.75 6.67 ± 2.05 0.010
Mother’s education attainment
Primary school 12 (4.98%) 6 (6.12%) 6 (4.20%) 0.032
Secondary school 37 (15.35%) 17 (17.35%) 20 (13.99%)
High school 30 (12.44%) 19 (19.39%) 11 (7.69%)
College/university 147 (61.00%) 52 (53.06%) 95 (66.43%)
Graduate and above 15 (6.22%) 4 (4.08%) 11 (7.69%)
Father’s education attainment
Primary school 6 (2.49%) 2 (2.04%) 4 (2.80%) 0.971
Secondary school 53 (21.99%) 24 (24.49%) 29 (20.28%)
High school 36 (14.94%) 12 (12.24%) 24 (16.78%)
College/university 133 (55.19%) 50 (51.02%) 73 (51.05%)
Graduate and above 23 (9.54%) 10 (10.20%) 13 (9.09%)
Language ability
Pre-verbal/single words 42 (17.43%) 28 (28.57%) 14 (9.79%) <0.001
Phrase speech 122 (50.62%) 65 (66.33%) 57 (39.86%)
Fluent speech 77 (31.95%) 5 (5.10%) 72 (50.35%)
Behavioral observation
Stereotyped speech 1.00 (0.00, 1.00) 1.00 (0.00, 1.25) 1.00 (0.00, 1.00) 0.002
Pointing/gestures 2.00 (1.00, 2.00) 2.00 (2.00, 2.00) 2.00 (1.00, 2.00) <0.001
Unusual eye contact 2.00 (0.00, 2.00) 2.00 (2.00, 2.00) 2.00 (0.00, 2.00) 0.002
Facial expression 1.00 (0.00, 1.00) 1.00 (0.00, 1.00) 1.00 (0.00, 1.00) 0.002
Social quality 1.00 (1.00, 2.00) 2.00 (1.00, 2.00) 1.00 (1.00, 1.00) <0.001
Unusual sensory interest 0.00 (0.00, 1.00) 0.00 (0.00, 1.25) 1.00 (0.00, 1.00) 0.001
Complex mannerisms 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.026
Repetitive stereotyped behaviors 1.00 (0.00, 2.00) 1.00 (0.00, 1.00) 2.00 (1.00, 2.00) <0.001
Overactivity 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.059
Negative behaviors 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.319
Anxiety 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.00 (0.00, 0.00) 0.721

strategy, so weakly correlated predictors can also be Diagnostic models

preserved (43).
To quantitatively evaluate the impact of feature selection Logistics regression
on the model, we separately constructed models based on all This study compares traditional LR to other ML methods
variables and the variables of feature selection and evaluated using it as a reference model. The binary logistic regression used
the contribution of feature selection to the diagnostic model in the study had only two possible outcome values (low versus
using integrated discrimination improvement index (IDI) high function). The log-odds of values marked “1” in LR are
(44). Performance improved when IDI was greater than linear combinations of predictors, and LR can be expressed as:
0. Performance was reduced when IDI was less than 0.
exp(wT x + b)
When IDI was set to 0, the performance remained constant. p(y = 1|x) = .
1 + exp(wT x + b)
Among them, hypothesis testing was used to identify whether
changes in model performance were statistically significant where x is the predicted value, y is the result, w is the weight of
(two-sided P < 0.05 was considered statistically significant). each predicted value, and b is the intercept.

Frontiers in Psychiatry 04 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

FIGURE 1
Boruta-based feature selection results.

TABLE 3 Comparison of full variables and variables after feature selection.

Type Quantity Variable list

Full variable 16 Gender, age, mother.edu, father.edu, language ability, stereotyped speech, pointing/gestures, unusual eye
contact, facial expression, social quality, unusual sensory interest, complex mannerisms, repetitive
stereotyped behaviors, overactivity, negative behaviors, and anxiety
Feature selection 10 Age, mother.edu, language ability, stereotyped speech, pointing/gestures, facial expression, social quality,
unusual sensory interest, negative behaviors, and repetitive stereotyped behaviors

Support vector machine and having a stable performance. Unfortunately, linear kernels
are difficult to analyze in non-linear data, and radial basis
The final decision function of SVM is only determined function kernels are the most commonly used. Therefore, this
by a very small number of support vectors (35). Thus, it has study adopts the radial basis function kernel for modeling.
good performance on small samples. However, most research
problems in the real world are often non-linear. To solve this Ensemble learning method
problem, SVM maps non-linear data to high latitudes through This study selects RF and XGBoost as representative
a kernel function, making it linearly separable at high latitudes ensemble learning models. RF is one of the commonly used

Frontiers in Psychiatry 05 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

TABLE 4 Improvement index before and after feature selection.

Statistical analysis
Model IDI 95% CI P-value
Continuous variables were expressed as mean ± SD (normal
LR 0.049 −0.018 to 0.117 0.150 distribution) and median of the interquartile range (IQR,
RF 0.043 0.012–0.075 0.006 skewed distribution), and categorical variables were expressed
SVM 0.382 0.244–0.520 <0.001 as percentages. The χ2 test or Fisher’s exact test was used
XGBoost 0.103 0.022–0.184 0.011 to compare the rates, the t-test was used to compare the
means of normal distribution, the rank-sum test was used
to compare the means of non-normal distribution, and the
Bagging methods. Breiman proposed it in 2001 (32). It univariate analysis was used to compare the multi-category
can parallelly generate multiple decision tree classifiers and distribution. Two-sided P < 0.05 was considered statistically
integrate them through voting to realize data classification. significant. All methods in this study were implemented with
The random characteristics are mainly reflected in (1) using R 3.6.0 with a random seed set to 123. The Boruta feature
the bootstrap method to generate multiple datasets of the selection method is derived from Boruta version 7.0.0. pROC
same size. (2) The variables in each decision tree node are version 1.17.0.1 is used for pairwise comparison of AUC, ROC
randomly combined from the list. XGBoost, known as the calculation, and drawing. PredictABEL version 1.2-2 is used
boost policy algorithm, is another ensemble learning method. for the implementation of the IDI evaluation method; e1071
The main difference between Boosting and Bagging is that version is used for model parameter tuning 1.7-6; randomForest
constructing the following weak classifier depends on its version 4.6-14 is used to build random forest; e1071 version
previous classifier. XGBoost was proposed by Chen et al. (45), 1.7-6 is used to build SVM; XGBoost version 1.2.0.1 is used
aiming to solve the computational problem when gradient to build XGBoost.
boosting decision tree (GBDT) encounters large data sets.
Compared to GBDT, XGBoost improvement uses both the
first and second derivatives, which significantly speeds up
Results
the computation. Furthermore, XGBoost uses multiple weak
classifiers to fit the prediction residuals and integrates all the
Patient characteristics
weak learners to obtain a robust ML model.

A total of 241 children with ASD were included, including

98 with ID, with an average age of 6.41 ± 1.96 (Table 2).
Model derivation and internal
validation
Selection of predictors using Boruta
The dataset is divided into the training set and test
set with a ratio of 7:3, which are used for modeling and The Boruta-based feature selection results are displayed
model performance evaluation. In the modeling phase, Boruta’s in Figure 1. The three blue features represent the maximum
method filters variables, and 10-fold cross-validation and grid Z-score, average Z-score, and minimum Z-score of the
search were used to tune hyperparameters. The subjects’
accuracy, precision, sensitivity, specificity, and area under curve TABLE 5 Diagnostic models performance.
(AUC) were used to evaluate the model’s discrimination. The
Bootstrap method was used to calculate the 95% confidence Model Accuracy Precision Sensitivity Specificity AUC
(95% CI)
interval (95% CI) of AUC, and De-long test was used to compare
AUC between diagnostic models. Additionally, each diagnostic LR 0.712 0.672 0.976 0.355 0.858
model is evaluated for its agreement with the ground truth (0.770–0.944)
using a calibration plot. Finally, we performed DCA to provide
a reference for selecting the best diagnostic model and clinical RF 0.726 0.789 0.714 0.742 0.829
practice. DCA is a simple method for evaluating diagnostic (0.738–0.920)

models that consider both accuracy and clinical utility, as

Andrew et al. proposed in 2006 (46). SVM 0.836 0.800 0.952 0.677 0.845
(0.747–0.944)
Furthermore, we use the SHapley Additive exPlanations
(SHAP) method to evaluate the interpretability of the optimal
XGBoost 0.767 0.838 0.738 0.806 0.845
model. The SHAP method is derived from game theory based
(0.734–0.937)
on the SHAP value, which shows the importance of variables and
determines the direction of effects (47, 48).

Frontiers in Psychiatry 06 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

FIGURE 2
Performance of the diagnostic models.

shadow feature. Green features are accepted (Language ability, Comparisons between models in
Mother.edu, Stereotyped Speech, Pointing/Gestures, Facial internal validation
Expression, Social Quality, Unusual Sensory Interest, Negative
Behaviors, and Repetitive Stereotyped Behaviors), and red Comparisons of discrimination and calibration
features are rejected (gender, Father.edu, Unusual Eye Contact, The performance is displayed in Table 5 and Figure 2. We
Complex Mannerisms, Overactivity, and Anxiety). The yellow found that the accuracy of SVM was the best (0.836), and the
features are regarded as edge features (age). Finally, we included other three models were more consistent in accuracy; in terms
10 accepted variables and edge variables (Language ability, of precision, SVM and XGBoost performed better (0.800, 0.838).
Mother.edu, age, Stereotyped Speech, Pointing/Gestures, Facial
Expression, Social Quality, Unusual Sensory Interest, Negative
Behaviors, and Repetitive Stereotyped Behaviors) (Table 3). TABLE 6 De-long test, pairwise comparison between models.
In addition, IDI was calculated to evaluate the contribution
of feature processing features (Table 4). The results Model Z P-value
demonstrated that after feature selection, the prediction
LR vs. RF 1.016 0.309
performance of SVM, RF, and XGBoost improved to varying
LR vs. SVM 0.491 0.623
degrees (38.20, 4.30, and 10.30%), and the difference was
LR vs. XGBoost 0.190 0.848
statistically significant (P < 0.05). On the other hand, LR model
RF vs. XGBoost −0.497 0.618
has a certain degree of improvement, but the difference is not
RF vs. XGBoost −0.244 0.807
statistically significant (P > 0.05). Therefore, this study selects
SVM vs. XGBoost 0.011 0.990
10 variables for the final modeling.

Frontiers in Psychiatry 07 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

FIGURE 3
Calibration diagram.

LR (0.939) is the best in sensitivity, followed by SVM (0.952). more consistent with the actual situation, and the prediction
Regarding specificity, SVM, RF, and XGBoost performed more consistency between LR, RF, and XGBoost is poor (Figure 3).
prominently, all higher than LR (0.355). AUC of ML (SVM,
0.835 [95% CI: 0.747–0.944]; RF, 0.829 [95% CI: 0.738–0.920]; Comparisons of decision curve analysis
XGBoost, 0.845 [95% CI: 0.734–0.937]) vs. traditional LR (0.858 Decision curve analysis curve results are illustrated in
[95% CI: 0.770–0.944]) was not significantly different (Figure 2). Figure 4. The none line on the X-axis indicates that all autistic
Concurrently, no statistical difference was found between the children were non-ID and intervened, with a net benefit of 0.
models by De-long test, as revealed in Table 6. The ALL symbol represents the net benefit at various thresholds,
The calibration graph evaluates the consistency between the with an assumption that all children with ASD have ID. In
model prediction results and the actual situation. The SVM is most cases, we found that the net benefit of LR was higher

Frontiers in Psychiatry 08 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

FIGURE 4
DCA of the four models.

TABLE 7 Variable importance order of LR, SVM, RF, and XGBoost.

Variable Logistic regression SVM Random forest XGBoost Average rank

OR Rank Rank IncNodePurity Rank Gain Rank

Language ability 27.16 2 3 6.07 2 0.35 1 1

Repetitive stereotyped behaviors 4.84 3 1 3.15 5 0.1 3 2
Mother.edu 27.69 1 7 3.18 4 0.06 8 3
Social quality 0.11 7 5 2.46 6 0.09 4 4
Stereotyped speech 0.27 6 2 2.12 8 0.04 9 5
Pointing/gestures 0.07 8 8 3.25 3 0.07 6 5
Negative behaviors 0 10 4 0.41 10 0.13 2 7
Age 5.03 9 10 8.58 1 0.06 7 8
Facial expression 0.64 4 9 1.51 9 0.08 5 8
Unusual sensory interest 0.37 5 6 2.46 7 0.02 10 10

than that of SVM, RF, and XGBoost models. However, when finally choose SVM as the predictive model for whether autistic
the threshold exceeded 0.70, the net benefit of the two ensemble children are combined with ID.
methods returned to zero, while LR and SVM still had an
enormous net benefit.
In practice, the threshold is set according to actual Variable importance for diagnostic
requirements. We can use the proportion of non-ID and models
combined ID in autistic children with different demographic
characteristics to calculate and determine the net benefit of The results for the different importance of each model
different diagnostic models under the set conditions to evaluate are demonstrated in Table 7. The SVM model in this study
the practical value of the model. used a non-linear kernel function, and its variable weights
To summarize, SVM has higher accuracy, precision, were no longer linear, so the SVM importance could not
sensitivity, better calibration, and higher benefit over a be directly obtained. Therefore, SHAP method was used to
more extensive threshold range. Therefore, combining evaluate the variable importance of SVM model. For LR, the top
discrimination, calibration, and clinical decision curve, we five predictors were Mother.edu, Language ability, Repetitive

Frontiers in Psychiatry 09 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

Stereotyped Behaviors, Facial Expression, and Unusual Sensory effectively explain the complex non-linear relationships in
Interest. It demonstrated that the increase of Mother.edu, the data content.
Language ability, and Repetitive Stereotyped Behaviors would
increase the risk of autistic children being diagnosed with ID.
In SVM, the top five predictors are Repetitive Stereotyped Discussion
Behaviors, Speech, Language ability, Negative Behaviors, and
Social Quality. In RF, age, Language ability, Pointing/Gestures, In this study, based on the socio-demographic information
Mother.edu, and Repetitive Stereotyped Behaviors. In XGBoost, and behavioral observation data of children with ASD, the
the top five predictors were Father.edu, gender, Language ML methods were used to construct a diagnostic model
ability, Facial Expression, and Unusual Eye Contact. When the for whether autistic children were combined with ID. The
importance of predictors in the four models was examined, it results demonstrated that the ML methods could effectively
was discovered that language ability was vital in each model. distinguish whether autistic children combined ID. Before
We further used SHAP plots to understand the building a model requires sufficient data preprocessing, such
interpretability of SVM (Figure 5). The results revealed that as outlier identification, missing value filling, normalization,
the top five important predictors are: Repetitive Stereotyped feature selection, etc. The data integrity in this study was all
Behaviors, Stereotyped Speech, Language ability, Negative above 99%, indicating that the quality of the data used for
Behaviors, and Social Quality. Among them, the Stereotyped diagnosis was good. For variables, this study uses the methods of
Speech score was more complex in predicting whether autistic outlier identification, missing value imputation, normalization,
children were combined with ID. The outcome is positively and feature selection to deal with variables. Some studies have
correlated with the above predictors within a certain range and pointed out that sufficient data preprocessing should be carried
negatively correlated beyond this range. The effects of other out before training the diagnostic model (49).
predictors on long-term outcomes were mainly unidirectional, Taking the feature selection of this study as an example,
such as decreasing Repetitive Stereotyped Behaviors score, AUC-based discrimination in the full-variable prediction model,
decreasing Stereotyped Speech score, decreasing Language SVM, RF, and XGBoost all performed lower than the model
ability, increasing Negative Behaviors score, Social Quality after feature processing using the Boruta method. This also
score, and Unusual Sensory. Furthermore, an increase in the depicts that feature processing is essential to obtain a more
Interest score, a decrease in maternal education, an increase concise and effective classifier. LR, SVM, RF, and XGBoost all
in the Facial Expression score, and an increase in age all show good performance in this study. The AUCs of LR, SVM,
increase the likelihood of autistic children being diagnosed RF, and XGBoost are 0.858 (0.770–0.944), 0.829 (0.738–0.920),
with ID. The above results demonstrate that ML method can 0.845 (0.747–0.944), and 0.845 (0.734–0.937), respectively. In

FIGURE 5
SHAP value and importance of each feature in SVM.

Frontiers in Psychiatry 10 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

the existing evaluation system, there is no evaluation of AUC. onset of anxiety (62, 63). Fortunately, the severity of repetitive
The accepted threshold for assessing good classifiers. Rice et al. stereotypes can be alleviated through intervention training (64),
proposed that AUC can be converted to effect sizes, such as but whether RRB intervention can improve IQ levels requires
Cohen’s d and Pearson’s rpb (50). further research.
In this study, the Cohen’s d values of LR, SVM, RF, and Although language disorder (LD) is no longer a core
XGBoost were 1.515, 1.344, 1.436, 1.436, and rpb were 0.604, symptom of ASD, it is one of the common comorbidities of ASD
0.558, 0.583, and 0.583, respectively. According to Cohen’s d (1). About 63% of autistic children have LD (65). Interestingly,
intensity criteria, our diagnostic models are equivalent to high in all four models of this study, Language ability demonstrated
impact levels. According to the standard of impact strength significant significance for the diagnosis of ASD comorbid ID.
on rpb , traditional LR demonstrated high correlation, and In other words, language ability is an important diagnostic factor
SVM, RF, and XGBoost revealed moderate correlation levels. for whether children with ASD have combined ID, and the worse
In addition, some studies also show that the performance the language ability, the greater the possibility of combined ID.
of ML and regression models are comparable (27, 51, 52). In a broad sense, language ability includes language perception
However, the difference between ML and traditional regression and expression ability (66). Language ability in this model refers
models in this study is not obvious, which is also different to the latter, while the Verbal Comprehension Index (VCI)
from most current studies. Possible explanations are as follows: is directly included in WISC-IV, and the score of language
ML is good at processing big data, so complex rules may comprehension will become calculate the part of total IQ (67).
not be found in the case of limited data. At the same time, Many studies have revealed that non-verbal IQ may be a strong
research on autism-related diagnostic models focuses on high- positive predictor of language ability in children with ASD (68,
cost imaging materials such as MRI, while this study focuses 69). This study suggests that language ability also positively
on relatively easily available behavioral observations and socio- impacts IQ, so strengthening language intervention for children
demographic data (37, 53, 54). In addition, selecting the best with ASD may also have a positive effect.
variable combination also has certain difficulties. Pepe et al. The average level of the mother’s educational level ranks
suggest that even with the same study category and study data, third in the four models. That is to say, whether autistic children
the variable combinations of influencing factors and diagnostic are combined with ID may have a certain relationship with the
models may be extremely contradictory (55). In other words, a mother’s educational level. The higher the mother’s education
factor may be closely related to the disease, but its contribution level, the lower the likelihood of autistic children with ID. The
to the diagnostic model may be minor. As a result, we can lower education level of parents may affect their awareness
conclude that the diagnostic model obtained in this study of ASD, thereby delaying the diagnosis of children with ASD
is not optimal but only performs well in this sample. At (70), and mothers with lower education levels have greater
the same time, interpreting the final influencing factors still parenting pressure and are more likely to develop anxiety
requires relevant clinical knowledge and experience to solve and and depression (71). Moreover, mothers with higher education
explain the problem. levels can better regulate their emotions, actively carry out
Our research found that among the best SVM models, rehabilitation training for their children, and improve their
Repetitive Stereotyped Behaviors, Stereotyped Speech, and children’s abilities (72, 73). This conclusion is not a judgment
Language ability were the top three key variables. Repetitive of causality, and the specific mechanism remains to be further
Stereotyped Behaviors and Stereotyped Speech belong to RRBs studied, which is different from the overturned “refrigerator
(Restrictive and Repetitive Behaviors). Although RRB is one mother” theory (74).
of the two core symptoms of children with ASD, it is not Assessing the clinical utility of diagnostic models can
unique to autism. Other neurodevelopmental disorders and guide clinical practice. DCA was performed in this study,
even ordinary children may also have RRB manifestations combining discrimination, calibration, and DCA, and the
(56). Therefore, the diagnostic model constructed in this results showed that SVM was relatively superior in terms
study is only suitable for children diagnosed with autism. of clinical benefit and ability to discriminate against autistic
For a long time, the relationship between RRBs and IQ children with ID. Medical and health workers can collect
has not reached a consensus (57, 58). The diagnostic model socio-demographic data and behavioral observations of autistic
constructed by ML in this study suggests that the higher children and use the recommended SVM model for children
the score of repetitive, stereotyped behavior in children with with autism in grassroots units that cannot conduct systematic
autism, the greater the possibility of combined ID. However, intelligence assessments (such as the WISC-IV test, etc.).
the precise reason is unknown due to its complex biological Preliminary diagnosis of ID is the premise of a stepped care and
and psychological mechanism (59). In addition to impairing personalized health approach for children with different types
intelligence, RRBs can impair the physical and mental health of ASD (75).
of individuals with autism (59), affect social and daily living Our study may have some potential advantages. First, we
skills in children (60) and adolescents (61), and predict the used demographic and behavioral observational data from

Frontiers in Psychiatry 11 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

a population cross-section to construct a diagnostic model with Data availability statement

low costs of model construction and easy access to diagnostic
data. Therefore, initial diagnostic screening may be applicable The raw data supporting the conclusions of this article will
to primary care physicians. Second, the optimal SVM model is be made available by the authors, without undue reservation.
computationally efficient and can be quickly computed in real
time during practical application. In addition, we conducted
a relatively complete data pre-processing work to ensure the
Author contributions
processing effect of the prediction model and provide a solid
foundation for the model architecture. Finally, we conducted
CS conceived the study and wrote the draft of the
a comprehensive evaluation of the prediction model, including
manuscript. Z-QJ conducted literature searches and completed
traditional metrics (identification and calibration), and clinical
the data analysis. L-FH and X-LL engaged in behavioral
utility analysis (decision curves and prediction curves). The
observation. W-HL, Y-YW, and W-YJ conducted the IQ test.
SHAP method was also used with a view to discussing in further
Z-WZ critically revised the manuscript. All authors contributed
depth the interpretability of the ML model. Also we followed the
to the article and approved the submitted version.
standard reporting procedures for prediction models described
in TRIPOD (76).
The limitation of this study is that the sample size is
relatively small. If the sample size continues to increase, the
Funding
model trained by ML will be more convincing. Concurrently,
This study was supported by the Zhejiang Provincial Natural
this study only evaluates the generalization ability of the
Science Foundation of China (LGF20H090015).
diagnostic model through internal verification. Further
research must be conducted in other perform external
validation on ASD population in the region or subsequent
children on the ASD in the region. The diagnostic model Acknowledgments
is based on the current behavioral observation to evaluate
the current IQ, which solves the current situation that We thank International Science Editing (http://www.
the intelligence test cannot be fully promoted in real internationalscienceediting.com) for editing this manuscript.
situations. However, children with ASD are frequently
diagnosed in early childhood, and intelligence evaluation
results at this stage are not stable. Therefore, predicting Conflict of interest
school-age IQ through behavioral observations of children
with ASD in early childhood is more in line with clinical The authors declare that the research was conducted in the
reality, which needs to establish a follow-up cohort of absence of any commercial or financial relationships that could
autistic children. be construed as a potential conflict of interest.

Conclusion Publisher’s note

Based on the data of children with ASD in Zhejiang, All claims expressed in this article are solely those of the
China, the ML model can effectively distinguish whether authors and do not necessarily represent those of their affiliated
autistic children are combined with ID. Given the degree organizations, or those of the publisher, the editors and the
of discrimination, calibration, and clinical usefulness, we reviewers. Any product that may be evaluated in this article, or
believe SVM is the best model for screening autistic claim that may be made by its manufacturer, is not guaranteed
children with ID. or endorsed by the publisher.

References
1. American Psychiatric Association. Diagnostic and statistical manual of mental United States, 2018. MMWR Surveill Summ. (2021) 70:1. doi: 10.15585/mmwr.
disorders. Am Psychiat Assoc. (2013) 5:612–3. ss7011a1
2. Maenner MJ, Shaw KA, Bakian AV, Bilder DA, Durkin MS, Esler A, et al. 3. Zeidan J, Fombonne E, Scorah J, Ibrahim A, Durkin MS, Saxena S, et al. Global
Prevalence and characteristics of autism spectrum disorder among children aged prevalence of autism: a systematic review update. Autism Res. (2022) 15:778–90.
8 years—autism and developmental disabilities monitoring network, 11 sites, doi: 10.1002/aur.2696

Frontiers in Psychiatry 12 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

4. Bougeard C, Picarel-Blanchot F, Schmid R, Campbell R, Buitelaar J. . 23. Zhou H, Xu X, Yan W, Zou X, Wu L, Luo X, et al. Prevalence of autism
Prevalence of autism spectrum disorder and co-morbidities in children and spectrum disorder in China: a nationwide multi-center population-based study
adolescents: a systematic literature review. Front Psychiatry. (2021) 12:744709. among children aged 6 to 12 years. Neurosci Bull. (2020) 36:961–71. doi: 10.1007/
doi: 10.3389/fpsyt.2021.744709 s12264-020-00530-6
5. Tsiplova K, Jegathisawaran J, Mirenda P, Kalynchuk K, Colozzo P, Smith V, 24. Simeoli R, Milano N, Rega A, Marocco D. Using technology to identify
et al. Parent coaching intervention for children with suspected autism spectrum children with autism through motor abnormalities. Front Psychol. (2021)
disorder: cost analysis. Res Autism Spect Disord. (2022) 93:101949. doi: 10.1016/j. 12:635696. doi: 10.3389/fpsyg.2021.635696
rasd.2022.101949
25. Kohli PS, Arora S. Application of machine learning in disease prediction.
6. Maenner MJ, Shaw KA, Baio J, Washington A, Patrick M, DiRienzo M, et al. Proceedings of the 2018 4th International Conference on Computing Communication
Prevalence of autism spectrum disorder among children aged 8 years—autism and Automation (ICCCA): 2018. Greater Noida: IEEE (2018). p. 1–4. doi: 10.1109/
and developmental disabilities monitoring network, 11 sites, United States, 2016. CCAA.2018.8777449
MMWR Surveill Summ. (2020) 69:1. doi: 10.15585/mmwr.ss6904a1
26. Syafrullah M. Diagnosis of smear-negative pulmonary tuberculosis using
7. Tonnsen BL, Boan AD, Bradley CC, Charles J, Cohen A, Carpenter LA. ensemble method: a preliminary research. Proceedings of the 2019 6th International
Prevalence of autism spectrum disorders among children with intellectual Conference on Electrical Engineering, Computer Science and Informatics (EECSI):
disability. Am J Intellect Dev Disabil. (2016) 121:487–500. doi: 10.1352/1944-7558- 2019. Jakarta: IEEE (2019). p. 112–6. doi: 10.23919/EECSI48112.2019.8976920
121.6.487
27. Battineni G, Sagaro GG, Chinatalapudi N, Amenta F. Applications of
8. Miller JS, Bilder D, Farley M, Coon H, Pinborough-Zimmerman J, Jenson W, machine learning predictive models in the chronic disease diagnosis. J Pers Med.
et al. Autism spectrum disorder reclassified: a second look at the 1980s Utah/UCLA (2020) 10:21. doi: 10.3390/jpm10020021
autism epidemiologic study. J Autism Dev Disord. (2013) 43:200–10. doi: 10.1007/
s10803-012-1566-0 28. Shipe ME, Deppen SA, Farjah F. Grogan EL. Developing prediction models
for clinical use using logistic regression: an overview. J Thorac Dis. (2019) 1 1:S574.
9. Leibson C, Weaver A, Myers S, Long K, Ransom J, Voigt R, et al. Objective doi: 10.21037/jtd.2019.01.25
estimates of direct-medical costs among persons aged 3 to 38 years with and
without research-defined autism spectrum disorder ascertained during childhood: 29. Wang S, Ma P, Zhang S, Song S, Wang Z, Ma Y, et al. Fasting blood glucose
a population-based birth-cohort study. Value Health. (2020) 23:595–605. doi: 10. at admission is an independent predictor for 28-day mortality in patients with
1016/j.jval.2019.12.006 COVID-19 without previous diagnosis of diabetes: a multi-centre retrospective
study. Diabetologia. (2020) 63:2102–11. doi: 10.1007/s00125-020-05209-1
10. Yum YN, Lau WK-W, Poon K, Ho FC. Music therapy as social skill
intervention for children with comorbid ASD and ID: study protocol for a 30. Liu L. Research on logistic regression algorithm of breast cancer diagnose
randomized controlled trial. BMC Pediatr. (2020) 20:545. doi: 10.1186/s12887-020- data by machine learning. Proceedings of the 2018 International Conference on
02454-6 Robots & Intelligent System (ICRIS): 2018. Madrid: IEEE (2018). p. 157–60. doi:
10.1109/ICRIS.2018.00049
11. Taheri A, Meghdari A, Alemi M, Pouretemad H. Clinical interventions of
social humanoid robots in the treatment of a pair of high-and low-functioning 31. Thabtah F, Abdelhamid N, Peebles D. A machine learning autism
autistic Iranian twins. Sci Iran. (2018) 25:1197–214. doi: 10.24200/SCI.2017. classification based on logistic regression analysis. Health Inf Sci Syst. (2019) 7:1–11.
4337 doi: 10.1007/s13755-019-0073-5

12. Eldevik S, Titlestad KB, Aarlie H, Tønnesen R. Community implementation 32. Yang Q, Zhang P, Wu R, Lu K, Zhou H. Identifying the best marker
of early behavioral intervention: higher intensity gives better outcome. Eur J Behav combination in CEA, CA125, CY211, NSE, and SCC for lung cancer screening by
Anal. (2020) 21:92–109. doi: 10.1080/15021149.2019.1629781 combining ROC curve and logistic regression analyses: is it feasible? Dis Markers.
(2018) 2018:2082840. doi: 10.1155/2018/2082840
13. Charman T, Pickles A, Simonoff E, Chandler S, Loucas T, Baird G. IQ in
children with autism spectrum disorders: data from the Special Needs and Autism 33. Hatton CM, Paton LW, McMillan D, Cussens J, Gilbody S, Tiffin PA.
Project (SNAP). Psychol Med. (2011) 41:619–27. doi: 10.1017/S0033291710000991 Predicting persistent depressive symptoms in older adults: a machine learning
approach to personalised mental healthcare. J Affect Disord. (2019) 246:857–60.
14. Kraper CK, Kenworthy L, Popal H, Martin A, Wallace GL. The gap between doi: 10.1016/j.jad.2018.12.095
adaptive behavior and intelligence in autism persists into young adulthood and
is linked to psychiatric co-morbidities. J Autism Dev Disord. (2017) 47:3007–17. 34. Lane T, Russo DP, Zorn KM, Clark AM, Korotcov A, Tkachenko V,
doi: 10.1007/s10803-017-3213-2 et al. Comparing and validating machine learning models for mycobacterium
tuberculosis drug discovery. Mol Pharmaceut. (2018) 15:4346–60. doi: 10.1021/acs.
15. Alvares GA, Bebbington K, Cleary D, Evans K, Glasson EJ, Maybery MT, molpharmaceut.8b00083
et al. The misnomer of ‘high functioning autism’: intelligence is an imprecise
predictor of functional abilities at diagnosis. Autism. (2020) 24:221–32. doi: 10. 35. Cortes C, Vapnik V. Support-vector networks. Mach Learn. (1995) 20:273–97.
1177/1362361319852831 doi: 10.1007/BF00994018
16. Katusic MZ, Myers SM, Weaver AL, Voigt RG. IQ in autism 36. Breiman L. Random forests. Mach Learn. (2001) 45:5–32. doi: 10.1023/A:
spectrum disorder: a population-based birth cohort study. Pediatrics. (2021) 1010933404324
148:e2020049899. doi: 10.1542/peds.2020-049899
37. Kim JI, Bang S, Yang J-J, Kwon H, Jang S, Roh S, et al. Classification of
17. White SW, Oswald D, Ollendick T, Scahill L. Anxiety in children and preschoolers with low-functioning autism spectrum disorder using multimodal
adolescents with autism spectrum disorders. Clin Psychol Rev. (2009) 29:216–29. MRI data. J Autism Dev Disord. (2022) [Online ahead of print]. doi: 10.3389/fninf.
doi: 10.1016/j.cpr.2009.01.003 2014.00014
18. Waugh C, Peskin J. Improving the social skills of children with HFASD: an 38. Aslam AR, Altaf MAB. An on-chip processor for chronic neurological
intervention study. J Autism Dev Disord. (2015) 45:2961–80. doi: 10.1007/s10803- disorders assistance using negative affectivity classification. IEEE Trans Biomed
015-2459-9 Circuits Syst. (2020) 14:838–51. doi: 10.1109/TBCAS.2020.3008766
19. Bakken TL, Helverschou S, Høidal SH, Martinsen H. Mental illness with 39. Aslam AR, Altaf MAB. A 10.13µJ/classification 2-channel deep neural
intellectual disabilities and autism spectrum disorders. In: Hemmings C, Bouras network based SOC for negative emotion outburst detection of autistic children.
N editors. Psychiatric and behavioural disorders in intellectual and developmental IEEE Trans Biomed Circuits Syst. (2021) 15:1039–52. doi: 10.1109/TBCAS.2021.
disabilities. Cambridge: Cambridge University Press (2016). p. 119–28. 3113613
20. Takayanagi M, Kawasaki Y, Shinomiya M, Hiroshi H, Okada S, Ino T, 40. McCarty P, Frye RE. Early detection and diagnosis of autism spectrum
et al. Review of cognitive characteristics of autism spectrum disorder using disorder: why is it so difficult? Semin Pediatr Neurol. (2020) 35:100831. doi: 10.
performance on six subtests on four versions of the Wechsler intelligence scale 1016/j.spen.2020.100831
for children. J Autism Dev Disord. (2022) 52:240–53. doi: 10.1007/s10803-021- 41. Wolff N, Eberlein M, Stroth S, Poustka L, Roepke S, Kamp-Becker I, et al.
04932-x Abilities and disabilities-applying machine learning to disentangle the role of
21. Peristeri E, Silleresi S, Tsimpli IM. Bilingualism effects on cognition in autistic intelligence in diagnosing autism spectrum disorders. Front Psychiatry. (2022)
children are not all-or-nothing: the role of socioeconomic status in intellectual 13:826043. doi: 10.3389/fpsyt.2022.826043
skills in bilingual autistic children. Autism. (2022) [Online ahead of print]. doi: 42. Aggarwal G, Singh L. Evaluation of supervised learning algorithms based
10.1177/13623613221075097 on speech features as predictors to the diagnosis of mild to moderate intellectual
22. Chen J, Lin Z, Li L-A, Li J, Wang Y, Pan Y, et al. Ten years of China’s new disability. 3D Res. (2018) 9:55. doi: 10.1007/s13319-018-0207-6
healthcare reform: a longitudinal study on changes in health resources. BMC Public 43. Kursa MB. Rudnicki WR. Feature selection with the Boruta package. J Stat
Health. (2021) 21:2272. doi: 10.1186/s12889-021-12248-9 Softw. (2010) 36:1–13. doi: 10.18637/jss.v036.i11

Frontiers in Psychiatry 13 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

44. Leening MJ, Steyerberg EW, Van Calster B, D’Agostino RB, Pencina MJ. Net spectrum disorder. J Autism Dev Disord. (2021) 51:4644–54. doi: 10.1007/s10803-
reclassification improvement and integrated discrimination improvement require 021-04905-0
calibrated models: relevance from a marker and model perspective. Stat Med.
61. Siracusano M, Postorino V, Riccioni A, Emberti Gialloreti L, Terribili M,
(2014) 33:3415–8. doi: 10.1002/sim.6133
Curatolo P, et al. Sex differences in autism spectrum disorder: repetitive behaviors
45. Chen T, He T, Benesty M, Khotilovich V, Tang Y, Cho H, et al. and adaptive functioning. Children. (2021) 8:325. doi: 10.3390/children8050325
Xgboost: Extreme Gradient Boosting. (2015). Available online at: https://cran.r-
62. Baribeau DA, Vigod S, Pullenayegum E, Kerns CM, Mirenda P, Smith IM,
project.org/web/packages/xgboost/vignettes/xgboost.pdf (accessed April 16, 2022).
et al. Repetitive behavior severity as an early indicator of risk for elevated anxiety
46. Capogrosso P, Vickers AJ. A systematic review of the literature demonstrates symptoms in autism spectrum disorder. J Am Acad Child Psychol. (2020) 59:890–9.
some errors in the use of decision curve analysis but generally correct interpretation doi: 10.1016/j.jaac.2019.08.478
of findings. Med Decis Mak. (2019) 39:493–8. doi: 10.1177/0272989X19832881
63. Kuzminskaite E, Begeer S, Hoekstra RA, Grove R. Social communication
47. Lundberg SM, Lee S-I. A unified approach to interpreting model predictions. difficulties and restricted repetitive behaviors as predictors of anxiety in adults
Adv Neural Inf Process Syst. (2017) 30:1. with autism spectrum disorder. Autism. (2020) 24:1917–23. doi: 10.1177/
1362361320934218
48. Lundberg SM, Lee S-I. Consistent feature attribution for tree ensembles.
arXiv [Preprint]. (2017) Available online at: https://arxiv.org/abs/1706.06060 64. Park JH, Kim Y-S, Koh Y-J, Leventhal BL. Differences in the severity and
(accessed June 19, 2017). doi: 10.48550/arXiv.1706.06060 variability of restricted and repetitive behaviors in ASD children with and without
service experiences. Res Autism Spectr Disord. (2020) 79:101672. doi: 10.1016/j.rasd.
49. Hossain MD, Kabir MA, Anwar A, Islam MZ. Detecting autism spectrum
2020.101672
disorder using machine learning techniques. Health Inf Sci Syst. (2021) 9:1–13.
doi: 10.1007/s13755-021-00145-9 65. Levy SE, Giarelli E, Lee L-C, Schieve LA, Kirby RS, Cunniff C, et al. Autism
spectrum disorder and co-occurring developmental, psychiatric, and medical
50. Rice ME, Harris GT. Comparing effect sizes in follow-up studies: ROC Area,
conditions among children in multiple populations of the United States. J Dev
Cohen’s d, and r. Law Hum. Behav. (2005) 29:615–20. doi: 10.1007/s10979-005-
Behav Pediatr. (2010) 31:267–75. doi: 10.1097/DBP.0b013e3181d5d03b
6832-7
66. Arutiunian V, Lopukhina A, Minnigulova A, Shlyakhova A, Davydova E,
51. Dadang E, Emanuel AWR. Machine learning classifiers for autism spectrum
Pereverzeva D, et al. Expressive and receptive language in Russian primary-school-
disorder: a review. Proceedings of the 2019 4th International Conference on
aged children with Autism Spectrum Disorder. Res Dev Disabil. (2021) 117:104042.
Information Technology, Information Systems and Electrical Engineering (icitisee):
doi: 10.1016/j.ridd.2021.104042
2019. Yogyakarta: IEEE (2019). p. 255–60. doi: 10.1109/ICITISEE48480.2019.
9003807 67. Wanderer S, Roessner V, Strobel A, Martini J. WISC-IV performance of
children with chronic tic disorder, obsessive–compulsive disorder and attention-
52. Thabtah F. Machine learning in autistic spectrum disorder behavioral
deficit/hyperactivity disorder: results from a German clinical study. Child Adolesc
research: a review and ways forward. Inform Health Soc Care. (2019) 44:278–97.
Psychiatry Ment Health. (2021) 15:1–13. doi: 10.1186/s13034-021-00392-4
doi: 10.1080/17538157.2017.1399132
68. Anderson DK, Lord C, Risi S, DiLavore PS, Shulman C, Thurm A, et al.
53. Kang J, Han X, Song J, Niu Z, Li X. The identification of children
Patterns of growth in verbal abilities among children with autism spectrum
with autism spectrum disorder by SVM approach on EEG and eye-tracking
disorder. J Consult Clin Psychol. (2007) 75:594. doi: 10.1037/0022-006X.75.4.594
data. Comput Biol Med. (2020) 120:103722. doi: 10.1016/j.compbiomed.2020.10
3722 69. Ellis Weismer S, Kover ST. Preschool language variation, growth, and
predictors in children on the autism spectrum. J Child Psychol Psychol. (2015)
54. Kang J, Jin Y, Liang G, Li X. Accurate assessment of low-function autistic
56:1327–37. doi: 10.1111/jcpp.12406
children based on EEG feature fusion. J Clin Neurosci. (2021) 90:351–8. doi: 10.
1016/j.jocn.2021.06.022 70. Chiang T-L, Lin S-J, Lee M-C, Shu B-C. Advanced maternal age and maternal
education disparity in children with autism spectrum disorder. Matern Child Health
55. Pepe MS, Feng Z, Huang Y, Longton G, Prentice R, Thompson IM, et al.
J. (2018) 22:941–9. doi: 10.1007/s10995-018-2470-9
Integrating the predictiveness of a marker with its performance as a classifier. Am J
Epidemiol. (2008) 167:362–8. doi: 10.1093/aje/kwm305 71. Krakovich TM, McGrew JH, Yu Y, Ruble LA. Stress in parents of
children with autism spectrum disorder: an exploration of demands and
56. Brunetti S, Rossi A, Galli J, Gitti F, Nardocci N, Giordano L, et al. Repetitive
resources. J Autism Dev Disord. (2016) 46:2042–53. doi: 10.1007/s10803-016-2
and stereotyped behaviors in neurodevelopmental disorders: an observational
728-2
analysis of four diagnostic groups. Minerva Pediatr. (2022) [Online ahead of print].
doi: 10.23736/S2724-5276.22.06835-5. 72. Zhou W, Liu D, Xiong X, Xu H. Emotional problems in mothers of
autistic children and their correlation with socioeconomic status and the children’s
57. Brierley NJ, McDonnell CG, Parks K, Schulz SE, Dalal TC, Kelley E, et al.
core symptoms. Medicine. (2019) 98:e16794. doi: 10.1097/MD.000000000001
Factor Structure of repetitive behaviors across autism spectrum disorder and
6794
attention-deficit/hyperactivity disorder. J Autism Dev Disord. (2021) 51:3391–400.
doi: 10.1007/s10803-020-04800-0 73. Zhao C, Liu J, Liu J, Jia M, Sang W. The anxiety evaluation of parents of
children with autism. Clin Med China. (2011) 27:878–9.
58. Courchesne V, Bedford R, Pickles A, Duku E, Kerns C, Mirenda P, et al. Non-
verbal IQ and change in restricted and repetitive behavior throughout childhood 74. Van Rosmalen L, Van der Veer R, van der Horst FC. The nature of love:
in autism: a longitudinal study using the Autism diagnostic interview-revised. Mol harlow, bowlby and bettelheim on affectionless mothers. Hist Psychiatry. (2020)
Autism. (2021) 12:1–10. doi: 10.1186/s13229-021-00461-7 31:227–31. doi: 10.1177/0957154X19898997
59. Comparan-Meza M, Vargas de la Cruz I, Jauregui-Huerta F, Gonzalez- 75. Lord C, Charman T, Havdahl A, Carbone P, Anagnostou E, Boyd B, et al. The
Casta?eda RE, Gonzalez-Perez O, Galvez-Contreras AY. Biopsychological lancet commission on the future of care and clinical research in autism. Lancet.
correlates of repetitive and restricted behaviors in autism spectrum disorders. (2022) 399:271–334. doi: 10.1016/S0140-6736(21)01541-5
Brain Behav. (2021) 11:e2341. doi: 10.1002/brb3.2341
76. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a
60. Hong E, Matson JL. Factors associated with restricted, repetitive behaviors multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD):
and interests and diagnostic severity level ratings in young children with autism the TRIPOD statement. Br. J. Surg. (2015) 102:148–58. doi: 10.1002/bjs.9736

Frontiers in Psychiatry 14 frontiersin.org

Song et al. 10.3389/fpsyt.2022.993077

Appendix

Schedule 1 Assignment table.

Variable Category Assignment

Gender Categorical Male = 1;

Female = 0
Language ability Categorical Pre-verbal/single words = 1;
Phrase speech = 2;
Fluent speech = 3.
Father’s education attainment Categorical Primary school = 1;
Secondary school = 2;
High school = 3;
College/university = 4;
Graduate and above = 5.
Mother’s education attainment Categorical Primary school = 1;
Secondary school = 2;
High school = 3;
College/university = 4;
Graduate and above = 5.
Age Continuous –
Stereotyped speech Continuous –
Pointing/gestures Continuous –
Unusual eye contact Continuous –
Facial expression Continuous –
Social quality Continuous –
Unusual sensory interest Continuous –
Complex mannerisms Continuous –
Repetitive stereotyped behaviors Continuous –
Overactivity Continuous –
Negative behaviors Continuous –
Anxiety Continuous –
Outcome Categorical Non-ID = 1;
ID = 0

Frontiers in Psychiatry 15 frontiersin.org