Received December 7, 2020, accepted December 27, 2020, date of publication January 5, 2021, date of current version January

13, 2021.
Digital Object Identifier 10.1109/ACCESS.2021.3049446

Predicting at-Risk Students at Different

Percentages of Course Length for Early
Intervention Using Machine
Learning Models
MUHAMMAD ADNAN 1 , ASAD HABIB 1 , JAWAD ASHRAF1 , SHAFAQ MUSSADIQ 1 ,
ARSALAN ALI RAZA2 , MUHAMMAD ABID1 , MARYAM BASHIR1 , AND SANA ULLAH KHAN1
1 Institute of Computing, Kohat University of Science and Technology, Kohat 26000, Pakistan
2 Department of Computer Science, COMSATS University Islamabad, Vehari Campus, Islamabad 45550, Pakistan
Corresponding author: Muhammad Adnan ([email protected])

ABSTRACT Online learning platforms such as Massive Open Online Course (MOOC), Virtual Learning
Environments (VLEs), and Learning Management Systems (LMS) facilitate thousands or even millions
of students to learn according to their interests without spatial and temporal constraints. Besides many
advantages, online learning platforms face several challenges such as students’ lack of interest, high dropouts,
low engagement, students’ self-regulated behavior, and compelling students to take responsibility for settings
their own goals. In this study, we propose a predictive model that analyzes the problems faced by at-risk
students, subsequently, facilitating instructors for timely intervention to persuade students to increase their
study engagements and improve their study performance. The predictive model is trained and tested using
various machine learning (ML) and deep learning (DL) algorithms to characterize the learning behavior
of students according to their study variables. The performance of various ML algorithms is compared
by using accuracy, precision, support, and f-score. The ML algorithm that gives the best result in terms
of accuracy, precision, recall, support, and f-score metric is ultimately selected for creating the predictive
model at different percentages of course length. The predictive model can help instructors in identifying
at-risk students early in the course for timely intervention thus avoiding student dropouts. Our results showed
that students’ assessment scores, engagement intensity i.e. clickstream data, and time-dependent variables
are important factors in online learning. The experimental results revealed that the predictive model trained
using Random Forest (RF) gives the best results with averaged precision = 0.60%, 0.79%, 0.84%, 0.88%,
0.90%, 0.92%, averaged recall = 0.59%, 0.79%, 0.84%, 0.88%, 0.90%, 0.91%, averaged F-score = 0.59%,
0.79%, 0.84%, 0.88%, 0.90%, 0.91%, and average accuracy = 0.59%, 0.79%, 0.84%, 0.88%, 0.90%, 0.91%
at 0%, 20%, 40%, 60%, 80% and 100% of course length.

INDEX TERMS Predictive model, earliest possible prediction, at-risk students, machine learning, feed-
forward neural network, random forest, early intervention.

I. INTRODUCTION analyzing the students’ data generated from online learning

Rapid innovations in the design and development of online platforms can help instructors to understand and monitor stu-
learning platforms such as Massive Open Online Course dents learning progress. The earlier the students’ performance
(MOOC), Virtual Learning Environments (VLEs), and is detected in the VLEs, the better it is for the instructor to
Learning Management System (LMS) not only have over- persuade and warn students for keeping them on the right
come the limitations of space and time but have also made track. Earlier studies report that the students learning vari-
access to education easy and affordable. Evaluating and ables stored in the database records can help instructors in
predicting the performance of students in the future [1]–[3].
The associate editor coordinating the review of this manuscript and But predicting students’ performance earlier in the course
approving it for publication was Utku Kose. would be more helpful as compared to predicting students’
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 9, 2021 7519
 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

performance once they have completed the course and have •Developing and evaluating predictive models using sev-
given the final exam. However, developing a predictive model eral ML/DL algorithms to predict students’ performance
that can identify the exact learning behavior of students ear- scores.
lier in the course by analyzing their behavior data is a chal- • Earliest possible identification of students in VLE who
lenging task. In an online learning environment, where a large are at-risk of dropout during the course.
amount of data is generated every day, machine learning (ML) • Integrating personalized feedbacks with a predictive
techniques could help in analyzing the variables that define model to help instructors in interventing students at an
the students and come up with the results that better describe optimal time.
their learning behavior, thus, ML may reveal information that • Discussing various persuasion techniques that may help
is beneficial for both instructors and students [4], [5]. students in increasing their study performance.
Identifying students at-risk of dropout and failure as early The rest of the paper is organized as follows: Section II
as possible during a course could help the instructors to exe- presents background and related work. Section III is about
cute timely and necessary interventions/persuasions to help the Open University Learning Analytics (OULA) dataset,
students to remain steady during their studies [6]. selected for predicting at-risk students at a different percent-
Generally, in traditional classroom settings and online age of course length. Section IV presents details about the
learning settings, a general approach is followed where the experimental setup for training various predictive models.
same guidelines are defined for all students ignoring indi- Section V discusses the experimental results. Various tech-
vidual discontentment. To provide personalized feedback and niques related to interventing students through persuasion is
support right from the start of the semester, VLE designers discussed in section VI. Conclusion, limitations, and future
require the development of a predictive model that makes work are presented in section VII.
rapid decisions about how and when to intervene students for
support. Educational Data Mining (EDM) tools, techniques, II. BACKGROUND AND RELATED WORK
and products have progressed significantly, helping educators A. EDUCATIONAL DATA MINING (EDM)
to make education easy and effective [7]. However, these EDM is an emerging discipline aimed at using statistical and
techniques lack in identifying at-risk students earlier in the ML methods to analyze large repository of educational data
course timeline, compelling instructors to perform significant for a better understanding of student’s behavior patterns and
manual work for students problem identification to keep them the learning environments. Several EDM studies have been
on track. carried out that leverages ML techniques to discover variables
The emergence of Artificial Intelligence (AI), machine that significantly influence students’ performance, dropout,
learning (ML), and deep learning (DL) techniques have facil- engagement, and interaction in online learning platforms.
itated and enabled researchers to develop series of predictive Most of these studies target analyzing variables that are
models to reveal hidden study patterns which explain the generated from students’ online activities [10]–[12] whereas
strength and weakness of online students [8], [9]. To reduce some studies also use demographics variables to observe
the dropout rates, researchers can use ML techniques to their effect on students’ study behavior [13], [14]. Earlier the
study different variables that significantly affect the students’ prime variables to be considered for analysis were study time,
dropout. Predictive models powers by ML techniques can study duration, learning content type, and variables derived
present the accurate picture of students that are likely to from social interaction activities. As the online learning plat-
quit their study thus facilitating instructors to come up with form became more stable and interactive, variables such as
preventive measures before dropout behavior occurs. The assessment scores, assignment scores, clickstream, online
prime objective of our research study is the earliest possible forum interaction, and location were added in the analysis
identification of students who are at-risk of dropouts by process [15]. Identification of significant variables becomes
leveraging ML techniques to understand variables associated a challenge for researchers due to diversity in LMS, VLEs,
with the learning behavior of students and how they interact MOOCs, courses offered, and course activity types.
with the VLE. In literature, the majority of studies target collecting vari-
By analyzing Open University Learning Analytics ables and predicting students’ performance at the end of
(OULA) dataset, it was observed that students are incon- the course. The results obtained from those studies were
sistent in their online learning activities throughout course useful in identifying the significant variables that influence
weeks resulting in high dropouts at the end of the course. the students’ performance the most; however, no solutions
Based on our observations, we developed a predictive model were provided to prevent students from dropout and failure.
that can identify students at-risk of dropout at the very start On the other hand, online learning platforms generate enor-
of the course. The predictive model is capable of facilitating mous data associated with student interaction, clickstreams,
instructors to intervene students through persuasive messages and courses, etc right from the start of the course. A com-
that encourage students to keep themselves on the right prehensive predictive model can be developed by analyzing
track thus avoiding dropouts. The contribution of this study variables data right from the start of the course that would
include: be effective in preventing failures/dropouts and facilitating

7520 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

instructors to make effective interventions at the optimal A Time-series clustering approach was used by [31] for
time. the earliest possible identification of at-risk students tak-
A study carried out by [16] implemented four ML algo- ing online courses. As compared to traditional aggregation
rithms for the early identification of students who have a high approaches, the time-series clustering approach generated
possibility of failure. The results revealed that the Support predictive models with higher accuracy. Sofer and Cohen
Vector Machine (SVM) was the most effective algorithm used various learning analytics techniques on engagement
in the earlier identification of students with 83% accuracy. variables generated from online courses to find their impact
The study also indicated that the process of data prepro- on students’ performance achievements [32]. The results indi-
cessing is very essential in increasing the performance of cated that students achieving good academic positions have
ML algorithms. Results of previous studies showed that a higher engagement percentage compared to the students
the development of predictive models is possible earlier in achieving lesser academic positions. As an example, students
the course however many challenges limit their application who completed the course were twice as much interactive as
to a specific learning platform. One of the major chal- students who did not complete it.
lenges for researchers is making predictive models flexible A study carried out by [33] argued that clickstream vari-
capable of adjusting/adapting in different learning environ- ables representing students’ online engagement are more
ments. Major reasons that are limiting predictive models to accurate, objective, and comprehensive than self-reported
become flexible, general, and transferrable is because of the data in measuring students’ learning behavior. The click-
different course structures, instruction designs, and online stream data is more reliable as it is collected in an authen-
platforms [17]. tic learning environment while the learning behavior is
In the last few years, research studies used both statistical happening as compared to self-reported data which is often
and predictive models to find insights in a large repository of generated from decontextualized and unreliable students’
data both formal and informal educational settings [13], [18], memories. Moreover, clickstream data are unobtrusive and
[19]. For example, several research studies [20]–[23] investi- did not require students full attention as they can be col-
gated the role of demographics variables towards contributing lected seamlessly without interrupting students learning pro-
to successful learning performance or students’ retention. cess [34]. Finally, intuitively collected clickstream data can
A study carried out by [24] investigated and compared more provide large-scale and timely measures of students learning
than 120 different datasets related to economics and business behavior which could assist instructors in knowing about the
undergraduate students’ demographics and interaction behav- students’ online engagements daily.
ior in online settings. The study observed the influence of a Recently, a large number of research studies have been
variety of variables such as educational background, click- undertaken to analyze clickstream data generated from online
stream data, assessment scores, entry test scores, and learning learning platforms (MOOCs, LMS, VLE) to measure stu-
personality data on students’ performance. While most of the dents’ online engagements. While most of the studies try to
studies targets discovering the impact of key variables on explore the relationship between clickstream data and stu-
students performance, there are other studies [25]–[27] that dents’ online engagements, limited studies have taken a step
encourage early intervention, informed support and timely further to facilitate instructors in when and how to intervene
feedback to guide at-risk students. Numerous studies carried students at the optimal time e.g., [35]–[38]
out at Open University, UK [28], [29] tried to identify at-risk Shivangi Gupta and A. Sai Sabitha in their research study
students by using several predictor variables. The studies attempted to decipher those variables that are responsible for
mapped the study behavior of students to predict 1) low per- student retention in e-learning [39]. Decision Tree (DT) algo-
formance (when performance falls below a threshold value) rithm was used to determine the significant features to help
2) whether the students are successful or unsuccessful at the MOOC learners and designers in improving course content,
end of the course. The studies also indicated that demographic course design, and delivery. Several data mining techniques
variables used along with students’ behavior variables pro- were applied to three MOOCs datasets to analyze the online
vided improved predictive models in terms of performance students’ in-course behavior. Finally, the authors claimed that
and accuracy. the models they used could be useful in predicting significant
Lee and Choi [30] carried out a study in which they tried to features to minimize the attrition rate.
identify relevant variables that compel students to decide to Akçapınar Gökhan developed an early warning system
quit the course. The variable responsible for student dropouts that used students’ eBook reading data to predict students
were classified into three categories. 1) students’ demo- at-risk of academic failure [40]. To develop the best predictive
graphic variables such as gender, background, relevant expe- model, 13 ML algorithms were used to train the model using
rience, skills, psychological attributes, earlier education, etc data from different weeks of the semester. The best predictive
2) variable related to course structure and requirement such as model was selected according to the accuracy/Kappa metric
the number of assessments, institutional support, interaction, and recommending optimal time for the instructors to inter-
difficulty level, time duration, etc 3) environments/context vene. The study revealed that all predictive models improved
factors such as technology used, location, external noise, their performance results when more and more weekly data
work environment, home environment, etc. was used during the training process. The early warning

VOLUME 9, 2021 7521

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

system predictive models were successful in classifying low performance. As compared to the feed-forward neural net-
and high-performance students with an accuracy of 79% work (FFNN), Support Vector Machine (SVM), a system for
starting from the 3rd week. When complete 15 weeks data educational data mining (SEDM), and Probabilistic Ensem-
were provided to various algorithms, Random Forest (RF) ble Simplified Fuzzy ARTMAP - Adaptive Resonance The-
outperformed other algorithms whereas with the transformed ory Mapping PESFAM techniques, the proposed drop out
data J48 outperformed all other algorithms. Moreover, upon detection technique exhibited higher performance score in
using categorical data, the Naïve Bayes (NB) showed better terms of precision, recall, specificity, and accuracy. The tutor-
performance. ing action plan based on logistic regression was able to reduce
Foretelling students’ performance early in the course is the dropout rate by 14%.
a challenging problem in online learning environments due Lara J.A. et al., proposed the use of knowledge discov-
to diversity in course structure and MOOCs design. While ery in databases (KDD) to extract information that might
the popularity of LMS/MOOCs is increasing rapidly, there help teachers in knowing about the interaction of students
is a need for an automated intervention system that can with e-learning systems [44]. The proposed technique builds
provide timely feedback to students. To integrate automated historical reference models of students that can be used
intervention system with LMS/MOOCs, researchers have to classify students in dropout or non-dropout classes. The
implemented various ML algorithms that can support instruc- proposed system called System for Educational Data Min-
tors in providing informed assistance to students during the ing (SEDM) analyzes two groups of students for a single
learning process. ML algorithms such as K-Nearest Neigh- course i.e. dropout students who are not able to sit in the
bors (KNN), Support Vector Machine (SVM), Decision Trees final examination and non-dropout students who are passed
(DT), and Random Forest (RF), etc are commonly trained the course assessments and can sit in the final examination.
using daily, weekly, or monthly students log data to find SEDM was able to generate study patterns for the two groups
students’ learning patterns. Deep learning (DL) algorithms that can be very helpful for instructors to improve students’
are now also used in creating predictive models as they can study performance. Table 1 presents summary of the recent
process raw data directly. Kőrösi, Gábor and Richard Farkas research studies that uses ML/DL techniques for creating
used the Recurrent Neural Network (RNN) algorithm trained predictive models, performing classification according to
on raw log student records to predict students’ learning per- students’ performance, dropouts prediction and performing
formance at the end of the course [41]. The results showed early intervention
the dominance of RNN in providing superior performance as The studies discussed aforementioned are related to avoid-
compared to standard baseline methods. ing students’ dropout, interventing students at the optimal
Alberto C. and John D. L. used multi-view genetic pro- time, predicting students at-risk of dropout, and classifying
gramming approach to develop classification rules to analyze students into different performance groups, however, none
students learning behavior, predict their academic perfor- of these studies predicts students at-risk of dropout at a dif-
mance, and trigger alerts at the optimal time to encourage ferent percentage of course length. The proposed predictive
at-risk students to improve their study performance [42]. model could help educational institutions and instructors in
The genetic programming technique was employed as it the earliest possible identification of at-risk students thereby
works nicely with multi-view learning. The prediction model intervening students through proper persuasive techniques
learned and evolved is directly explainable without further to encourage them to be on track and improve their study
adjustment. Moreover, while using genetic programming performance.
approach results in the natural evolution of classification
rules with time as more and more data is available i.e. the III. DATA DESCRIPTION
classification rules evolved with the availability of new data. We used a freely available Open University Learning
The early warning system built with comprehensible Genetic Analytics Dataset (OULAD), provided by the Open Uni-
Programming classification rules specifically targets under- versity, UK. Students’ data is spread across 7 tables
performing and underrepresented students. Comprehensible each containing students centered information such as
feedbacks are provided to students, instructors, and adminis- students’ demographics, students’ Virtual Learning Envi-
tration staff using three interfaces to provide timely support to ronment (VLE) interaction, assessments, course registration,
students to keep them on the right track. The main drawback and courses offered. Tables relate to each other through
of this study was that the author did not mention explicitly key identifiers. Students’ daily activities and VLE inter-
the various semester stages at which the performance metrics action are represented as clickstreams data (number of
such as accuracy, sensitivity, specificity, and Kappa, etc were clicks) stored in the student VLE table. Students’ assessment
calculated using multi-view genetic programming algorithm scores are stored in a dataset triplet called student-module-
along with other machine learning algorithms. presentation. The OULAD was generated for the year
A research study carried out by [43] used a ML algo- 2013 and 2014 containing 7 courses, 22 module-presentations
rithm called logistic regression to pinpoint students that are with 32,593 registered students. OULAD is freely accessible
expected to drop out in an e-learning course. The algorithm at https://analyse.kmi.open.ac.uk/open_dataset and has been
uses students’ historical grades as an input to model students’ certified by the Open Data Institute http://theodi.org/ [63].

7522 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

TABLE 1. Summary of research studies using ML/DL techniques for student dropout and performance prediction along with the particular algorithms
used, their performance, problem addressed and limitations.

When properly analyzed and modeled, OULAD can pro- nulls, or noise were removed or replaced by their mean
vide a very suitable platform for an early forecast of at-risk values from the OULAD. As an example, the date values
students. were missing in the assessments table which represents the
date the assessments were taken/submitted. As the date is an
A. DATA PREPROCESSING important variable in the early prediction of at-risk students,
To enhance the performance efficiency of the predictive all the date instances having N/A, null, or missing values were
models, all missing variables instances in the form of replaced by the date mean value.
VOLUME 9, 2021 7523
 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

TABLE 2. Student-Assessment-Clickstream triplet.

FIGURE 1. Predicting and intervening at-risk students at different percentages of the course length.

B. FEATURE ENGINEERING students’ interaction with VLE learning contents during a

For the earliest possible prediction of students’ performance, course module. More information about the triplet student-
we divided the course length into 5 parts i.e. 20%, 40%, 60%, assessment-clickstream table is detailed in table 2. The triplet
80%, and 100% of course completed. We also assumed that student-assessment-clickstream table contains 31 columns
demographic data solely can also be used to predict students’ with 32593 rows.
upcoming performance in assessments and final exams. The
students’ future performance prediction was determined by IV. EXPERIMENTAL SETUP FOR PREDICTIVE MODELING
modeling the predictive models using only demographics For predicting the performance of at-risk students at a dif-
data, using demographics and 20% course completion data, ferent percentage of the course length, the variables about
demographics, and 40% course completion data, and so on. students’ demographics, VLE interaction, and assessments
To predict students’ performance at different times of course were used. This workflow is shown in figure 1 where the
module, several new variables were created from the exist- length of the course is divided into 6 periods i.e. course starts,
ing variables. Relative Score (RS) variables were created to 20%, 40%, 60%, 80%, and 100% of the course studied.
represent student relative performance at 20%, 40%, 60%, Six ML algorithms and one DL algorithm were selected for
80%, and 100% of course module completion (RS20, RS40, training/testing the predictive models during different stages
RS60, RS80, RS100). Variables indicating the number of late of the course. For modeling Open University Learning Ana-
submissions were created when 20%, 40%, 60%, 80%, and lytics Dataset (OULAD), these algorithms were designated
100% of the course module was completed (LS20, LS40, for classifying students’ performance into four categories
LS60, LS80, LS100). Variables representing the raw assess- i.e. Withdrawn (students who were not able to complete the
ment scores were also created at 20%, 40%, 60%, 80%, and course), Fail (students who completed the course but were
100% of the course module completion (AS20%, AS40%, not able to secure passing marks), Pass (completed courses
AS60%, AS80%, AS100%). Variables representing students’ with a passing score), Distinction (completed courses with
VLE interaction in the form of clickstreams were created excellent grades). The Python 3.7.8 scripts were used for the
for the different percentage of course module length. Two construction of predictive models. The Python libraries used
types of variables namely sum_clicks and mean_clicks were were TensorFlow, Keras, sklearn, numpy, and seaborn.
created to indicate the sum of clicks and average clicks at
20%, 40%, 60%, 80%, and 100% of the course module A. EVALUATION CRITERIA
completion (SC20%, SC40%, SC60%, SC80%, SC100%, Before training and testing predictive models at various stages
AC20%, AC40%, AC60%, AC80%, AC100%). Students’ of course module length, the dataset at hand was split into
demographics table was merged with the students’ assess- training and testing set using K-fold cross-validation tech-
ment table to get demographics and assessment data in one nique where the value of k was set to 10. By using the k-fold
table. Moreover, the VLE information i.e. students click- cross-validation technique, the dataset is divided into k sets
stream data was also merged with demographics data to know where k-1 sets are used for model training and the remaining

7524 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 2. Heatmap showing correlation between demographics variables and final result.

1 set is using for model testing i.e. measuring the prediction would correctly predict Withdrawn, Fail, or Pass grades with
performance of the model on the unseen data not used during 99% accuracy. The predictive model will have an accuracy
the model training. The metrics selected for measuring the of 99% and students having a Distinction position will likely
performance of predictive models include the following: be categorized among Withdrawn, Fail, and Pass students.
Recall ensures that we are not overlooking those 1% of stu-
1) ACCURACY dents having a Distinction position. Recall = TP/(TP + FN).
The accuracy is calculated by dividing the number of correct
classes predicted by the total number of classes i.e. accu- 4) F-SCORE
racy = (True Positives + True Negatives)/all. Determines the harmonic mean of recall and precision of a
predictive model. F1 measure is good for classification prob-
2) PRECISION lems where the target labels are imbalanced. F-measure = 2
Determines the fraction of true positives among true pos- * (Precision * Recall)/(Precision + Recall).
itives and false positives predicted i.e. precision = true
positives/(true positives + false positives). If a small per- V. EXPERIMENTAL RESULTS AND DISCUSSION
centage of students (1%) are withdrawing or failing the A. PHASE 1: USING ONLY DEMOGRAPHICS DATA FOR
course, we could build a predictive model that always accu- CONSTRUCTING PREDICTIVE MODELS
rately predicts whether students are getting withdrawn, fail, Before training the 7 predictive models using only demo-
pass, or distinction. This predictive model would be 99% graphics data, a heatmap was constructed to know the
accurate but 0% useful and reliable. demographics variables correlations with the final result.
As shown in figure 2, we noticed that there is no signif-
3) RECALL icant positive or negative correlation between demograph-
Recall ensures that the predictive model is not overlook- ics variables and the final result. Only studied_credtis and
ing a few VLE students who are getting Withdrawn, Fail, num_of_prev_attempts have a weak positive correlation
Pass, or Distinction grades. Suppose if only 1% of students between them. All the demographics variables were consid-
are getting the Distinction position and the rest are getting ered for training RF, SVM, K-NN, ET, AdaBoost classifier,
Withdrawn, Fail, or Pass grades, then the predictive model Gradient boosting classifier, and ANN models.

VOLUME 9, 2021 7525

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

TABLE 3. Performance score of the 7 predictive models when trained only on demographics data.

Table 3 presents the results of 7 predictive models when data. In the case of Pass class, the RF, ET, AdaBoost, and
trained only on the demographics data using K-fold cross- Gradient Boost classifiers are showing satisfactory results
validation techniques where the value of k was set to 10. whereas, for the Fail and Distinction class, the perfor-
The final result variable was set as the target variable which mance scores are still very low. Though the performance
the predictive models will try to predict whereas all other scores of the predictive models are better than when trained
demographics variables acted as input the predictive models. only on demographics data, but much farther from being
The values of precision, recall, accuracy, and f-score of the acceptable.
predictive models for various positions of students’ final
results when trained only on the demographics data indicates C. PHASE III: CONSIDERING DEMOGRAPHICS,
the very low performance of predictive models. Moreover, CLICKSTREAM, AND ASSESSMENT SCORES FOR
the performance of all predictive models for the Fail position DEVELOPING PREDICTIVE MODELS
is very bad. In early intervention systems, where identifica- The assessment scores were further added to demographics
tion of at-risk students is vital, the performance results of and clickstream for possible predictive model performance
predictive models for Fail students become more crucial as improvement. From the heatmap in figure 4, we observed a
such the students at-risks can be intervened earlier in the moderate correlation between assessment scores and the final
course for improving study behavior. result. Interestingly the average score variable (AS100) has
a moderate negative correlation with the final result which
B. PHASE II: USING DEMOGRAPHICS AND CLICKSTREAM implies an increase in the final result score when the aver-
DATA FOR CONSTRUCTING PREDICTIVE MODELS age assessment score decreases. As expected, a strong pos-
To improve the performance of predictive models, click- itive correlation (.87) was noted between the average score
stream data (students’ interaction with VLE in the form of and the relative score variable. Moreover, the mean clicks
numbers of clicks during the course timeline) along with (MC100) and sum clicks (SC100) variables were also having
demographics was considered to train the predictive models. a significant positive correlation with the assessment score
From the heatmap in figure 3, we noticed that the correlation variables. Finally, a weak correlation was observed between
of the final result with all other variables is similar and there the late submission variable (LS100) and the final result.
is still no significant negative/positive correlation of demo- Table 5 presents the performance scores of the predictive
graphics and clickstream variables with the final result. The models when trained on the demographics, clickstream, and
correlation between sum_clicks100 and mean_clicks100 is assessment data. A substantial improvement was noted in the
moderate but still far away from significant. Next, we con- performance of predictive models for Pass, Withdrawn, Dis-
sidered all the demographics and clickstream variables for tinction, and Fail classes when assessment data was added for
training and testing the predictive models. constructing the predictive models. The performance results
Table 4 shows the performance results of the predic- of SVM and K-NN models were still very low with the
tive models developed using demographics and clickstream accuracy of .32 and .38.

7526 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 3. Heatmap showing correlation between demographics plus clickstream variables and final result.

TABLE 4. Performance score of the 7 predictive models when trained demographics plus clickstream data.

D. FEATURE ENGINEERING merged into Fail class as these classes are of the same types
To further improve the performance results, a merge oper- and portray similar information. The goal of performing the
ation was performed where Distinction-Pass classes were feature engineering technique was to improve the perfor-
combined into Pass class and Withdrawn-Fail classes were mance of the predictive models especially for the Fail class as

VOLUME 9, 2021 7527

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 4. Heatmap showing correlation between demographics, clickstream, and assessment variables with
final result.

TABLE 5. Performance score of the 7 predictive models when trained demographics, clickstream, and assessment variables.

the students belonging to the Fail class are at-risk and need than 80% performance score for precision, recall, F-score,
informed guidance. Table 6 a decent increase in the predic- and accuracy. Overall, RF outperformed all other baseline
tive model performances after performing feature engineer- models whereas the SVM showed the lowest performance.
ing. On average all the predictive models achieved greater The performance scores of GradientBoosting, AdaBoost, and

7528 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

TABLE 6. Predictive models performance after performing feature engineering.

ExtraTree classifiers were almost similar and closer to the were used, the eight categorical variables (‘code_module’,
RF. Ultimately, the RF classifier was selected for training and ‘code_presentation’, ‘gender’, ‘region’, ‘highest_education’,
testing the predictive model for the different duration (0%, ‘imd_band’, ‘age_band’, ‘disability’) were converted into
20%, 40%, 60%, 80%, 100%) of the course module. dummy variables using one-hot encoding technique and
one numerical variable (‘num_of_prev_attempts’) was scaled
E. PHASE IV: TRAINING PREDICTIVE MODEL USING DEEP using MinMax scaler. After performing a one-hot encoding
FEED FORWARD NEURAL NETWORK (DFFNN) and scaling operation, a total of 47 variables were created
After training the predictive models using traditional ML ready to be fed to the DFFNN. Therefore, based on input
methods, next, we used a deep learning technique called variables, 47 neurons were used in the input layer, 24 were
Deep Feed Forward Neural Network (DFFNN) to train the used in the first hidden layer, 12 were used in the sec-
predictive model using OULAD. DL techniques differ from ond hidden layer and four neurons were used at the output
the techniques used in traditional ML algorithms in that they layer for representing Withdrawn, Fail, Pass, and Distinc-
structure algorithms in layers that can learn and make sensible tion classes. Similarly, as new clickstream and assessment
judgments on their own. DL algorithms process data by using variables were added to the demographics variables, new
graphs of neurons in the input, hidden, and output layers. Like DFFNN layers structures were created for a smooth training
traditional ML algorithms, DFFNN was trained repeatedly process. The hypermeters used for training the predictive
using demographics data, demographics + clickstream data, model using DFFNN were loss=‘categorical_crossentropy’,
demographics + clickstream + assessment data, and lastly optimizer=‘adam’, metrics=‘accuracy’, activation function
on all the variables of OULAD, setting the final result as at hidden layers = ReLU, activation function at the out-
the target variable and the rest of the variables as the pre- put layer = softmax, epochs = 100, training set = 85%,
dictor variables. Predictive models based on DFFNN were and testing set = 15%. Tables 7, 8, 9 and 10 present the
trained by first initializing the neurons’ edges weights to precision, recall, f-score, support, average accuracy, macro
numbers close to zero using TensorFlow Dense Class. Uti- average, and weighted average scores of the prediction mod-
lizing the TensorFlow Sequential class, the first observation els when trained using demographics variables, demograph-
of OULAD was fed to the DFFNN by applying the forward ics + clickstream variables, demographics + clickstream +
propagation technique. Applying the Rectified Linear Unit assessment variables, and all variables. The results indicated
(ReLU), the neurons during the forward propagation process that the predictive model generated mediocre scores when
were activated in such a way that the impact of each neu- trained only on demographics data which entails that demo-
ron activation was restricted by the weights of the edges. graphics data only cannot be used for the earliest possible
For training predictive models, the Keras API offers fully- prediction of at-risk students. When trained only demo-
configurable components that can be plugged simultaneously graphics + clickstream variables, the predictive model per-
with as little constraints as possible. In particular, the Keras formance score improved which suggests that clickstream
API provides the feed-forward neural networks initialization variables are an important indicator of students’ performance.
schemes, activation functions, cost function optimizers, and Upon adding the assessment variables with demographics
regularization schemes components that can be combined to and clickstream variables, the accuracy score improved 8%
create predictive models. meanwhile the precision, recall, and f-score remained low
The structure (input, hidden, and output layers) of DFFNN for Distinction, Fail, and Withdrawn classes. Finally, when
was modified according to the number of variables pro- all the predictor variables were used in the training process,
vided. For example, when only demographics variables a slight improvement was noticed in the metrics variables

VOLUME 9, 2021 7529

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

TABLE 7. DFFNN performance results when trained only on TABLE 11. DFFNN performance after feature engineering step for Fail
demographics variables. and Pass classes.

24 neurons at hidden layer 1, 12 neurons at hidden layer

TABLE 8. DFFNN performance results when trained on demographics and
2, and 2 neurons at the output layer. The hyperparameters
clickstream variables. used were loss=‘binary_crossentropy’, optimizer=‘adam’,
metrics=‘accuracy’, activation functions at hidden layers =
ReLU, activation function at output layer = sigmoid, and
epochs = 100. Table 11 presents the performance score of
DFFNN with much improved scores than the multiclass clas-
sification arrangement. Overall, an 18% improvement was
noticed in the average accuracy score. Figure 6 illustrates the
confusion matrix for the predictive model when the predic-
tions were evaluated using the testing set considering only
Pass and Fail classes. After the merged operation, the pre-
TABLE 9. DFFNN performance results when trained on demographics,
clickstream and assessment variables. dictive model accuracy substantially improved but we also
noticed that as compared to the RF-based predictive model
accuracy, it is still 2% less, therefore for predicting students’
performance at different lengths of course module, the RF
method was finally selected.

F. PHASE V: CONSTRUCTING RF PREDICTIVE MODEL AT

DIFFERENT PERCENTAGE OF COURSE MODULE LENGTH
Table 12 shows the result of the RF predictive model when
TABLE 10. DFFNN performance results when trained on all variables.
trained and tested repeatedly on demographics data, 20%
course data, 40% course data, 60% course data, 80% course
data, 100% course data. Random Forest (RF) uses ensembling
techniques by constructing a multitude of decision trees (DT)
for classification and regression tasks. During the RF training
process, multiple DTs are constructed by using different sub-
samples of training data, and the mean scores are used to
improve prediction accuracy and avoid over-fitting.
RF predictive model, when trained only on demographics
variables, gave the inferior performance scores with averaged
with only a 1% increase in the average accuracy score. precision = .60, recall = .59, f-score = .59, and accuracy =
Figure 5a, 5b, 5c, and 5d present the confusion matrices for .59 for both Fail and Pass classes. When trained on 20% of
the predictive models when the predictions were evaluated course data, the RF predictive model performance improved
using the testing set. The diagonal elements are the correctly with averaged precision = .79, recall = .79, f-score = .79 and
predicted classes where it can be noticed that the correct accuracy = .79 for both Fail and Pass classes. It was also
predictions increased upon adding more and more data during observed that the performance score for Pass class was high
the predictive model training process. as compare to Fail class when more clickstream and assess-
ment data was provided to the RF predictive model. More
1) USING FEATURE ENGINEERING TO IMPROVE DFFNN improvement was observed in the performance score of the
PERFORMANCE RF predictive model when clickstream data and assessment
To further improve the performance of DFFNN, fea- data were provided at 40% of course completion with aver-
ture engineering technique was applied to the final result aged precision = .84, recall = .84, f-score = .84, and accu-
variable where Pass-Distinction classes were combined racy = .84. The performance score results revealed that RF
into Pass class and Withdrawn-Fail classes were merged predictive model performance improved when more and more
into Fail class. Once again the DFFNN based predictive clickstream and assessment data was provided which infers
model was trained with 81 neurons at the input layer, that the model was learning which variables have a significant

7530 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 5. DFFNN confusion matrices when tested on different variables of OULAD.

TABLE 12. Performance scores of RF predictive model at different percentage of course length.

VOLUME 9, 2021 7531

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 7. Different types of triggers for students having different

performance state.

of the predictive model is to support instructors as early as

FIGURE 6. Normalized confusion matrix showing the DFFNN prediction possible in the course timeline to help students at-risk of
for Fail and Pass classes. failure or dropouts.

relationship with the students’ final result. At 60% of course VI. INTERVENING STUDENTS THROUGH PERSUASIVE
completion, an overall 4% improvement in the performance TECHNIQUES
score of the RF predictive model was observed with average Fogg Behavior Model (FBM) suggests that three factors (abil-
precision = .88, recall = .88, f-score =.88 and accuracy = ity, motivation, and triggers) must be present at the same time
.88. At 80% of course completion, an increase of 2% in the to increase people’s attitude positively [64]. For intervening
performance score was observed with averaged precision = and persuading students to improve their study behavior,
.90, recall = .90, f-score = .90 and accuracy = .90 for both the optimal time selection is important. Based on the satisfac-
Fail and Pass classes. Finally, at 100% course completed, tory results (with 79% accuracy, precision, recall, f-score) of
we observed the highest RF predictive model performance the RF predictive model, students can be intervened after 20%
score with averaged precision = .92, recall = .91, f-score = of the course length. Furthermore, if comprehensive details
.91 and accuracy = .91 for both Fail and Pass classes. are included in the students’ demographic data, then inter-
The precision, recall, f-score, and accuracy scores vention and persuasion can be carried out at the start of the
improved when more and more course data was provided to course. Figure 7 summarizes the types of triggers presented
the RF predicted model. Overall the precision score improved to students having fragile, improving, and consistent study
from .60 to .92, the recall score improved from .59 to .91, performance. The factors of fear, hope, and suggestion are
the F-score improved from .59 to .91 and the accuracy score added to the triggers directed to at-risk students. Similarly,
improved from .59 to .91. The performance metric results for triggers intended for improving and consistent students,
concluded that the performance of the RF predictive model the factors of praise, reward, appreciation, and social accep-
improved with the length of the course. Even at 20% of course tance are added. The optimal time for sending triggers to
length, the RF predictive model showed a decent performance students depends upon the results of the predictive model dur-
score with precision for Fail class = .75, Pass class = .84, ing different stages of the course. As an example, the trigger
recall for Fail class = .85, Pass class = .75, f-score for Fail having hope factor for at-risk students for improving their
class = .79, Pass class = .79, and accuracy for Fail class = study behavior might be as follows:
.84, Pass class = .75 which indicated the model could be According to our predictive model, your assessment
very useful in interventing at-risk students as early as 20% of score achieved a 50% success rate. You made a commend-
course length. As a result, the RF predictive model at 20% able effort this week. Your class position can be further
of course completion can assist instructors in interventing improved if you cautiously follow the next week lectures.
students and assisting them during their studies. Moreover, If you submit all your assessments on time next week, you
the performance scores further improved with 40%, 60%, will achieve a cumulative score of 60%.
80%, and 100% of the course length which indicated that the Similarly, trigger having a fear factor for at-risk students
RF predictive model can support instructors with higher accu- for improving their study behavior might be as follows:
racy to intervene students and provide the needed feedback. According to our predictive model, you are consistently
Intervening students at 100% of course length is useless as getting a low assessment score. It would be great if nobody
the student would have completed the course but the target is dropped from the course due to a low assessment score.

7532 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 8. Assessment score at 100% course vs sum of clicks at 100% course.

FIGURE 9. Late submission vs sum of clicks at 100% course.

FIGURE 10. Assessment score vs sum of clicks at 100% course.

VII. CONCLUSION, LIMITATION, AND FUTURE WORK instructors. It provides instructors an opportunity to assist
Predicting and interventing students during different stages students at-risk of dropout and make an intervention at
of course length provides benefits to both students and the optimal time to improve their study behavior. In the
VOLUME 9, 2021 7533
 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 11. Assessment performance at different percentage of course length.

FIGURE 12. Sum of clicks at different percentage of course length.

present study, we proposed various predictive models trained performance based on demographics variables, demograph-
on several ML and DL algorithms for predicting students’ ics + clickstream variables, and demographics + clickstream

7534 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 13. Students’ VLE interaction vs final grade positions at different percentage of course length.

+ assessment variables. RF predictive model with the high- students’ performance at the different lengths of course. Such
est performance scores was finally selected for predicting a predictive model can facilitate instructors to make timely

VOLUME 9, 2021 7535

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

FIGURE 14. Relationship of Code module and imd_band with the final result where highest performance is observed in course
BBB, FFF and lowest performance in course AAA.

interventions and persuade at-risk students to improve their intervention techniques can be implemented in the online
study performance. Out of many variables, clickstream, and learning environment to encourage students to keep on the
assessment variables were having the most significant impact right track. In the future, we plan to examine the activity-
on the final result of the students. wise significance having a prominent influence on the stu-
This study revealed that techniques such as feature engi- dents’ performance by modeling textual variables relating to
neering momentously improve the performance of predictive students’ feedbacks by utilizing deep learning models and
models. During the course module timeline, the perfor- natural language processing techniques.
mance of students was predicted at the very beginning when
only demographics variables were available. Subsequently, APPENDIX
the students’ performance was predicted at 20%, 40%, 60%, OULAD EXPLORATORY DATA ANALYSIS (EDA)
80%, and 100% of the course length. Even at 20% of Before training the predictive models, EDA was performed
the course length, the RF predictive model was producing on OULAD to understand relationships among different vari-
promising results with 79% average precision score, 79% ables. Figure 8 presents the comparison of clickstream data
average recall score, 79% average f-score, and 79% average with assessment score at the end of the course indicating that
accuracy score. At 60% of the course length, the RF pre- with a higher assessment score the sum clicks per student
dictive model performance improved significantly with 88% increased.
of average precision, recall, f-score, and accuracy. Finally, Figure 9 shows that students having more late assessment
at 100% course completion, the highest RF predictive model submitted were interacting more with the VLE but the rela-
performance scores were observed with an average precision tionship is not very significant. From the line chart, it can
of 92%, average recall of 91%, average f-score of 91%, be observed that students having 12 late assessment sub-
and average accuracy of 91 percent. We also observed that missions were involved more than any of the remaining late
the performance scores for Fail class individually after the submissions.
feature engineering process were better than the Pass class Figure 10 reveals that students getting Distinction (com-
performance scores. Getting a higher performance score for pleted courses with excellent grades) and Pass (completed
the Fail class was due to an imbalanced class problem with courses with a passing score) grades in the final result were
17,208 Fail students and 15,385 Pass students. involved more with the VLE as compare to the student getting
Overall, the results of the RF predictive model demon- Fail (students who completed the course but were not able to
strated effectiveness in the earliest possible prediction of secure passing marks) and Withdrawn (students who were not
the performance of at-risk students. Such data-driven studies able to complete the course) score.
can assist VLE administrators and instructors in the for- The five subplots in figure 11 shows students’ assessment
mulation of the online learning framework which can con- performance at 20%, 40%, 60%, 80%, and 100% of course
tribute to the decision-making process. We also deemed that length. The subplots reveal similar students’ assessment
more in-depth studies are required to evaluate various online scores at a different percentage of course length. Students
activities in the OULAD. Particularly, how various early showing a lower assessment performance at the beginning of

7536 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

the course have a similar score during the rest of the course [16] E. B. Costa, B. Fonseca, M. A. Santana, F. F. de Araájo, and J. Rego,
length. ‘‘Evaluating the effectiveness of educational data mining techniques for
early prediction of students’ academic failure in introductory programming
Similarly, the subplots in figure 12 shows clickstream courses,’’ Comput. Hum. Behav., vol. 73, pp. 247–256, Aug. 2017.
pattern at different stages of the course length. It can be [17] D. Gašević, S. Dawson, T. Rogers, and D. Gasevic, ‘‘Learning analytics
observed that the clickstream pattern remains the same for should not promote one size fits all: The effects of instructional conditions
in predicting academic success,’’ Internet Higher Edu., vol. 28, pp. 68–84,
most students during a different percentage of the course Jan. 2016.
length. However, we can also observe that VLE interaction [18] A. Bozkurt and I. E. Aydin, ‘‘Cultural diversity and its implications
at the beginning of the course (at 20% and 40% of course in online networked learning spaces,’’ in Supporting Multiculturalism
Open Distance Learning spaces. Hershey, PA, USA: IGI Global, 2018,
length) is higher but remain almost the same after the 50% of pp. 56–81.
course completion (at 60%, 80%, and 100% of course length). [19] R. S. Baker and P. S. Inventado, ‘‘Educational data mining and learning
analytics,’’ in Learning Analytics. New York, NY, USA: Springer, 2014,
Similarly, the subplots in figure 13 shows students similar
pp. 61–75.
clickstream patterns at a different percentage of course length [20] L. Cen, D. Ruta, L. Powell, B. Hirsch, and J. Ng, ‘‘Quantitative approach
for the Pass, Withdrawn, Fail, and Distinction cases. to collaborative learning: Performance prediction, individual assessment,
and group composition,’’ Int. J. Comput.-Supported Collaborative Learn.,
Finally, figure 14 presents a relationship between the final vol. 11, no. 2, pp. 187–225, Jun. 2016.
result and code module/imd_band where it can be observed [21] A. Mueen, B. Zafar, and U. Manzoor, ‘‘Modeling and predicting students’
that students showed the lowest performance in code module academic performance using data mining techniques,’’ Int. J. Mod. Edu.
Comput. Sci., vol. 8, no. 11, p. 36, 2016.
AAA and highest performance in code module BBB and FFF. [22] S. Huang and N. Fang, ‘‘Predicting student academic performance in an
engineering dynamics course: A comparison of four types of predictive
REFERENCES mathematical models,’’ Comput. Edu., vol. 61, pp. 133–145, Feb. 2013.
[1] L. P. Macfadyen and S. Dawson, ‘‘Mining LMS data to develop an ‘early [23] F. Marbouti, H. A. Diefes-Dux, and K. Madhavan, ‘‘Models for early
warning system’ for educators: A proof of concept,’’ Comput. Edu., vol. 54, prediction of at-risk students in a course using standards-based grading,’’
no. 2, pp. 588–599, Feb. 2010. Comput. Edu., vol. 103, pp. 1–15, Dec. 2016.
[2] C. Romero, S. Ventura, and E. García, ‘‘Data mining in course management [24] D. T. Tempelaar, B. Rienties, and B. Giesbers, ‘‘In search for the most
systems: Moodle case study and tutorial,’’ Comput. Edu., vol. 51, no. 1, informative data for feedback generation: Learning analytics in a data-rich
pp. 368–384, Aug. 2008. context,’’ Comput. Hum. Behav., vol. 47, pp. 157–167, Jun. 2015.
[3] S. Valsamidis, S. Kontogiannis, I. Kazanidis, T. Theodosiou, and [25] R. F. Kizilcec, M. Pérez-Sanagustín, and J. J. Maldonado, ‘‘Self-
A. Karakos, ‘‘A clustering methodology of Web log data for learn- regulated learning strategies predict learner behavior and goal attainment
ing management systems,’’ J. Educ. Technol. Soc., vol. 15, no. 2, in massive open online courses,’’ Comput. Edu., vol. 104, pp. 18–33,
pp. 154–167, 2012. Jan. 2017.
[4] M. Hussain, W. Zhu, W. Zhang, S. M. R. Abidi, and S. Ali, ‘‘Using machine [26] J. Kuzilek, M. Hlosta, D. Herrmannova, Z. Zdrahal, and A. Wolff, ‘‘Ou
learning to predict student difficulties from learning session data,’’ Artif. analyse: Analysing at-risk students at the open university,’’ Learn. Analyt-
Intell. Rev., vol. 52, no. 1, pp. 381–407, Jun. 2019. ics Rev., vol. 8, pp. 1–16, Mar. 2015.
[5] O. E. Aissaoui, Y. E. A. El Madani, L. Oughdir, and Y. E. Allioui, ‘‘Com- [27] A. Wolff, Z. Zdrahal, A. Nikolov, and M. Pantucek, ‘‘Improving retention:
bining supervised and unsupervised machine learning algorithms to predict Predicting at-risk students by analysing clicking behaviour in a virtual
the learners’ learning styles,’’ Procedia Comput. Sci., vol. 148, pp. 87–96, learning environment,’’ in Proc. 3rd Int. Conf. Learn. Analytics Knowl.,
Jan. 2019. 2013, pp. 145–149.
[6] J. Y. Chung and S. Lee, ‘‘Dropout early warning systems for high school [28] M. Hlosta, D. Herrmannova, L. Vachova, J. Kuzilek, Z. Zdrahal,
students using machine learning,’’ Children Youth Services Rev., vol. 96, and A. Wolff, ‘‘Modelling student online behaviour in a virtual
pp. 346–353, Jan. 2019. learning environment,’’ 2018, arXiv:1811.06369. [Online]. Available:
[7] S. A. Salloum, M. Alshurideh, A. Elnagar, and K. Shaalan, ‘‘Mining in
http://arxiv.org/abs/1811.06369
educational data: Review and future directions,’’ in Proc. Joint European-
[29] Y. Cui, F. Chen, and A. Shiri, ‘‘Scale up predictive models for early
US Workshop Appl. Invariance Comput. Vis. Cairo, Egypt: Springer, 2020,
detection of at-risk students: A feasibility study,’’ Inf. Learn. Sci., vol. 121,
pp. 92–102.
nos. 3–4, pp. 97–116, Feb. 2020.
[8] A. Hernández-Blanco, B. Herrera-Flores, D. Tomás, and
[30] Y. Lee and J. Choi, ‘‘A review of online course dropout research: Impli-
B. Navarro-Colorado, ‘‘A systematic review of deep learning approaches
cations for practice and future research,’’ Educ. Technol. Res. Develop.,
to educational data mining,’’ Complexity, vol. 2019, May 2019,
vol. 59, no. 5, pp. 593–618, Oct. 2011.
Art. no. 1306039.
[31] J.-L. Hung, M. C. Wang, S. Wang, M. Abdelrasoul, Y. Li, and W. He,
[9] K. S. Rawat and I. Malhan, ‘‘A hybrid classification method based on
‘‘Identifying at-risk students for early interventions—A time-series clus-
machine learning classifiers to predict performance in educational data
tering approach,’’ IEEE Trans. Emerg. Topics Comput., vol. 5, no. 1,
mining,’’ in Proc. 2nd Int. Conf. Commun., Comput. Netw. Chandigarh,
pp. 45–55, Nov. 2015.
India: National Institute of Technical Teachers Training and Research,
[32] T. Soffer and A. Cohen, ‘‘Students’ engagement characteristics predict
Department of Computer Science and Engineering, 2019, pp. 677–684.
[10] S. M. Jayaprakash, E. W. Moody, E. J. M. Lauría, J. R. Regan, and success and completion of online courses,’’ J. Comput. Assist. Learn.,
J. D. Baron, ‘‘Early alert of academically at-risk students: An open source vol. 35, no. 3, pp. 378–389, Jun. 2019.
analytics initiative,’’ J. Learn. Analytics, vol. 1, no. 1, pp. 6–47, May 2014. [33] P. H. Winne, ‘‘Improving measurements of self-regulated learning,’’ Educ.
[11] C. Márquez-Vera, A. Cano, C. Romero, A. Y. M. Noaman, H. Mousa Psychologist, vol. 45, no. 4, pp. 267–276, Oct. 2010.
Fardoun, and S. Ventura, ‘‘Early dropout prediction using data mining: [34] L. Sha, C.-K. Looi, W. Chen, and B. H. Zhang, ‘‘Understanding mobile
A case study with high school students,’’ Expert Syst., vol. 33, no. 1, learning from the perspective of self-regulated learning,’’ J. Comput. Assist.
pp. 107–124, Feb. 2016. Learn., vol. 28, no. 4, pp. 366–378, Aug. 2012.
[12] S. Palmer, ‘‘Modelling engineering student academic performance using [35] R. Baker, B. Evans, Q. Li, and B. Cung, ‘‘Does inducing students to
academic analytics,’’ Int. J. Eng. Edu., vol. 29, no. 1, pp. 132–138, 2013. schedule lecture watching in online classes improve their academic per-
[13] Z. Papamitsiou and A. Economides, ‘‘Learning analytics and educational formance? An experimental analysis of a time management intervention,’’
data mining in practice: A systematic literature review of empirical evi- Res. Higher Edu., vol. 60, no. 4, pp. 521–552, 2019.
dence,’’ Edu. Technol. Soc., vol. 17, no. 4, pp. 49–64, 2014. [36] A. Cicchinelli, E. Veas, A. Pardo, V. Pammer-Schindler, A. Fessl,
[14] A. Peña-Ayala, ‘‘Educational data mining: A survey and a data mining- C. Barreiros, and S. Lindstädt, ‘‘Finding traces of self-regulated learning in
based analysis of recent works,’’ Expert Syst. Appl., vol. 41, no. 4, activity streams,’’ in Proc. 8th Int. Conf. Learn. Anal. Knowl., Mar. 2018,
pp. 1432–1462, Mar. 2014. pp. 191–200.
[15] N. Z. Zacharis, ‘‘A multivariate approach to predicting student outcomes [37] J. M. Lim, ‘‘Predicting successful completion using student delay indica-
in Web-enabled blended learning courses,’’ Internet Higher Edu., vol. 27, tors in undergraduate self-paced online courses,’’ Distance Edu., vol. 37,
pp. 44–53, Oct. 2015. no. 3, pp. 317–332, Sep. 2016.

VOLUME 9, 2021 7537

 M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

[38] J. Park, K. Denaro, F. Rodriguez, P. Smyth, and M. Warschauer, ‘‘Detecting [61] H. L. Fwa and L. Marshall, ‘‘Modeling engagement of programming stu-
changes in student behavior from clickstream data,’’ in Proc. 7th Int. Learn. dents using unsupervised machine learning technique,’’ GSTF J. Comput.,
Anal. Knowl. Conf., Mar. 2017, pp. 21–30. vol. 6, no. 1, pp. 1–6, 2018.
[39] S. Gupta and A. S. Sabitha, ‘‘Deciphering the attributes of student retention [62] J. Xu, K. H. Moon, and M. van der Schaar, ‘‘A machine learning approach
in massive open online courses using data mining techniques,’’ Edu. Inf. for tracking and predicting student performance in degree programs,’’
Technol., vol. 24, no. 3, pp. 1973–1994, May 2019. IEEE J. Sel. Topics Signal Process., vol. 11, no. 5, pp. 742–753, Aug. 2017.
[40] G. Akçapçnar, M. N. Hasnine, R. Majumdar, B. Flanagan, and H. Ogata, [63] J. Kuzilek, M. Hlosta, and Z. Zdrahal, ‘‘Open university learning analytics
‘‘Developing an early-warning system for spotting at-risk students by dataset,’’ Sci. Data, vol. 4, no. 1, Dec. 2017, Art. no. 170171.
using eBook interaction logs,’’ Smart Learn. Environ., vol. 6, no. 1, p. 4, [64] B. J. Fogg, ‘‘Fogg behavior model,’’ Behav. Des. Lab., Stanford
Dec. 2019. Univ., Stanford, CA, USA, Tech. Rep., 2019. [Online]. Available:
[41] G. Kőrösi and R. Farkas, ‘‘Mooc performance prediction by deep learning https://behaviormodel.org/
from raw clickstream data,’’ in Proc. Int. Conf. Adv. Comput. Data Sci.
Valletta, Malta: Springer, 2020, pp. 474–485.
[42] A. Cano and J. D. Leonard, ‘‘Interpretable multiview early warning system MUHAMMAD ADNAN received the bachelor’s
adapted to underrepresented student populations,’’ IEEE Trans. Learn. degree in computer science from the University of
Technol., vol. 12, no. 2, pp. 198–211, Apr. 2019. Peshawar and the master’s degree in information
[43] C. Burgos, M. L. Campanario, D. D. L. Peña, J. A. Lara, D. Lizcano,
technology from the School of Electrical Engi-
and M. A. Martínez, ‘‘Data mining for modeling students’ performance:
neering and Computer Science, National Univer-
A tutoring action plan to prevent academic dropout,’’ Comput. Electr. Eng.,
vol. 66, pp. 541–556, Feb. 2018. sity of Science and Technology, Islamabad, where
[44] J. A. Lara, D. Lizcano, M. A. Martínez, J. Pazos, and T. Riera, ‘‘A system he is currently pursuing the Ph.D. degree. He is
for knowledge discovery in e-learning environments within the European currently a Lecturer with the Institute of Infor-
higher education area–application to student data from open university of mation Technology, Kohat University of Science
Madrid, UDIMA,’’ Comput. Edu., vol. 72, pp. 23–36, Mar. 2014. and Technology. His research area includes mobile
[45] C. C. Gray and D. Perkins, ‘‘Utilizing early engagement and machine learning, machine learning, adaptive learning, and ubiquitous systems.
learning to predict student outcomes,’’ Comput. Edu., vol. 131, pp. 22–32,
Apr. 2019.
[46] R. Al-Shabandar, A. J. Hussain, P. Liatsis, and R. Keight, ‘‘Detecting at- ASAD HABIB received the Ph.D. degree from
risk students with early interventions using machine learning techniques,’’ the Nara Institute of Science and Technology,
IEEE Access, vol. 7, pp. 149464–149478, 2019. Nara, Japan. He is currently an Assistant Profes-
[47] S. Lee and J. Y. Chung, ‘‘The machine learning-based dropout early sor with the Institute of Information Technology,
warning system for improving the performance of dropout prediction,’’ Kohat University of Science and Technology. He
Appl. Sci., vol. 9, no. 15, p. 3093, Jul. 2019. is involved in the area of natural language process-
[48] A. Behr, M. Giese, and K. Theune, ‘‘Early prediction of university
ing, human–computer interaction, artificial intelli-
dropouts—A random forest approach,’’ J. Nat. Stat., vol. 1, pp. 743–789,
gence, learning technologies, mobile learning, and
Feb. 2020.
[49] L. C. B. Martins, R. N. Carvalho, R. S. Carvalho, M. C. Victorino, and adaptive interface design.
M. Holanda, ‘‘Early prediction of college attrition using data mining,’’
in Proc. 16th IEEE Int. Conf. Mach. Learn. Appl. (ICMLA), Dec. 2017,
pp. 1075–1078.
[50] N. Mduma, K. Kalegele, and D. Machuve, ‘‘Machine learning approach for JAWAD ASHRAF received the Ph.D. degree from
reducing students dropout rates,’’ Int. J. Adv. Comput. Res., vol. 9, no. 42, the Department of Computer Science, University
2019, doi: 10.19101/IJACR.2018.839045. of Leicester, U.K. He is currently with the Insti-
[51] J. Figueroa-Ca nas and T. Sancho-Vinuesa, ‘‘Predicting early dropout tute of Information Technology, Kohat University
student is a matter of checking completed quizzes: The case of an online of Science and Technology, where he is working
statistics module,’’ in Proc. LASI-SPAIN, 2019, pp. 100–111. on partner-based scheduling algorithm for grid
[52] A. Ortigosa, R. M. Carro, J. Bravo-Agapito, D. Lizcano, J. J. Alcolea, workflows in advance reservation environment,
and O. Blanco, ‘‘From lab to production: Lessons learnt and real-life K-shortest path variant for routing in advance
challenges of an early student-dropout prevention system,’’ IEEE Trans. reservation environment, and novel workflow job
Learn. Technol., vol. 12, no. 2, pp. 264–277, Apr. 2019. selection technique.
[53] A. S. Imran, F. Dalipi, and Z. Kastrati, ‘‘Predicting student dropout in a
MOOC: An evaluation of a deep neural network model,’’ in Proc. 5th Int. SHAFAQ MUSSADIQ received the M.S. degree
Conf. Comput. Artif. Intell. (ICCAI), 2019, pp. 190–195. in computer science and the Ph.D. degree from the
[54] N. Wu, L. Zhang, Y. Gao, M. Zhang, X. Sun, and J. Feng, ‘‘CLMS-net:
School of Electrical Engineering and Computer
Dropout prediction in MOOCs with deep learning,’’ in Proc. ACM Turing
Celebration Conf., May 2019, pp. 1–6. Science, NUST, Islamabad. She is currently an
[55] C. P. Rosé, E. A. McLaughlin, R. Liu, and K. R. Koedinger, ‘‘Explanatory Assistant Professor with the Institute of Informa-
learner models: Why machine learning (alone) is not the answer,’’ Brit. tion Technology, Kohat University of Science and
J. Educ. Technol., vol. 50, no. 6, pp. 2943–2958, 2019. Technology, Kohat. Her research work includes
[56] A. A. Mubarak, H. Cao, and S. A. M. Ahmed, ‘‘Predictive learning machine learning, deep neural networks, multime-
analytics using deep learning model in MOOCs’ courses videos,’’ Edu. Inf. dia technologies, and big data in education.
Technol., vol. 6, pp. 1–22, Jul. 2020.
[57] S. N. Liao, D. Zingaro, K. Thai, C. Alvarado, W. G. Griswold,
and L. Porter, ‘‘A robust machine learning technique to predict low-
performing students,’’ ACM Trans. Comput. Edu., vol. 19, no. 3, pp. 1–19, ARSALAN ALI RAZA is currently pursuing the
Jun. 2019. Ph.D. degree in computer science with the Kohat
[58] B. Sekeroglu, K. Dimililer, and K. Tuncal, ‘‘Student performance predic- University of Science and Technology, Kohat,
tion and classification using machine learning algorithms,’’ in Proc. 8th Pakistan. He is currently a Lecturer in computer
Int. Conf. Educ. Inf. Technol., Mar. 2019, pp. 7–11. science with the COMSATS Institute of Informa-
[59] Y. Mao, ‘‘Deep learning vs. Bayesian knowledge tracing: Student models
tion Technology, Pakistan. His research interest
for interventions,’’ J. Educ. Data Mining, vol. 10, no. 2, pp. 1–27, 2018.
[60] Z. Iqbal, J. Qadir, A. Noor Mian, and F. Kamiran, ‘‘Machine learning
includes natural language engineering, data min-
based student grade prediction: A case study,’’ 2017, arXiv:1708.08744. ing, machine learning, and artificial intelligence.
[Online]. Available: http://arxiv.org/abs/1708.08744

7538 VOLUME 9, 2021

M. Adnan et al.: Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention

MUHAMMAD ABID received the Ph.D. degree SANA ULLAH KHAN received the M.S. degree in
from the Institute of Computing, Kohat University computer science from the Islamia College Uni-
of Science and Technology, Kohat. He is currently versity Peshawar in 2014, where he is currently
working on various projects related to automa- pursuing the Ph.D. degree in the field of medical
tion and transparency of educational institutes. His image processing. He is also a permanent Faculty
research area includes natural language process- Member with the Institute of Computing, Kohat
ing, machine learning, and data mining. University of Science and Technology (KUST),
Kohat, serving as a Lecturer for the past three
years. He has authored various research articles
of journals and conferences. His research interest
includes computer vision, medical imaging, machine learning, cloud com-
puting, and deep learning.

MARYAM BASHIR is currently pursuing the Ph.D. degree with the Institute
of Computing, Kohat University of Science and Technology, Pakistan. Her
Ph.D. thesis is on trajectory data mining. She is currently a Visiting Lecturer
with the Government Post Graduate College, Kohat.

VOLUME 9, 2021 7539