SN Computer Science (2024) 5:488

https://doi.org/10.1007/s42979-024-02830-4

ORIGINAL RESEARCH

Explainable Anomaly Detection of Synthetic Medical IoT Traffic Using

Machine Learning
Lerina Aversano1 · Mario Luca Bernardi2 · Marta Cimitile3 · Debora Montano4 · Riccardo Pecori5,6 · Luca Veltri7

Received: 21 November 2023 / Accepted: 25 March 2024

© The Author(s) 2024

Abstract
In the context of the Internet of Things (IoT), particularly within medical facilities, the detection and categorization of
Internet traffic remain significant challenges. While conventional methods for IoT traffic analysis can be applied, obtaining
suitable medical traffic data is challenging due to the stringent privacy constraints associated with the health domain. To
address this, this study proposes a network traffic simulation approach using an open-source tool called IoT Flock, which
supports both CoAP and MQTT protocols. The tool is used to create a synthetic dataset, to simulate IoT traffic originating
from various smart devices in different hospital rooms. The study shows a complete anomaly detection analysis of IoT-Flock-
generated traffic, both normal and malicious, by leveraging and comparing traditional machine learning techniques, deep
learning models with multiple hidden layers, and explainable artificial intelligence techniques. The results are very promising.
For the binary classification, for example, the obtained accuracy is close to 100% in the case of the CoAP protocol. Good
results are also obtained when the multinomial classification is performed, observing that CoAP packets are classified better
than MQTT packets, even if the identification of the different MQTT packets reaches very high metrics for the most of the
considered algorithms. Moreover, the obtained classification rules are also meaningful in the considered IoT context. The
results indicate that IoT-Flock synthetic data can effectively be used to train and test machine and deep learning models for
detecting abnormal IoT traffic in medical scenarios. This research attempts also to bridge the gap between IoT security and
healthcare, providing useful insights into securing medical IoT networks in general.

Keywords Medical internet of things · Anomaly detection · Intrusion detection systems · Machine learning · Explainable
artificial intelligence · Deep neural networks

Introduction

The Internet of Things (IoT) refers to the integration of dif-

This article is part of the topical collection “Recent Trends on Data ferent devices across various technologies, enabling them
Science, Technology and Applications” guest edited by Slimane
Hammoudi, Alfredo Cuzzocrea and Oleg Gusikhin. to connect and communicate autonomously, without human

2
* Riccardo Pecori Department of Engineering, University of Sannio,
[email protected] 82100 Benevento, Italy
3
Lerina Aversano Department of Law and Digital Society, Unitelma Sapienza
[email protected] University, 00161 Rome, Italy
4
Mario Luca Bernardi CeRICT Scrl, 82100 Benevento, Italy
[email protected] 5
SMARTEST Research Centre, eCampus University,
Marta Cimitile 22060 Novedrate, CO, Italy
[email protected] 6
Institute of Materials for Electronics and Magnetism,
Luca Veltri National Research Council of Italy, 43124 Parma, Italy
[email protected] 7
Department of Engineering and Architecture, University
1 of Parma, 43124 Parma, Italy
Department of Agricultural Science, Food, Natural
Resources and Engineering, University of Foggia,
71122 Foggia, Italy

SN Computer Science
Vol.:(0123456789)
488 Page 2 of 15 SN Computer Science (2024) 5:488

intervention. These IoT devices not only interact with each ing features of single packets, through both a binary and
other, but also with a wide array of equipment, including a multinomial classification;
industrial machinery, robots, drones, and energy genera- • The application of an inherent explainable machine
tion systems, making them applicable in numerous sec- learning technique, i.e., fuzzy Hoeffding decision tree,
tors, including healthcare. IoT devices, especially when to extract useful rules to interpret the classification out-
deployed in critical settings like healthcare, raise signifi- comes and evaluate if the obtained results have a signifi-
cant security concerns [1]. Traditional security measures, cant meaning in the real-world context.
such as firewalls, intrusion detection systems (IDSs), and
intrusion prevention systems (IPSs), are commonly used to The final aim is to encourage the large usage of IoT-Flock
safeguard devices and networks against cyber-threats. Pres- as a tool to create IoT-based traffic datasets useful to train
ently, many firewalls and IPSs rely on predefined rules to real-world effective AI-based IDSs in both hospitals and
filter out abnormal or malicious traffic. However, certain healthcare-related buildings.
IDS and IPS systems also leverage Artificial Intelligence
(AI) techniques to identify malicious traffic effectively [2]. Structure of the work
As a matter of fact, combining AI methods with predefined
rules enhances the ability to detect attack traffic compared This work is structured as follows:
with using only static rules. AI-based IDS and IPS systems
are typically trained and tested using datasets that encom- • Section “ Background” offers background information
pass both normal and attacking traffic. These datasets can concerning the IoT attacks that are under consideration
be acquired through two approaches: (i) collecting actual in this paper and the algorithms used for the analyses of
traffic data from live systems to capture both malicious and the considered IoT traffic;
normal traffic, or (ii) utilizing synthetic traffic generators • Section “Related Work” discusses related researches on
that simulate real-time network traffic patterns. This research the application of machine and deep learning algorithms
aims at a medical IoT environment, where obtaining real IoT for IoT attack categorization;
traffic data proves challenging due to the stringent privacy • Section “IoT-Flock” explains the utilization of IoT-Flock
constraints associated with patients’ health information, and outlines the considered use-cases;
even when anonymized. To address this issue and study this • Section Experimental Settings” provides a detailed expla-
critical IoT scenario, we employed a network traffic simula- nation of the regarded attributes, the considered data,
tion approach using an open-source tool called “IoT-Flock” the hyper parameters of the machine and deep learning
[3]. IoT-Flock is capable of generating IoT traffic originat- models that we used for the classification analysis, as
ing from various smart devices and supports two distinct well as the evaluated performance metrics;
application layer protocols commonly used in IoT scenarios: • Section “Results” reports the results of all the analyses
CoAP and MQTT. Furthermore, IoT-Flock offers the flex- we conducted, using both machine and deep learning
ibility to create customized IoT use cases, allowing the inclu- models, and shows some explainable rules we extracted;
sion of as many custom IoT devices as needed and the gen- • Section “Discussion and Threats to the Validity” dis-
eration of both malicious and normal IoT traffic associated cusses the obtained results, the meaning of the output
with these scenarios. This approach was chosen to overcome rules and some threats to the validity of the study;
the challenges of acquiring real medical IoT traffic data and • Section “Conclusions and Future work” concludes the
facilitate the study of this complex environment. article by offering guidance and proposals for future
research developments.
Major Contributions of the Work

The major contributions of this paper, which is an extension Background

of the preliminary paper presented at DATA 2023 confer-
ence [4], include: Attack Description

• The identification, for the first time as we know, of IoT- In this study, we examine the traffic generated by two widely
Flock-generated CoAP traffic; used IoT protocols, namely MQTT (Message Queue Telem-
• The comparison, for the first time as we know, of both etry Transfer) [5] and CoAP (Constrained Application Pro-
machine and deep learning models on synthetic traffic tocol) [6]. Additionally, we investigate two types of attack
generated by IoT-Flock; for each protocol currently supported by IoT-Flock,1 an
• A thorough analysis of IoT-Flock-generated traffic by
means of a packet-based feature model, thus consider- 1
https://​github.​com/​Thing​zDefe​nse/​IoT-​Flock.

SN Computer Science
SN Computer Science (2024) 5:488 Page 3 of 15 488

open-source tool, designed for generating synthetic IoT traf- the variations of the data. In recent years, many studies have
fic, whose users can construct various IoT use-case scenarios been conducted on the classification of IoT traffic using often
with specific IoT devices and produce both legitimate and a supervised ML technique [4, 10].
malignant IoT packets over a real-time synthetic network. This study, in particular, employs several algorithms such
A brief summary of the attacks analyzed in this paper is as:
provided hereinafter:
• Naïve Bayes [11], a popular algorithm used for classifica-
• MQTT packet crafting attack: in this attack, MQTT pack- tion and probabilistic modeling. It is based on the Bayes’
ets are deliberately crafted to disrupt the MQTT broker. theorem, which is a probability theory that calculates the
The attacker initiates a connection with the MQTT bro- likelihood of an event occurring given prior knowledge.
ker at the transport level and sends a malformed MQTT Naïve Bayes is particularly well-suited for scenarios with
packet. This malicious packet has the potential to trigger high-dimensional data, where other algorithms might
a buffer overflow, causing certain MQTT broker imple- struggle. However, its performance can be sub-optimal
mentations to crash, thereby rendering a Denial of Ser- when the independence assumption is strongly violated,
vice (DoS) attack possible [7]. or when dealing with complex relationships within the
• MQTT publish flood attack: in this attack, MQTT publish data.
messages are directed at a broker with high rate, with • Support vector machine (SVM) [12], a powerful and
the intention of potentially initiating a Denial of Service versatile supervised algorithm used for classification,
(DoS) attack, by filling all the memory and/or processing regression, and outlier detection tasks. It works by find-
capabilities of the broker. ing an optimal hyperplane that best separates data-points
• CoAP segmentation fault attack: this attack utilizes a vul- in different classes. The term “support vector” refers to
nerability of a specific CoAP library, the LibNyoci,2 for the data-points that are closest to the decision boundary
embedded systems that let a malformed Uri-Path option (the hyperplane), in fact, they play a crucial role in SVM
cause a possible segmentation fault, leading to a denial operations. SVM is widely used due to its flexibility and
of service attack versus the received CoAP server device ability to handle both linear and non-linear problems
[8]. effectively.
• CoAP memory leak attack: like the previous one, also this • Logistic regression [13] is a statistical and ML technique
attack exploits a vulnerability in a CoAP implementation, used for binary classification, which is employed to pre-
the Eclipse Wakaama,3 which gets a crafted packet with dict one of two possible outcomes based on one or more
invalid options leading to a possible memory leak/waste predictor features. Despite its name, it is not a regression
(of 24 bytes) on the server side. This second-hand attack algorithm, but a classification algorithm. Its simplicity,
can be also used to exhaust memory resources leading to efficiency, and interpretability make it a really valuable
a DoS [9]. classification technique.
• Decision tree [14], an algorithm that learns to make deci-
Employed Machine and Deep Learning Techniques sions by recursively splitting data into subsets based on
the most significant features. All the decision rules form
In this subsection, we summarize the main characteristics a tree-like structure, where each internal node represents
of the used machine and deep learning techniques, focus- a decision or test on an attribute, and each leaf node rep-
ing also on the Fuzzy Hoeffding Decision Tree chosen as resents the outcome or class label.
explainable AI model.
More details about the settings of ML algorithms used in this
Machine Learning work are discussed in subsection “Models Settings”.

Machine Learning (ML) is composed of algorithms and Deep Learning

techniques capable of creating systems that learn and
enhance their performance over time by leveraging the Deep learning (DL) is a subset of ML, a field of AI that
information inside the data themselves. The quality and focuses on algorithms and models inspired by the structure
nature of the data significantly impact the effectiveness of and function of the human brain’s neural networks. DL algo-
ML algorithms, as various data representations can either rithms, also known as deep neural networks, are designed
reveal or obscure the underlying explanatory factors driving to model and understand complex patterns in large datasets.
They have gained significant attention and success in various
2
https://​github.​com/​darco​neous/​libny​oci. AI applications due to their ability to automatically learn
3
https://​www.​eclip​se.​org/​wakaa​ma/. hierarchical representations from data. DL is applied in a

SN Computer Science
488 Page 4 of 15 SN Computer Science (2024) 5:488

wide range of fields, such as image and speech recognition, of IoT-based smart health monitoring systems are being
autonomous vehicles, recommendation systems, healthcare, introduced and integrated into daily healthcare practices.
finance, and natural language understanding. In recent years, However, in terms of security of healthcare systems, the
there has been a growing interest in utilizing deep learning IoT is still in its early stages of development. Unfortu-
techniques also for the classification of IoT traffic [15–17]. nately, given that the resources of IoT smart devices are
In our work, we use a Deep Neural Network (DNN) to truly limited, and given that the security requirements to
classify synthetic IoT traffic. A DNN has multiple layers be met are increasingly greater, especially in the health-
between the input and output layers. These intermediate lay- care sector, it is not possible to use the normal security
ers, the hidden layers, enable the network to learn increas- solutions usually deployed for the protection of network
ingly abstract and complex representations of the data, mak- traffic in a scenario dedicated to the medical IoT [20, 21].
ing it capable of solving more intricate tasks, and they are Furthermore, IoT intelligent objects record information
a fundamental component of DL. More details about the based on the specific use case, and, therefore, normal
architecture of our DNN model are reported in subsection security solutions and protocols must be adapted, from
“Models Settings”. time to time, based on the specific use and needs of the
IoT scenario to be protected [20]. Therefore, it is neces-
Fuzzy Hoeffding Decision Tree sary to ensure that there are no external attacks on the
network of smart health devices, taking into account that
The fuzzified version of the Hoeffding Decision Tree IoT solutions applied in the medical field have limited
(FHDT) has been also used in this research work. This model resources. There are many studies regarding the security
entails a fuzzy discretization of the input features, resulting solutions to be adopted to protect IoT healthcare systems
into a linguistic fuzzy partition [18]. FHDT can help in tack- from external attacks. A first proposal in this sense details
ling explainability and interpretability issues because (i) it a technique for identifying replay attacks against battery-
guarantees a balance between classification performance and powered IoT healthcare equipment [22]. To detect and pre-
the complexity of the model, (ii) fuzzy partitions associated vent such attacks, the authors suggest examining battery
with linguistic terms can make the interpretability of the drain behavior, unique device IDs, and timestamps. The
extracted rules easier for a human being [19]. solution proposed by Rathore et al. [21] involves instead
Two details make FHDT different from its traditional ver- the adoption of the semi-supervised Fuzzy C-Means tech-
sion [18]: nique, based on Extreme Learning Machine (ELM), so that
cyber-attacks can be detected in fog-based IoT systems.
• The update process of the statistics in the internal nodes, The use of Fuzzy C-Means is effective in addressing the
because in this variant a training instance can reach difficulties associated with labeling datasets, while ELM
more than one leaf, encompassing the fuzzy membership techniques are useful for quickly and effectively detecting
degree, the local fuzzy cardinality of the whole node, and cyber-attacks. Furthermore, Alradashi et al. [23] propose
the fuzzy cardinalities per class in a node; a strategy capable of identifying malicious devices in a
• The computation of the fuzzy Information Gain and of fog-based IoT healthcare scenario; in particular, this is
the fuzzy Hoeffding bound that exploit the fuzzy cardi- a smart home equipped with a remote patient monitor-
nality instead of the traditional cardinality. ing system. The authors use a system of online sequential
ELMs capable of detecting different types of attacks such
Moreover, the FHDT model can be explained through lin- as distributed denial of service, man-in-the-middle, and
guistic IF-THEN rules extracted from the tree model, by fol- other potential threats to the patient’s health. As already
lowing the paths from the initial node (root) to the terminal said, IoT security is currently a major concern. Currently,
nodes (leaves). Each rule is composed of some antecedents, the most used solutions to protect IoT flows are anti-intru-
regarding the attributes described through the linguistic sion IDSs and IPSs. In fact, if you use datasets relating to
terms associated with the fuzzy partitions, and a consequent both malicious and normal IoT network traffic it is pos-
returning the decided class. sible to carry out an evaluation of the traffic. However,
this involves the use of a large amount of data relating
to IoT traffic to better train IDS and IPS. Multiple recent
Related Work studies provide collections of real-world IoT data [15–17];
however, in these studies, it is possible to detect one of
The concept of smart health and smart devices has under- the main difficulties of the medical IoT scenario, as it is
gone a radical transformation with the rapid growth of the hard to collect adequate data, which follows all the rigor-
IoT. As IoT-based health monitoring systems continue to ous delivery requirements that all hospitals and healthcare
evolve and become more intelligent, an increasing number institutions are required to comply with. Furthermore, it is

SN Computer Science
SN Computer Science (2024) 5:488 Page 5 of 15 488

very difficult to collect useful traffic data via IoT systems AI models to be tested in real-world scenarios afterwards.
in the real world, this is due to the very low throughput We have chosen to use the fuzzy version of the Hoeffding
of some IoT devices and the various difficulties in col- Decision Tree as an explainable ML algorithm, an algorithm
lecting a large-enough number of instances of both nor- usually adopted in stream mining and so perfectly matching
mal and malicious traffic. Furthermore, various datasets the variable nature of IoT traffic flows.
are available online, mostly used for training and testing
IDSs and IPSs, and they refer to both traditional networks
and IoT networks. Indeed, the major studies in the sec- IoT‑Flock
tor in recent years use such data precisely. In particular,
some are generated using real-time systems, and can be In order to generate the data used in this study, we lever-
considered on a par with real datasets, while others are aged IoT-Flock [3], an open-source IoT traffic generator
generated through simulation techniques, thus they are that operates in real-time and through which different use
labeled as simulated and/or synthetic datasets [24–30]. cases may be designed, characterizing them with different
Concerning the application of explainable models to the intelligent devices. IoT-Flock manages to generate both
anomaly detection of IoT traffic, such a research topic has regular and anomalous IoT traffic, a function most com-
been investigated more and more in the last years [31]. For mercial and open-source tools lack; more precisely, they
example, in [32], explainable deep learning, specifically, do not support the creation of malicious devices in the
autoencoder-based LSTM and DNNs, is applied to an same use case. This functionality is particularly useful
industrial IoT scenario and to a real world GSP system. In because it allows one to provide more adequate IDSs and
[33], an explainable variational autoencoder is applied to IPSs. Furthermore, IoT-Flock provides for the possibility
detect anomalies in NetFlow traffic. The explanations are of exporting the designed use cases in XML format and of
provided through a gradient based fingerprinting technique importing the XML generated using both IoT-Flock itself
applied to the UGR16 dataset containing anonymized and other tools. Furthermore, the tool also allows one to
NetFlow traces. The work in [34] is the closest to ours generate a series of recent attacks, specifying them based
in terms of explainability applied to an IoT scenario: the on the protocol in use, MQTT and COAP; this feature is
authors apply a fuzzy decision tree to a real-world data- not present in any of the open-source IoT traffic generators.
set containing flow-based features, while we focused on IoT-Flock can work in two ways, i.e., via a GUI or via
a synthetic dataset with packet-based attributes. In [35], a command line. In both modes, to generate a single intel-
the focus is on explaining the best anomaly detection tech- ligent device, it is necessary to provide a series of relevant
niques for a certain IoT dataset, while in [36], there is the information about it, both functional and non-functional.
claim that explainable neural networks have been applied In particular, the first type of information concerns the
to an Internet of Vehicles scenario by exploiting K-means working behavior of the device, such as the type of device
clustering for feature scoring and ranking using two public (normal or malicious), the used protocol (MQTT and/or
datasets, namely CICIDS2019 and UNSW-NB15. How- CoAP), the data profile (type and scope of transmitted
ever, explainable results are not discussed, focusing on data), the time profile (periodic or random), the type of
performance metrics instead. In [37], the focus is again command (Subscribe or Publish in MQTT, Post or Get in
on an industrial scenario, specifically industrial control CoAP). It is the latest information that distinguishes one
systems, and the explainability is provided through an IoT device from another one by IP address, device name,
LSTM-based autoencoder, with one class SVM and gra- number of devices of the same type, etc.
dient SHAP values applied to a SCADA dataset.
For the work we propose, we have taken inspiration from
the Bot-IoT dataset [38], which simulates IoT traffic pro- Considered Use Case
duced by some normal and attacker virtual machines. Unlike
this, however, the traffic we generated uses IoT-Flock as a In our research, we explored a scenario similar to the one
simulator, which, unlike Bot-IoT, takes the traffic gener- discussed in a previous study [39]. This scenario involves
ated by both MQTT and CoAP protocols into account. This two hospital rooms, each equipped with various devices
allowed us not only to recognize the type of traffic, be it that are meticulously controlled through smart sensors and
normal or malicious, but also to classify the types of attacks actuators connected to the Internet. In both rooms, we can
based on the used IoT protocol. Moreover, we applied, for distinguish two categories of devices:
the first time, ante-hoc explainable ML to the synthetic traf-
fic generated by IoT-flock, in order to verify whether the • The devices responsible for monitoring and regulating
resulting rules have a real-world meaning and thus confirm- the environment of the room. These devices use the
ing the feasibility of using IoT-Flock as a tool for training MQTT protocol to communicate through a dedicated

SN Computer Science
488 Page 6 of 15 SN Computer Science (2024) 5:488

broker. Their primary role is to autonomously manage • Fan speed controller: it is an actuator and every second
and maintain the comfort of the room and environmen- it receives data on the topic “Fan Speed”, related to the
tal conditions. speed of the fan present in the room. It receives also
• The devices designed for monitoring the physical sta- data from the topics “Smoke”, “CO-GAS”, “Humid-
tus of the patients occupying the room. These devices ity”, “Door Lock”, and “Temperatures”;
employ the CoAP protocol to communicate with a des- • Lock: it publishes data relating to the status of the lock
ignated server, which can be accessed by healthcare with a random frequency between 1 and 5 seconds in
personnel. Their purpose is to supervise and report on the “Door Lock” topic.
the health and well-being of the patients.
In the case of CoAP devices, each bed in the rooms is
This setup illustrates how IoT technology is applied in a equipped with nine smart devices, and there is a CoAP
healthcare environment, facilitating both patient care and server that works as central control unit. The CoAP server
the management of the hospital environmental conditions. has various duties, including: managing time profiles for
Specifically, the smart devices that use the MQTT pro- patients, regulating the quantity of medication adminis-
tocol establish communication with a broker that functions tered to the patient through an infusion pump, initiating
as an environment control unit. These MQTT-based smart alarms for the medical staff based on the patient’s health
devices encompass nine distinct types and operate at three status as monitored by the smart sensors. The nine smart
different Quality-of-Service (QoS) levels. To provide a devices that utilize CoAP for communication are the fol-
concise overview, the nine types of MQTT-based smart lowing and are described below:
devices are summarized hereinafter:
• ECG sensor: it provides information about the heart
• Light intensity sensor/actuator: publisher/subscriber beat rhythm every 1 second;
type devices. The sensor publishes periodically, namely • Infusion pump: an actuator used to deliver possible
every second, data on the light detected in the envi- nutrients and drugs to the patients, retrieving data from
ronment on the “Light Intensity” topic. The actuator the server every 10 minutes;
receives data from both the light publisher device and • Pulsoximeter: a smart sensor furnishing the oxygen
the movement sensor to fade in or out the light on the saturation in the blood every 1 second;
basis of the external illumination of the room and the • Mouth airflow sensor: a smart sensor providing the
movements in the room itself; breathing rate of the patient every 1 second;
• Temperature sensor/actuator: publisher/subscriber type • Blood pressure sensor: a smart sensor conveying infor-
devices. The sensor publishes periodically, every 2 sec- mation about blood pressure every 2 seconds;
onds, ambient temperature data on the topic “Tempera- • Glucometer: a smart sensor providing information
tures”. The actuator receives the temperature values about the glucose in the blood every 10 minutes;
and tries to keep the room temperature at a constant • Body temperature sensor: a smart sensor measuring the
level, i.e., about 20◦C; temperature of the patient every 1 hour;
• Humidity sensor: every 1 second it publishes ambient • EMG sensor: a smart sensor measuring the electromi-
humidity data in the “Humidity” topic. It is a publisher- ography, i.e., the potential produced by the body mus-
type device; cles, every 5 minutes;
• Motion sensor: publishes data on movements, occur- • GSR sensor: a smart sensor measuring the galvanic skin
ring in the room, in the “Movement” topic. Unlike the response, i.e., the electrical conductance of the skin,
other sensors that publish data periodically at constant every 5 minutes.
time intervals, the sensor of motion publishes data
pseudo-randomly in the 1 − 5-second interval; In Fig. 1, we illustrate the medical IoT scenario simu-
• CO-GAS sensor: it receives and publishes data relat- lated in this study. In this scenario, all devices share a
ing to gases detected in the room in the “CO-GAS” uniform configuration, utilizing the same range of private
topic. The frequency of the publications is random in IP addresses for both MQTT and CoAP devices. The sen-
the range between 1 and 5 seconds; sor network is confined to a secure and controlled access
• Smoke sensor: it is a publisher device. On the topic area, where the devices establish communication with the
“Smoke”, it shares in random intervals, in the range MQTT broker or the CoAP server. Notably, the simulated
between 1 and 5 seconds, the data relating to the sur- network does not include additional components such as
veys on the presence of smoke; firewalls or routers. The traffic within the network is moni-
• Fan sensor: it publishes data every 3 seconds related to tored and captured by a Tshark process running in the
fan operations in the “Fan” topic;

SN Computer Science
SN Computer Science (2024) 5:488 Page 7 of 15 488

Fig. 1  The smart health sce-

nario we considered: MQTT Hospital Room
devices are on the left and
monitor the environment, CoAP
devices can be seen on the
right and control the patient
conditions. MQTT publish
messages or CoAP POST and
GET packets are represented
by means of blue dashed lines, CoAP
while MQTT delivery messages Server
or CoAP response messages
are represented through orange MQTT
solid lines. Malicious nodes are Broker
present in both networks

MQTT devices CoAP devices

background on the computer that is executing IoT-Flock,

Table 1  Description of the considered IoT traffic dataset
facilitating the collection and analysis of IoT traffic data.
Protocol Number of instances Type of traffic
Simulated Attacks CoAP 834,881 Normal traffic
200,003 Segmentation fault attack
For the malicious traffic simulated in this study, we assumed 198,090 Memory leak attack
that there is a specific number of malicious devices within MQTT 593,363 Normal traffic
the private network, controlled remotely by an attacker, and 29,406 Publish flood attack
that this is not adequately protected. 1532 Packet crafting attack
During the simulation of an attack, malicious intelligent
devices are directly connected to the MQTT broker or CoAP
server to execute one of the considered attacks. In this study,
hours per considered protocol, on a machine equipped with
we focused on the analysis of the type of attack and the traf-
an Intel Core i7 7th generation CPU, 16GB of RAM, and
fic corresponding to it, while the methodology used to carry
one NVIDIA GeForce GPU. More precisely, the duration of
out the attacks and/or the way in which the attacker gained
capturing normal packets was 12 hours, while MQTT and
control of the node were not discussed as they go beyond the
CoAP attack packets have been captured for 6 hours each.
scope of this work.
When considering the attacks, we supposed the presence of
4 malicious smart devices, in the corresponding private net-
work, that were sending packets following either a periodic
Experimental Settings (1s) or random (1 − 5s range) time profile.
In order to discriminate between malicious and benign
The IoT Traffic Dataset
traffic, we have extracted, by using a BASH script outputting
the final.csv file, all the features available in Tshark for both
After the setup of the scenarios, we ran IoT-Flock for the
the MQTT5 and the CoAP6 protocols.
actual generation of IoT packets. While the traffic was being
Thereafter, we performed, using a proper Python script,
generated, we captured IoT packets, with the help of Tshark,4
a cleaning phase in order to tear out the features having no
a tool capable of analyzing the traffic generated through
values (NaN), the source and destination IP addresses, as
IoT-Flock in real-time. The output of running Tshark was a
well as the non-significant, and to divide features incorporat-
standard.pcap file which we have further processed in order
ing multiple meaning into different columns.
to obtain the considered features and a final.csv file. The
capturing phase of IoT packets lasted about 48 hours, 24
5
https://​www.​wires​hark.​org/​docs/​dfref/m/​mqtt.​html.
4 6
https://​tshark.​dev/​setup/​insta​ll/. https://​www.​wires​hark.​org/​docs/​dfref/c/​coap.​html.

SN Computer Science
488 Page 8 of 15 SN Computer Science (2024) 5:488

Table 2  Considered MQTT features inspections. The outcome of this preliminary phase is shown
Name Type in Tables 2 and 3, reporting the selected features for MQTT
and CoAP, respectively, as well as their inherent type
mqtt.packet_type1,2,3 Categorical (numeric or categorical).
mqtt.hdr_flags1,2,3 Categorical As concerns MQTT, we considered the following 6
mqtt.kalive1,2,3 Numeric features:
mqtt.conflag.cleansess1,2,3 Categorical
mqtt.conack.flags1,2,3 Categorical • Packet type (CONNECT, AUTH, PUBLISH, SUB-
mqtt.len1,2,3 Numeric SCRIBE, etc.): this feature and the next one have been
extracted from the Tshark feature ‘mqtt.hdrflags’;
• Header flags, that is the flag bits present in the first byte
Table 3  Considered CoAP features of the fixed header of an MQTT packet;
Name Type
• Keep alive interval (measured in seconds), that is the
maximum time interval between two MQTT control
coap.code Categorical packets sent by the client;
coap.opt.type1,2,3 Categorical • Clean session flag of CONNECT message;
coap.opt.length1,2,3 Numeric • Connect acknowledge flags, that is flag bits of CON-
coap.opt.desc1A,2A,3A Categorical NACK packets;
coap.opt.desc1B,2B,3B Categorical • Packet length.
coap.opt.block_size Categorical
coap.opt.observe Categorical Since each captured IP packet concerning the MQTT traf-
coap.opt.end_marker Categorical fic, that corresponds to a TCP segment in turn, may con-
coap.payload_length Numeric tain more than one MQTT packet, for most of the features
mentioned above Tshark provides three different values
corresponding to possible three different MQTT packets
The final dataset, used for our analyses, consists of within the same IP packet (it considers a maximum of
1, 857, 275 instances, each of them corresponding to a three concatenated packets). Therefore, concerning MQTT
packet. Table 1 describes the dataset, in particular, the first traffic we considered a total of 3 × 6 = 18 features per sin-
column reports the application protocol, the second one gle IP packet.
shows the number of records per type of traffic, shown in As regards CoAP, we considered the following 9
the third column. The type of traffic represents the classifica- features:
tion label considered in the following analyses, performed
independently for each considered application protocols, i.e., • CoAP code: it discriminates between requests and
MQTT and CoAP. responses and provides either the type of request (GET,
Hence, the dataset described in Table 1 has been split in 2 POST, PUT, or DELETE) or response code;
different subdatasets, one for CoAP traffic with 1, 232, 974 • Option type: when one or more options are present, it
packets, and one for MQTT traffic with 624, 301 packets. indicates the type of each option;
Moreover, we used both a binary dataset, with instances/ • Option length: the size (in bytes) of the option;
packets assigned only to the “Normal” or “Attack” class, and • Option descA: it distinguishes between critical or elec-
a multinomial dataset, to classify the four particular types tive options;
of attacks as described in Subsection “Attack Description” • Option descB: it distinguishes between safe or unsafe
and the two types of normal traffic (MQTT and CoAP). options;
In the binary classification we had 1, 428, 244 packets for • Option block size: the transfer block size specified in
the “Normal” class and 429, 031 packets for the general the Block1 and Block2 options in case the CoAP block-
“Attack” class. wise transfer extension is used [40];
• Option observe value: the number used to identify a
The Feature Set given notification within a sequence of resource obser-
vations, when the CoAP Observe extension is used
Since Tshark extracts a total of 78 features for MQTT pack- [41];
ets and 86 features for CoAP packets, we decided to reduce • Option end marker: it indicates the end of the options
them by trying to consider only the most significant ones. field when a payload is present;
This was done by means of a thorough analysis campaign • Payload length: the payload size in bytes.
leveraging both automated statistical analyses and manual

SN Computer Science
SN Computer Science (2024) 5:488 Page 9 of 15 488

Table 4  Hyper-parameters of the ML models we considered

Classifier Hyper-parameters Description Used value

Naïve Bayes Priors This is the prior probability of the classes. If the value is ’none’ the prior probabilities are None
calculated based on the data
Var_smoothing It is the portion of the biggest function variance of all features, and shows how it contrib- 10−9
uted to the others for computation stability
SVM Gamma Kernel coefficient 2
C It is a parameter of regularization; in particular, regularization is inversely proportional to Squared L2
the value of C
Logistic Penalty The value of the considered penalty L2 penalty
Class_weight The value of the class-specific weights 1 (for all classes)
Decision tree Max_depth The maximum depth of the tree 10
Criterion The criterion selected to perform splits at each internal node of the tree Gini impurity

Since more than one option may be present within the same
CoAP message, the Tshark tool provides the feature values Table 5  The DNN model Layer Type Neurons
for the first three options of each packet. This allowed us architecture we used
to consider triples of features for the option type, option Ia Dense 1024
length, option descA, and option descB, leading to a total of Ib Drop out –
17 features for each packet. IIa Dense 512
IIb Drop out –
IIIa Dense 128
Models Settings IIIb Drop out –
IVa Dense 32
For the classification tasks, both binary and multinomial, we IVa Drop out –
analyze different ML models and one Deep Neural Network V Softmax 4
(DNN) architecture. Moreover we have also applied an ante-
hoc explainable model, the already mentioned FHDT.
In Table 4, we present the hyper-parameters used for the in the previous layer. Its output value becomes the input
ML models of the study. The table is structured as follows: for the next neuron layer;
the first column shows the ML classifier, the second col- • Dropout layer, a layer used to set input units to 0 at a
umn lists the name of the hyper-parameter, the third col- chosen rate at each step during training. When the input
umn offers a short description of the hyper-parameter itself is not set to 0, it is increased according to the formula
and its purpose, while the last column indicates the specific 1∕(1 − rate) so that the total number of input is constant.
values that have been employed in the analysis. The hyper-
parameter settings presented in the table were chosen fol-
lowing a grid-search optimization process. These settings The other components of the DNN models can be detailed
were determined using an 80/20 ratio to divide the dataset as follows:
into training and test sets, and they represent the configura-
tions that yielded the best performance in the experiments. • As an Activation function we chose the rectified linear
On the contrary, the provided DNN model architecture is unit activation function via ‘relu activation’ to all neu-
outlined in Table 5. In particular, the first column designates rons in all considered layers;
the hidden layer level, the second column specifies the type • As an Optimizer we applied the Adam optimizer, a par-
of the hidden layer used at that particular level, while the last ticular type of stochastic gradient descent. It is useful
column details the number of neurons within that specific and widely used because it converges in a short time
layer. In this architecture, two different types of layers have and, therefore, makes the network less demanding from
been employed, each serving distinct purposes within the a computational point of view compared with the classic
DNN model: SDG which converges to ‘flat minima’ [42];
• Concerning the Dropout rate of the relative layer, we set
• Dense layer, a fully connected layer, in which every neu- it to 0.20.
ron in the next layer is connected to every other neuron

SN Computer Science
 488 Page 10 of 15 SN Computer Science (2024) 5:488

Table 6  Values of the main Parameter Value in the case of FHDT, we used as a training set 80% of the
hyper-parameters for FHDT whole dataset and 20% for testing.
Split confidence (𝛿) 10−7
Tie threshold (TT) 2.5 Evaluation Metrics
Grace period (GP) 25
Minimum fraction (MF) 0.01 In order to evaluate the performance, we used several spe-
cific metrics to gain a comprehensive understanding. In this
work, the following evaluation metrics, in addition to confu-
The architectural design of the DNN was implemented using sion matrices, have been taken into account:
Python, with a specific focus on TensorFlow7 and Keras8
libraries: • Accuracy, which provides an overall measure of how
often the model correctly classifies items in the dataset
• TensorFlow is an open-source software framework concerning the total number of instances. It is a useful
widely used in the various sub-fields of AI. It is particu- general indicator of the performance of a model.
larly valuable for tasks related to training and inference • Weighted precision, which is calculated by dividing the
in DNNs. number of true positives by the total number of instances
• Keras is a Python package that serves as a user-friendly marked as belonging to a particular class. It provides
interface to the TensorFlow library. It simplifies the pro- insight into the precision of the classifier, taking into
cess of building and working with artificial neural net- account class imbalances.
works, making it easier for developers and researchers to • Weighted recall, which is defined as the number of true
create complex models efficiently. positives divided by the total number of instances that
genuinely belong to a specific class. It assesses the abil-
These libraries provide the tools and resources necessary for ity of the classifier to correctly identify instances of a
developing and training DNNs for a variety of applications, particular class while considering the class distribution.
including the analysis and classification of IoT traffic in the • Weighted F1-score, which is the harmonic mean between
context of the proposed research. In this study, all the neu- precision and recall. It balances the trade-off between
ral network models were trained for a total of 100 epochs. precision and recall, offering a more comprehensive
The training data have been divided into three sets using a measure of the performance of a model, especially in
60/20/20 splitting ratio for the training, validation, and test cases of class imbalance.
sets, respectively. The specific hyper-parameters, including
the dropout rate, number of epochs, and batch size, were The considered performance metrics, in combination with
meticulously selected after a process of grid-search opti- confusion matrices, provide a well-rounded view of how
mization, involving the evaluation of various combinations well each classifier is performing, especially in our case
to determine the most effective configuration for the neural study with unbalanced datasets wherein it is essential to
network models. The chosen settings reflect the combination account for the differences in class distribution.
that yielded the best performance and results for the objec- Beside the aforementioned metrics, when considering
tives of the study. FHDT, we have also regarded the total number of nodes
As concerns the FHDT, only triangular, strong, and uni- in the tree and the number of leaves as metrics useful to
form fuzzy partitions have been used, considering 3, 5, and assess the complexity of the model and so of its interpret-
7 triangles in the [0 − 1] range. Each partition triangle has ability: less nodes and leaves allow for better interpretabil-
been associated with a particular linguistic term, e.g., Low, ity of the corresponding rules. Moreover, the criterion we
Medium, and High in case of three triangles, or Very Low, used to perform inference in the FHDT, given that a testing
Low, Medium, High, and Very High in case of five trian- instance could reach more than one leaf, was the majority
gles. As for the categorical features, we took care to assign voting strategy.
fixed numerical values in the considered range [0 − 1], cor-
responding to the upper vertex of a triangle, while numeric
features were simply normalized accordingly. In Table 6, Results
we detail the main hyper-parameters of the FHDT model we
considered as well as their corresponding considered values, Hereunder we presents the results obtained in the analyses
obtained after a grid search optimization procedure. Also carried out for this research. The evaluation of all models is
carried out separately for both MQTT and CoAP protocols.
7
https://​www.​tenso​rflow.​org/. To enhance the readability of the presented results, this sec-
8
https://​keras.​io/. tion is divided into two subsections. First, we present the

SN Computer Science
SN Computer Science (2024) 5:488 Page 11 of 15 488

Table 7  Results of the binary AI model CoAP MQTT

classification
Accuracy Precision Recall F1-score Accuracy Precision Recall F1-score

Naïve Bayes 0.915 0.966 0.937 0.939 0.917 0.866 0.942 0.905
SVM 0.991 0.990 0.988 0.989 0.975 0.979 0.975 0.977
Logistic reg. 0.969 0.977 0.981 0.979 0.981 0.988 0.984 0.986
Decision tree 0.988 0.989 0.989 0.989 0.979 0.979 0.965 0.977
DNN 0.993 0.992 0.995 0.994 0.989 0.990 0.989 0.990
FHDT-3FS 0.864 0.983 0.981 0.981 0.984 0.997 0.997 0.997
FHDT-5FS 0.885 0.993 0.991 0.991 0.994 0.998 0.998 0.998
FHDT-7FS 0.893 0.994 0.992 0.992 0.995 0.998 0.997 0.998

Table 8  Results of the AI model CoAP MQTT

multinomial classification
Accuracy Precision Recall F1-score Accuracy Precision Recall F1-score

Naïve Bayes 0.948 0.936 0.929 0.948 0.835 0.670 0.921 0.642
SVM 0.971 0.982 0.970 0.983 0.928 0.929 0.948 0.949
Logistic reg. 0.966 0.945 0.956 0.966 0.929 0.918 0.969 0.939
Decision tree 0.991 0.992 0.996 0.997 0.999 0.999 0.979 0.989
DNN 0.999 0.999 1.00 0.999 0.999 0.993 0.994 0.993
FHDT-3FS 0.866 0.981 0.983 0.983 0.983 0.961 0.997 0.961
FHDT-5FS 0.871 0.99 0.989 0.989 0.984 0.971 0.996 0.972
FHDT-7FS 0.895 0.991 0.99 0.99 0.985 0.973 0.995 0.982

Fig. 2  Confusion matrices of both MQTT and CoAP multinomial classification as regards some considered classification models

results of the binary classification (see subsection “Binary Table 7 shows the values of the evaluation metrics con-
Classification”), followed by the results obtained through sidering the CoAP and MQTT datasets separately. The
the multiclassification (see subsection “Classification analysis shows that the classification accuracy is very high,
of Attacks»). This division facilitates a clear and organ- reaching, in the case of the CoAP protocol, values close to
ized presentation of the outcomes for each classification 100% for both some ML models and the DNN, while FHDT
approach. performs slightly worse, considering all three types of fuzzy
sets (FSs), but, anyway, close to 90% of accuracy and with
Binary Classification much more explainable and compact models as described
in Subsection “Explainable Rules”. Furthermore, all the
This subsection discusses the results of the binary classifica- considered metrics achieve close to optimal results, indicat-
tion; hence, in this phase, we have considered the packets ing that the considered algorithms classify the data really
referring to the studied attacks as a single malicious class. well even in cases of unbalanced classes. The best model,
if one considers the CoAP protocol and takes a look at all

SN Computer Science
488 Page 12 of 15 SN Computer Science (2024) 5:488

presents the worst performance, also worse than all versions

desc2A
of FHDT.
VL VH Finally, Fig. 2 shows the confusion matrices for the mul-
tinomial classification of the MQTT and CoAP protocols.
length1 Normal In particular, we have only reported the results obtained for
VL VH Naïve Bayes, Decision Tree, and the DNN model, for the
L M
sake of brevity. From the figure, it can be inferred that for the
desc3A Attack Normal Normal CoAP protocol, the only relevant misclassifications occur
VL VH
when using the Naïve Bayes classifier; in particular, the nor-
mal packets are those most confused with the segmentation
attack packets. In contrast, the decision tree classifier and
Attack Normal
the DNN architecture we considered are the ones with fewer
classification errors: for these algorithms, errors between
Fig. 3  FHDT resulting after training on the CoAP binary dataset predicted data and real data are very rare and mainly involve
when considering a partition of 5 fuzzy sets normal traffic when classified with decision trees.
By the same token, when considering MQTT, no model
produces a perfect diagonal matrix and when using Naïve
the metrics, is the DNN we have taken into account, but one Bayes we obtain the worst outcome whose main criticality
can see that the difference with SVM or Decision Tree is entails the identification of Publish Flood packets as normal
very small. When considering the MQTT protocol alone, ones. In the decision tree model, the main criticality con-
the models that best classify are the DNN and the FHDT, cerns the confusion of packet crafting packets with publish
in this latter case with a slight increase in the performance, flood packets or vice versa. This reciprocal misclassifca-
experienced also in the CoAP case, moving from 3 fuzzy tion of attack packets does not happen in the case of the
sets (3FS) to seven fuzzy sets (7FS). considered DNN model, which has only a few problems in
discriminating packet-crafting packets from publish-flood
Classification of Attacks packets.

This subsection reports the results arising from the multino-

mial classification analysis for both MQTT and CoAP. The
Explainable Rules
specific results are shown in Table 8. CoAP packets, both
In this subsection, we report the graphical representation
normal and malicious, are classified better than MQTT pack-
of some of the obtained FHDTs, limiting the analysis to the
ets. Furthermore, the identification of the different MQTT
binary case because the multinomial trees are exactly the
packets reaches very high figures for all the algorithms we
same but, obviously, with more than two outcomes. We have
considered, except for the Naïve Bayes classifier, which

Fig. 4  FHDT resulting after

training on the MQTT binary
hdrflags2
dataset when considering a
partition of 3 fuzzy sets
Low High

hdrflags3
len1 packet
Type
Low High Low High

Medium
Normal Attack
length2 Attack
hdrflags1
packet
Type Low Medium

Low High
Normal Attack

Normal Attack

SN Computer Science
SN Computer Science (2024) 5:488 Page 13 of 15 488

not reported the traditional decision trees, because they were have a minimum of 33 bytes for a request and 16B for a
poorly interpretable for their complexity as well as for the response and that are usually significantly longer (hun-
lack of associated linguistic terms. dreds or thousands of bytes). Additionally, during the
In Fig. 3, we report the binary FHDT when considering testing phase of the hyper-parameter configuration for the
5 fuzzy sets applied to the features of the CoAP protocol, regarded models, the CoAP protocol returned results with
while, in Fig. 4, we show the binary FHDT applied to the metrics always close to 100% , regardless of the adopted
features of the MQTT dataset when considering 3 fuzzy configuration. Therefore, we have been able to notice that
sets. The trees we have shown are the most compact and packets related to the CoAP protocol can be subject to
interpretable we obtained among the various combinations the so-called “benign overfitting”: this condition occurs
of fuzzy sets, although they all have achieved very similar when a classifier adapts perfectly to noisy training data,
performance results as shown in Tables 7 and 8. Indeed, the while keeping the error between predicted data and real
5-fuzzy-set FHDT for CoAP has 9 total nodes and 6 leaves, data very low [43]. As a corroborating information, in our
while the 3-fuzzy-set version has 13 total nodes and 8 leaves, study, the loss function related to CoAP always tends to 0
and the 7-fuzzy-set version features 29 total nodes and 24 when using the DNN model.
leaves. As regards MQTT, the 3-fuzzy-set FHDT has 12 total Contrary to what occurred for the CoAP protocol, the
nodes and 7 leaves, while the 5-fuzzy-set version has 21 total MQTT protocol is classified with a higher error between
nodes, and 16 leaves and the 7-fuzzy-set version has 29 total predicted data and observed data. However, we have to
nodes and 24 leaves. remember that the MQTT protocol involves both a connec-
In the case of CoAP, a very high value of the desc2A tion, being encapsulated in TCP, and three distinct entities
attribute leads to the normal classification, while a very (the publisher, the broker, and the subscriber), compared
low value requires a test of the length1 feature. Attacks take with the only two involved in CoAP (client and server);
place when length1 is low as well as desc2A is very low, or this makes it a much more complex protocol to identify.
when length1 is very low and desc3A is very low as well as As regards the obtained rules, for CoAP traffic we see
desc2A. that a deciding aspect appears to be whether the packet
In the case of MQTT, the most discriminating feature is contains some options (at least two), and they belong to
hdrflags2 and attacks can take place when it is both low or the ‘critical’ class (that corresponds to the features descA
high. In the first case len1 has to be high or medium, but with value very low), that means that the given option
with a high value associated to hdrflags1_packet_Type . In needs to be understood and processed by the message des-
the second case when hdrflags3_packet_Type is high or it is tination. This requires at least a processing overhead at
low, but with a medium value for length2. the receiver side and may be used for exploiting possible
Two examples of the extracted rules, for both CoAP and software vulnerabilities in case they are present.
MQTT, are the following: On the other hand, for detecting MQTT attacks, we have
that it is important whether the IP packet contains different
R1: IF desc2A is VERY LOW AND length1
MQTT messages; this is evident because of the presence
is LOW THEN CoAP Attack of features marked with 1, 2, and 3, related to three dif-
ferent MQTT messages within a single TCP/IP packet.
R2: IF hdrflags2 is LOW AND length1 This conditions may be more probable in case of flooding/
is HIGH THEN MQTT Attack DoS attacks. In addition, also the type of MQTT packet
is relevant, since some methods like UNSUBSCRIBE or
DISCONNECT, corresponding to the ‘packet_type’ fea-
ture with high values, are mainly used in some attacks.
As for the threats to the validity of our study, we dis-
Discussion and Threats to the Validity cuss construct, internal, and external validity threats in
the following.
When considering both the binary and multinomial clas- Regarding construct validity threats, they usually entail
sification, we may see that the CoAP protocol is classified how well a set of indicators represents or reflects a con-
better than the MQTT protocol, except when using the cept that is not directly measurable. In our study, we have
FHDT. However, this could arise from the extreme light- focused on the main features of the packets of both MQTT
ness and simplicity implicit in CoAP implementations. and CoAP created during various simulation sessions with
As a matter of fact, such a protocol is routinely used on different random seeds and we have accurately removed
battery-powered devices and in case of limited CPU and statistically non-significant features.
RAM. Moreover, it is fair to say that a CoAP packet can
be only 4B long, as opposed to a HTTP messages that can

SN Computer Science
 488 Page 14 of 15 SN Computer Science (2024) 5:488

When focusing on the internal validity, we exclude any Funding Open access funding provided by Consiglio Nazionale Delle
labeling issues because the traffic was synthetically created Ricerche (CNR) within the CRUI-CARE Agreement. The authors
declare not to have received any funding for carrying out this research
through IoT-Flock in a controlled environment.
Concerning the threats to external validity, which affect Data Availability the obtained dataset will be available upon request
the generalization of the discussed outcomes, we still have
to test whether training on the synthetic traffic generated Declarations
through IoT-Flock and testing on real-world data could
Conflict of interest The authors declare no competing nor financial
lead to similar very good results. However, the rules conflicts of interest.
extracted from the FHDT confirm a real-world significant
meaning in the identification of the CoAP and MQTT Research Involving Human and /or Animals ’Not Applicable’
attacks and thus we can envisage its useful application
Informed Consent ’Not Applicable’
also to a real-world scenario.
Open Access This article is licensed under a Creative Commons Attri-
bution 4.0 International License, which permits use, sharing, adapta-
Conclusions and Future work tion, distribution and reproduction in any medium or format, as long
as you give appropriate credit to the original author(s) and the source,
In this paper, we have applied a complete and explainable provide a link to the Creative Commons licence, and indicate if changes
were made. The images or other third party material in this article are
anomaly detection analysis, performed via machine and included in the article’s Creative Commons licence, unless indicated
deep learning techniques, to the synthetic traffic produced otherwise in a credit line to the material. If material is not included in
trough IoT-Flock when considering a smart health sce- the article’s Creative Commons licence and your intended use is not
nario. The research has encompassed both a binary and a permitted by statutory regulation or exceeds the permitted use, you will
need to obtain permission directly from the copyright holder. To view a
multinomial classification; moreover, we have considered copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
both MQTT and CoAP messages, i.e., the most used appli-
cation protocols in the IoT scenario, and both normal and
malicious traffic, trying to identify four different attacks References
by using application-layer packet features. Furthermore we
have also applied an explainable machine learning tech- 1. Hossain E, Khan I, Un-Noor F, Sikander SS, Sunny MSH.
nique in order to extract potential classification rules to be Application of big data and machine learning in smart grid,
applied in a real-world scenario. The results we obtained and associated security concerns: a review. IEEE Access.
2019;7:13960–88. https://​doi.​org/​10.​1109/​ACCESS.​2019.​28948​
have demonstrated the full feasibility in using synthetic 19.
traffic produced by IoT-Flock as a base for the anomaly 2. Ajagbe SA, Awotunde JB, Florez H. Ensuring intrusion detec-
detection of IoT medical traffic, providing also meaningful tion for iot services through an improved CNN. SN Comput Sci.
classification rules. 2023;5(1):49. https://​doi.​org/​10.​1007/​s42979-​023-​02448-y.
3. Ghazanfar S, Hussain F, Rehman AU, Fayyaz UU, Shahzad F,
As for future research directions, we are committed Shah GA. IoT-Flock: an open-source framework for IoT traf-
to train the models on the synthetic traffic coming from fic generation. In: 2020 International Conference on Emerging
IoT-Flock and perform the testing phase on real labeled Trends in Smart Technologies (ICETST), 2020;1–6. https://​doi.​
medical IoT traffic. Moreover, we intend also to perform a org/​10.​1109/​ICETS​T49965.​2020.​90807​32.
4. Aversano L, Bernardi M, Cimitile M, Montano D, Pecori R,
per-flow analysis considering features related to a certain Veltri L. anomaly detection of medical IoT traffic using machine
traffic flow rather than to the single packets, and to enrich learning. In: Proceedings of the 12th International Confer-
the overall study with a feature selection analysis, in order ence on Data Science, Technology and Applications-DATA,
to verify if, even with a reduction of the considered fea- 2023:173–182. SciTePress
5. OASIS Standard: MQTT Version 5.0. OASIS Standard. Version
tures, we could output similar outcomes. 5. (2019). https://​docs.​oasis-​open.​org/​mqtt/​mqtt/​v5.0/​os/​mqtt-​
v5.0-​os.​html. Accessed Jan 2023
Acknowledgements Riccardo Pecori is a member of the INdAM 6. Internet Engineering Task Force (IETF): The Constrained
GNCS research group. The authors would like to thank Mr. Antonio Application Protocol (CoAP). Internet Engineering Task Force
Enrico Buonocore, for carefully proofreading the whole manuscript, (IETF). Updated by: RFC 7959, 8613, 8974, 9175. (2019).
and Guido Iannone for creating and testing the IoT-Flock scenarios. https://​www.​r fc-​editor.​org/​r fc/​r fc72​52. Accessed Jan 2023
7. CVE-2016-10523, Common Enumeration of Vulnerabili-
Author Contributions Debora Montano collected the data and per- ties. https://www.cve.org/CVERecord?id=CVE-2016-10523.
formed the analyses using ML and DL techniques. She also contributed Accessed 30 Jan 2023.
in writing the paper. Riccardo Pecori performed the analyses using 8. CVE-2019-12101, Common Enumeration of Vulnerabili-
FHDT and contributed in writing the paper. Luca Veltri helped in ana- ties. https://www.cve.org/CVERecord?id=CVE-2019-12101.
lyzing the features and provide an interpretation of the extracted rules, Accessed 30 Jan 2023.
contributing also in writing some parts of the paper. Lerina Aversano,
Mario Luca Bernardi, and Marta Cimitile supervised the work.

SN Computer Science
SN Computer Science (2024) 5:488 Page 15 of 15 488

9. CVE-2019-9004, Common Enumeration of Vulnerabili- 30. UNIBS: Data Sharing. (2009). http://​netweb.​ing.​unibs.​it/​~ntw/​
ties. https://www.cve.org/CVERecord?id=CVE-2019-9004. tools/​traces/​index.​php. Accessed Jan 2023
Accessed 30 Jan 2023. 31. Moustafa N, Koroniotis N, Keshk M, Zomaya AY, Tari Z.
10. Aversano L, Bernardi ML, Cimitile M, Pecori R. A systematic Explainable intrusion detection for cyber defences in the internet
review on Deep Learning approaches for IoT security. Comput of things: opportunities and solutions. IEEE Commun Surv Tutor.
Sci Rev. 2021;40: 100389. 2023;3:1775–807. https://​doi.​org/​10.​1109/​COMST.​2023.​32804​
11. Rish, I. An empirical study of the naive bayes classifier. In: 65.
IJCAI 2001 Workshop on Empirical Methods in Artificial Intel- 32. Khan IA, Moustafa N, Pi D, Sallam KM, Zomaya AY, Li B. A new
ligence, 2001;3:41–46. explainable deep learning framework for cyber threat discovery in
12. Suthaharan, S. Machine learning models and algorithms for industrial IoT networks. IEEE Internet Things J. 2022;13:11604–
big data classification. In: Integrated Series in Information 13. https://​doi.​org/​10.​1109/​JIOT.​2021.​31301​56.
Systems, Springer, 2016;36:1–12. https://​d oi.​o rg/​1 0.​1 007/​ 33. Nguyen QP, Lim KW, Divakaran DM, Low KH, Chan MC. GEE:
978-1-​4899-​7641-3 A gradient-based explainable variational autoencoder for network
13. Wright, RE. Logistic regression. (1995). anomaly detection. In: 2019 IEEE Conference on Communica-
14. Magee JF. Decision Trees for Decision Making. MA, USA: Har- tions and Network Security (CNS), 2019:91–99. https://​doi.​org/​
vard Business Review Brighton; 1964. 10.​1109/​CNS.​2019.​88028​33.
15. Aversano L, Bernardi ML, Cimitile M, Pecori R, Veltri L. effec- 34. Fazzolari M, Ducange P, Marcelloni F. An explainable intrusion
tive anomaly detection using deep learning in IoT systems. detection system for IoT networks. In: 2023 IEEE International
Wirel Commun Mobile Comput. 2021. https://​doi.​org/​10.​1155/​ Conference on Fuzzy Systems (FUZZ), 2023:1–6. https://​doi.​org/​
2021/​90543​36. 10.​1109/​FUZZ5​2849.​2023.​10309​785.
16. Pecori R, Tayebi A, Vannucci A, Veltri L. IoT Attack detection 35. Khelifati A, Khayati M, Cudré-Mauroux P, Hänni A, Liu Q,
with deep learning analysis. In: 2020 International Joint Confer- Hauswirth M. VADETIS: an explainable evaluator for anomaly
ence on Neural Networks (IJCNN), 2020:1–8. https://​doi.​org/​ detection techniques. In: 2021 IEEE 37th International Confer-
10.​1109/​IJCNN​48605.​2020.​92071​71. ence on Data Engineering (ICDE), 2021;2661–2664. https://​doi.​
17. Aversano L, Bernardi ML, Cimitile M, Pecori R. Anomaly org/​10.​1109/​ICDE5​1399.​2021.​00298.
detection of actual IoT traffic flows through deep learning. In: 36. Aziz S, Faiz MT, Adeniyi AM, Loo K-H, Hasan KN, Xu L, Irshad
2021 20th IEEE International Conference on Machine Learning M. Anomaly detection in the internet of vehicular networks using
and Applications (ICMLA), 2021:1736–1741. https://​doi.​org/​ explainable neural networks (xNN). Mathematics. 2022. https://​
10.​1109/​ICMLA​52953.​2021.​00275. doi.​org/​10.​3390/​math1​00812​67.
18. Ducange P, Marcelloni F, Pecori R. Fuzzy Hoeffding decision 37. Ha DT, Hoang NX, Hoang NV, Du NH, Huong TT, Tran KP.
tree for data stream classification. Int J Comput Intell Syst. Explainable anomaly detection for industrial control system
2021;14:946–64. cybersecurity. In: 10th IFAC Conference on Manufacturing Mod-
19. Gacto MJ, Alcalá R, Herrera F. Interpretability of linguistic fuzzy elling, Management and Control MIM 2022, IFAC-PapersOnLine
rule-based systems: an overview of interpretability measures. Inf 2022;(10):1183–1188. . https://​doi.​org/​10.​1016/j.​ifacol.​2022.​09.​
Sci. 2011;181(20):4340–60. 550
20. Pundir S, Wazid M, Singh DP, Das AK, Rodrigues JJ, Park Y. 38. Koroniotis N, Moustafa N, Sitnikova E, Turnbull B. Towards the
Intrusion detection protocols in wireless sensor networks inte- development of realistic botnet dataset in the internet of things for
grated to Internet of Things deployment: survey and future chal- network forensic analytics: Bot-IoT dataset. Future Gener Comput
lenges. IEEE Access. 2019;8:3343–63. Syst. 2019;100:779–96.
21. Rathore S, Park JH. Semi-supervised learning based distrib- 39. Hussain F, Abbas SG, Shah GA, Pires IM, Fayyaz UU, Shahzad
uted attack detection framework for IoT. Appl Soft Comput. F, Garcia NM, Zdravevski E. a framework for malicious traffic
2018;72:79–89. detection in IoT healthcare environment. Sensors. 2021;9:3025.
22. Rughoobur P, Nagowah L. A lightweight replay attack detection https://​doi.​org/​10.​3390/​s2109​3025.
framework for battery depended IoT devices designed for health- 40. Bormann C. Block-wise transfers in the constrained applica-
care. In: 2017 International Conference on Infocom Technologies tion protocol (CoAP). Internet Engineering Task Force (IETF).
and Unmanned Systems (Trends and Future Directions)(ICTUS), Internet Engineering Task Force (IETF). Updated by: RFC 8323.
2017:811–817. IEEE. (2016). https://​www.​r fc-​editor.​org/​r fc/​r fc79​59. Accessed Jan
23. Alrashdi I, Alqazzaz A, Alharthi R, Aloufi E, Zohdy MA, Ming 2023.
H. FBAD: fog-based attack detection for IoT healthcare in smart 41. Hartke K. Observing resources in the constrained application pro-
cities. In: 2019 IEEE 10th Annual Ubiquitous Computing, Elec- tocol (CoAP). Internet Engineering Task Force (IETF). Internet
tronics & Mobile Communication Conference (UEMCON), Engineering Task Force (IETF). Updated by: RFC 8323. (2015).
2019:0515–0522. IEEE. https://​www.​rfc-​editor.​org/​rfc/​rfc76​41. Accessed Jan 2023.
24. DARPA Intrusion Detection Evaluation Dataset. (1998). https://​ 42. Kingma DP, Ba J. Adam: A method for stochastic optimization.
www.l​ l.m​ it.e​ du/r-d​ /d​ atase​ ts/1​ 998-d​ arpa-i​ ntrus​ ion-d​ etect​ ion-e​ valu​ arXiv preprint. (2014). arXiv:​1412.​6980.
ation-​datas​et. Accessed Jan 2023. 43. Shamir O. the implicit bias of benign overfitting. In: Loh, P.-L.,
25. KDD Cup 1999 Data. (1998). (http://​kdd.​ics.​uci.​edu/​datab​ases/​ Raginsky, M. (eds.) Proceedings of Thirty Fifth Conference on
kddcu​p99/​kddcu​p99.​html. Accessed Jan 2023. Learning Theory. Proceedings of Machine Learning Research,
26. NSL-KDD Dataset. (1999). htps://​www.​unb.​ca/​cic/​datas​ets/​nsl.​ vol. 178, pp. 448–478. PMLR, USA 2022.
html. Accessed Jan 2023.
27. (2023). https://​defcon.​org/​html/​links/​dc-​ctf.​html. Accessed Jan Publisher's Note Springer Nature remains neutral with regard to
2023. jurisdictional claims in published maps and institutional affiliations.
28. LBNL/ICSI Enterprise Tracing Project. (2023). (http://​www.​icir.​
org/​enter​prise-​traci​ng/. Accessed Jan 2023.
29. Center for Applied Internet Data Analysis (CAIDA). (2023).
https://​catal​og.​caida.​org/. Accessed Jan 2023

SN Computer Science