Received 9 December 2022, accepted 22 December 2022, date of publication 26 December 2022,

date of current version 29 December 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3232131

Node-Based Horizontal Pod Autoscaler in

KubeEdge-Based Edge Computing Infrastructure
LE HOANG PHUC 1 , MAJID KUNDROO 1 , DAE-HEON PARK 2, SEHAN KIM 2,

AND TAEHONG KIM 1 , (Senior Member, IEEE)

1 School of Information and Communication Engineering, Chungbuk National University, Cheongju 28644, South Korea
2 Electronics and Telecommunications Research Institute, Daejeon 34129, South Korea
Corresponding author: Taehong Kim ([email protected])
This work was partially supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded
by the Korea government (MSIT) (No. 2018-0-00387, Development of ICT based Intelligent Smart Welfare Housing System for the
Prevention and Control of Livestock Disease, 80%) and Basic Science Research Program through the National Research Foundation of
Korea (NRF) funded by the Ministry of Education (No. NRF-2022R1I1A3072355, 20%).

ABSTRACT KubeEdge (KE) is a container orchestration platform for deploying and managing containerized
IoT applications in an edge computing environment based on Kubernetes. It is intended to be hosted at
the edge and provides seamless cloud-edge coordination as well as an offline mode that allows the edge
to function independently of the cloud. However, there are unreliable communication links between edge
nodes in edge computing environments, implying that load balancing in an edge computing environment
is not guaranteed while using KE. Furthermore, KE lacks Horizontal Pod Autoscaling (HPA), implying
that KE cannot dynamically deploy new resources to efficiently handle increasing requests. Both of the
aforementioned issues have a significant impact on the performance of the KE-based edge computing system,
particularly when traffic volumes vary over time and geographical location. In this study, a node-based
horizontal pod autoscaler (NHPA) is proposed to provide dynamical adjustment for the number of pods of
individual nodes independently from each other in an edge computing environment where the traffic volume
fluctuates over time and location, and the communication links between edge nodes are not stable. The
proposed NHPA can dynamically adjust the number of pods depending on the incoming traffic at each node,
which will improve the overall performance of the KubeEdge-based edge computing environment. In the
KubeEdge-based edge computing environment, the experimental findings reveal that NHPA outperforms
KE in terms of throughput and response time by a factor of about 3 and 25, respectively.

INDEX TERMS Kubernetes, KubeEdge, horizontal pod autoscaler, dynamic resource provisioning, edge
computing, IoT.

I. INTRODUCTION latency, sensitivity, resource restriction of IoT devices, and

Over the past few years, the Internet of Things (IoT) has network bandwidth [4], [5].
emerged as an important technology for addressing numerous Cloud computing has been widely considered in vari-
social challenges such as agriculture, healthcare, and trans- ous industrial fields, and its applicability has attracted the
portation [1], [2]. IoT enables the communication between attention of researchers [6], [7], [8]. However, the archi-
billions of smart devices [3] for collecting and sharing sensi- tecture of cloud computing has fundamental limitations
tive data. However, deploying and managing IoT applications for deploying and managing IoT applications. The primary
are challenging owing to application requirements such as limitation is that cloud computing resources are primarily
located far away from IoT devices. In other words, high
round-trip latency is added to the IoT application response
The associate editor coordinating the review of this manuscript and time. Moreover, because of the massive communication traf-
approving it for publication was Wei-Wen Hu . fic between devices and the cloud, which increases bandwidth

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 10, 2022 134417
 L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

consumption; hence, communication bottlenecks cannot be communication links. Here the main idea is to dynamically
avoided in cloud computing. auto-scale the number of pods based on resource metrics
To overcome the limitations of cloud computing, edge at each node independently to handle the incoming traffic
computing was introduced to move the computational requirements at the node level. The primary concept of NHPA
resources closer to the data sources (devices) and minimize is to apply HPA to each node to ensure that each node
the propagation distance of packets from end devices to can handle incoming requests locally regardless of unstable
the computational resources. By allocating computational communication links to the other nodes.
resources, IoT-based edge computing applications can main- The remainder of this paper is organized as follows.
tain low latency and low bandwidth usage [9], [10] because Section II presents related work, and Section III describes the
incoming requests do not need to be transferred to the cloud. fundamental background of KubeEdge. Section IV describes
In addition, containerization, represented by Docker, is a the proposed NHPA and how it solves the problem of
lightweight virtualization technology that packages applica- KubeEdge in an edge computing environment. The perfor-
tions and dependent libraries so that they can be run in an mance evaluations of NHPA compared with KubeEdge in
independent environment. It has been widely used for deploy- various traffic scenarios are reported in Section V. Finally,
ing and managing applications in both cloud and edge com- we conclude the paper in Section VI.
puting environments due to its portable, flexible, and easy
deployment features [11]. However, deploying and managing
containerized applications requires a container orchestration II. RELATED WORKS
platform, such as Kubernetes and KubeEdge (KE), to max- Edge computing has been considered a new computing
imize application performance under various circumstances. paradigm that moves computing and storage capabilities to
Kubernetes [12], [13] is a well-known container orchestration the network’s edge, i.e., nearer to the end devices. Since
platform that offers several features, such as service man- its advent, edge computing has received significant attention
agement and resource provisioning of edge nodes, as well as from researchers. More specifically, edge computing has been
assuring service availability. A pod is the smallest deployable applied to improve the performance of several technologies,
unit in Kubernetes that contains one or more containers. such as voice/vision recognition [16] and 5G [17]. Moreover,
In Kubernetes, IoT applications are deployed and run on several studies have focused on resolving obstacles in deploy-
worker nodes in the form of pods [14]. Furthermore, hori- ing applications in an edge computing environment. These
zontal pod autoscaling (HPA) provided by Kubernetes can studies have demonstrated that resource provisioning and
optimize the application performance and resource cost by dynamic resource allocation play a vital role in the successful
dynamically adjusting the computational resources according deployment of the application.
to the actual demand. Nguyen et al. [18] proposed ElasticFog, which re-allocates
KE provides better resource orchestration in an edge com- computational resources between edge nodes based on the
puting environment because it enables containerized appli- network traffic accessing them. To be precise, ElasticFog
cations to operate properly even when the communications distributes more resources to nodes that have more incoming
between the cloud node and edge nodes are disrupted [15]. traffic. Similarly, NetMARKS [19] enables resource provi-
Furthermore, the applications can be deployed to the desired sioning with network traffic awareness to improve the quality
nodes based on the intentions of administrative personnel. of IoT services. It improves the efficiency of the Kuber-
Load balancing is an important aspect to be considered for netes scheduling mechanism by extending the Kube sched-
providing seamless services in any computing environment. uler and applying the dynamic network metrics collected with
In the KE-based edge computing environment, the EdgeMesh the Istio Service Mesh. Santos et al. [20], [21] proposed a
agent distributes incoming requests to all pods in the cluster network-aware scheduling mechanism (NAS), which aimed
for load balancing at each node. But in an edge computing to improve the efficiency of the scheduling mechanism of
environment, there are no guaranteed communication links Kubernetes by extending the Kube-scheduler to reduce the
between edge nodes, which is necessary for load balancing application response time and avoid bandwidth usage viola-
using EdgeMesh. Load balancing between edge nodes fails tion. Some studies suggest using a leader to coordinate tasks
due to a lack of stable communication links between geo- among pods to achieve good performance. Because of its
graphically dispersed edge nodes. Furthermore, because KE inherent design, the leader often carries heavy workloads.
lacks an HPA functionality, edge nodes cannot allocate more Having the leaders of multiple applications concentrated in
resources dynamically to handle heavy traffic demands. As a a particular node in a Kubernetes cluster may cause the
result of unstable communications links and the lack of HPA, system to bottleneck. So some studies like [22] provide leader
KE is constrained to providing high scalability in an edge selection algorithms that solve the bottleneck problem by
computing environment. uniformly distributing the leaders among nodes in the cluster.
To solve the problems mentioned above, this study pro- Although the experimental results of the study mentioned
poses a node-based horizontal pod autoscaler (NHPA) that above show that the developed mechanisms are highly effec-
allocates new computational resources independently for tive, they cannot be thoroughly applied in such an edge com-
each edge node to eliminate the effect of the absence of puting environment because they do not consider unstable or

134418 VOLUME 10, 2022

L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

unestablished communication links resulting from the geo- and its working, as well as the issues that Horizontal Pod
graphical distribution of the edge nodes. autoscaling faces, particularly in Edge Computing contexts.
Phuc et al. [4] proposed a traffic-aware horizontal pod
autoscaler (THPA) that provides proper resource autoscal- A. KubeEdge
ing according to the actual network traffic accessing edge KE is an open-source orchestration platform built on Kuber-
nodes in an edge computing environment. In other words, netes [15] for extending native containerized applications to
THPA collects network traffic information and calculates edge computing. One of the primary goals of KE is to pro-
the number of pods that need to be adjusted on each edge vide a variety of functions in the network infrastructure that
node to ensure that IoT service performance can be improved connects the cloud and the edges, such as service deployment
compared with Kubernetes default autoscaling (KHPA). Sim- and metadata synchronization.
ilarly, Libra [23] provides a hybrid resource autoscaling Like Kubernetes, KE forms a cluster with a master node
mechanism by leveraging both the vertical and horizon- in the cloud and edge nodes at the edges. The main com-
tal autoscaling mechanisms of Kubernetes. Libra aims to ponents of KE are Edged, EdgeHub, CloudHub, EdgeCon-
improve autoscaling by calculating and adjusting the appro- troller, EventBus, DeviceTwin, MetaManager, ServiceBus,
priate resource limit for every pod in the cluster. and DeviceController. EdgeD is an agent on edge nodes that
In addition to conventional approaches, several studies manages the pod life cycle. It helps users to deploy container-
have applied machine learning (ML) to improve the effi- ized workloads or applications at the edge node. EdgeHub
ciency of existing autoscaling mechanisms. One such mech- is a web socket client responsible for updating cloud-side
anism that uses ML is HPA+ [24]. HPA+ was developed resources to the edge and reporting status changes from the
to provide a proactive autoscaling mechanism to improve edge-side host and device to the cloud. In contrast, CloudHub
the quality of IoT services by employing a multi-forecast is a web socket server responsible for updating cloud-side
ML model to make scaling decisions. Tenfei Hu et al. [25] changes, caching, and sending update messages to EdgeHub.
applied a forecast model to precisely predict the num- In KE, the data can be targeted to a specific edge node
ber of pods in the cluster; thus, the application perfor- by the EdgeController, an extended Kubernetes controller.
mance can be optimized based on the load fluctuation. In contrast to Kubernetes, KE supports components’ pub/sub
Dong et al. [26] proposes JCETD (Joint Cloud-Edge Task capabilities by implementing the EventBus, an MQTT
Deployment), a reinforcement learning and pruning algo- client/server model. DeviceTwin is responsible for storing
rithm based resource management and task deployment device statuses and updating their statuses on the cloud
strategy. side. Devices metadata can be stored in MetaManager in a
In addition, the KE platform has also been considered lightweight database (SQLite) that enables applications eas-
in certain studies to improve the application performance ily retrieve necessary information. Finally, DeviceController
in an edge computing environment. For instance, in [16], manages devices to ensure that the device metadata/status
KE was applied to improve the efficiency of spectrum intelli- data can be synchronized between the edge and cloud by
gent applications by solving terminal storage and computing extending the Kubernetes controller. In contrast, ServiceBus
resource problems. The researchers proposed that the algo- offers a reachable HTTP connection from cloud components
rithm can be trained on the cloud side based on KE spectrum to the servers running at the edge. In KE, the network con-
sensing architecture while interference occurs at edge nodes. nectivity is provided by EdgeMesh [28], which ensures the
In [27], the authors proposed a novel edgemesh framework continuity of applications even when the connection between
developed on the KE platform to fit dynamic latency-sensitive the edge and cloud is disrupted. The EdgeMesh component
multiple service circumstances. comprises two main sub-components: the EdgeMesh Server
The studies mentioned above have made numerous con- and EdgeMesh Agent. While EdgeMesh Server is responsible
tributions to solving the problem of the existing resource for establishing a connection with the EdgeMesh Agent to
allocation mechanisms, such as KHPA and Kube scheduler. provide relay and hole punching capability, EdgeMesh Agent
However, they continue to ignore the unstable or estab- ensures that incoming traffic can reach the backend pods
lished communication links between nodes, which is one running on the edges.
of the essential characteristics of an edge computing envi- Load balancing allows you to spread the workload evenly
ronment. Therefore, in this study, we propose the NHPA among available resources. Its goal is to provide contin-
mechanism that independently adjusts the number of pods uous service if any component fails by provisioning and
on nodes to maximize the application performance regarding de-provisioning application instances to utilize resources
response time and throughput in a KE-based edge computing properly. Furthermore, load balancing tries to reduce task
environment. response time and optimize resource usage, which improves
system performance at a reduced cost. Load balancing also
III. PRELIMINARIES offers scalability and flexibility for applications whose size
This section introduces KubeEdge principles, the primary may rise in the future and demand additional resources,
components of KubeEdge, and how KubeEdge works. as well as prioritize jobs that require immediate execution
We also address the Horizontal Pod autoscaling algorithm above other processes.

VOLUME 10, 2022 134419

 L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

Load balancing is achieved in KubeEdge-based edge com- Algorithm 1 HPA Algorithm [4]
puting systems through the EdgeMesh agent, which acts as pods : list of application pods in cluster.
a network load balancer by forwarding incoming requests curPods : current number of application pods.
using the random, round-robin, and session persistence meth- dPods : desired number of application pods.
ods. As a result, device requests can be efficiently handled curMetVal : current metric value.
by distributing them to local and remote backend pods on dMetVal : desired metric value.
the cluster’s edge nodes where the requests are processed. HPA_Sync_Period : HPA sync period.
However, in edge computing environments, this approach has ——————————————————————-
several disadvantages, including: 1: while true do
1) In Edge computing environments, the pods of an appli- 2: curPods = getCurPods()
cation can be geographically dispersed, which can 3: curMetVal = getCurMetricValue(app)
incur significant delays when the requests are for- 4: dMetVal = getDesiredMetricValue(app)
warded from one node to another. Also, in geograph- 5: ratio = curMetricVal ÷ dMetricVal
ically distributed systems, the traffic load varies by 6: dPods = ceil[ratio × curPods]
location and time [29], resulting in a demand imbalance 7: if dPods ! = curPods then
across nodes. 8: setDesiredPods(app, dPods)
2) Forwarding requests to nodes without knowing the sta- 9: end if
tus of their resources can result in non-optimal results. 10: time.sleep(HPA_Sync_Period)
Some nodes are too busy to manage a high traffic 11: end while
volume, while others remain idle [20], resulting in
degraded overall performance.
3) In an environment such as edge computing, where number of nodes in the cluster. Lines 2-6 will take one unit of
communication links between nodes might be unstable time(constant) and the function setDesiredPods(app, dpods)
or even absent, any attempt to distribute the incoming in line 8 sets the number of pods by allocating new pods for
requests between nodes can reduce the overall perfor- each edge node (upscaling) or terminating pods for each node
mance of the system. (downscaling), it needs to send commands to n nodes. There-
fore, we can say that the time complexity of Algorithm 1
B. HORIZONTAL POD AUTOSCALER is O(n). Although HPA is an essential feature for dynamic
The Horizontal Pod Autoscaler (HPA) is a representative resource orchestration, it is not a default option of KubeEdge.
technology for providing high scalability and availability in Thus, it cannot handle incoming traffic dynamically if the
Kubernetes that regularly alters the number of computational amount of traffic is high.
resources (pods) in the cluster. It ensures that the cluster’s
state always meets the desired state according to the specified IV. NODE-BASED HORIZONTAL POD AUTOSCALER
metric values [30]. Furthermore, the HPA is configurable In this section, we will go through the KubeEdge-based edge
with metrics such as CPU, memory, and custom metrics to computing architecture and highlight some of the current
support various usage circumstances. issues that KHPA is dealing with. Then, we discuss NHPA,
The workflow of HPA is illustrated in Algorithm 1 [4]. its working, and the detailed algorithm of NHPA.
In each period denoted by HPA_Sync_Period, HPA collects
the number of current pods in the cluster (curPods), the A. KubeEdge-BASED EDGE COMPUTING ARCHITECTURE
current metric value (curMetricVal), and the desired metric Figure 1 (a) illustrates the architecture of a KE-based edge
value (dMetricVal). Subsequently, it calculates the desired computing system. The application is deployed on edge nodes
number of pods in the cluster (dPods) based on the ratio in the form of pods and can be accessed by IoT devices.
(in line 5). If the desired number of pods in the cluster differs Notably, the EdgeMesh agent is enabled and configured with
from the current number of pods in the cluster, the number of a round-robin-based traffic forwarding mechanism on each
application pods in the cluster is adjusted by the HPA (line 8 node to ensure that the traffic loads are evenly distributed
in Algorithm 1). among application pods in the cluster. In other words, local
The upscaling process is invoked if the desired number and remote pods handle incoming traffic at each node. Thus,
of pods is greater than the current number of pods in the none of the nodes becomes overloaded solely because traffic
cluster, whereas the downscaling process is invoked in the distribution of the EdgeMesh agent.
opposite case. For instance, given that the CPU usage metric However, in an edge computing environment, edge
is enabled, assume that all pods’ current average CPU usage nodes are geographically dispersed. Thus, their communi-
is 100 m (where m stands for millicore) and the desired CPU cation links might not be stable or even existent. Because
metric value is 50 m. HPA doubles the number of pods in EdgeMesh’s load balancing works on connected worker
the cluster. In contrast, when the current CPU metric value nodes, it cannot provide round-robin-based traffic forwarding
is 50 m and the desired CPU metric value is 100 m, HPA in an edge computing environment where the links between
halves the number of pods in the cluster. Let n denote the nodes are unstable or absent, as shown in Figure 1 (a).

134420 VOLUME 10, 2022

L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

FIGURE 1. (a) KubeEdge-based edge computing architecture, and (b) Node-based HPA in
KubeEdge-based edge computing architecture.

VOLUME 10, 2022 134421

 L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

For example, although the volume of incoming traffic at Algorithm 2 NHPA Algorithm
Node 1 is high as it deals simultaneously with seven devices, Nodes : list of edge nodes in cluster.
the EdgeMesh agent on Node 1 cannot spread incoming pods : list of application pods in cluster.
traffic because of unstable or absent links. Thus, traffic at curPods : current number of application pods of each
Node 1 can only be handled by local pods, which might node.
lead to an overload at Node 1 when handling heavy traffic dPods : desired number of application pods of each
concentrated from seven devices. node.
In addition, the number of pods on edge nodes is fixed curMetVal : current metric value of each node.
because KE does not necessarily include an HPA feature. dMetVal : desired metric value of each node.
However, to adapt to the actual demand from IoT devices HPA_Sync_Period : HPA sync period.
that fluctuates over time and locations in an edge computing ——————————————————————-
environment, the number of pods on edge nodes can be 1: while true do
adjusted based on their workload in real-time and locations. 2: for node ∈ Nodes do
In particular, as shown in Figure 1 (a), more application pods 3: curPods = getCurPods(node)
should be allocated to Node 1 to adapt to the demand of 4: curMetVal = getCurMetricValue(app, node)
seven IoT devices to reduce resource overload and improve 5: dMetVal = getDesiredMetricValue(app, node)
application performance. 6: ratio = curMetricVal ÷ dMetricVal
By carefully considering the abovementioned problems 7: dPods = ceil[ratio × curPods]
in an edge computing environment, we propose an NHPA 8: if dPods ! = curPods then
that independently adjusts the number of pods on edge 9: setDesiredPods(app, dPods, node)
nodes to maximize the application performance. As shown in 10: end if
Figure 1 (b), because the communication links between edge 11: end for
nodes are not established, the NHPA adjusts the number of 12: time.sleep(HPA_Sync_Period)
pods at each edge node independently based on resource 13: end while
usage. Thus, the number of pods on Node 1 becomes nine,
while other nodes retain the same number of pods as earlier.
Therefore, the number of incoming requests at Node 1 can metric value (curMetVal), the desired metric value (dMetVal)
be handled effectively and locally, even without cooperation and the desired number of pods in the cluster (dPods) are
with other nodes. measured and calculated from the perspective of an individual
node in Algorithm 2. Subsequently, if the desired number of
B. NODE-BASED HORIZONTAL POD AUTOSCALER pods differs from the current number of pods, the desired
This subsection describes the details of the proposed NHPA. number of pods in the cluster will be updated, and scaling
Generally, KE does not provide HPA, although it is inher- will occur. For instance, in Figure 1 (b), NHPA can adjust
ited from Kubernetes. Thus, it cannot provide scalability the number of pods at Node 1 to nine while retaining one at
for workload fluctuations. For example, in Figure 1 (a), Nodes 2 and 3 to handle the increasing traffic. Consequently,
we assume that a KE system comprises three edge nodes the nine pods at Node 1 can efficiently handle incoming
with one application pod, which are not mutually connected traffic from seven devices without the need to distribute it to
because of the geographical distribution. Although the num- other nodes. Let n denote the number of nodes in the cluster.
ber of devices accessing Node 1 increases to seven, the The loop started in line 2 of Algorithm 2 will be executed
number of pods on Node 1 remains one because of the n times and the rest of the lines 3-10 will take one unit of
lack of autoscaling features. Consequently, the application time(constant). Therefore, we can say that the time complex-
performance at Node 1 will deteriorate as Node 1 becomes ity of Algorithm 2 is O(n). Here we need to consider that the
overloaded with traffic from seven devices. Furthermore, setDesiredPods() function in line 8 of Algorithm 2 is executed
it cannot offload the traffic to the remote edge nodes because one time for a specific node, while the setDesiredPods()
of the absence of connecting links. function in Algorithm 1 requires a repetitive process for n
Therefore, to adapt to fluctuations in traffic for each loca- nodes. Hence, the proposed approach can ensure an effective
tion (node), we propose NHPA that can independently adjust pod autoscaling feature in an edge computing environment
the number of pods for each node based on its workload. with unstable or absent links between edge nodes.
This approach is more flexible and efficient than HPA as each
node can independently adjust the number of pods as per its V. PERFORMANCE EVALUATIONS
requirement, hence it also doesn’t require any coordination In this section, we first define the experimental setup and the
between edge nodes. Details of the underlying mechanism of performance assessment criteria. Then we enlisted some sce-
NHPA are presented in Algorithm 2. The primary benefit of narios to be tested for performance comparison in normal and
NHPA is that it works for each node individually in contrast overloaded behaviours. Further, we examine the outcomes of
to the HPA in Algorithm 1. Thus, the information such as pod distribution with and without NHPA. We also test the
the current number of pods in the cluster (curPods), current performance of NHPA utilizing throughput and latency at one

134422 VOLUME 10, 2022

L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

TABLE 1. Pods distribution according to network traffic accessing cluster.

is applied to KE in various scenarios. From here on, we use

the format x:y:z to represent the proportion of concurrent
requests accessing nodes and x-y-z to represent the distribu-
tion of the pods in the cluster. For example, 1:1:1 indicates
the case with one request accessing each node, while 1-1-1
denotes each node having one application pod. Furthermore,
Table 1 shows that the number of pods in KE remains constant
regardless of traffic types, whereas NHPA adjusts them to
adapt to the traffic distribution. For example, in 1:1:1 traffic,
although one application pod cannot effectively handle one
concurrent request, KE maintains one pod on each node.
In contrast, NHPA adjusts the number of pods in each node
to 2-2-2 to effectively handle the traffic demand based on
the average CPU usage. In particular, NHPA scales up the
number of pods at the node scale to ensure that each node has
a sufficient number of pods to handle the incoming requests
locally. For example, when the distribution of traffic is bal-
anced (e.g., 3:3:3), NHPA increases the number of pods on
each node to 5. In contrast, the pod distributions in the cluster
FIGURE 2. Experimental setup. are changed to 8-4-4 and 9-2-2, respectively, for imbalanced
traffic in cases such as 5:2:2 and 7:1:1. Consequently, appli-
cation performance when using NHPA is considerably higher
edge node. Finally, we analyze the performance of NHPA at
than that of KE, which will be scrutinized in the following
the cluster level.
sections.
A. EXPERIMENTAL SETUP
C. APPLICATION PERFORMANCE AT ONE EDGE NODE
In this section, we evaluate the performance of NHPA
To demonstrate the efficiency of NHPA in an edge computing
in terms of throughput and latency in various scenarios.
environment, we estimated the application throughput and
A KE cluster was set up as shown in Figure 2, which includes
response time of NHPA and KE at Node 1, as shown in
one master node, three edge nodes, and one for generating
Figure 3. Because the communication links between nodes
traffic.
are not established, EdgeMesh Agent at Node 1 only for-
The communication links between edge nodes are unestab-
wards incoming traffic to the local pods of Node 1. Thus, the
lished to simulate actual edge computing environment con-
ditions. All machines were equipped with six CPU cores
and 8 GB of RAM, and the KE version used in the cluster
was 1.9.1. A hey tool is installed on the traffic generator
machine to send requests to the application. The number
of pods in the KE cluster was set to three by default, and
they were evenly distributed across the three edge nodes. For
NHPA, the minimum and the maximum number of pods and
average CPU usage threshold were set as 3, 12, and 80%,
respectively.

B. PODS DISTRIBUTION ACCORDING TO NETWORK

TRAFFIC
Table 1 compares the pod distribution between the cases in FIGURE 3. Application performance at Node 1 in terms of response
which KE is used without the NHPA feature and when NHPA time (a), and throughput (b).

VOLUME 10, 2022 134423

 L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

incoming traffic is handled locally by the pods on Node 1.

Furthermore, KE enables the configuration of the number
of pods to be changed according to traffic demand. Thus,
we used the 1, 3, and 6 pods configuration in this evaluation.
In NHPA, we initially set the number of pods to one. In any
case, it can dynamically autoscale the number of pods based
on the traffic volume. To compare the performance according
to the amount of traffic, we increased the number of concur- FIGURE 4. Application response time at three nodes.
rent requests accessing the application service on the node
from 1 to 9.
As shown by the observed average response time in
Figure 3 (a), a higher number of pods at Node 1 enables as listed in Table 1, but we extend the evaluation cases such
the system to handle more incoming requests without delay, as KE (3-3-3) and KE (6-6-6) that each edge node is allo-
thereby lowering the application response time. However, cated three and six pods respectively for the fair comparison
KE lacks dynamic scaling of the number of pods, which with NHPA.
limits its ability to handle incoming requests effectively. For Figure 4 shows that the response time of KE varies accord-
example, as it requires two pods to handle one concurrent ing to the amount of incoming traffic because it maintains
request, as already discussed in Table 1, KE (one pod) results a constant number of pods at each node. In contrast, NHPA
in a high response time because it cannot increase the number maintains a low response time regardless of the type of traf-
of pods. KE (three pods) and KE (six pods) achieved low fic. In other words, the application response times in both
response times for one concurrent request based on sufficient KE and NHPA tend to be reduced for smaller traffic and
resources; however, the response times tended to increase higher resources. This result can be attributed to the fact
as the number of concurrent requests increased. In contrast, that the incoming requests are handled by local edge nodes
NHPA exhibits a steady increment in response time from owing to the lack of communication links between nodes.
2.6 ms to 4.9 ms because it allocates a sufficient number of When the traffic accessing the cluster is balanced (e.g.,3:3:3),
pods, i.e., 2, 5, 9, and 11 for 1, 3, 5, and 9 concurrent requests, all nodes have the same response time because they have
respectively. the same number of pods. However, KE (1-1-1) exhibits a
Figure 3 (b) compares the throughput of NHPA and KE response time greater than 50 ms, whereas others exhibit
according to the increase in traffic accessing Node 1. It shows a low response time of less than 10 ms. This discrepancy
that the throughput increases as more pods are allocated in the in response time indicates that one pod at each node can-
case of KE; however, the maximum achievable throughput not efficiently handle three concurrent requests immediately,
is limited by the number of pods. For example, the maxi- resulting in a processing delay. For imbalanced traffic, such
mum throughput with one pod, three pods, and six pods is as 5:2:2 and 7:1:1, we observe that the response time at
approximately 65, 144, and 284 req/s, respectively, and it Node 1 in KE increases as the incoming traffic increases.
cannot increase further, even in the case of higher network This result demonstrates that each node in the KE has a
traffic. Furthermore, the number of available resources lim- predetermined and fixed number of pods. Thus, the system
its the maximum achievable throughput. In contrast, NHPA cannot dynamically adapt to increasing traffic. In contrast,
dynamically adjusts the number of pods according to network NHPA shows a low response time regardless of the traffic
traffic. For instance, it allocates 2, 5, 9, and 11 pods for pattern and amount of traffic accessing each node. This low
one, three, six, and nine concurrent requests, respectively. response time can be achieved because each node in the
Therefore, we can conclude that NHPA improves throughput NHPA adjusts the number of pods dynamically and inde-
by allocating pod resources proportionally to the network pendently, according to the traffic load from the other edge
traffic volume that accesses each node. nodes.
Figure 5 shows the cumulative throughput of KE and
D. APPLICATION PERFORMANCE AT CLUSTER SCALE NHPA against diverse network traffic patterns. In general,
To analyze the effect of pod distribution on the overall system the cumulative throughput of the KE tends to increase for the
performance in an edge computing environment, this subsec- allocated pod resources, but it decreases as the traffic con-
tion evaluates the application response time and cumulative centrates on Node 1. For example, the aggregated throughput
throughput of KE and NHPA according to scenarios 2, 3, for 7:1:1 decreases compared with 3:3:3 for all pod distribu-
and 4 of Table 1. We evaluated the network traffic access- tions in the KE. However, in this case, the total amount of
ing each edge node such as 3:3:3, 5:2:2, and 7:1:1, and incoming traffic of 3:3:3, 5:2:2, and 7:1:1 is the same, and
NHPA dynamically allocates the number of pods to 5-5- only the traffic distribution accessing each node is different.
5, 8-4-4, and 7-1-1 to accommodate each traffic pattern as In KE, as the traffic concentrates on Node 1, the throughput
reported in Table 1. KE, on the other hand, cannot dynam- of Nodes 2 and 3 decreases owing to the reduced incoming
ically adjust and has a fixed number of pods. By default, traffic, whereas that of Node 1 is bounded owing to the fixed
KE allocates one pod to each node, noted as KE (1-1-1), number of pod resources. In contrast, NHPA independently

134424 VOLUME 10, 2022

L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

REFERENCES
[1] B. Ahlgren, M. Hidell, and E. C.-H. Ngai, ‘‘Internet of Things for smart
cities: Interoperability and open data,’’ IEEE Internet Comput., vol. 20,
no. 6, pp. 52–56, Nov. 2016.
[2] A. A. Sadri, A. M. Rahmani, M. Saberikamarposhti, and M. Hosseinzadeh,
‘‘Fog data management: A vision, challenges, and future directions,’’
J. Netw. Comput. Appl., vol. 174, Jan. 2021, Art. no. 102882.
[3] H. Zhang, J. Yu, C. Tian, L. Tong, J. Lin, L. Ge, and H. Wang, ‘‘Efficient
and secure outsourcing scheme for RSA decryption in Internet of Things,’’
IEEE Internet Things J., vol. 7, no. 8, pp. 6868–6881, Aug. 2020.
[4] L. H. Phuc, L.-A. Phan, and T. Kim, ‘‘Traffic-aware horizontal pod
autoscaler in Kubernetes-based edge computing infrastructure,’’ IEEE
Access, vol. 10, pp. 18966–18977, 2022.
[5] G. Peng, H. Wu, H. Wu, and K. Wolter, ‘‘Constrained multiobjective
optimization for IoT-enabled computation offloading in collaborative
edge and cloud computing,’’ IEEE Internet Things J., vol. 8, no. 17,
FIGURE 5. Application cumulative throughput at three nodes. pp. 13723–13736, Sep. 2021.
[6] Y.-H. Hung, ‘‘Investigating how the cloud computing transforms the devel-
opment of industries,’’ IEEE Access, vol. 7, pp. 181505–181517, 2019.
[7] S. Aljanabi and A. Chalechale, ‘‘Improving IoT services using a hybrid
fog-cloud offloading,’’ IEEE Access, vol. 9, pp. 13775–13788, 2021.
adjusts the number of pods of each node in all traffic cases;
[8] N. Al-Nabhan, S. Alenazi, S. Alquwaifili, S. Alzamzami, L. Altwayan,
thus, it achieves a high aggregated throughput regardless N. Alaloula, R. Alowaini, and A. B. M. A. A. Islam, ‘‘An intelligent
of the traffic distribution. Especially in imbalanced traffic, IoT approach for analyzing and managing crowds,’’ IEEE Access, vol. 9,
such as in the case of 7:1:1, NHPA achieves approximately pp. 104874–104886, 2021.
[9] M. Bukhsh, S. Abdullah, and I. S. Bajwa, ‘‘A decentralized edge computing
219%, 81%, and 18% higher throughput than KE (1-1-1), latency-aware task management method with high availability for IoT
KE (3-3-3), and KE (6-6-6), respectively, by allocating more applications,’’ IEEE Access, vol. 9, pp. 138994–139008, 2021.
pods to Node 1. Therefore, we can conclude that NHPA [10] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, ‘‘Fog computing and its
role in the Internet of Things,’’ in Proc. 1st, Ed., MCC Workshop Mobile
improves the overall performance for all types of traffic dis- Cloud Comput. (MCC), 2012, pp. 13–16.
tribution accessing the edge computing environment by fully [11] J. Zhang, X. Zhou, T. Ge, X. Wang, and T. Hwang, ‘‘Joint task scheduling
utilizing the available resources of each node independently and containerizing for efficient edge computing,’’ IEEE Trans. Parallel
Distrib. Syst., vol. 32, no. 8, pp. 2086–2100, Aug. 2021.
and dynamically. [12] K. Kaur, S. Garg, G. Kaddoum, S. H. Ahmed, and M. Atiquzzaman,
‘‘KEIDS: Kubernetes-based energy and interference driven scheduler for
VI. CONCLUSION industrial IoT in edge-cloud ecosystem,’’ IEEE Internet Things J., vol. 7,
no. 5, pp. 4228–4237, May 2020.
KubeEdge-based edge computing environments are widely [13] E. Truyen, N. Kratzke, D. Van Landuyt, B. Lagaisse, and W. Joosen,
used in the distribution and administration of containerized ‘‘Managing feature compatibility in Kubernetes: Vendor comparison and
IoT applications because they allow applications to operate analysis,’’ IEEE Access, vol. 8, pp. 228420–228439, 2020.
[14] Z. Cai and R. Buyya, ‘‘Inverse queuing model-based feedback control for
smoothly even when connections between the cloud and the elastic container provisioning of web systems in Kubernetes,’’ IEEE Trans.
edge are disrupted. However, communication links between Comput., vol. 71, no. 2, pp. 337–348, Feb. 2022.
nodes may be weak or even nonexistent in edge comput- [15] Y. Xiong, Y. Sun, L. Xing, and Y. Huang, ‘‘Extend cloud to edge with
kubeedge,’’ in Proc. IEEE/ACM Symp. Edge Comput. (SEC), Oct. 2018,
ing environment where edge nodes are geographically dis- pp. 373–377.
tributed, therefore KubeEdge’s load balancing that forwards [16] D. Wang, Y. Zhou, J. Chen, and C. Zhang, ‘‘Research on spectrum intel-
requests to remote nodes may fail. Although the edge node ligent recognition technology based on an edge processing framework
KubeEdge,’’ in Proc. 6th Int. Conf. Intell. Comput. Signal Process. (ICSP),
needs to handle the incoming request locally in this case, Apr. 2021, pp. 847–850.
KubeEdge lacks a mechanism for dynamically allocating [17] S. Douch, M. R. Abid, K. Zine-Dine, D. Bouzidi, and D. Benhaddou,
computing resources. In this study, we proposed NHPA, ‘‘Edge computing technology enablers: A systematic lecture study,’’ IEEE
Access, vol. 10, pp. 69264–69302, 2022.
which independently adjusts the number of pods in the cluster
[18] N. D. Nguyen, L.-A. Phan, D.-H. Park, S. Kim, and T. Kim, ‘‘ElasticFog:
for each edge node to maximize application performance Elastic resource provisioning in container-based fog computing,’’ IEEE
regarding response time and throughput. The experimental Access, vol. 8, pp. 183879–183890, 2020.
results showed that the NHPA performed better than KE [19] L. Wojciechowski, K. Opasiak, J. Latusek, M. Wereski, V. Morales,
T. Kim, and M. Hong, ‘‘NetMARKS: Network metrics-AwaRe Kubernetes
in the KubeEdge-based edge computing environment. When scheduler powered by service mesh,’’ in Proc. IEEE INFOCOM Conf.
compared to KE, the throughput and response time of NHPA Comput. Commun., May 2021, pp. 1–9.
were reduced by a factor of 3 and 25, respectively. Therefore, [20] J. Santos, T. Wauters, B. Volckaert, and F. D. Turck, ‘‘Resource provision-
ing in fog computing: From theory to practice,’’ Sensors, vol. 19, no. 10,
we concluded that providing resource autoscaling indepen- p. 2238, May 2019. [Online]. Available: https://www.mdpi.com/1424-
dently for each node can significantly improve application 8220/19/10/2238
performance by removing the effect of unstable communi- [21] N. A. Cv and R. Lavanya, ‘‘Fog computing and its role in the Internet of
Things,’’ in Advancing Consumer-Centric Fog Computing Architectures.
cation links in an edge computing environment. Hershey, PA, USA: IGI Global, 2019, pp. 63–71.
[22] N. D. Nguyen and T. Kim, ‘‘Balanced leader distribution algorithm in
ACKNOWLEDGMENT Kubernetes clusters,’’ Sensors, vol. 21, no. 3, pp. 1–15, 2021.
[23] D. Balla, C. Simon, and M. Maliosz, ‘‘Adaptive scaling of Kubernetes
(Le Hoang Phuc and Majid Kundroo contributed equally to pods,’’ in Proc. NOMS IEEE/IFIP Netw. Oper. Manage. Symp., Apr. 2020,
this work.) pp. 1–5.

VOLUME 10, 2022 134425

 L. H. Phuc et al.: Node-Based Horizontal Pod Autoscaler in KE-Based Edge Computing Infrastructure

[24] L. Toka, G. Dobreff, B. Fodor, and B. Sonkoly, ‘‘Machine learning-based DAE-HEON PARK received the B.S., M.S., and
scaling management for Kubernetes edge clusters,’’ IEEE Trans. Netw. Ph.D. degrees in communication and information
Service Manage., vol. 18, no. 1, pp. 958–972, Mar. 2021. engineering from Sunchon National University,
[25] T. Hu and Y. Wang, ‘‘A kubernetes autoscaler based on pod replicas in 2006, 2008, and 2015, respectively. Since 2011,
prediction,’’ in Proc. Asia–Pacific Conf. Commun. Technol. Comput. Sci. he has been a Senior Research with the Elec-
(ACCTCS), Jan. 2021, pp. 238–241. tronics and Telecommunications Research Insti-
[26] Y. Dong, G. Xu, M. Zhang, and X. Meng, ‘‘A high-efficient joint ’cloud- tute (ETRI), South Korea. His research interests
edge’ aware strategy for task deployment and load balancing,’’ IEEE
include the IoT, AI, cloud, bigdata, and ICT con-
Access, vol. 9, pp. 12791–12802, 2021.
vergence with agriculture.
[27] T. Yang, J. Ning, D. Lan, J. Zhang, Y. Yang, X. Wang, and A. Taherkordi,
‘‘Kubeedge wireless for integrated communication and computing ser-
vices everywhere,’’ IEEE Wireless Commun., vol. 29, no. 2, pp. 140–145,
Apr. 2022.
[28] GitHub Kubeedge/Edgemesh: Simplified Network and Services for
Edge Applications. Accessed: Nov. 26, 2022. [Online]. Available:
https://github.com/kubeedge/edgemesh SEHAN KIM received the B.S. and M.S. degrees
[29] Z. Ma, S. Shao, S. Guo, Z. Wang, F. Qi, and A. Xiong, ‘‘Container in computer engineering from Korea Aerospace
migration mechanism for load balancing in edge network under power University, South Korea, in 1998 and 2000, respec-
Internet of Things,’’ IEEE Access, vol. 8, pp. 118405–118416, 2020. tively. He was a Research Staff at the Samsung
[30] T.-T. Nguyen, Y.-J. Yeom, T. Kim, D.-H. Park, and S. Kim, ‘‘Hor- Advanced Institute of Technology, in 2000. Since
izontal pod autoscaling in Kubernetes for elastic container orchestra- 2001, he has been the Principal Researcher and the
tion,’’ Sensors, vol. 20, no. 16, p. 4621, Aug. 2020. [Online]. Available: Director of the Electronics and Telecommunica-
https://www.mdpi.com/1424-8220/20/16/4621 tions Research Institute (ETRI), South Korea. His
research interests include digital twin, platform,
data science with the IoT, AI, cloud, bigdata, and
intelligent ICT convergence with agriculture, food, and fisheries.
LE HOANG PHUC received the B.S. degree
in electronics and communication engineering
from Can Tho University, in 2018. He is cur-
rently pursuing the M.S. degree with the School
of Information and Communication Engineering,
TAEHONG KIM (Senior Member, IEEE) received
Chungbuk National University, South Korea. His
the B.S. degree in computer science from Ajou
research interests include the IoT applications,
University, South Korea, in 2005, and the
container orchestration, open-source software, and
M.S. degree in information and communication
cloud/edge computing.
engineering and the Ph.D. degree in computer sci-
ence from the Korea Advanced Institute of Sci-
ence and Technology (KAIST), in 2007 and 2012,
respectively. He was a Research Staff Member
MAJID KUNDROO received the M.S. degree in at the Samsung Advanced Institute of Technol-
computer science from the Islamic University of ogy (SAIT) and the Samsung DMC Research and
Science and Technology, Jammu and Kashmir, Development Center, from 2012 to 2014. He was a Senior Researcher
India, in 2019. He is currently pursuing the Ph.D. at the Electronics and Telecommunications Research Institute (ETRI),
degree with the School of Information and Com- South Korea, from 2014 to 2016. Since 2016, he has been an Associate
munication Engineering, Chungbuk National Uni- Professor with the School of Information and Communication Engineering,
versity, South Korea. His research interests include Chungbuk National University, South Korea. His research interests include
edge computing, edge AI, the Internet of Things, edge computing, container orchestration, the Internet of Things, and feder-
and federated learning. ated learning. He has been an Associate Editor of IEEE ACCESS, since 2020.

134426 VOLUME 10, 2022