Future Generation Computer Systems 97 (2019) 50–60

Contents lists available at ScienceDirect

Future Generation Computer Systems

journal homepage: www.elsevier.com/locate/fgcs

Container-based fog computing architecture and energy-balancing

scheduling algorithm for energy IoT
∗
Juan Luo a , Luxiu Yin a , Jinyu Hu a , Chun Wang a , Xuan Liu a , , Xin Fan a , Haibo Luo b
a
College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China
b
Electronic Information and Control of Fujian University Engineering Research Center, Minjiang University, Fujian, China

article info a b s t r a c t

Article history: The traditional architecture of fog computing is for one data center and multiple fog nodes. It is unable
Received 1 June 2018 to adapt to the current development of private cloud. In addition, virtual machines used for cloud
Received in revised form 8 October 2018 computing, are also used for fog computing as the resource unit, cannot satisfy the requirement of fog
Accepted 29 December 2018
computing. Furthermore, the limited capacity of battery power has been one of the major constraints
Available online 11 January 2019
when considering cloud-to-fog-to-sensor pattern in the scenario of Energy Internet. We propose a multi-
Keywords: cloud to multi-fog architecture and design two kinds of service models by employing containers to
Container improve the resource utilization of fog nodes and reduce the service delay. According to the two service
Docker models, we present a task scheduling algorithm for energy balancing. The algorithm is based on the
Energy balancing transmission energy consumption of terminal devices and uses a dynamic threshold strategy to schedule
Fog computing requests in real time, thereby guaranteeing the energy balancing of terminal devices without increasing
Multi-cloud the transmission delay. Experimental results show that our proposed service models and scheduling
algorithm can reduce service latency, improve fog node efficiency, and prolong WSNs life cycle through
energy balancing.
© 2018 Published by Elsevier B.V.

1. Introduction from the core of a network to the network edge to support real-
time applications and reduce the network burden of data centers.
1.1. Background
1.2. Motivation
Cloud computing has been applied widely in many fields. How-
ever, many problems with cloud computing have emerged with Although there is extensive research on fog computing, there
the development of IoT. In front of a mass of IoT devices, cloud are still several issues. First, most of the resource management of
computing has difficulty satisfying delay-sensitive and location- fog computing is based on virtual machines (VMs). However, using
aware applications because it cannot be built extensively due to VMs as the resource is not adaptable to fog computing platforms.
the expensive cost of construction. With the number of IoT devices For example, the boot-up time of a VM is several minutes, which is
growing (Cisco predicts that there will be 50 billion connected too long for real-time applications. For a large numbers of access
devices by 2020 [1]), an ocean of data will be transferred to data points, the characteristics of fog computing are affected by the
centers for processing (annual global data center IP traffic will number of VMs because the performance of physical machines is
reach 15.3 ZB by the end of 2020). If IoT still uses the current cloud degraded when the number of VMs increases. The overhead of a
computing paradigm to handle the enormous amount of devices
hypervisor exponentially increases when more VMs run within
and data, it will give rise to high latency and network congestion.
the same machine [4]. Therefore, a new resource form is required
Based on these issues, fog computing was proposed by Cisco in
to support the features of fog computing. Second, the conven-
2012 [2], and defined by L. M. et al. [3]. Fog computing is considered
tional IaaS platform is not adapted to fog computing. The service
as extension of cloud computing to fix defects of cloud computing.
paradigm of conventional IaaS is that users send their demands
In fog computing, the computation and storage will be migrated
for virtual resources to the data center, and the data center sub-
sequently creates the VM in terms of the demand and returns
∗ Corresponding author.
an IP address to the user. Users manage the VM by accessing
E-mail addresses: [email protected] (J. Luo), [email protected] (L. Yin),
[email protected] (J. Hu), [email protected] (C. Wang),
the IP, and last, the user installs various binaries and libraries to
[email protected] (X. Liu), [email protected] (X. Fan), lay the groundwork for whatever application needs to run. For
[email protected] (H. Luo). example, if an application uses Elasticsearch, then the VM has

https://doi.org/10.1016/j.future.2018.12.063
0167-739X/© 2018 Published by Elsevier B.V.
 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60 51

to install Java and Lucene before deploying Elasticsearch. This 2. Related work
procedure is tedious and inappropriate for the real-time response
to fog computing. Third, as shown in Fig. 1, most fog computing At present, the researches on the fog computing architecture
researches [5–7] are aimed at one data center with multiple fog are mainly focus on the architecture, resource management, task
nodes. However, increasingly, enterprises are building their own scheduling of fog computing.
private data centers in order to store and analyze data. At the same Sun et al. [12] first proposed an SDN-based network architec-
time, the enterprises need to deploy their applications on fog nodes ture. The network architecture places the SDN in the middle of the
to collect data and provide real-time response to terminal devices. fog node and the cloud data center, which improves the communi-
Despite the low cost of building a single fog node, it is necessary to cation efficiency between the fog node and the cloud. In addition,
deploy a host of fog nodes to cover a wide range. The construction the article also gives the fog node service architecture. Its service
and maintenance cost of these fog nodes is tremendous for enter- architecture is divided into two layers, application layer and user
prises, especially for those enterprises whose services only run for layer. At the user data layer, a user’s private virtual machine is
a short term or periodically. deployed, the original data sent by the user is processed, and
As an improved version of the IoT, Energy Internet (EI) in- the processed data is sent to the application-level service virtual
tegrates energy and information in a distributed and renewable machine. Through the processing of the original data, the user’s
energy system, which has been well studied by numerous re- privacy is guaranteed. Lu Hou et al. [13] combined HTTP and MQTT
to implement a request response model and a message subscrip-
searchers [8–11] in recent years. And when considering cloud-to-
tion and release model, respectively, to satisfy the service diversity
fog-to-sensor pattern in the scenario of Energy Internet, energy
of fog computing. In addition, the implementation of the message
efficiency of sensor node has still been the most crucial issue on fog
broker is given and a three-level QoS level is designed to improve
computing. Because of the renewable nature of EI, energy manage-
service reliability. The architecture uses several components such
ment is an essential part on fog computing, which has significant
as application servers, database clusters and agents, load balancers
effects on energy balancing among sensor nodes. In this paper, our
and so forth to ensure its high performance.
objective is to balance the distribution of residual energy of the
Mao et al. [14,15] developed an online joint radio and com-
overall network nodes. Therefore, one of the greatest challenges putational resource management algorithm for multi-user MEC
in energy management is to select an optimal way to balance the systems, and the objective was to minimize the long-term av-
energy consumption on offloading the tasks from energy-limited erage weighted sum power consumption of mobile devices and
sensors to energy-sufficient clouds/fogs. MEC servers, subject to a task buffer stability constraint. Fur-
thermore, a green MEC system with energy harvesting devices
1.3. Contributions was investigated, an effective computation offloading strategy was
developed, and a low-complexity online algorithm (i.e., Lyapunov
Aiming to solve the problems mentioned above, we propose a optimization-based dynamic computation offloading algorithm)
novel hierarchical architecture that is called multi-cloud to multi- was proposed. The algorithm could select offloading decisions for
fog architecture, and design a temporary service model and a long- CPU cycle frequencies during mobile execution and power trans-
term service model. Furthermore, we propose a task scheduling mission for offloading computation. Ma et al. [16] proposed a cloud
algorithm for energy balancing. The main contributions of this assisted mobile edge computing (CAME) framework to enhance
paper are described as following: system computing capacity and a workload scheduling mechanism
to balance the tradeoff between system delay and cost.
• We propose a novel architecture and design several com- Container-based virtualization presents an interesting alter-
ponents to solve potential security issues caused by multi- native to VMs in the cloud. In the cloud, containers have been
cloud to multi-fog as well as to reduce the construction and proposed as a lightweight virtualized technology. There are several
maintenance cost of service developers. studies comparing the performance of containers and VMs [17,18].
• We design two service models, a temporary service model We contacted these studies to discuss which virtualization tech-
and a long-term service model, based on containers which nology is more suitable to fog computing for deployment conve-
are used as the resource units. The temporary service model nience, performance.
provides real-time request and response services, while the Deployment convenience: Using containers, everything re-
long-term service model is responsible for subscripting and quired to make a piece of software run is packaged into isolated
publishing service. Our experiments show that the service containers. Unlike VMs, containers do not bundle a full oper-
models based on containers can effectively improve the re- ating system, and only libraries and settings required to make
source utilization of fog nodes and reduce service delays. the software work are needed. This approach makes for efficient,
• We propose a task scheduling algorithm based on energy lightweight, self-contained systems and guarantees that software
balancing to prolong the life of WSNs and reducing the delay will always run the same, regardless of where it is deployed. This
constraint of tasks. In the scheduling algorithm, we design an property makes the deployment of a container simpler and easier
energy balancing strategy to control the transmission power than a VM and hides the heterogeneity of fog computing. Further-
of terminal devices and use a dynamic threshold to schedule more, due to the service object of fog computing being mobile, live
arrived requests in real time. The algorithm improves the migration has to be considered. To date, the container is weak for
number of concurrent tasks in the fog node and balances the live migration because of nested process migration. Although the
energy consumption of terminal devices without increasing live migration of containers can be implemented by CRIU [19], the
the data transmission time. nested relationship of process groups among multiple containers
is not sustainable. However, for a single container, live migration
The rest of this paper is structured as follows. Section 2 is is supported well. In comparison to containers, the live migration
the related work. Section 3 details the architecture and presents of VMs has developed maturely. At present, there are solutions
the implementation of components in this architecture. Section 4 based on memory [20] and disk [21]. Moreover, it is easy to execute
elaborates the task scheduling algorithm for energy balancing. The without nested process problems since VMs are migrated with an
experimental results of containers and VMs are evaluated and dis- integrated object.
cussed in Section 5. Section 6 summarizes our study and outlines Performance: The container has approximately native perfor-
our future research. mance compared to the virtual machine because the container can
52 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60

Fig. 1. Traditional architecture of fog computing.

run in the operating system directly, while the virtual machine and access control. A request is processed from the top tier to the
has a hypervisor layer. IBM undertook the research [17] comparing lower tier. When the fog node receives a request, first, the request
the performance between virtual machines and Linux containers. will be validated for permission and resolved into the service
In the aspects of memory and CPU, both virtual machines and catalog at the access tier. Next, the service catalog is transferred
containers introduce negligible overhead. For I/O, containers are to the control tier. The control tier dispatches the services in the
much better than virtual machines when they perform intensive service catalog to the corresponding manager, either temporary or
I/O operations. Regarding network latency, despite container la- long-term according to the service type. Last, the corresponding
tency being double the native, compared to virtual machines, it is a manager calls the API provided by the infrastructure tier to create
negligible difference (only a difference of 10 µs-20 µs). This paper the container. As shown in Fig. 2, there are three SDs and three fog
also compares the service performance including Redis and Mysql. nodes. Each fog node provisions the physical resource for TDs near
The results show that the latency and throughput of a container the node location, as well as SDs. There are several services running
are similar to native, while the virtual machine is inferior to the in fog nodes; these fog nodes connect to multiple private clouds of
container. Overall, containers are better than virtual machines for SD with the Internet. The services are encapsulated to image-file
the real-time characteristic of fog computing. by the SD, and the image-file is actively pushed to fog nodes by the
SD or downloaded by users to fog nodes. The fog node starts the
3. The architecture of fog computing services by running image-files. TDs upload data to the fog node
for computing and storing. The data that must be uploaded to the
3.1. Roles cloud is packaged by the fog node and regularly transferred to the
cloud.
Different from service developers and infrastructure providers
as one role in conventional fog computing, we divide it into two 3.3. Components of multi-cloud to multi-fog architecture based on
roles. In conventional fog computing, a service developer not only containers
develops service but also constructs fog nodes in an extensive
range. However, the cost of constructing fog nodes in an exten- The lower tier is the infrastructure tier. In the infrastructure tier,
sive range is expensive for service developers if their services are the object to be managed is the container. The physical resources
running just for few months; therefore, renting to the resource of are provided by one or more physical machines. These PMs have in-
fog nodes to run their services can efficiently reduce the cost to stalled hypervisor, which means that these PMs will be segmented
the service developer. There are three roles in our architecture, to a number of VMs. However, as mentioned earlier, using a VM
infrastructure provider (InP), service developer (SD) and terminal as the unit of resource is not efficient. For temporary service, a
device (TD). InPs construct fog nodes for service developers and container is a better alternative because the container has absolute
terminal devices, service developers develop services that can run superiority relative to VMs as a resource unit in fog computing.
in the fog nodes, terminal devices use the service via submitting a Containers take up less disk space. A container is represented by
request to a fog node and paying for it. lightweight image-files. VMs are also based on images but fuller
monolithic ones than a container. Note that rather than deploying
3.2. Multi-cloud to multi-fog architecture a container directly on a physical machine, we use the VM as the
host of containers because the incomplete isolation of a container,
In this paper, we propose a multi-cloud to multi-fog archi- which shares the system kernel, can cause insecurity. We are
tecture based on containers. The architecture is designed with a unable to create a VM for each container since the performance
hierarchy using containers as the resource unit. In our architecture, of the physical machine will decrease with the number of VMs
a fog node consists of three tiers, that is, the infrastructure tier, the increasing. Consequently, containers-on-VM is adopted to alleviate
control tier and the access tier. The infrastructure tier virtualizes the damage from a system failure caused by a container so that
the physical resource and offers the API of managing a container even a container breakout or system failure only affects the current
to the control tier. The control tier orchestrates services for en- VM, reducing the influence of the risk of containers. In summary,
hancing resource utilization and provides the interaction between the resource manager should be introduced in this tier. The re-
fog nodes and SDs. The access tier is used for request management source manager provides a managed interface of containers to the
 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60 53

Fig. 2. Multi-cloud to multi-fog architecture.

Fig. 3. Components of multi-cloud to multi-fog architecture based on container.

above tier and offers the managed interface of VMs for InPs. The management tools, such as virsh, virt-install, virt-manager and the
resource manager is described as follows: cloud computing framework are using it at the foundation.
Resource manager: In the resource manager, we choose Docker The middle tier is control tier. In this tier, the object to be
as the container engine and KVM (Kernel-based Virtual Machine) managed is service. In general, long-term services rent the resource
as the hypervisor. Docker is the most popular container engine. It of a fog node until the contract expires. Temporary services are
uses Docker Container as the basic unit for resource segmentation submitted with the data processed by the service. When the service
and scheduling, encapsulates the entire software runtime environ- is finished, the resource that is used by the service will be released.
ment and is designed for developers to build, publish and run dis-
Therefore, because of the difference between long-term service
tributed applications. The core of Docker is Docker daemon; it will
and temporary service, there are two managed components, Long-
start the API Server that provides interfaces, including container
term Service Manager (LTSM) and Temporary Service Manager
creating, updating, deleting and searching to receive requests from
the Docker client (the client refers to those components of the (TSM), for the life cycle of containers of long-term service and
above tier). QEMU-KVM is employed as a hypervisor. KVM only temporary service.
supports the CPU and memory virtualization; therefore, it needs to LTSM: Long-term services are deployed by SD on the fog node
combine QEMU to provide a complete virtualization. In addition, that is constructed by InP. Thus, a terminal user is unaware of
InPs manage virtual machines via Libvirt that is the most widely whether the service has been deployed in the fog node and does not
used tool and application programming interface (API) for manag- know the port of the service when the user submits their requests.
ing KVM virtual machines. Several commonly used virtual machine The service registration and discovery must to be leveraged by
54 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60

the LTSM. LTSMs manage the service in their own fog node, and should be migrated to clouds that registered their cloud service
collaborate with the LTSM of neighbor nodes. Since the resources in in the fog node. The cloud agent will periodically send heartbeat
a fog node are finite, the service will be migrated to other fog nodes information to those clouds in order to detect the network delay
or suspended when resources are not sufficient. For example, if for each registered cloud. When TSM receives the request, if the
all neighbor nodes of a fog node host a service that is accessed TSM finds that the request cannot be hosted in the fog node, then
infrequently and occupies considerable resource in this fog node, the TSM transmits the service request to the cloud agent. The cloud
then the fog node should suspend the service and release resources agent should select a cloud service provider, which minimizes the
and subsequently forward those requests that access this service delay to reduce the latency.
to neighbor nodes with current fog node ID. When a service is The top tier is the access tier. The object to be managed is a
accessed frequently by this fog node, the fog node should restart request. The request is responsible for authorizing users, as well
the service to reduce the overhead of communication. However, as receiving requests. The request uses JSON to transfer the mes-
which service should be migrated or suspended and how to select
sage to hide the diversity of IoT devices. A request for temporary
the forwarding destination are still open issues. InP can design
service consists of an image-file URL list, container number, to-
various heuristic schedules in terms of an InP’s demand such as
be-processed data; the request that accesses long-term service
minimizing communication cost or optimizing performance.
consists of a unique identifier of service, to-be-processed data, and
TSM: Temporary services are more sensitive than long-term
service token.
services for the latency of communication, and most temporary
Request queue: Due to fog computing needs to process high
services are applied by TDs that are mobile. Therefore, in order
to minimize latency, the service should be migrated when the concurrency requests, which have different priorities. Fog nodes
terminal user roams out of the coverage of the fog node. The TSM of require a component to assign the request according to the pri-
the current fog node notices the TSM of the destination fog node in ority of the request. Therefore, we choose the priority queue in
advance according to the path of the TD to ensure a seamless hand fog computing, rather than a traditional FIFO queue. The request
over and take over. Because the size of some image-files is too large manager has a request queue that stores forthcoming requests and
relative to a TD from a few KB to a few GB, TSM has a database addresses the priority of these requests. If a request has high prior-
for storing image-files, which are used frequently. For example, ity, then the request queue will deliver this request to LTSM/TSM
the ubuntu image-file requested by many users as a base image priority.
should be stored in the fog node, the fog node creates the container Access controller: The access controller is divided into network
directly instead of downloading the image-file from an image-file access control and service access control. The network access con-
repository such as DockHub to alleviate the cost of communication trol is responsible for managing the terminal user connection to
between the fog node and image-file warehouse. For an image- a fog node. The network access control uses the username and
file that is not stored in the repository, users just submit the URL password to verify that the terminal user can connect to the fog
of the image-file in the repository, and then the fog node will node. The service access control is employed to manage which
automatically download this image and run it. The manager uses services the terminal user can access; it works primarily with long-
a database to store the MD5 of image-files. The image manager term services. For service access control, the access controller has
periodically deletes image-files, which are accessed infrequently. a table for storing the token of each service running in the fog
Cloud agent: The cloud agent plays an important role in this node. The service token is provided by SD. The SD also provides
architecture. The cloud agent builds a bridge between fog nodes the corresponding token to the terminal user with devices or
and clouds to improve efficiency and protect the system. We design applications, the terminal user submits the token with the request
the cloud agent based on both long-term service and temporary to the access controller. If the token in the request matches the
service.
token in the access controller, the access controller pushes the
For long-term service, the cloud agent offers two functionalities.
request to request queue; otherwise the access controller returns
The first is the service proxy. The long-term service should be
an error to terminal user.
under SD control, whereas a fog node is shared by a multitude of
SDs. Considering the security of the fog node, an SD is unable to
operate Docker directly. The only way to manage their services is 4. Energy balancing for terminal devices
by the cloud agent. The management includes creating, updating
and deleting service, increasing or reducing the service resources. 4.1. System model
When the SD needs to manage its services, it will send the request
to the cloud agent, then the cloud agent calls the API provided by
The fog computing system is modeled as Fig. 3, with set N of
the Resource Manager to respond to the request. For the long-term
TDs and a fog node for the high-performing servers around those
service that runs periodically, the SD sends its service schedule to
devices. TDs will periodically collect the environment data and
the cloud agent and the cloud agent works with LTSM to imple-
harvest energy. Because of the limitation of computation capacity
ment the schedule. The cloud agent can avoid malicious behavior
of TDs, the collected data need to be transferred to the fog node
or incorrect operation that causes potential insecurity. The second
for processing and analyzing. In addition, TDs will send the heart-
is supervising. SD supervises long-term services; however, due
to running conditions, such as the utilization of CPU, memory, beat messages regularly to the fog node which connect with those
I/O and the network of service that is offered by Docker in the TDs. A heart-beat message includes the ID of the TD, the energy
infrastructure tier, the service is unable to perceive its own running collected by the current time slot and the energy consumption
condition. Therefore, the running condition is submitted to the and battery capacity at the current time slot. In each time slot, the
cloud agent by Docker, and the cloud agent delivers it according fog node calculates the available transmitting power for each TD
to the service affiliation. according to its own heart-beat messages. However, if the current
Temporary service will commonly be hosted in the fog node. battery capacity of the device does not meet the energy demand
However, the fog node serves a sea of users with limited resources for data transmitting at the next slot, then it is determined that
in the fog node. If a single temporary service applies for excessive the energy consumption of the device is insufficient and the device
resources that will cause other users wait too long or not be is adjusted to a dormancy state, waiting for the energy harvesting
satisfied by the fog node and thus affect the user experience they equipment to be charged.
 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60 55

that the transmission delay for feedback is negligible. Hence, the

execution time in the fog node j of task t can be calculated as
j j
Exj (t) = Exjcom (t) + Eximg (t) + Extra (t). (3)

In Eq. (3), similar to the [22,23], but there are some differences
j
due to use of containers. The computation time Excom (t) of the task
t in the fog node j can be calculated by:
vt ,data
Exjcom (t) = ∑ j
, (4)
t
1≤p≤Pmax fp (t)
j
where fp (t) represents allocated computation resources by the
t
fog node j for the task t during the period p. Pmax represents the
maximum number of period that the task can stay. In order to
t
satisfy the delay constraint, Pmax can be calculated by:

t D(t)
Pmax =⌊ ⌋, (5)
ρj
where ρj is a constant, which is expressed as the time of a period
and D(t) represents the delay of the task t.
Fig. 4. The harvested solar energy.
The resource amount which task t need can be calculated by:
j
F (t) = εt · vt ,data (6)
4.1.1. Energy model of TDs
vt ,data represents the data volume need to be processed of the task
Because the fog node can support wired power supply, the main j
consideration is the energy model of TDs. A TD consists of several t, εt represents the number of instructions needed to process per
bit of data for image which needs by the task t. Its unit is MIPS and
sensors and a energy harvesting equipment. In this paper, we use
it can be measured [24].
solar panel to harvest energy. The battery capacity of the TD in the
The data transmission time of task t can be calculated by:
time slot k can be calculated as follow:
j vi (k)
Ei (k + 1) = Ei (k) + Ei,har (t) − Ei,cost (k), (1) Extra (t) = ht +pow eri,k
, (7)
wnj × log2 (1 + o
)
where Ei,cost (k) and Ei,har (k) denote the energy cost and the har-
The image transmission time of the task t is
vested energy of the TD i in time slot t respectively. ⎧
Since the TD cannot process the collected data within a very ⎨0, Imaget ∈ I j ;
Eximg (t) = vt ,img
low delay, it needs to upload the collected data to the fog node for j
/ I j,
Imaget ∈ (8)
processing. Therefore, the energy cost of the TD can be obtained by ⎩ wc
j
:
where vt ,img represents the image volume needs the bandwidth
vi (k)
Ei,cost (k) = pow eri,k × hk +pow eri,k
, (2) used to connect to the cloud when the fog node j allocates to the
wnj × log2 (1 + o
) task t.
In addition, there is a constraint in the fog execution model:
where pow eri,k denotes the transmission power. The second item
of the right side of the equation is the data transmission delay, vi (k)
∑
fpj (t) ≤ F j , (9)
represents the collected data volume in the time slot k, w is system j
t ∈Tp
bandwidth, hk is the channel power gain at the kth time slot, o is
noise power at the TD. j
where Tp represents the task set which in the fog node j during the
The energy harvested by solar panels varies with time, season, period p and F j represents the total computation resource of the fog
and other factors. At 10 am, energy harvesting equipment is the node j. The constraint indicates that the resources occupied by the
most efficient. When entering the night, the energy collected by tasks running on the fog node cannot exceed the total resources of
the collecting device is 0. From our experiment, we know that the node in any period.
the energy harvested by solar panels at different time periods, as
shown in Fig. 4. Therefore, in this paper, we will use the energy 4.1.3. Cloud execution model
harvesting value in our following experiments. In the cloud execution model, we assume that the computation
resource of the cloud is infinite. Therefore, when a task arrival, it
4.1.2. Fog execution model can be executed immediately without waiting for computation re-
When a fog node receives a data sent by TDs, the fog node deter- source. In addition, the all image file will be stored in the cloud and
mines whether the task can run locally or transmitted to the cloud hence the image transmission time can be ignored. The execution
according to its execution time. Each task t ∈ T is characterized time of a task t that is run in the cloud can be calculated as
by a three-tuple of parameters, t ≜ ⟨Data, Delay, Image⟩, where j j,c
Exc (t) = Extra (t) + Exdata (t), (10)
the Data represents the processing data. The unit of data is kB.
j,c
The Delay represents the deadline of the task t which is generated where Exdata (t) represents the data transmission time of the task t
by the TD. The Image parameter refers to a Uniform Resource from the fog node j to the cloud. Because computation resource of
Identifier. The execution time of a task in a fog node depends cloud is assumed to be infinite, all image files can be stored without
j
on computation time denoted by Excom (t) and its data and image regard to the transfer time of the image files. According to our
j j
transmission time denoted by Extra (t) and Eximg (t) respectively. computational model, if the cloud can provide enough resources,
We assume that the output of the computation is of small size so the computing time of its tasks in the cloud will be negligible.
56 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60

The data transmission time of task t which from the fog node j immediately. This approach makes the scheduler unable to obtain
to cloud can be calculated by: all tasks and perform optimal scheduling. Therefore, we design a
j,c vt ,data threshold and adjust it according to the current set of tasks, to
Exdata (t) = , (11)
wtj,c maximize the number of active tasks on the fog node. However,
this threshold is difficult to determine.
j,c
where wt represents the data transmission rate from the fog node Task scheduler maximizes the number of active tasks on a
j to cloud. fog node j at each period as the optimization objectives for task
The delay constraint should be met, i.e., scheduling. However, it is online problem, which means the tasks
j are scheduled one by one and hence the task scheduler cannot
xt Exj (t) + xct Exc (t) ≤ D(t), (12)
obtain all the tasks and perform the optimal scheduling. Therefore,
j j
xt xct
and are binary variables, which indicate whether the task t is the task scheduler needs to set a decision threshold δp for each
running on the fog node j and whether it is running in the cloud period, indicating the resource threshold of the fog node j in the
respectively. If running in the fog node or the cloud, their value is period p. When the task is accepted, the task scheduler first calcu-
j
1, otherwise 0. Based on the indivisible assumption of the task, and lates the resource demand Fav g (t) per period of the task, which is
j
hence, xt ̸ = xct . calculated as:
F (t)
4.2. Algorithm description Faj v g (t) = t
(14)
Pmax
We need to balance the energy consumption of TDs within the j
And then compared with the threshold δp of current period,
entire network for increasing the life cycle of a wireless sensor j j
network. If the energy of a TD is exhausted prematurely, the entire if the Fav g (t) > δp , then the task shall be distributed to the
network life cycle will end prematurely. However, a TD cannot ob- cloud; otherwise, the task will be distributed to the fog node for
tain the energy status information of other devices and thus cannot execution. It should be noted that this paper calculates the resource
make decisions on transmission power. In order to ensure that the requirements per period rather than the total amount of resources.
remaining battery capacity of TDs within the entire network can Since it will not cause too much impact on the node density in the
be maintained on the same level, all devices to periodically send case of a task with a large amount of resources and longer delay
their own energy status information to the fog node. Then the fog constraint, which means that each period of resource demand is
node plans the calculation power and transmission power of each not too much on average. However, for a task with large amount of
TD. This ensures that the remaining battery capacity of each node resources but a short delay constraint, the cloud can be scheduled
can be kept at the same level. Note that all data must be completed because the cloud is extremely rich in resources relative to the fog
within the specified delay. nodes. Scheduling such tasks to the cloud can greatly reduce the
In order to balance the energy consumption of each device, we computing time.
need to limit the standard deviation of the energy consumption Instead of always using a fixed threshold, the task scheduler
of each TD and the entire network energy consumption within a j
adjusts δp in each period based on the number of accepted tasks
specific range. When the remaining battery capacity of a device is in last period. We assume that tasks will be continually sent to the
lower than the network average, we need to reduce its transmis- task scheduler, therefore, if the number of tasks accepted in the
sion power so that it can reduce the energy overhead. When the
current period is less than the previous period, it means that too
remaining battery capacity of a device is higher than the average
many tasks are allocated to the cloud due to the threshold value
value, we can increase its transmission power, thereby reducing
of the current period being too small, so that the tasks that can
the delay of data transmission. Since there is no guarantee that
be accepted by the fog node exceed the threshold will be rejected.
the energy consumption of all devices is exactly the same, we
define an offset value θ that allows a certain amount of deviation. If the number of tasks accepted in the current period is greater
Consequently, energy balancing can be expressed as follow: than the number of tasks in the previous period, it means that the
∑ threshold of the current period may not realize the optimal value.
j∈N |Ej (k + 1) − Ej (k)| The task scheduler continues to decrease the threshold according
|Ei (k + 1) − Ei (k)| ≤ + θ, (13)
N to the maximal resource quota of all active tasks in the current
then we bring it in Eq. (1) to obtain the energy consumption that period,
the device i can use in the next time slot. According to the energy ⎧ j j j
consumption in next time slot, we can calculate the transmission ⎨δp−1 , p > 1, Tp < Tp−1
j
power via Eq. (2). δ p+1 = j
max(Fav g (t) |t ∈ j
Tp ) , p > 1,
j
Tp ≥ j
Tp−1 (15)
F,
j
p = 1,
⎩
Since the energy balance of the WSN needs to be ensured, the
transmission energy cost of the TD needs to be adjusted, resulting
in an increase of the data transmission delay. For those delay-
sensitive applications, delay-sensitive requirements need to be
met by reducing the scheduling of fog nodes. If the uploaded data 5. Experimental result
is large, the fog node needs to make a decision that upload the
uploaded data locally or forward it to the cloud for processing to
In this section, we compare the performance of VMs and con-
meet the delay of tasks.
Task scheduler in TSM will distribute the task when the task tainers for the concurrency condition that is the most significant
has arrived. Fog nodes are used to assist the cloud to accomplish characteristic of fog computing. The experiences verify that con-
those tasks that are less computationally intensive, so tasks that tainers, which are the foundation of our architecture, are more
are overly computationally expensive, that is, overly resourced, suitable for fog computing than VMs. We run multiple Linpack
need to be forwarded to the cloud. However, the data need to programs to test the performance loss of multi-program parallel
be processed of these tasks are very small, but spent a lot of runtime to simulate temporary services. Next, we simulate long-
time in data transmission. Since the task scheduler needs real- term services with MySQL and use SysBench as the TD to access
time scheduling, that is, whenever a task arrives, it is scheduled MySQL.
 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60 57

Algorithm 1 The Threshold Update Algorithm GNOME is used to start multiple terminals simultaneously in order
j
to simulate concurrency. We set up all containers/VMs to share all
Input: δ , receiv edRecordList;
p physical resources. For example, the resource of a container will be
j
Output: δp+1 ; 16 GB memory and 8 cores CPU when there is only one container
1: Calculate the amount of received tasks in current period; or VMs. If there are two containers or VMs, each container or VM
2: Get the amount of received tasks in last period from receive- will have, for instance, 4 GB memory and 4 cores CPU. Containers
dRecordList; use Ubuntu 16.04 as the base image and the operating system of
j
3: Tp ← the amount of received tasks in current period; VMs is also Ubuntu 16.04. The array size of Linpack is 200 × 200.
j Each level repeats for ten experiments to calculate the average
4: Tp−1 ← the amount of received tasks in last period;
j j value of the result to avoid result occasionality. As we expected,
5: if Tp ≥ Tp−1 then
Fig. 5 shows that the performance of containers/VMs decreases
6: Calculate the average per period resource quote of each
with an increase in the number. Containers and VMs decrease with
task;
j an equal rate from one to three, while from four to ten, the decrease
7: Calculate δp+1 using Eq. (15);
in the containers is slower than the VMs, which proves that in the
8: else
case of high concurrency, the CPU performance of a container is
9: δpj +1 ← δpj −1 ; better than VMs’. Due to the performance limitations of the server,
10: end if
j when we create 11 VMs, some VM failures render the complete
11: return δp+1 ;
result of 11 VMs unable to obtain.
Lastly, we simulate long-term services. In the long-term ex-
perimental environment, the resource allocation is the same as
the configuration of the Linpack test. We employ MySQL as the
long-term service application. We set up one container/VM to host
one MySQL service. Note that we are using MySQL images directly
instead of Ubuntu image as the base image. The database engine is
configured to use InnoDB. Before the test, we use SysBench to insert
1,000,000 items in each database for a pressure test. Each MySQL
has been tested by 100 threads to send requests concurrently and
address a total of 100,000 requests.
Because of the average allocation of resources, the same level
of the container or VM results are very close. Figs. 6(a) and 6(b)
adapt the averaged value for comparison. We also did not test
11 containers/VMs, since some VM failures occurred when we
are creating 11 virtual machines. Fig. 6(a) shows the per-request
Fig. 5. KFLOPS of containers/VMs. execution time as a function of the number of containers/VMs
simulated by the OLTP of SysBench. The average execution time
5.1. Experimental configuration for each request increases with the number of containers/VMs
increasing. However, containers perform is better than VMs from
We use a server with 16 GB of memory and an E5-2620v4@ 1 to 10; the increase rate is slower than VMs. Fig. 6(b) plots the
average number of transactions per second for all containers/VMs
2.10 GHz (a total of 8 cores and 16 threads) processor as a fog
on each level. Initially, containers are considerably better than
node, the operating system of the server is ubuntu16.04. In addi-
VMs, and VMs later gradually narrow the gap with containers.
tion, we use Docker 1.12.6, QEMU 2.5.0 and Libvirt 1.3.1 for our
experiments.
5.3. Task scheduling and resource management

5.2. Results In our simulation, there are 20 TDs to connect with a fog node,
and each TD sends 50,100, 150, 200, 250 requests to the fog node
First, we used Linpack [25] to test the performance of containers in different experiments respectively. The configuration of the fog
and VMs under the concurrency condition. In this experiment, node is set with a schedule cycle of 100 ms. There are three types

Fig. 6. Per-request execute time and transactions of containers/VMs.

58 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60

Fig. 7. Residual energy of TDs.

3. Dynamic threshold + Energy balancing (DT–EB): DT–EB

uses dynamic threshold for task scheduling, which proposed
by this paper and energy balancing for power management.
This approach considers the high concurrency of fog nodes
as well as the average transmission power of TDs.
4. Dynamic threshold + Maximum power (DT–MP): DT–MP
uses dynamic threshold and maximized transmission power.
This approach considers the high concurrency of tasks with-
out energy balancing.

Figs. 7 and 8 show the residual energy and the transmission

time of TDs. Through the energy-balanced scheduling strategy
proposed in this paper, the residual energy of each TD is evenly
distributed. For the maximum transmission energy strategy, the
energy of some TDs is exhausted prematurely. When other TDs
have a large amount of energy, these TDs can only enter the
Fig. 8. Average transmission time.
dormant state and wait for their charge. And for the transmission
time, they only differ by a few milliseconds and do not have a great
impact on the entire data transmission.
of data requests, 10, 20, and 30 kB, respectively. In addition, hk = Figs. 9(a) and 9(b) plot the number of task concurrency by 10
−40 · (1/50)4 , where −40 represents the path-loss constant and MIPS and 20 MIPS respectively. The results suggest that DT–EB
1 represents the distance between TDs and the fog node, 50 is and DT–MP, which employ the dynamic threshold, can increase
maximum transmission distance of TDs. The bandwidth w of each the number of accepted tasks. Due to the lack of ability to learn
TD is 1 MHz, o = 10−13 W, θ = 5. The compared task scheduling and adjust in terms of situation of requests, the approaches of
algorithms are as follows: fixed threshold causes that most of tasks will be distributed to
the fog node and quickly run out of its resources in the first few
1. Fixed threshold + Energy balancing (FT–EB): FT–EB uses periods, leading the upcoming tasks with insufficient resources to
be assigned while many of those tasks can be completed using
fixed thresholds which is the most direct and simplest way
fewer resources. However, the approach of dynamic threshold
for online task scheduling and energy balancing proposed in can constantly adjust the threshold to find the best value for the
this paper. current situation of requests. By updating threshold, some tasks
2. Fixed threshold + Maximum power (FT–MP): FT–MP uses that require a lot of resources will be distributed to the cloud, and
fixed thresholds for task scheduling and maximizes trans- thus the fog node can save the resources for those tasks that require
mitting power for data uploading. smaller resources.

Fig. 9. The total number of accepted tasks.

 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60 59

6. Conclusion [19] S. Nadgowda, S. Suneja, N. Bila, et al., Voyager: Complete container state
migration, in: IEEE 37th International Conference on Distributed Computing
Systems, 2017, pp. 2137–2142.
This paper proposes multi-cloud to multi-fog architecture based [20] S. Osman, D. Subhraveti, G. Su, et al., The design and implementation of zap: a
on container and its corresponding implementation. It aims to system for migrating computing environments, Oper. Syst. Rev. 36 (6) (2002)
address the challenge of private cloud developing. The key point 361–376.
of this architecture is using container as the resource unit to [21] Y. Luo, B. Zhang, X. Wang, et al., Live and incremental whole-system migration
of virtual machines using block-bitmap, IEEE International Conference on
reduce the response time of requests and designing the cloud Cluster Computing (2008) 99–106.
agent component to solve the problem caused by multiple ten- [22] Y. Xiao, M. Krunz, Qoe and power efficiency tradeoff for fog computing
ants. In addition, a task scheduling algorithm based on energy networks with fog node cooperation, in: IEEE Conference on Computer Com-
balancing has been proposed to advance the life of WSNs without munications, Atlanta, GA, USA, 2017.
[23] A.-C. Pang, W.-H. Chung, T.-C. Chiu, et al., Latency-driven cooperative task
increasing the delay of tasks. We evaluate the performance of computing in multi-user fog-radio access networks, in: IEEE 37th Interna-
virtual machines and containers under the high concurrency, and tional Conference on Distributed Computing Systems, Atlanta, GA, USA, 2017.
the result shows that containers is better than virtual machines. [24] A. Miettinen, J. Nurminen, Energy efficiency of mobile clients in cloud com-
Furthermore, we simulate the energy cost and task scheduling in puting, in: USENIX Association Berkeley, HotCloud’10 Proceedings of the 2nd
USENIX Conference on Hot Topics in Cloud Computing, Boston, 2010, 4–4.
the TDs and fog node respectively, the algorithm proposed in this [25] Linpack, https://github.com/ereyes01/linpack.
paper can effectively balance the energy of TDs in the network.
Juan Luo (M’10) received the Bachelor’s degree in elec-
Acknowledgments tronic engineering from the National University of De-
fense Technology, Changsha, China, in 1997, and the Mas-
ter’s and Ph.D. degrees in communication and informa-
This work was supported by the National Natural Science Foun- tion systems from Wuhan University, Wuhan, China, in
dation of China (61672220), Key scientific and technological re- 2000 and 2005, respectively. Currently, she is a Professor
search and development plan of Hunan Province (2017GK2030), and Doctoral Supervisor with the College of Computer
Electronic Information and Control of Fujian University Engineer- Science and Electronic Engineering, Hunan University,
Changsha. She is also the Director with the Department
ing Research Center, Minjiang University (EIC1701).
of Network and Information Security, Hunan University.
From 2008 to 2009, she was a Visiting Scholar with the
References University of California at Irvine, Irvine, CA, USA. She has authored/coauthored more
than 40 papers. Her research interests include wireless networks, cloud computing,
[1] Cisco, Cisco global cloud index: Forecast and methodology, 2015–2020. and wireless sensor networks. Dr. Luo is a Member of the Association for Computing
[2] F. Bonomi, R. Milito, J. Zhu, et al., Fog computing and its role in the internet Machinery (ACM) and a Senior Member of China Computer Federation (CCF).
of things, in: Edition of the Mcc Workshop on Mobile Cloud Computing ACM,
2012, pp. 13–16. Luxiu Yin received the bachelor’s degree in Computer
[3] L.M. Vaquero, L. Rodero-Merino, Finding your way in the fog: Towards a com- Science and Technology in 2011 from Changsha Univer-
prehensive definition of fog computing, ACM Sigcomm Comput. Commun. sity, Hunan, China. He achieved masters degree in Com-
Rev. 44 (5) (2014) 27–32. puter software and theory in 2015 from Hunan Normal
[4] B.I. Ismail, D. Jagadisan, M.F. Khalid, Determining overhead, variance & isola- University, Hunan, China. He is currently receiving for a
tion metrics in virtualization for iaas cloud, in: Data Driven e-Science, 2011, Ph.D. degree in college of information science and en-
pp. 315–330. gineering, Hunan University, Hunan, China. His research
[5] M. Peng, S. Yan, K. Zhang, et al., Fog-computing-based radio access networks: is focused on cloud computing and wireless virtual net-
Issues and challenges, IEEE Netw. 30 (1) (2016) 46–53. work.
[6] K. Liang, L. Zhao, X. Chu, et al., An integrated architecture for software defined
and virtualized radio access networks with fog computing, IEEE Netw. 31 (1)
(2015) 80–87.
[7] X. Hou, Y. Li, M. Chen, et al., Vehicular fog computing: A viewpoint of vehicles Jiuyu Hu received the B.E. degree in communication engi-
as the infrastructures, IEEE Trans. Veh. Technol. 65 (6) (2016) 3860–3873. neering from Hunan University, Changsha, China, in 2009,
[8] Z. Li, B. Chang, S. Wang, et al., Dynamic compressive wideband spectrum and the M.E. degree in computer applications technology
sensing based on channel energy reconstruction in cognitive internet of from Chongqing Jiaotong University, Chongqing, China, in
things, IEEE Trans. Ind. Inf. (2018) 1–1. 2013. Currently, she is pursuing the Ph.D. degree at the
[9] J. Nie, J. Luo, L. Yin, Energy-aware multidimensional resource allocation algo- College of Computer Science and Electronic Engineering,
rithm in cloud data center, KSII Trans. Internet Inf. Syst. 11 (9) (2017) 4165– Hunan University. Her research interests include wireless
4187. networks.
[10] K. Xie, J. Cao, X. Wang, et al., Optimal resource allocation for reliable and
energy efficient cooperative communications, IEEE Trans. Wirel. Commun. 12
(10) (2013) 4994–5007.
[11] B. Yang, Z. Li, S. Chen, et al., Stackelberg game approach for energyaware Chun Wang received the B.E. degree in China University
resource allocation in data center, IEEE Trans. Parallel Distrib. Comput. 27 (12) of Mining and Technology in 2013 and 2017from College
(2016) 3646–3658. of computer science and technology, Xuzhou, China. She
[12] X. Sun, N. Ansari, Edgeiot: Mobile edge computing for the internet of things, is currently receiving for a Ph.D. degree in college of
IEEE Commun. Mag. 54 (12) (2016) 22–29. information science and engineering, Hunan University,
[13] L. Hou, S. Zhao, X. Xiong, et al., Internet of things cloud: Architecture and Changsha, China. Her research is focused on wireless
implementation, IEEE Commun. Mag. 54 (12) (2016) 32–39. networks.
[14] Y. Mao, J. Zhang, S. Song, et al., Stochastic joint radio and computational
resource management for multi-user mobile-edge computing systems, IEEE
Trans. Wirel. Commun. 16 (9) (2017) 5994–6009.
[15] Y. Mao, J. Zhang, K.B. Letaief, Dynamic computation offloading for mobile-
edge computing with energy harvesting devices, IEEE Journal on Selected Xuan Liu received the B.E. degree and M.E. degree in
Areas in Communications 34 (12). electronic engineering from the National University of
[16] X. Ma, S. Zhang, W. Li, et al., Cost-efficient workload scheduling in cloud Defense Technology, Changsha, China, and the Ph.D. de-
assisted mobile edge computing, in: IEEE/ACM International Symposium on grees in Hong Kong Polytech University, respectively.
Quality of Service, Vilanova i la Geltru, Spain, 2017, pp. 1–10. Currently, she is a Professor with the College of Computer
[17] W. Felter, A. Ferreira, R. Rajamony, et al., An updated performance comparison Science and Electronic Engineering, Hunan University,
of virtual machines and linux containers, in: 2015 IEEE International Sympo- Changsha. Her research interests include wireless net-
sium on Performance Analysis of Systems and Software, 2015, pp. 171–172. works, cloud computing, and wireless sensor networks.
[18] F. Ramalho, A. Neto, Virtualization at the network edge: A performance Dr. Luo is a Member of the Association for Computing Ma-
comparison, in: IEEE 17th International Symposium on A World of Wireless, chinery (ACM) and a Senior Member of China Computer
Mobile and Multimedia Networks, 2016, pp. 1–6. Federation (CCF).
60 J. Luo, L. Yin, J. Hu et al. / Future Generation Computer Systems 97 (2019) 50–60

Xin Fan received a bachelor’s degree in science and Haibo Luo received the B.E. degree in Communication En-
technology from Anhui Jianzhu University in 2017. He gineering from Wuhan University of Technology, China,
is currently acquiring a master’s degree from the Hu- in 2006, the M.E. degree in Information and Communica-
nan University for information science and engineering. tion Engineering from Hunan University, China, in 2009.
His research focuses on cloud computing and resource He is currently studying for a Ph.D. degree in College of
scheduling in wireless data centers. Physics and Information Engineering, Fuzhou University,
Fuzhou, China. His research interests include Internet of
Things, Edge computing and Mobile computing.