Performance Analysis of Hyperledger Fabric 2.0 Blockchain Platform
Performance Analysis of Hyperledger Fabric 2.0 Blockchain Platform
0 Blockchain
Platform
Julian Dreyer, Marten Fischer, Ralf Tönjes
Lab for RF-Technology and Mobile Communications
University of Applied Sciences Osnabrueck
Osnabrueck, Germany
[email protected]
[email protected]
[email protected]
ABSTRACT generation of Blockchain technology arrived. The most notable
Hyperledger Fabric is currently one of the most popular business difference to the first Blockchain generation is the introduction of
Blockchain platforms. With its included functionality of executing so-called "smart contracts". Software Developers use these to imple-
custom smart contracts, Fabric has become one of the most widely ment their own functionality and execute their custom programs
used frameworks, e.g. for industry 4.0 applications. Though, most on the Blockchain and thus interacting directly with it. Using this
applications require a previously known data throughput. For a novel paradigm, countless opportunities for custom applications
potentially interested developer, it is not trivial to decide, whether are possible.
a given Fabric network configuration will meet the required ex- Hyperledger Fabric makes use of the smart contract paradigm
pectations in regards to performance. Thus this work shows a and offers a fully operational private Blockchain platform, which is
performance analysis for Hyperledger Fabric v2.0 and focusses on currently one of the most established publicly available Blockchain
the evaluation of throughput, latency and error rate, together with solutions. Firstly introduced by IBM and later handed to the open-
the overall scalability of the Fabric Blockchain platform. The results source community, Hyperledger Fabric published the official ver-
show, that Fabric v2.0 outperforms previous versions in almost any sion 2.0 in February 2020. Recent work gave a broad evaluation of
regard in addition to an improved smart contract lifecycle. the performance e.g. of Hyperledger Fabric v0.6, v1.0 and v1.4 [4].
The importance of the performance evaluation is given because
CCS CONCEPTS the real-world applications require in-depth planning of the net-
work configuration. A smart contract developer needs to know in
• Networks → Network performance analysis.
advance, whether the requirements in regards to throughput and
KEYWORDS delay of the Blockchain network can be fulfilled.
Thus, this paper addresses the performance of Hyperledger Fab-
Blockchain, Hyperledger Fabric v2.0, Performance analysis, Scaling, ric v2.0 in regards to the most common evaluation characteristics,
Distributed Ledger Technology, throughput, latency, error rate namely throughput, latency and error rate. According to the au-
ACM Reference Format: thors knowledge this is the first evaluation of the Hyperledger v2.0
Julian Dreyer, Marten Fischer, Ralf Tönjes. 2020. Performance Analysis of framework. The particular focus lies on the performance analysis of
Hyperledger Fabric 2.0 Blockchain Platform. In Cloud Continuum Services varying network parameters such as the number of peers, orderers,
for Smart IoT Systems (CCIoT ’20), November 16–19, 2020, Virtual Event, Japan. organizations and block size. This paper therefore provides a large
ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3417310.3431398
performance result set for various Hyperledger Fabric v2.0 network
configurations.
1 INTRODUCTION
The rest of this paper is organized as follows: Section 2 pro-
The Blockchain Technology offers countless opportunities to in- vides an overview of the previously conducted studies in regards to
crease the data security and redundancy of modern information Hyperledger Fabric performance for previously released versions.
systems. With its introduction by Nakamoto [5], the first generation It also gives an overview of recent publications that address the
of Blockchain systems came up. Its main focus lied on the cryptocur- performance and improvements of Hyperledger Fabric in general.
rency domain. With its inherent transparency and tamper-proof The particular components and improvements within Hyperledger
data structure, the Blockchain offered new ways for digital payment. Fabric v2.0 are described within section 3. It also introduces the test
With the introduction of Ethereum by Buterin in [1], the second strategies and key performance indicators (KPI), which will later be
Permission to make digital or hard copies of all or part of this work for personal or used to evaluate the performance of the Blockchain network. Said
classroom use is granted without fee provided that copies are not made or distributed evaluation is being described in section 4 and evaluated in section
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
5. Finally, Section 6 concludes the paper.
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from [email protected]. 2 RELATED WORK
CCIoT ’20, November 16–19, 2020, Virtual Event, Japan Various performance evaluations of the Hyperledger Fabric Blockchain
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-8131-4/20/11. . . $15.00 platform exist in general. One central problem which applies to
https://doi.org/10.1145/3417310.3431398 all of the performance evaluations is the objectivity of the smart
CCIoT ’20, November 16–19, 2020, Virtual Event, Japan J. Dreyer, M.Fischer, R. Tönjes
contracts. To evaluate the performance, most often a custom smart 3 TECHNOLOGIES AND TEST METHODS
contract is developed and executed multiple times. Depending on Hyperledger Fabric offers a a range of functionalities in regards
the concrete implementation, the test results may vary by a large to customization and possible use cases. It enables a developer to
extent. Thus the objectivity and the comparability of the results are set up a custom Blockchain business network, consisting of het-
limited. erogenous clients with different roles assigned to them. It also al-
Nasir et. al conducted a study by utilizing the previously de- lows to create, deploy and execute smart contracts on the platform.
scribed methodology in [6]. They developed and deployed a custom Fabric ensures a flexible configuration by deploying an execute-
smart contract, which is capable of transferring virtual money. The order-commit transaction flow. An endorsement peer node will first
overall goal of their work is to evaluate the performance of one execute a transaction and determine its read-write set. After every
single Blockchain peer node without the inclusion of consensus endorsement peer executed the transaction, the results get sent
algorithms. For the evaluation, they also used the common KPIs to the ordering service, which orders and sorts the transactions,
of "execution time", "latency" and "throughput". The results show, depending on their timestamps and hierarchical order. Finally, the
that previous versions of Hyperledger Fabric, namely v0.6 and v1.0 ordered results get broadcasted via the gRPC Protocol to all peers
improved in regards to throughput and latency. With an increas- within the Fabric Blockchain channel [12]. Each one will then com-
ing number of incoming transactions, the performance diminishes mit the results to their own copy of the ledger and thus achieving
rapidly. With 10,000 incoming transactions per second (tps), the consensus. The specific roles involved in the transaction flow will
execution time of a single transaction lies well above 60s for v0.6 be described in the following.
and 59s for v1.0 respectively.
1. Peers
Kuzlu et. al evaluated the performance of Hyperledger Fabric
A peer within the Fabric Blockchain network can either be an
v1.4 by varying the incoming workloads in terms of transaction
endorsement or a committing peer. An endorsement peer is respon-
speed, number of transactions and transaction types [4]. The net-
sible for endorsing the transactions which is sent to them by the
work was set up without regard to the consensus mechanism and
ordering service. By executing the invoked smart contract code, a
no network configuration has been given. For the benchmarking
read-write set is created which is unique to the particular transac-
operation, Hyperledger Caliper [10] was used since it is an officially
tion. A committing peer on the contrary will not execute the smart
provided benchmarking tool for Hyperledger Fabric v1.4. The re-
contract at all. It will only receive the ordered transactions from the
sults show that the average throughput lies below 200 tps for an
orderers and commit them to its copy of the ledger. Any peer can
"Open" operation when applying non-simultaneous transactions.
be chosen to be an anchor peer, which is then responsible for the
The latency also increases rapidly after applying a transaction rate
communication with the ordering service or other organizations
above 200 tps. When considering simultaneous transactions the
within the same channel. There must exist at least one anchor peer
throughput does not exceed 50 tps overall.
per organization.
Pongnumkul et. al investigated the differences in regards to the
performance of Ethereum and Hyperledger Fabric v0.6 [8]. They 2. Organizations
describe a basic principle for evaluating the performance of the re- Every Fabric Blochcain channel can be separated into several
spective Blockchain platform by requiring a common smart contract organizations, each of which owns a certification authority (CA).
functionality and scaling the network afterward. For the evaluation, There can be multiple peers within each organization that execute
they also used the common performance indicators throughput and validate the transactions. Though, a single peer can be assigned
and latency. The deployed smart contracts revolve around the cre- to one organization only. It has to issue specific keys and certificates
ation of a money transfer infrastructure. While Fabric exceeds its from the CA of the organization to communicate with the other
maximum throughput at 100 incoming transactions at 299.85 tps, participants of the network.
Ethereum only reaches a maximum throughput of 38.93 tps. By 3. Orderers
increasing the incoming transactions per second, the latency also The ordering service is the central component to ensure that the
increases linearly in case of Fabric and flattens out at 400s in case transactions within the Fabric Blockchain channel get sorted cor-
of Ethereum. rectly. Fabric v2.0 recommends the use of the "Raft" ordering service
Various attempts to improve the performance of Hyperledger and thus replaces the now deprecated Kafka ordering service [11].
Fabric have been performed e.g. by Gorenflo et al. [3]. By intro- Each orderer node will order and append the endorsed transactions
ducing different improvements to the fundamental architecture to a new block for the Blockchain. After either the upper "block
of Hyperledger Fabric, the authors managed to scale the through- size" limit is reached, which describes the maximum number of
put performance from 3,000 tps to almost 20,000 tps while also transactions a single block can hold, or the "block timeout" is met,
decreasing the latency. the block is being validated and appended to the Blockchain. This
Thakkar et al. [9] investigated multiple performance bottlenecks is done by broadcasting the block information to all anchor peers
Hyperledger Fabric v1.0 which can be improved. By making use of of each organization, which then broadcast the information to all
aggressive caching or parallelizing the verification of endorsement peers within the particular organization internally.
policies, the authors were able to scale the performance of Hyper- 4. Fabric v2.0 innovations
ledger Fabric v1.0 by a factor of 16. Said improvements were later With the v2.0 release of Hyperledger Fabric, new features and
included into later versions of Hyperledger Fabric. improvements were introduced [13]. Most notable in regards to
performance is the new lifecycle management of the peers. New
Performance Analysis of Hyperledger Fabric 2.0 Blockchain Platform CCIoT ’20, November 16–19, 2020, Virtual Event, Japan
smart contracts, which are developed against the Fabric v2.0 SDKs Though, all these performance indicators are also depending on
are now packaged in a common tape archive (TAR) format, unlike the smart contract, which gets executed to evaluate their perfor-
previous versions of Fabric which used a proprietary packaging mance. If the complexity is too high, the comparability to other
format for every language. Every organization is now responsible performance evaluations, e.g. of previous Fabric versions, is not
for approving the metadata of the to-be-installed smart contract, given. Thus, for this paper a custom smart contract has been created
after it has been loaded onto every endorsement peer within the with the interface in listing 1, using the Fabric Java Chaincode SDK.
channel. If every organization approved the smart contract, it gets The DefaultAsset class is only containing a single value at-
committed to the ledger of each peer and is then executable. By tribute, which holds the particular random data of the object itself.
utilizing specific configuration parameters within the configuration Each DefaultAsset object also contains a unique id object for
files, the approval process can be customized. querying. The sole functionality of the smart contract is to imple-
5. Test methods ment a basic Create and Read (CR) scheme, which can be easily
The performance of a Hyperledger Fabric business network can compared to existing performance evaluations. Future work can
be measured by issuing multiple transactions in parallel which the also implement a smart contract based on this scheme and use it
network has to validate. By measuring the number of transactions for comparable testing. Considering the smart contract, which has
that get validated per second, the "throughput" can be identified. been used by Pongnumkul in [8], the complexity has been higher
This metric can be primarily used to evaluate the superficial per- thus making it not comparable with the smart contract interface
formance of a network, by comparing it to other Blockchain plat- used in this paper.
forms like Ethereum. Such a comparison has been described by While the Kafka ordering service is now deprecated since Fabric
Pongnumkul in [8]. Figure 1 depicts the general concept of the v2.0, the Raft ordering service should be used. Fundamentally, the
throughput metric. ordering algorithm has not changed, thus resulting in no theoretical
impact on the performance of the network itself.
4 PERFORMANCE ANALYSIS
The purpose of this work is to get a definitive evaluation of how
the scaling of different network participants influences the perfor-
mance of the Fabric Blockchain network. In particular, the scaling
of the number of peers, organizations, orderers and block size has
been considered, since the number of each component is config-
urable. A custom tool [2], purposely made for Fabric 2.0, has been
developed to automatically generate the required business network
containing the desired number of individual network components.
Figure 1: Throughput performance metric It also installs the test smart contract automatically. Using this tool,
a broad spectrum of custom made network configurations can be
Another metric is the latency/delay of the Blockchain network. generated in an instant.
It gives a concrete measurement, when the data, which gets added After the setup is complete, the network can be used and tested.
to the Blockchain, is available for query. If the latency was too high, By utilizing the generated invoke and query scripts, the network
the data could not be read within a short time interval after adding can be directly addressed and interacted with. To measure the dif-
the data to the Blockchain, thus resulting in errors. ferent impacts on the performance metrics, four different incoming
The last metric which gets measured within this work is the transaction rates, ranging from one transaction per second (tps)
error rate. There are multiple ways for a transaction to fail within a to 1000 tps, have been considered. The incoming transaction rate
Blockchain network. This paper focusses on measuring the software- will be called T PSin in the following and the transaction rate of the
side errors, meaning that external errors due to signal interference network itself will be calledT PSout . The transactions are sent in par-
or packet loss get excluded from the overall error rate. The soft- allel to generate a real-world application scenario where multiple
ware errors mainly revolve around measuring the number of race- nodes issue different transactions at the same time. All transactions
conditions, which cause a validation error due to an inconsistent will call the createDefaultAsset(String value) method of the
read-write set. Other software errors like validation errors are also installed smart contract and pass a random value as an argument, in
included in the overall error rate. order to append new data. All transaction types will be invoke since
only invoke calls will trigger the ordering and consensus process
Listing 1: Java Interface for the test smart contract
within the network, thus generating the required load. For every
class DefaultContract implements ContractInterface { transaction rate, each external client will issue 100 transactions. The
@Transaction() network will then validate the transactions and append new blocks
public boolean defaultAssetExists(String _id); to the Blockchain, thereby generating the performance metrics.
The performance metrics cannot be measured directly and have
@Transaction() to be derived by measuring the execution time of the transactions,
public boolean createDefaultAsset(String value); the commit times of the individual blocks and the error rate. The
} following formulas can be used to get the performance metrics
CCIoT ’20, November 16–19, 2020, Virtual Event, Japan J. Dreyer, M.Fischer, R. Tönjes
Table 1: Test network configurations transactions graphically. Notice that the x-Axis also includes the
scaling of the block size and tps. In particular, the scaling of the
Organizations Peers Orderer/Kafka nodes peers can be best seen within batches of four bars beginning with
1 2,4,8,16 1 the blue bar on the left and ending with the yellow bar on the right.
2 2,4,8,16 1 Each bar shows which effects and impacts a doubling of the peer
4 4,8,16 1 count has on the execution times. By increasing the number of
8 8 1 peer nodes within a given organization, the execution time of the
2 2 2,4,8 transaction increases significantly. With higher T PSin rates above
4 4 2,4,8 100 tps, the difference between the lowest configuration containing
two peers and the highest configurations with 16 peers is as large as
Each test network configuration has been evaluated against the 3535ms in the scenario bs50-100tps. This behavior is expected since
previously described transaction rates, which are summarized in the introduction of more peers leads to a higher communication
Table 2. The following section will describe the evaluation results und synchronization overhead within the network. When analyz-
in detail. ing the same scenario with two organizations, the overall tendency
in regards to the performance impact continues. Figure 3 shows the
5 EVALUATION same evaluation as in figure 2, but now includes two organizations.
To derive a complete evaluation of the different performance im- All the peers are equally distributed among all organizations. In
pacts of the various network components, each of them will get the particular scenario for 16 peers, each organization therefore
Performance Analysis of Hyperledger Fabric 2.0 Blockchain Platform CCIoT ’20, November 16–19, 2020, Virtual Event, Japan
Figure 7: Transaction times with scaling orderer nodes orderers or organizations does influence the error rate only within
with four organizations the margin of error. The results show, that a larger block size will
result in less errors than a smaller block size.
Nasir et al. showed in [6] a performance comparison between
Fabric v0.6 and v1.0. Those results can be directly compared with the
transaction times from this work. Thus, figure 10 extends the figure
8 of [6] with the measurements for Fabric v2.0. The comparison
shows, that Fabric v2.0 improved the overall performance of the
system even more.
is derivable (s. figure 8). Also the error rate does not show any clear
Figure 10: Transaction times of Fabric v0.6, v1.0 and v2.0
notable impacts when scaling the orderer nodes.
The previous figures for the block commit times also hint towards
a performance impact which is caused by the block size. A higher
block size will lead to a higher block commit time. This is due to 6 CONCLUSION
the block including more transactions and thus requiring more The performance of Hyperledger Fabric has been further improved
validation time. So the block size stands in direct correlation with with the release of version 2.0. The conducted performance study in
the overall latency of the system. this work showed, that the internal improvements to v2.0 of Fabric
3. Performance metrics lead to an increase in the overall throughput of the network. The
By applying the formulas of section 4, the impact of the peers measured performance factors such as latency and error rate also
in conjunction with a high T PSin rate gets emphasized even more. experienced an improvement in comparison to previous versions
Figure 9 shows the throughput T PSout in dependency of the T PSin of Fabric. With the new lifecycle management of Fabric 2.0, the
rate. The throughput is highly dependent on the number of peers, validation of transactions is faster than before. Also, the probability
as can be seen from the transaction times. With more endorsement of errors occurring during transaction validation is also lowered.
peer nodes being active within the network, less throughput is to This work also showed that by scaling the different components
be expected. On average the throughput reached 22,071 tps among within the network such as peers, orderers and organizations, the
all test cases, including those with high T PSin rates of over 100 tps. performance varies by a large factor. It is not trivial to decide
The latency is further dependent on the block size, as well as which configuration is fitting for a given requirement in regards to
on the number of endorsement peer nodes. Smaller block sizes throughput expectations.
generally allow a significantly lower latency. Though, it is to be Future work may focus on examining the real-world performance
noted that the block size should be selected for the expected T PSin of Fabric e.g. in an Industry 4.0 appliance. A guideline for an ideal
rate, as a fast filling of the block may result in a frequent validation configuration of the network might be provided. This work did
process thus causing additional latency. not include the performance of query calls, which are generally
The previous subsection already discussed the error rate for each faster, as they do not require any changes to the ledger. Thus future
particular component. Generally, only the block size and the T PSin work may focus on evaluating the overall query performance of
rate are highly influential to the error rate. The number of peers, Hyperledger Fabric v2.x or above.
Performance Analysis of Hyperledger Fabric 2.0 Blockchain Platform CCIoT ’20, November 16–19, 2020, Virtual Event, Japan