Resource Analysis of Ethereum 2.0 Clients: Mikel Cortes-Goicoechea Luca Franceschini Leonardo Bautista-Gomez
Resource Analysis of Ethereum 2.0 Clients: Mikel Cortes-Goicoechea Luca Franceschini Leonardo Bautista-Gomez
0 Clients
Mikel Cortes-Goicoechea Luca Franceschini Leonardo Bautista-Gomez
Barcelona Supercomputing Center Barcelona Supercomputing Center Barcelona Supercomputing Center
Barcelona, Spain Barcelona, Spain Barcelona, Spain
[email protected] [email protected] [email protected]
Abstract—Scalability is a common issue among the most Proof-of-Work (PoW) to Proof-of-Stake (PoS). This change
used permissionless blockchains, and several approaches have allows blocks to be produced in a more sustainable way, saving
been proposed accordingly. As Ethereum is set to be a solid electricity while implementing a more exhaustive network
arXiv:2012.14718v1 [cs.CR] 29 Dec 2020
1
to save clients’ logs to a text file. The logs were parsed through
python scripts [5], allowing to extrapolate the syncing speed
of the clients. Metrics and slot times were plotted through
python-matplotlib scripts [6], and saved into CSV files.
A. Evaluation platform
Tests have been run on two nodes, each node with an
Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz with 1 core
and two threads. Each client has been run on a single node
with 4.8 GB of total memory available and 34GB of storage
(40GB total, of which 6GB used for the operating system).
Clients have been run on the Medalla Test Network using
their default configuration. No additional flags or configuration
have been used with the exception of the flag on Teku that
limits the JVM memory and rising the heap size of Lodestar Fig. 1: Slot synchronization of Eth2 clients
on the package.json. Teku ran allocating 2 GB to the JVM,
as allocating more than 2GB would make the server crash. Figure 1 shows the synchronization slot as the client runs
Similar issues occurred with Lodestar. from genesis. We tried to run all clients for over 24 hours,
with the exception of Lodestar, which suffered from multiple
B. Client versions and running period
crashes and had to be restarted several times. We can see that
Clients have been run with the following versions: Lighthouse appears to be the fastest syncing client, though
• Teku: v0.12.14-dev-6883451c [7] its syncing process stops around hour 24 due to reaching the
• Prysm: v1.0.0-beta.1-4bc7cb6959a1ea5b [8] maximum storage capacity of the server (See Section III-G).
• Lighthouse: v0.3.0-95c96ac5 [9] After Lighthouse stopped its syncing process, Prysm was able
• Nimbus: 0.5.0-9255945f [10] to catch up and sync faster than the other clients with a steady
• Lodestar: commit 40a561483119c14751 [11] progress throughout the whole test. Nimbus and Lodestar seem
Clients have been running during the periods depicted in to be the ones that take more time to sync. More importantly,
Table I. Teku and Lighthouse were run several times in order it can be seen that all clients experience a sharp increase in
to compare the resource usage of two different executions. syncing speed around slot 100,000. The fact that all clients
observe the same change of steepness in the syncing curve
TABLE I: Client Running Periods around this slot might suggest that this phenomena is inherited
from the network conditions (See Section III-H).
Client Start Time End Time
Teku 2020-11-09 17:25:12 2020-11-10 17:34:45
Prysm 2020-11-04 18:34:12 2020-11-06 09:34:34 B. Peer connections
Lighthouse 2020-11-02 17:17:51 2020-11-04 02:57:38
Nimbus 2020-11-04 18:40:35 2020-11-06 10:23:04
We kept track of the number of peer connections for each
Lodestar 2020-11-08 20:19:02 2020-11-09 08:54:04 client during the experiment. While this is a parameter that
can be tuned, we used the default values for each client, as
this is what most user would do on a regular basis. We can
III. A NALYSIS see in Figure 2 that Teku connects to the highest number of
In this section we go over the results obtained during the peers (i.e., over 70 peers) than any other client and keeps a
runs of the five clients and we analyse their differences in stable number of connections during the entire experiment.
resource consumption and behaviour. Lighthouse is the client that connects with more peers after
Teku with about 50 peers. After 20 hours of execution, the
A. Client syncing number of connected peers drastically decreased (oscillating
First, we start by studying the synchronization time of all around 25 peers), and later rises its peer connection back to 50
clients. It is important to note that the synchronization time peers (hour 24). After hour 24 Lighthouse experienced sharp
of a client is not really a determining factor of its efficiency. drops and recoveries in peer connections. It is important to
First, this is a cost that it should be paid only once in the notice that Lighthouse was the first client to be run, from
lifetime of a client (assuming no hard failures that force a November 2dn to November 4th, while the Medalla Testnet
restart from scratch), but also because most clients can now was experiencing some issues related to non-finality [13]
start from a weak subjectivity state and be up and running which might explain the peer connection drops as well as
in under a minute. Nonetheless, synchronization time is very other strange phenomena observed in the Lighthouse run. This
relevant when comparing it with other resources metrics to try allowed us to do a more detailed analysis of the data. Nimbus
to uncover hidden effects or understand unexpected behaviour peers connections oscillate significantly, usually connecting
and this is the way we use it in this research. Other studies with over 15 peers and arriving up to 42 peers. Prysm
have also used syncing metrics in a similar way [12]. distinguishes itself for its stability of peer connections, as
2
it succeeds in achieving 30 peer connections with almost no
variation. When it comes to Lodestar, the client does not report
the number of peers until the moment in which it retrieves the
genesis block (hour 4), and from this point Lodestar’s peer
connections oscillate in the range of 20-30 peers.
3
usage begins to diminish and then becomes flat. This is due
to the storage limitations in the node.
E. Memory
Fig. 8: Memory consumption of Eth2 clients
When it comes to memory consumption, we can see in
Figure 8 how Teku and Prysm tend to have a similar behaviour,
by starting with a steep usage of memory consumption and F. CPU
settling with a similar high memory consumption in the long
run. Interestingly, we can notice sharp drops of memory usage From monitoring the CPU usage across the different clients,
for Prysm at somehow regular intervals, indicating some kind we noticed in Figure 10 that Nimbus has the lowest CPU
of periodic cleaning process, while the behaviour of Teku consumption overall, staying always under 50%. The CPU
is more constant. Nimbus distinguishes itself with a very usage of Prysm and Lodestar concentrates around 50%, while
low memory consumption with minimal changes. Lodestar the CPU usage of Lighthouse is most of the time above
memory usage is characterized by periodic spikes, and appears 50% and Teku constantly oscillates near 100%. Teku’s CPU
to be less demanding than the memory usage of Teku and usage is always high and stable in our experiments. There
Prysm. It can also be noticed that the first relevant spike might be several reasons for this. First, we performed our
corresponds to the moment in which the client actually starts experiments on a virtual machine with just one core of an
the syncing process. Lighthouse memory consumption is low Intel(R) Xeon(R) E5-2620 CPU (See Section II), which might
overall, but it showed some unstable behaviour. be significantly more limited than a contemporary node with
a more recent CPU generation and multiple cores. Second, we
limited the JVM heap size to only 2GB because larger JVM
heap sizes kept leading to crashes. With 2GB for the JVM
heap, Teku managed to run smoothly without issues. However,
2GB is quite tight for Medalla testnet and it is possible that a
significant portion of the time was spent in garbage collection,
leading to the observed high CPU utilization.
4
Prysm also shows a relatively constant CPU utilization
except for hour 35, where it drops to 0, as shown in Figure
11. During that time, we also observe a strong reduction on
network incoming traffic, which explains the drop on CPU
utilization. It is interesting to notice that this happens even
while Prysm keeps stable peer connections (See Figure 2).
G. Disk Usage
We plot the disk usage of all clients in Figure 14. This plot
shows some unexpected behaviour for some of the clients.
The most stable behaviour, in terms of disk usage among
Fig. 10: CPU usage of Eth2 clients all clients, seems to be Prysm according to this experiment.
Nimbus maintains a low storage usage for most of the time,
Nimbus also showed a constant CPU utilization most of the although it does experience a sharp increase in storage at
time, except for a short period of time around hour 7 where it some point. We investigate this increase in storage by isolating
witnessed a quite sharp drop in CPU usage, as it can be seen Nimbus’ disk utilization and comparing it with its CPU usage.
in Figure 12. We also can observe that this drop comes just We can see in Figure 15 that the increase in storage is quite
after a steep increase in blocks synced. In the other clients, dramatic, it goes from 1GB to over 4GB in a short period of
such an increase in synced blocks in a period of time is not time. Curiously, just after this storage increase, the CPU usage
followed by a CPU drop. This drop in CPU usage will be of Nimbus drops substantially. While it is true that periods of
further investigated later (See Section III-G). high storage intensity are often followed by idle CPU time,
note that the x-axis is denominated in hours. Such idle CPU
times just last for milliseconds or seconds after storage bursts,
so this solely it is unlikely to explain the whole drop of CPU
utilization. In addition, this is not observed in other clients
that also experience large increases in disk utilization.
5
to increment at the same speed than initially doing, keeping
the same growth rate for several hours, to then drop sharply at
around hour 8. Interestingly, extrapolating the initial storage
growth rate would place Lighthouse at the same spot after
9 hours of syncing, as if the whole storage used during the
perturbation was just temporary data that once everything
solved, could be discarded without losing any real information.
After 20 hours, however, Lighthouse observes another sharp
increase in disk usage, but this time it keeps going up until
it consumes the whole 34GB of space in the node, at which
point it becomes a flat line for the next 10 hours.
6
part of the chain where there are no forks to explore and During this period, we observe a sharp increase in the
everything is deterministic. We also ran Lighthouse twice to disk usage of Teku, Lighthouse and Nimbus, while Lodestar
check if the disk behaviour was deterministic and it is for most shows the exact opposite and the behaviour of Prysm seems
of the period between slots 75,000 and 120,000, as shown in to be unaffected. For Lighthouse and Teku, we can see that
Figure 18. However, Lighthouse behaviour does change a bit the sharp increase in Disk usage is followed by a similarly
after slot 210,000: the second run managed to reduce storage sharp drop towards the end of this period. Considering the
usage way before the first run. Although, this shows that this relatively similar behaviour of both clients, it is interesting to
behaviour is not just an artifact of our experiments, this does notice how Teku reduced the time of higher disk usage, while
not explain why the disk grows so fast and what is the root- Lighthouse keeps running several hours with additional data
cause of the perturbation. before dumping it. This is due to the dual-database system that
Lighthouse and some other clients use: a Hot database that
stores unfinalized BeaconState objects, and a cold database
that stores finalized BeaconStates. As they sync through a large
patch of non-finality, their hot databases grow large until they
reach finality and then migrate this state into the cold database.
7
To look deeper into this question, we used Teku logs to analyse finality period, we have demonstrated in this paper that it
the number of times a block was queued and/or processed for is possible to identify such a network disturbance by simply
each slot during the non-finality period. The results, depicted looking at the resource utilization of the clients.
in Figure 21, show that during this period there were almost
no blocks queued, which seems to be consistent with the IV. R ELATED W ORK
accelerated syncing speed. However we also notice that just at
In recent years, there have been significant efforts to study
the beginning of the non-finality period, at exactly slot 73,248,
the scalability and security of the Ethereum network from mul-
there were 219 blocks queued (note the logarithmic Y axis).
tiple perspectives. The importance of a new consensus mech-
This clearly shows a huge perturbation in the network.
anism for Ethereum can be understood through the scalability
challenges that the protocol faces [15]. Peers participation of
Ethereum network has been studied previously [16].
Eth2 clients rely on the Proof-of-Stake (PoS) consensus
mechanism called Casper [17] and some works have shown
insights on the Casper’s robustness depending on the network
latency [18]. However, Casper and the Beacon Chain is just the
first step (i.e., Phase 0) of a fully sharded Eth2 protocol, and its
following phase concerns the problem of data availability [19].
In addition, the current security threats of Ethereum have
also been analysed recently [20]. While all these analyses
are very valuable for the community, they all focus on the
protocol security aspects and none of them evaluates the actual
implementation of the protocol.
Eth2 consensus mechanism requires validators to lock ETH
Fig. 21: Number of times a block is queued into a deposit smart-contract, whose security is essential. In
order to verify and increase the security of the deposit contract,
We assume that the accelerating disk usage is related to an formal verification of the Eth2 deposit contract has been
increase in the state stored in the database of the client, and done [21]. This study does verify the implementation of the
this might be linked to a difficulty of pruning states during deposit contract but not the Eth2 clients. While Eth1 clients
a non-finality period. Thus, to corroborate our hypothesis, we have also been analysed in the past [22], to the best of our
analysed Lighthouse detailed logs and plot the frequency at knowledge there is no a study showing the resource utilization
which different events get executed. Figure 22 lists a total of of Eth2 clients and what can be understood from it.
11 different types of events. We can see that during the non-
finality period there are four type of events that almost never V. C ONCLUSION
get executed: Freezer migration started, Database pruning In this work, we have performed a detailed analysis of the
complete, Extra pruning information, and Starting database resource usage of Eth2 clients. Through this analysis, we have
pruning. This demonstrates that during this period the client been able to notice the effort of different teams to develop
is unable to prune the database, which is consistent with the functional software in order to tackle the transitioning to Eth2.
rapid disk usage increase. The necessity of relying on several types of clients is reflected
by the different objectives and resources of a broad variety
of users. Our analysis showed significant differences between
clients in terms of CPU, memory and disk utilization. We also
observed some network perturbations where the number of
peers dropped quickly and its impact on the other resources.
To the best of our knowledge, this is the first study of the Eth2
clients resource usage at this scale and to demonstrate that it
is possible to detect traces of non-finality periods by simply
looking at the clients’ resource usage. The diverse behaviour
of the analysed clients is a positive sign for the Ethereum
community, as it shows a solid variety of software available
for Eth2 validators, and this provides the network with further
decentralization, resilience and security.
Fig. 22: Events timeline for Lighthouse As future work, we would like to further study how non-
finality periods affect the resources usage of Eth2 clients, in
Although there are multiple of things that remain to be order to implement online detectors that use this information
understood about the behaviour of some clients during a non- to signal possible network perturbations in real time.
8
ACKNOWLEDGMENT
This work has been supported by the Ethereum Founda-
tion under Grant FY20-0198. We would like to thank the
researchers of the Ethereum Foundation for their feedback and
suggestions. We would also like to thank the developers and
researchers of the five client teams for all their help to set up
our experiments and their constructive feedback on this study.
R EFERENCES
[1] “Ethereum whitepaper,” https://ethereum.org/en/whitepaper/.
[2] “Phase-0,” https://notes.ethereum.org/@djrtwo/Bkn3zpwxB.
[3] “Medalla test network:,” https://github.com/goerli/medalla.
[4] “Monitoring scripts:,” https://github.com/leobago/BSC-ETH2/tree/
master/ETH2-clients-comparison/scripts/parsing scripts/metrics
monitoring scripts.
[5] “Logs parsing scripts:,” https://github.com/leobago/BSC-ETH2/tree/
master/ETH2-clients-comparison/scripts/parsing scripts/client logs
analysis scripts.
[6] “Plotting scripts:,” https://github.com/leobago/BSC-ETH2/tree/master/
ETH2-clients-comparison/scripts/plotting scripts.
[7] “Teku version:,” https://github.com/ConsenSys/teku/commit/
6883451c9f87a28c5923dbc8f291db237930cad0.
[8] “Prysm version:,” https://github.com/prysmaticlabs/prysm/commit/
4bc7cb6959a1ea5b4b4b53b42284900e3b117dea.
[9] “Lighthouse version:,” https://github.com/sigp/lighthouse/commit/
95c96ac567474df2abb4e9da9f5e771cf5a7426d.
[10] “Nimbus version:,” https://github.com/status-im/nimbus-eth2/commit/
9255945fb0ef1bf036d32b7cce5df42a8dd69be7.
[11] “Lodestar version:,” https://github.com/ChainSafe/lodestar/pull/1731.
[12] “Multi-client benchmark on medalla testnet 2020/10/01:,” https://github.
com/q9f/eth2-bench-2020-10.
[13] “Medalla non-finality october 2020:,” https://gist.github.com/
yorickdowne/ea9b18ac2b51a508080c9a810d978522.
[14] “Medalla non-finality period august
2020:,” https://docs.google.com/document/d/
11RmitNRui10LcLCyoXY6B1INCZZKq30gEU6BEg3EWfk.
[15] M. Bez, G. Fornari, and T. Vardanega, “The scalability challenge of
ethereum: An initial quantitative analysis,” in 2019 IEEE International
Conference on Service-Oriented System Engineering (SOSE), April
2019, pp. 167–176.
[16] S. K. Kim, Z. Ma, S. Murali, J. Mason, A. Miller, and M. Bailey,
“Measuring ethereum network peers,” in Proceedings of the Internet
Measurement Conference 2018, ser. IMC ’18. New York, NY, USA:
Association for Computing Machinery, 2018, p. 91–104. [Online].
Available: https://doi.org/10.1145/3278532.3278542
[17] V. Buterin and V. Griffith, “Casper the friendly finality gadget,” 2019.
[18] O. Moindrot, “Proof of stake made simple with casper,” 2017.
[19] D. Sel, K. Zhang, and H.-A. Jacobsen, “Towards solving the
data availability problem for sharded ethereum,” in Proceedings
of the 2nd Workshop on Scalable and Resilient Infrastructures
for Distributed Ledgers, ser. SERIAL’18. New York, NY, USA:
Association for Computing Machinery, 2018, p. 25–30. [Online].
Available: https://doi.org/10.1145/3284764.3284769
[20] H. Chen, M. Pendleton, L. Njilla, and S. Xu, “A survey on
ethereum systems security: Vulnerabilities, attacks, and defenses,”
ACM Comput. Surv., vol. 53, no. 3, Jun. 2020. [Online]. Available:
https://doi.org/10.1145/3391195
[21] D. Park, Y. Zhang, and G. Rosu, “End-to-end formal verification
of ethereum 2.0 deposit smart contract,” in Computer Aided Verifica-
tion, S. K. Lahiri and C. Wang, Eds. Cham: Springer International
Publishing, 2020, pp. 151–164.
[22] S. Rouhani and R. Deters, “Performance analysis of ethereum transac-
tions in private blockchain,” in 2017 8th IEEE International Conference
on Software Engineering and Service Science (ICSESS), 2017, pp. 70–
74.