Bioinformatics, 38(2), 2022, 453–460

doi: 10.1093/bioinformatics/btab649
Advance Access Publication Date: 16 September 2021
Original Paper

Systems biology

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
BioDynaMo: a modular platform for high-performance
agent-based simulation
Lukas Breitwieser 1,2,*, Ahmad Hesam1,3,*, Jean de Montigny1,
Vasileios Vavourakis4,5, Alexandros Iosif4, Jack Jennings6, Marcus Kaiser6,7,8,
Marco Manca 9, Alberto Di Meglio1, Zaid Al-Ars3, Fons Rademakers1,
Onur Mutlu2,10,* and Roman Bauer11,*
1
CERN openlab, IT Department, CERN, Geneva 1211, Switzerland, 2Department of Computer Science, ETH Zurich, Zurich 8092,
Switzerland, 3Department of Quantum & Computer Engineering, Delft University of Technology, Delft 2628CD, The Netherlands,
4
Department of Mechanical & Manufacturing Engineering, University of Cyprus, Nicosia 2109, Cyprus, 5Department of Medical
Physics & Biomedical Engineering, University College London, London WC1E 6BT, UK, 6School of Computing, Newcastle University,
Newcastle upon Tyne NE4 5TG, UK, 7Department of Functional Neurosurgery, Ruijin Hospital, Shanghai Jiao Tong University School of
Medicine, Shanghai 200025, China, 8Precision Imaging Beacon, School of Medicine, University of Nottingham, Nottingham NG7 2UH,
UK, 9SCimPulse Foundation, Geleen 6162 BC, The Netherlands, 10Department of Information Technology and Electrical Engineering,
ETH Zurich, Zurich 8092, Switzerland and 11Department of Computer Science, University of Surrey, Guildford GU2 7XH, UK
*To whom correspondence should be addressed.
Associate Editor: Jonathan Wren

Received on February 4, 2021; revised on September 2, 2021; editorial decision on September 3, 2021; accepted on September 13, 2021

Abstract
Motivation: Agent-based modeling is an indispensable tool for studying complex biological systems. However,
existing simulation platforms do not always take full advantage of modern hardware and often have a field-specific
software design.
Results: We present a novel simulation platform called BioDynaMo that alleviates both of these problems.
BioDynaMo features a modular and high-performance simulation engine. We demonstrate that BioDynaMo can be
used to simulate use cases in: neuroscience, oncology and epidemiology. For each use case, we validate our find-
ings with experimental data or an analytical solution. Our performance results show that BioDynaMo performs up to
three orders of magnitude faster than the state-of-the-art baselines. This improvement makes it feasible to simulate
each use case with one billion agents on a single server, showcasing the potential BioDynaMo has for computational
biology research.
Availability and implementation: BioDynaMo is an open-source project under the Apache 2.0 license and is avail-
able at www.biodynamo.org. Instructions to reproduce the results are available in the supplementary information.
Contact: [email protected] or [email protected] or [email protected] or [email protected]
Supplementary information: Available at https://doi.org/10.5281/zenodo.5121618.

1 Introduction First, after the slowing down of Moore’s law (Moore, 1965) and
Dennard scaling (Dennard et al., 1974), hardware has become in-
Agent-based simulation (ABS) is a powerful tool assisting life scien- creasingly parallel and heterogeneous. Most ABS platforms do not
tists in better understanding complex biological systems. In silico take full advantage of these hardware enhancements. The resulting
simulation is an inexpensive and efficient way to rapidly test limited computational power forces life scientists to compromise ei-
hypotheses about the (patho)physiology of cellular populations, tis- ther on the resolution of the model or on simulation size (Thorne
sues, organs or entire organisms (Ji et al., 2017; Yankeelov et al., et al., 2007). Second, existing ABS platforms have often been devel-
2016). oped with a specific use case in mind. This makes it challenging to
However, the effectiveness of such computer simulations for sci- implement the desired model, even if it deviates only slightly from
entific research is often limited, in part because of two reasons. its original purpose.

C The Author(s) 2021. Published by Oxford University Press.

V 453
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits
unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
454 L.Breitwieser et al.

To help researchers tackle these two major challenges, we pro- and scale each use case to one billion agents to demonstrate
pose a novel open-source platform for biology dynamics modeling, BioDynaMo’s performance.
BioDynaMo. We alleviate both of these problems by emphasizing
performance and modularity. BioDynaMo features a high- 1.1 Prior work
performance simulation engine that is fully parallelized to utilize The history of agent-based modeling and simulation significantly
multi-core CPUs and able to offload computation to hardware accel- precedes the 1990s; however, it was not widely adopted for biologic-
erators (e.g. a GPU). The software comprises a set of fundamental al systems until the 2000s. Several ABS platforms have been pub-

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
biological functions, and a flexible design that adapts to specific lished demonstrating the importance of agent-based models in
user requirements. Currently, BioDynaMo implements the neurite computational biology research (Collier and North, 2011; Cytowski
model and mechanical forces presented in Zubler and Douglas and Szymanska, 2014; Emonet et al., 2005; Ghaffarizadeh et al.,
(2009), but these components can easily be extended, modified or 2018; Kang et al., 2014; Koene et al., 2009; Lardon et al., 2011;
replaced. Hence, BioDynaMo is well-suited for simulating a wide Matyjaszkiewicz et al., 2017; Mirams et al., 2013; Richmond et al.,
range of biological processes in tissue modeling and beyond. 2010; Rudge et al., 2012; Torben-Nielsen and De Schutter, 2014;
BioDynaMo provides by design five system properties: Wilensky, 1999; Zubler and Douglas, 2009). In this section, we
compare BioDynaMo’s most crucial system properties with prior
• Agent-based. The BioDynaMo project is established to support
work.
an agent-based modeling approach which allows one to simulate
a wide range of developmental biological processes. A character-
Large-scale model support. The authors of BioCellion (Kang et al.,
istic property of agent-based simulations is the absence of a cen- 2014), PhysiCell (Ghaffarizadeh et al., 2018), Timothy (Cytowski
tral organizational unit that orchestrates the behavior of all and Szymanska, 2014) and Repast HPC (Collier and North, 2011)
agents. Quite to the contrary, each agent is an autonomous entity recognize the necessity for efficient implementations to enable large-
that individually determines its actions based on its current state, scale models. Although these tools can simulate a large number of
behavior and the surrounding environment. agents, they do not support neural development. The NeuroMaC
•
neuroscientific simulation platform (Torben-Nielsen and De
General purpose. BioDynaMo is developed to become a general-
Schutter, 2014) claims to be scalable, but the authors do not present
purpose tool for agent-based simulation. To simulate models performance data and present simulations with only 100 neurons.
from various fields, BioDynaMo’s software design is extensible Therefore, BioDynaMo’s ability to simulate large-scale neural devel-
and modular. opment, which we demonstrate in the results section, is, to our
• Large scale. Biological systems contain a large number of agents. knowledge, unrivaled.
The cerebral cortex, for example, comprises approximately 16
billion neurons (Azevedo et al., 2009). Biologists should not be General-purpose platform. Many ABS platforms focus on a specific
limited by the number of agents within a simulation. application area: bacterial colonies (Emonet et al., 2005; Lardon
et al., 2011; Matyjaszkiewicz et al., 2017; Rudge et al., 2012), cell
Consequently, BioDynaMo is designed to take full advantage of
colonies (Cytowski and Szymanska, 2014; Kang et al., 2014;
modern hardware and use memory efficiently to scale-up Mirams et al., 2013) and neural development (Koene et al., 2009;
simulations. Torben-Nielsen and De Schutter, 2014; Zubler and Douglas, 2009).
• Easily programmable. The success of an ABS platform depends, Pronounced specialization of an ABS platform may prevent its cap-
among other things, on how quickly a scientist, not necessarily acity to adapt to different use cases or simulation scenarios. In con-
an expert in computer science or high-performance program- trast, BioDynaMo can be adapted to many different fields due to its
modularity and extensibility.
ming, can translate an idea into a simulation. This characteristic
can be broken down into four key requirements that
BioDynaMo is designed to fullfil: first, BioDynaMo provides a Quality assurance. Automated software testing is the foundation of
a modern development workflow. Unfortunately, several simulation
wide range of common functionalities such as visualization, plot-
tools (Cytowski and Szymanska, 2014; Koene et al., 2009; Lardon
ting, parameter parsing, backups, etc. Second, BioDynaMo pro- et al., 2011; Rudge et al., 2012; Torben-Nielsen and De Schutter,
vides simulation primitives that minimize the programming 2014; Zubler and Douglas, 2009) omit these tests. Mirams et al.
effort necessary to build a use case. Third, as outlined in item (2013) recognize this shortcoming and describe a rigorous develop-
‘General purpose’, BioDynaMo has a modular and extensible de- ment workflow in their article. BioDynaMo has over 400 automated
sign. Fourth, BioDynaMo provides a coherent API and hides im- tests which are continuously executed on all supported operating
systems to ensure high code quality. BioDynaMo’s open-source code
plementation details that are irrelevant for a computational base, tutorials and documentation not only help users get started,
model (e.g. details such as parallelization strategy, synchroniza- but also enable validation by external examiners.
tion, load balancing or hardware optimizations).
• Quality assured. BioDynaMo establishes a rigorous, test-driven
development process to foster correctness, maintainability of the 2 Materials and methods
codebase and reproducibility of results. In this section, we present the main simulation concepts of
BioDynaMo and describe our approach to achieve modularity, ex-
The main contribution of this article is an open-source, high-per- tensibility and high performance.
formance and modular simulation platform for agent-based simula-
tions. We provide the following evidence to support this claim: (i) 2.1 Simulation concepts
we detail the user-facing features of BioDynaMo that enable users to BioDynaMo is implemented in the Cþþ programming language and
build a simulation based on predefined building blocks and to define supports simulations that follow an agent-based approach. Figure 1
a model tailored to their needs. (ii) We present three basic use cases gives an overview of BioDynaMo’s main concepts, Figure 2 presents
in the field of neuroscience, oncology and epidemiology to demon- the abstraction layers, while Figure 3 illustrates its object-oriented
strate BioDynaMo’s capabilities and modularity. (iii) We show that design.
BioDynaMo can produce biologically meaningful simulation results An agent (Figure 1A) has a 3D geometry, behavior and environ-
by validating these use cases against experimental data, or an analyt- ment. There is a broad spectrum of entities that can be modeled as
ical solution. (iv) We present performance data on different systems an agent. In the results section, we show examples where an agent
BioDynaMo 455

A B involved: initialization and execution. During initialization, the user

places agents in space, sets their attributes, and defines their behav-
ior. In the execution step, the simulation engine evaluates the
defined model in the simulated physical environment by executing a
series of operations. We distinguish between agent operations and
standalone operations (Figure 1D). At a high level, an agent oper-
ation is a function that: (i) alters the state of an agent and potentially

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
also its environment, (ii) creates a new agent or (iii) removes an
agent from the simulation. Examples for agent operations are: exe-
cute all behaviors and calculate mechanical forces. The simulation
C D engine executes agent operations for each agent for each time step.
Alternatively, standalone operations perform a specific task during
one time step and are therefore only invoked once. Examples include
the update of substance diffusion and the export of visualization
data.

2.2 BioDynaMo features

BioDynaMo is a simulation platform that can be used to develop
agent-based simulations in various computational biology fields
(e.g. neuroscience, oncology, epidemiology, etc.). Although agent-
based models in these different fields may intrinsically vary, there is
Fig. 1. Simulation concepts. Agents (A) have their own geometry, behavior (B) and
a set of functionalities and definitions that they have in common.
environment (C). (B) Agent behavior is defined in separate classes, which are
inserted into agents. The update of an agent is based on its current state and its sur-
These commonalities can be divided into low- and high-level
rounding environment. (C) The environment is determined by radius r and contains agent-based features and are an integral part of BioDynaMo.
other agents or extracellular substances. The simulation algorithm (D) can be div- BioDynaMo also provides model building blocks to accelerate the
ided into two main parts: the definition of the initial model and the execution of the development of agent-based models. The described layered architec-
simulation ture is shown in Figure 2.

2.2.1 Low-level features

The low-level features (Figure 2) form the foundation of
BioDynaMo and provide crucial functionality responsible for high-
performance execution and ease-of use. These features are mostly
hidden from the user and require control only in exceptional
situations.
In this section we will provide more details about parallelization
and thread safety and refer the reader to Supplementary File SF1
Section 1.1 for more details of the remaining low-level features.

Parallelism and thread safety. BioDynaMo exploits the inherent par-

allelism of agent-based models in which agents update themselves
based on their current state, behavior and local environment.
BioDynaMo’s implementation uses OpenMP (https://www.openmp.
org/) compiler directives to parallelize the loop over all agents
Fig. 2. BioDynaMo’s layered architecture. BioDynaMo is predominantly executed
(Figure 1D). In addition to parallelizing the execution of agent oper-
on multi-core CPUs, is able to offload computations to the GPU, and supports ations for each agent, standalone operations like updating the diffu-
Linux and MacOS operating systems. BioDynaMo provides a rich set of low- and sion grid and visualization are parallelized separately (Figure 1D).
high-level features commonly required in agent-based models. On top of these gen- Synchronization between threads is only needed if agents modify
eric features, BioDynaMo offers model building blocks to simplify the development their environment. In this case, two agents (handled by two different
of a simulation. Even if BioDynaMo does not provide the required building blocks, threads) might attempt to update the same resource in the local en-
users still benefit from all generic agent-based features (illustrated by the vertical ex- vironment. This scenario occurs in the neuroscience use case in
tension of the ‘Simulation’ layer) which neurite elements modify neighboring segments. Therefore,
BioDynaMo provides built-in synchronization mechanisms to en-
sure that even if two threads try to modify the same agent or re-
represents a subcellular structure (neuroscience use case), a cell (on- source, data is not corrupted. There are two thread safety
cology use case) or a person (epidemiology use case). Figure 1B mechanisms to protect agents from data corruption: automatic and
shows example agent behaviors such as growth factor secretion, user-defined. Automatic thread safety uses the environment to pre-
chemotaxis or cell division. Behaviors can be activated or inhibited. vent two threads from processing agents with overlapping local
BioDynaMo achieves this by attaching them to or removing them environments. This mechanism can be enabled with a single param-
from the corresponding agent. eter, but might be too restrictive for some use cases. User-defined
The Environment is the vicinity that the agent can interact with thread safety on the other hand offers more fine-grained control
(Figure 1C). It comprises agents and other resources like chemical over which agents must not be processed at the same time, but likely
substances in the extracellular matrix. Surrounding agents are, for requires additional input from the user.
example, needed to calculate mechanical interactions among agent Other resources that are modified by agents (e.g. the
pairs. DiffusionGrid to simulate extracellular diffusion) need their own
Currently, the user defines a simulation programmatically in protection mechanism. This feature is used in the soma clustering
Cþþ (see Figure 1D and Supplementary File SF1 Section 1.4). Cþþ benchmark where cells secrete a substance into the extracellular
is a great choice in terms of execution speed, efficiency, and inter- matrix.
operability with high-performance computing libraries, but is harder For typical BioDynaMo simulations users do not need to control
to program in due to its lower level of abstraction (versus higher parallel execution and thread synchronization. This holds particu-
level languages like Java or Python). There are two main steps larly true for all use cases and benchmarks presented in this article.
456 L.Breitwieser et al.

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
Fig. 3. Software design and modularity. Overview of selected classes and functions that are important from the users’ perspective. Classes in white (BioDynaMo core) and
green (BioDynaMo’s neuroscience module) are part of the current BioDynaMo installation. The remaining classes illustrate how we extended BioDynaMo to implement the
use cases and benchmarks presented in this article (purple: neuroscience use case, red: oncology use case, orange: epidemiology use case, blue: soma clustering benchmark, yel-
low: cell proliferation benchmark). A complete list of BioDynaMo classes can be found at https://biodynamo.org/api

Only for more advanced uses, like adding a new environment algo- Environment search. To determine the agents in the local environ-
rithm or adding a shared resource that is not an agent, users have to ment (neighbors), BioDynaMo uses an environment search algo-
consider parallel execution. rithm (Supplementary Tutorial ST06). BioDynaMo’s default
environment algorithm is based on a uniform grid implementation.
The implementation divides the total simulation space into uniform
boxes of the same size and assigns agents to boxes based on the cen-
2.2.2 High-level features
ter of mass of the agent. Hence, the agents in the environment can
The high-level layer (Figure 2) provides general functionality which
be obtained by iterating over the assigned box and all its surround-
is commonly required in agent-based models across many fields. ing boxes (27 boxes in total). The box size is chosen by the user or
determined automatically based on the largest agent in the simula-
Generation of agent populations. The first step in an agent-based tion to ensure all mechanical interactions are taken into account.
model is to specify the starting condition of the simulation. Therefore, Alternatively, BioDynaMo provides an octree and kd-tree environ-
BioDynaMo provides functionality to create agent populations with spe- ment implementation. The interface is kept generic enough to sup-
cific properties. Class ModelInitializer provides several functions to create port non-Euclidean space environment definitions.
agents in 3D space and to initialize extracellular substances (Figure 3 and
Supplementary File SF1 Section 1.4). Furthermore, to initialize the attrib- Multi-scale simulations. A biological simulation has to account for dy-
utes of an agent population, researchers can use one of the many prede- namic mechanisms that range from milliseconds to weeks (e.g. physical
fined random number generators that draw samples from a specific forces, reaction-diffusion processes, gene regulatory dynamics, etc.).
distribution (uniform, exponential, gaussian, binomial, etc.) or define their BioDynaMo supports processes at different time scales by providing a
own one. These features are demonstrated in Supplementary Tutorial parameter to specify the time interval between two time steps and an exe-
ST01, ST02 and ST08. cution frequency for each operation (Supplementary Tutorial ST07). An
execution frequency of one means that the corresponding operation is
Agent reproduction and mortality. The addition and removal of executed every time step. In contrast, a frequency of three would mean
agents during the execution of a simulation is an integral part of that the operation is executed every three time steps. This mechanism
agent-based simulations. Therefore, BioDynaMo provides a frame- allows BioDynaMo to simulate e.g. substance diffusion and neurite
growth in the same model.
work to create new agents during a simulation and initialize their
attributes. By default, agents that are created in iteration i will be
visible to other agents in iteration i þ 1. The removal of agents is Statistical analysis. Statistical analysis plays a fundamental role in
handled identically. The handling of when new or removed agents generating new insights from simulation data. BioDynaMo builds
become visible to the simulation is encapsulated in the execution upon the rich features of CERN’s primary data analysis framework
context. Therefore, a user could provide another implementation ROOT (Brun and Rademakers, 1997), which provides an extensive
where agents are visible immediately. mathematical, histogram, graphing, and fitting library. BioDynaMo
Besides adding and removing agents, a second major part is to complements this functionality by providing an easy mechanism to
provide a generic way to initialize the attributes of an agent. To this collect simulation data over time and a simplified API targeted to
end, BioDynaMo simplifies the regulation of behaviors if new agents the agent-based use case. These capabilities are demonstrated in
are created. The user can control whether a behavior will be copied Supplementary Tutorial ST08–ST11.
to a new agent or removed from the existing agent, based on the
underlying process (e.g. cell division). Similarly, agents have a func- Hierarchical model support. Railsback et al. (2006) describe an
tion Initialize which can be overridden by user-defined agents to ini- agent-based model in which large agents have to be executed before
tialize additional attributes. These concepts are demonstrated in smaller agents. BioDynaMo supports these hierarchical models with
Supplementary Tutorial ST03–ST05. several functions in the ResourceManager, Scheduler, and
BioDynaMo 457

Operation class. The described order can be implemented in using the arithmetic mean and rates such as speedup using the har-
BioDynaMo by adding three lines of code as demonstrated in monic mean.
Supplementary Tutorial ST12. Additionally, it is possible to execute
a different set of operations for large and small agents. 2.5 Reproducibility
We use the latest BioDynaMo version v1.01-55-gd05111e3 for all
Hybrid modeling support. Some models benefit from the combin- use cases and benchmarks in the result section. To help other
ation of multiple simulation methodologies—e.g. the combination researchers replicate our findings we provide the following

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
of an agent-based and continuum-based model. BioDynaMo’s flex- Supplementary Information for utmost transparency. First, we pub-
ible build system supports hybrid modeling capabilities and was lish all source code and data in Supplementary File SF3. The archive
demonstrated by de Montigny et al. (2021) to investigate cancer contains six shell scripts that execute all simulations, and generate
development. all plots, visualizations and videos shown in this article. Second, we
The remaining high-level features are detailed in Supplementary provide a ready-to-use self-contained Docker image to simplify the
File SF1 Section 1.2. process of executing our simulations and benchmarks and to guar-
antee long-term reproducibility (Supplementary File SF4). Third, we
add a step-by-step instruction in Supplementary File SF2.
2.2.3 Model building blocks
Currently, BioDynaMo’s building blocks (Figure 2) belong to the
(neural) tissue modeling domain. Similar to the biological model
presented in (Zubler and Douglas, 2009), BioDynaMo supports
3 Results
spherical and cylindrical agent geometries, mechanical interactions 3.1 Neuroscience use case
between agents and diffusion of extracellular substances. This example illustrates the use of BioDynaMo to model neurite
With these features, researchers can simulate cell body dynamics, growth of pyramidal cells using chemical cues. Initially, a pyramidal
neural growth and gene regulatory networks. We provide more cell, composed of a 10 lm cell body, three 0.5 lm long basal den-
details about the individual model building blocks in Supplementary drites and one 0.5 lm long apical dendrite (all of them considered
File SF1 Section 1.3. here as agents), is created in 3D space [L37–L51 (Line numbers in
Simulations to study the development of (neural) tissue are only Section 3.1 correspond to the code example in Supplementary File
one example of how BioDynaMo could be used in the future. By SF1 Listing 4.)]. Furthermore, two artificial growth factors were ini-
designing BioDynaMo in a modular and extensible way, we laid the tialized, following a Gaussian distribution along the z-axis (L54–
foundation to create new building blocks easily (Figure 3 and L65). The distribution of these growth factors guided dendrite
Section 3.3). growth and remained unchanged during the simulation.
Dendritic development was dictated by a behavior defining
2.3 Use cases growth direction, speed and branching behavior for apical and basal
We demonstrate BioDynaMo’s capacity to simulate disparate prob- dendrites (L12–L35). At each step of the simulation, the dendritic
lems in systems biology with simple yet representative use cases in growth direction depended on the local chemical growth factor gra-
neuroscience, oncology and epidemiology. Since BioDynaMo does dient, the dendrite’s previous direction and a randomly chosen direc-
not contain any epidemiological building blocks, this use case indi- tion. In addition, the dendrite’s diameter tapered as it grew
cates how easy it is to implement a model-based solely on features (shrinkage), until it reached a specified diameter, preventing it from
from the high- and low-level layer (Figure 2). growing any further. The weight of each element on the direction
For each use case we present the implemented model, validate varied between apical and basal dendrites.
the simulation results against verified experimental or analytical These simple rules gave rise to a straight long apical dendrite
data, and report performance data for different problem sizes on with a simple branching pattern and more dispersed basal den-
multiple hardware configurations. Furthermore, we provide pseudo- drites, as shown in Figure 4A, similar to what can be observed in
code for all agent behaviors, a table with model parameters and real pyramidal cell morphologies as shown in Figure 4B or
more detailed performance results in Supplementary File SF1 Spruston (2008) (Figure 1A CA1). Using our growth model, we
Section 2. were able to generate a large number of various realistic pyram-
idal cell morphologies. We used a publicly available database of
real pyramidal cells coming from Mellström et al. (2016) for
2.4 Performance analysis comparison and parameter tuning. Two measures were used to
We compare BioDynaMo’s performance with two established serial compare our simulated neurons and the 69 neurons composing
ABS platforms [Cortex3D (Zubler and Douglas, 2009) and NetLogo the real morphologies database: the average number of branching
(Wilensky, 1999)], analyze BioDynaMo’s scalability, and quantify points, and the average length of dendritic trees. No significant
the impact of GPU acceleration. Display updates are turned off on differences were observed between our simulated neurons and
all platforms for these evaluations. Cortex3D has the highest simi- the real ones (P < 0.001 using a T-test for two independent sam-
larity in terms of the underlying biological model out of all the ples). These results are shown in Figure 4D. The model specifica-
related works presented in Section 1.1. More specifically, tion of the pyramidal cell growth simulation consists of 127 lines
BioDynaMo and Cortex3D use the same method to determine of Cþþ code (Supplementary File SF1 Listing 4).
mechanical forces between agents and the same model to grow neur- Figure 4C shows a large-scale simulation incorporating 5000
al morphologies. This makes Cortex3D the best candidate with neurons similar to the one described above and demonstrates the po-
which to compare BioDynaMo and ensure a fair comparison. tential of BioDynaMo for developmental, anatomical and connectiv-
We quantify the performance of BioDynaMo with four simula- ity studies in the brain. This simulation contained nine million
tions: cell growth and division, soma clustering, pyramidal cell agents.
growth and the epidemiology use case. We compare the runtime of
the first three simulations with Cortex3D and the epidemiology use
3.2 Oncology use case
case with NetLogo 3D. These simulations have different properties
In this section, we present a tumor spheroid simulation to replicate
and are, therefore, well suited to evaluate BioDynaMo’s simulation
in vitro experiments from Gong et al. (2015). Tumor spheroid
engine under a broad set of conditions. Supplementary File SF1
experiments are typically employed to investigate the pathophysi-
Section 2.2 contains more details about these benchmarks.
ology of cancer, and are also being used for pre-clinical drug screen-
ing (Nunes et al., 2019). Here we considered three in vitro test cases
Statistical method. We perform five measurements for each pre- using a breast adenocarcinoma MCF-7 cell line (Gong et al., 2015)
sented data point in Figure 7 and Table 1. We summarize runtimes with different initial cell populations (2000, 4000 and 8000 MCF-7
458 L.Breitwieser et al.

Table 1. Performance data

Simulation Agents Diffusion volumes Iterations System Physical CPUs Runtime Memory

Neuroscience use case

Single (Figure 4A) 1494 250 500 A 1 0.16 s 382 MB
Large-scale (Figure 4C) 9 054 740 65 536 500 A 72 36 s 6.02 GB
Very-large-scale 1 018 644 154 5 606 442 500 B 72 1 h 26 min 436 GB

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
Oncology use case (Figure 5)
2000 initial cells 4177 0 312 A 1 1.05 s 382 MB
Large-scale 10 003 925 0 288 A 72 1 min 42 s 7.42 GB
Very-large-scale 986 054 868 0 288 B 72 6 h 21 min 604 GB
Epidemiology use case (Figure 6C)
Measles 2010 0 1000 A 1 0.53 s 381 MB
Seasonal Influenza 20 200 0 2500 A 1 16.41 s 383 MB
Large-scale (measles) 10 050 000 0 1000 A 72 59.19 s 5.87 GB
Very-large-scale (measles) 1 005 000 000 0 1000 B 72 2 h 0 min 495 GB

Note: The values in column ‘Agents’ and ‘Diffusion volumes’ are taken from the end of the simulation. Runtime measures the wall-clock time to simulate the
number of iterations. It excludes the time for simulation setup and visualization. The entries in column ‘System’ correspond to Supplementary File SF1 Table 5.
Supplementary File SF1 Table 6 contains more detailed performance data.

A B C A B

Fig. 5. Comparison between in vitro MCF-7 tumor spheroid experiments and our in
silico simulations using BioDynaMo. (A) Human breast adenocarcinoma tumor
spheroid (MCF-7 cell line) development during a 15 day period, where different ini-
tial cell populations were considered [see Fig. 3 in Gong et al. (2015)]. Error bars de-
note standard deviation to the experimental data. The mean of the in silico results is
shown as a solid black line with a gray band depicting minimum and maximum
Fig. 4. Pyramidal cell simulation. (A) Example pyramidal cell simulated with observed value. (B) A qualitative comparison between the microscopy images and
BioDynaMo. (B) Real neuron (R67nr67b-CEL1.CNG) taken from Mellström et al. simulation snapshots. Scale bars correspond to 100 lm. A video is available in the
(2016) and visualized with https://neuroinformatics.nl/HBP/morphology-viewer/. Supplementary Materials
(C) Large-scale simulation. The model started with 5000 initial pyramidal cell
bodies and contained 9 million agents after simulating 500 iterations. The simula-
tion execution time was 35 s on a server with 72 CPU cores. (D) Morphology com-
Figure 5B compares snapshots of the simulated tumor spheroids
parison between simulated neurons and experimental data from Mellström et al.
(2016). Error bars represent the standard deviation. (A, C) A video is available in (bottom row) against microscopy images of in vitro spheroids (top
the Supplementary Materials row) at different time points. The spheroid’s morphologies between
the in vitro experiments and the BioDynaMo simulations are in ex-
cellent agreement.
cells). Our goal was to simulate the growth of this mono cell culture
Model specification required 154 lines of Cþþ code.
embedded in a collagenous (extracellular) matrix.
The fundamental cellular mechanisms modeled here include cell
growth, cell duplication, cell migration and cell apoptosis. All these
processes are implemented in the class TumorCellBehavior. The cell 3.3 Epidemiology use case
growth rate was derived from the published data (Sutherland et al., This section presents an agent-based model that describes the
1983), while cell migration (cell movement speed), cell survival and spreading of infectious diseases between humans. The model divides
apoptosis were fine-tuned after trial and error testing. Since the the population into three groups: susceptible, infected and recovered
in vitro study considered the same agarose gel matrix composition (SIR) agents. We compare our simulation results with the solution
among the experiments, the BioDynaMo model assumes identical of the original SIR model from Kermack et al. (1927), which used
parameters for the cell–matrix interactions in the simulations. the following three differential equations to describe the model dy-
Considering the homogeneous ECM properties, tumor cell migra- namics: dS=dt ¼ bSI=N, dI=dt ¼ bSI=N  cI and dR=dt ¼ cI. S, I
tion was modeled as Brownian motion. and R are the number of susceptible, infected and recovered individ-
The in vitro experiments from Gong et al. (2015) and the simula- uals, N is the total number of individuals, b is the mean transmission
tions using BioDynaMo are depicted in Figure 5. Each line plot in rate and c is the recovery rate.
Figure 5A compares the mean diameter between the experiments For our agent-based implementation (Figure 6C) we created a
and the simulations over time, which demonstrates the validity and new agent (representing a person) that encompasses three new
accuracy of BioDynaMo. The diameter of the spheroids in the simu- behaviors (see Figure 3). First, a susceptible agent became infected
lations were deducted from the volume of the convex hull that with the infection probability if an infected agent was within the in-
enclosed all cancer cells. The in vitro experiments used microscopy fection radius. Second, an infected agent recovered with the recovery
imaging to measure the spheroid’s diameters (Gong et al., 2015). probability at every time step. Third, all agents moved randomly in
BioDynaMo 459

A B speedup between 19 and 74 for Cortex3D and 25 for NetLogo.
The speedup was larger, when the simulation was more dynamic or
more complex. Note that we set the number of threads available to
BioDynaMo to one since Cortex3D and NetLogo are not parallel-
ized. In the ‘epidemiology (medium-scale)’ benchmark we increased
the number of available physical CPU cores to 72 and observe a
three order of magnitude speedup of 945. This result clearly shows

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
the impact of parallelization on the overall performance. Although
C NetLogo is not parallelized, it benefits from parallel garbage collec-
Time [h]: 0 100 200 300 900
tion. We could not perform a medium-scale analysis with
Measles

Cortex3D, because it only supports simulations with a small number

of agents.
Second, to evaluate the scalability of BioDynaMo, we measured
Time [h]: 0 500 1000 1500 2400 the simulation time with an increasing number of threads. We
increased the number of agents used in the comparison with
Inf uenza

Cortex3D and reduced the number of simulation timesteps to 10.

Figure 7B shows the strong scaling analysis. We define the term
‘strong scaling’ as the property of a simulation platform to reduce
the runtime of a simulation with a fixed size x with an increasing
Fig. 6. Measles and seasonal influenza SIR model results. (A, B) Comparison be- number of CPU cores c: speedupðc; xÞ ¼ timeð1;xÞ
timeðc;xÞ (Hill, 1990).
tween agent-based (solid lines) and analytical (dashed lines) model for measles (A) All simulation parameters were kept constant, and the number
and seasonal influenza (B). The agent-based simulation was repeated ten times. The
of threads was increased from one to the number of logical cores
individual simulation results are shown as thin solid lines. The bold solid line repre-
sents the mean from all simulations. The legend is shared between the two plots. (C)
provided by the benchmark server. The maximum speedup ranged
Visualization of the measles and influenza model for different time steps in 3D between 62 and 77, which corresponds to a parallel efficiency of
space. Susceptible persons are shown in white, infected persons in red, and recov- 0.86 and 1.07. Performance improved even after all physical cores
ered persons in blue were utilized and hyper-threads were used. Hyper-threads are high-
lighted in gray in Figure 7B. We want to emphasize that even the
pyramidal cell growth benchmark scaled well, despite the challenges
of synchronization and load imbalance.
Third, we evaluated the impact of calculating the mechanical
forces on the GPU using the cell growth and division, and soma clus-
tering simulations. We excluded the pyramidal cell growth simula-
tion because the current GPU kernel does not yet support cylinder
geometry. The benchmarks were executed on System C (see
Supplementary File SF1 Table 4), comparing an NVidia Tesla V100
GPU with 32 CPU cores (64 threads). We observed a speedup of
1.01 for cell growth and division, and 4.16 for soma clustering.
Fig. 7. BioDynaMo performance analysis. (A) Speedup of BioDynaMo compared to The speedup correlated with the number of collisions in the simula-
the serial simulation platforms Cortex3D and NetLogo. Simulations use one CPU tion. The computational intensity is directly linked with the number
core except for the ‘epidemiology (medium-scale)’ benchmark, for which all 72 of collisions between agents.
physical cores were available. The comparison with NetLogo uses the same simula- In summary, in the scalability test, we observed a minimum
tion with different numbers of agents. Cortex3D supports only small-scale simula- speedup of 62. Furthermore, we measured a minimum speedup of
tions. (B) Strong scaling behavior of BioDynaMo on a server with 72 physical cores, 19 comparing BioDynaMo with Cortex3D both using a single
two threads per core and four NUMA domains. The gray area highlights hyper-
thread. Based on these two observations, we conclude that on
threads
System A (see Supplementary File SF1 Table 4) BioDynaMo is more
than three orders of magnitude faster than Cortex3D. In the com-
space with toroidal boundary condition. The absolute distance an parison with NetLogo we observed a 945 speedup directly.
agent could travel in every time step was limited. Based on these speedups, we executed the neuroscience, oncol-
We selected two infectious diseases with different characteristics ogy and epidemiology use cases with one billion agents. Using all 72
to verify our model: measles and seasonal influenza. We obtained physical CPUs on System B (see Supplementary File SF1 Table 4),
values for the basic reproduction number R0 and recovery duration we measured a runtime of 1 h 26 min, 6 h 22 min and 2 h, respective-
TR from the literature (Measles: R0 ¼ 12:9, TR ¼ 8 days (Guerra ly. One billion agents, however, are not the limit. The maximum
depends on the available memory and accepted execution duration.
et al., 2017; World Health Organization, 2020), Influenza: R0 ¼
To be consistent across all use cases and keep our pipeline’s total
1:3; TR ¼ 4:1 days (Chowell et al., 2008)) and determined the
execution time better manageable, we decided to run these bench-
parameters b and c for the analytical model, based on R0 ¼ b=c and
marks with one billion agents. Table 1 shows that available memory
c ¼ 1=TR . For the agent-based model we set the recovery probability would permit an epidemiological and neuroscience simulation with
to c, and determined the remaining parameters (infection radius, in- two billion agents. With enough memory, BioDynaMo is capable of
fection probability and maximum movement in one time step) using supporting hundreds of billions of agents.
particle swarm optimization (Kennedy and Eberhart, 1995).
Figure 6 shows that the agent-based model is in excellent agreement
with the equation-based approach from (Kermack et al., 1927) for 4 Discussion
measles and influenza.
Model specification required 181 lines of Cþþ code. This article presented BioDynaMo, a novel open-source platform
for agent-based simulations. Its modular software architecture
allows researchers to implement models of distinctly different fields,
3.4 Performance of which neuroscience, oncology and epidemiology were demon-
First, to demonstrate the performance improvements against estab- strated in this article. Although the implemented models follow a
lished ABS platforms, we compared BioDynaMo with Cortex3D simplistic set of rules, the results that emerge from the simulations
and NetLogo. Figure 7A shows the speedup of BioDynaMo for four are prominent and highlight BioDynaMo’s capabilities. We do not
simulations. We define speedup as the runtime of the compared ABS claim that these models are novel, but we rather want to emphasize
platform over the runtime of BioDynaMo. We observed a significant that BioDynaMo enables scientists to (i) develop models in various
460 L.Breitwieser et al.

computational biology fields in a modular fashion, (ii) obtain results de Montigny,J. et al. (2021) An in silico hybrid continuum-/agent-based pro-
rapidly with the parallelized execution engine, (iii) scale up the cedure to modelling cancer development: interrogating the interplay
model to billions of agents on a single server and (iv) produce results amongst glioma invasion, vascularity and necrosis. Methods, 185, 94– 104.
that are in agreement with validated experimental data. Although Dennard,R.H. et al. (1974) Design of ion-implanted MOSFET’s with very
BioDynaMo is modular, we currently offer a limited number of small physical dimensions. IEEE J. Solid State Circuits, 9, 256–268.
ready-to-use simulation primitives. We are currently expanding our Emonet,T. et al. (2005) AgentCell: a digital single-cell assay for bacterial
library of agents and behaviors to facilitate model development be- chemotaxis. Bioinformatics, 21, 2714–2721.
Ghaffarizadeh,A. et al. (2018) PhysiCell: an open source physics-based cell

Downloaded from https://academic.oup.com/bioinformatics/article/38/2/453/6371176 by Centre IRD de Montpellier (ex. Orstom) user on 28 February 2023
yond the current capacity.
simulator for 3-D multicellular systems. PLoS Comput. Biol., 14,
Ongoing work uses BioDynaMo to gain insights into retinal de-
e1005991.
velopment, cryopreservation, multiscale (organ-to-cell) cancer mod-
Gong,X. et al. (2015) Generation of multicellular tumor spheroids with
eling, radiation-induced tissue damage and more. Further efforts
microwell-based agarose scaffolds for drug testing. PLoS One, 10,
focus on accelerating drug development by replacing in vitro experi-
e0130348.
ments with in silico simulations using BioDynaMo. Guerra,F.M. et al. (2017) The basic reproduction number (R0) of measles: a
Our performance analysis showed improvements of up to three systematic review. Lancet Infect. Dis., 17, e420–e428.
orders of magnitude over state-of-the-art baseline simulation soft- Hill,M.D. (1990) What is scalability? ACM SIGARCH Comput. Archit.
ware, allowing us to scale up simulations to an unprecedented num- News, 18, 18–21.
ber of agents. To the best of our knowledge, BioDynaMo is the first Ji,Z. et al. (2017) Mathematical and computational modeling in complex bio-
scalable ABS platform for neural development that scales to more logical systems. BioMed Res. Int., 2017, 5958321.
than one billion agents. The same principles used to model axons Kang,S. et al. (2014) Biocellion: accelerating computer simulation of multicel-
and dendrites in the neuroscience use case could also be applied to lular biological system models. Bioinformatics, 30, 3101–3108.
simulate blood and lymphatic vessels. Kennedy,J. and Eberhart,R. (1995) Particle swarm optimization. In:
We envision BioDynaMo to become a valuable tool in computa- Proceedings of ICNN’95 – International Conference on Neural Networks,
tional biology, fostering faster and easier simulation of complex and Perth, WA, Australia, Vol. 4, pp. 1942–1948.
large-scale systems, interdisciplinary collaboration and scientific Kermack,W.O. et al. (1927) A contribution to the mathematical theory of epi-
reproducibility. demics. Proc. R. Soc. Lond. Ser. A Contain. Papers Math. Phys. Charact.,
115, 700–721.
Koene,R.A. et al. (2009) NETMORPH: a framework for the stochastic gener-
Funding ation of large scale neuronal networks with realistic neuron morphologies.
Neuroinformatics, 7, 195–210.
This work was supported by the CERN Knowledge Transfer office [to L.B. Lardon,L.A. et al. (2011) iDynoMiCS: next-generation individual-based mod-
and A.H.]; the Israeli Innovation Authority [to A.H.]; the Research elling of biofilms. Environ. Microbiol., 13, 2416–2434.
Excellence Academy from the Faculty of Medical Science of the Newcastle Matyjaszkiewicz,A. et al. (2017) BSim 2.0: an advanced agent-based cell simu-
University [to J.dM.]; the UCY StartUp Grant scheme [to V.V.]; the Medical lator. ACS Synth. Biol., 6, 1969–1972. [PMC][10.1021/acssynbio.7b00121]
Research Council of the United Kingdom [MR/N015037/1 to R.B., MR/ [28585809]
Mellström,B. et al. (2016) Specific cytoarchitectureal changes in hippocampal
T004347/1 to M.K.]; the Engineering and Physical Sciences Research Council
subareas in daDREAM mice. Mol. Brain, 9, 22.
of the UK [EP/S001433/1 to R.B., NS/A000026/1, EP/N031962/1 to M.K.]; a
Mirams,G.R. et al. (2013) Chaste: an open source Cþþ library for computa-
PhD studentship funded by Newcastle University’s School of Computing [to tional physiology and biology. PLoS Comput. Biol., 9, e1002970.
J.J.]; the Wellcome Trust [102037 to M.K.]; the Guangci Professorship Moore,G.E. (1965) Cramming more components onto integrated circuits.
Program of Ruijin Hospital (Shanghai Jiao Tong Univ.) [to M.K.]; and by sev- Electronics, 38, 114–117.
eral donations by SAFARI Research Group’s industrial partners including Nunes,A. et al. (2019) 3D tumor spheroids as in vitro models to mimic in vivo
Huawei, Intel, Microsoft, and VMware [to O.M.]. human solid tumors resistance to therapeutic drugs. Biotechnol. Bioeng.,
116, 206–226.
Conflict of Interest: none declared. Railsback,S.F. et al. (2006) Agent-based simulation platforms: review and de-
velopment recommendations. Simulation, 82, 609–623.
Richmond,P. et al. (2010) High performance cellular level agent-based simula-
Acknowledgements tion with FLAME for the GPU. Brief. Bioinf., 11, 334–347.
Rudge,T.J. et al. (2012) Computational modeling of synthetic microbial bio-
The authors thank Giovanni De Toni for his work on the BioDynaMo build
films. ACS Synth. Biol., 1, 345–352.
system.
Spruston,N. (2008) Pyramidal neurons: dendritic structure and synaptic inte-
gration. Nat. Rev. Neurosci., 9, 206–221.
Sutherland,R.L. et al. (1983) Cell proliferation kinetics of mcf-7 human mam-
References mary carcinoma cells in culture and effects of tamoxifen on exponentially
Azevedo,F.A.C. et al. (2009) Equal numbers of neuronal and nonneuronal growing and plateau-phase cells. Cancer Res., 43, 3998–4006.
cells make the human brain an isometrically scaled-up primate brain. J. Thorne,B.C. et al. (2007) Combining experiments with multi-cell agent-based
Compar. Neurol., 513, 532–541. modeling to study biological tissue patterning. Brief. Bioinf., 8, 245–257.
Brun,R. and Rademakers,F. (1997) ROOT — An object oriented data analysis Torben-Nielsen,B. and De Schutter,E. (2014) Context-aware modeling of
framework. Nuclear Instrum. Methods Phys. Res. Sect. A Accel. Spectrom. neuronal morphologies. Front. Neuroanat., 8, 92.
Detect. Assoc. Equip., 389, 81–86. Wilensky,U. (1999) Netlogo. Center for Connected Learning and
Chowell,G. et al. (2008) Seasonal influenza in the United States, France, and Computer-based Modeling, Northwestern University, Evanston, IL. http://
Australia: transmission and prospects for control. Epidemiol. Infect., 136, ccl.northwestern.edu/netlogo/ (30 April 2021, date last accessed).
852–864. World Health Organization (2020) Measles. https://www.who.int/news-
Collier,N. and North,M. (2011) Repast HPC: A Platform for Large-Scale room/fact-sheets/detail/measles (23 November 2020, date last accessed).
Agent-Based Modeling, Chapter 5. John Wiley & Sons, Ltd., New York, pp. Yankeelov,T.E. et al. (2016) Multi-scale modeling in clinical oncology: oppor-
81–109. tunities and barriers to success. Ann. Biomed. Eng., 44, 2626–2641.
Cytowski,M. and Szymanska,Z. (2014) Large-scale parallel simulations of 3D Zubler,F. and Douglas,R. (2009) A framework for modeling the growth and
cell colony dynamics. Comput. Sci. Eng., 16, 86–95. development of neurons and networks. Front. Comput. Neurosci., 3, 25.