Reconstruction of Ordinary Differential Equations From Time Series Data

Manuel Mai1 , Mark D. Shattuck2 , and Corey S. O’Hern3,1,4,5

1
Department of Physics, Yale University, New Haven, Connecticut 06520, USA
2
Benjamin Levich Institute and Physics Department,
The City College of New York, New York, New York 10031, USA
3
Department of Mechanical Engineering & Materials Science,
Yale University, New Haven, Connecticut 06520, USA
4
Department of Applied Physics, Yale University, New Haven, Connecticut 06520, USA and
5
Graduate Program in Computational Biology and Bioinformatics,
Yale University, New Haven, Connecticut 06520, USA
arXiv:1605.05420v1 [physics.data-an] 18 May 2016

We develop a numerical method to reconstruct systems of ordinary differential equations (ODEs)

from time series data without a priori knowledge of the underlying ODEs using sparse basis learn-
ing and sparse function reconstruction. We show that employing sparse representations provides
more accurate ODE reconstruction compared to least-squares reconstruction techniques for a given
amount of time series data. We test and validate the ODE reconstruction method on known 1D,
2D, and 3D systems of ODEs. The 1D system possesses two stable fixed points; the 2D system
possesses an oscillatory fixed point with closed orbits; and the 3D system displays chaotic dynamics
on a strange attractor. We determine the amount of data required to achieve an error in the re-
constructed functions to less than 0.1%. For the reconstructed 1D and 2D systems, we are able to
match the trajectories from the original ODEs even at long times. For the 3D system with chaotic
dynamics, as expected, the trajectories from the original and reconstructed systems do not match
at long times, but the reconstructed and original models possess similar Lyapunov exponents. Now
that we have validated this ODE reconstruction method on known models, it can be employed in
future studies to identify new systems of ODEs using time series data from deterministic systems
for which there is no currently known ODE model.

PACS numbers: 87.19.xd 87.19.xw 07.05.Kf 05.45.Tp 05.45.-a

I. INTRODUCTION generates random mathematical expressions for a candi-

date ODE model. At each iteration, the ODE model is
We will present a methodology to create ordinary solved and the solution is compared to the time series
differential equations (ODEs) that reproduce measured data to identify the parameters in the candidate model.
time series data from physical systems. In the past, The selected parameters minimize the distance between
physicists have constructed ODEs by writing down the the input trajectories and the solutions of the candidate
simplest mathematical expressions that are consistent model. Candidate models with small errors are then co-
with the symmetries and fixed points of the system. For evolved using a genetic algorithm to improve the fit to
example, E. N. Lorenz developed an ODE model for at- the input time series data [18–21]. The advantage of this
mospheric convection [1] by approximating solutions to method is that it yields an approximate analytical expres-
the Navier-Stokes equations for Rayleigh-Bénard convec- sion for the ODE model for the dynamical system. The
tion. Beyond its specific derivation, the Lorenz system disadvantages of this approach include the computational
of ODEs is employed to model a wide range of systems expense of repeatedly solving ODE models and the dif-
that display nonlinear and chaotic dynamics, including ficulty in finding optimal solutions for multi-dimensional
lasers [2], electrical circuits [3], and MEMS [4] devices. nonlinear regression.
ODE models are also used extensively in computa- Here, we develop a method to build numerical expres-
tional biology. For example, in systems biology, genetic sions of a system of ODEs that will recapitulate time
circuits are modeled as networks of electronic circuit el- series data of a dynamical system. This method has the
ements [5, 6]. In addition, systems of ODEs are often advantage of not needing any input except the time se-
employed to investigate viral dynamics (e.g. HIV [7–9], ries data, although a priori information about the fixed
hepatitis [10, 11], and influenza [12, 13]) and the immune point structure and basins of attraction of the dynamical
system response to infection [14, 15]. Population dynam- system would improve reconstruction. Our method in-
ics and epidemics have also been successfully modeled us- cludes several steps. We first identify a basis to sparsely
ing systems of ODEs [16]. In most of these cases, an ad represent the time series data using sparse dictionary
hoc ODE model with several parameters is posited [17], learning [22–24]. We then find the sparsest expansion
and solutions of the model are compared to experimental in the learned basis that is consistent with the measured
data to identify the relevant range of parameter values. data. This step can be formulated as solving an under-
A recent study has developed a more systematic com- determined system of linear equations. We will solve the
putational approach to identify the “best” ODE model underdetermined systems using L1 -norm regularized re-
to recapitulate time series data. The approach iteratively gression, which finds the solution to the system with the
 2

fewest nonzero expansion coefficients in the learned ba- namical system.

sis. We test our ODE reconstruction method on time The remainder of this manuscript is organized as fol-
series data generated from known ODE models in one-, lows. In the methods section (Sec. II), we provide a for-
two-, and three-dimensional systems, including both non- mal definition of sets of ODEs and details about obtain-
chaotic and chaotic dynamics. We quantify the accuracy ing the right-hand side functions of ODEs from numeri-
of the reconstruction for each system of ODEs as a func- cally differentiating time series data. We then introduce
tion of the amount of data used by the method. Further, the concept of L1 regularized regression and apply it to
we solve the reconstructed system of ODEs and compare the reconstruction of a sparse undersampled signal. We
the solution to the original time series data. The method introduce the concept of sparse basis learning to identify
developed and validated here can now be applied to large a basis in which the ODE can be represented sparsely. At
data sets for physical and biological systems for which the end of the methods section, we define the error met-
there is no known system of ODEs. ric that we will use to quantify the accuracy of the ODE
Identifying sparse representations of data (i.e. sparse reconstruction. In the results section (Sec. III), we per-
coding) is well studied. For example, sparse coding form ODE reconstruction on models in one-, two-, and
has been widely used for data compression, yielding the three-dimensional systems. For each system, we measure
JPEG, MPEG, and MP3 data formats. Sparse coding the reconstruction accuracy as a function of the amount
relies on the observation that for most signals a basis of data that is used for the reconstruction, showing ex-
can be identified for which only a few of the expansion amples of both accurate and inaccurate reconstructions.
coefficients are nonzero [25–27]. Sparse representations We end the manuscript in Sec. IV with a summary and
can provide accurate signal recovery, while at the same future applications of our method for ODE reconstruc-
time, reduce the amount of information required to de- tion.
fine the signal. For example, keeping only the ten largest
coefficients out of 64 possible coefficients in an 8 × 8 two-
dimensional discrete cosine basis (JPEG), leads to a size II. METHODS
reduction of approximately a factor of 6.
Recent studies have shown that in many cases perfect In this section, we first introduce the mathematical
recovery of a signal is possible from only a small num- expressions that define sets of ordinary differential equa-
ber of measurements of the signal [28–32]. This work tions (ODEs). We then describe how we obtain the sys-
provided a new lower bound for the amount of data re- tem of ODEs from time series data. In Secs. II B and II C,
quired for perfect reconstruction of a signal; for exam- we present the sparse reconstruction and sparse basis
ple, in many cases, one can take measurements at fre- learning methods that we employ to build sparse rep-
quencies much below the Nyquist sampling rate and still resentations of the ODE model. We also compare the
achieve perfect signal recovery. The related field of com- accuracy of sparse versus non-sparse methods for signal
pressed sensing emphasizes sampling the signal in com- reconstruction. In Sec. II D, we introduce the specific
pressed form to achieve perfect signal reconstruction [30– one-, two-, and three-dimensional ODE models that we
36]. Compressed sensing has a wide range of applications use to validate our ODE reconstruction methods.
from speed up of magnetic resonance image reconstruc-
tion [37–39] to more efficient and higher resolution cam-
eras [40, 41]. A. Systems of ordinary differential equations
Our ODE reconstruction method relies on the assump-
tion that the functions that comprise the “right-hand A general system of N nonlinear ordinary differential
sides” of the systems of ODEs can be sparsely repre- equations is given by
sented in some basis. A function f (~x) can be sparsely
representedPby a set of basis functions {φi }, i = 1, . . . , n dx1
n = f1 (x1 , x2 , . . . , xN )
if f (~x) = i=1 ci φi (~x) with only a small number s  n dt
of nonzero coefficients ci . This assumption is not as dx2
restrictive as it may seem at first. For example, sup- = f2 (x1 , x2 , . . . , xN )
dt (1)
pose we sample a two-dimensional function on a discrete ..
128 × 128 grid. Since there are 1282 = 16384 indepen- .
dent grid points, a complete basis would require at least dxN
n = 16384 basis functions. For most applications, we = fn (x1 , x2 , . . . , xN ),
dt
expect that a much smaller set of basis functions would
lead to accurate recovery of the function. In fact, the where fi with i = 1, . . . , N are arbitrary nonlinear func-
sparsest representation of the function is the basis that tions of all N variables xi and d/dt denotes a time deriva-
contains the function itself, where only one of the coef- tive. Although the functions fi (x1 , . . . , xN ) are defined
ficients ci is nonzero. Identifying sparse representations for all values of xi within a given domain, the time deriva-
of the system of ODEs is also consistent with the physics tives of the solution ~x(t) = (x1 (t), x2 (t), . . . , xN (t))T only
paradigm of finding the simplest model to explain a dy- sample the functions along particular trajectories. The
 3

functions fi can be obtained by taking numerical deriva- The solution to Eq. 5 can be obtained by minimizing
tives of the solutions with respect to time: the squared differences between the measurements of the
signal ~y and the reconstructed signal Φ~c subject to the
xi (t0 + ∆t) − xi (t0 ) constraint that the solution ĉ is sparse [43]:
fi (t0 ) ≈ . (2)
∆t
1
We will reconstruct the functions from a set of m mea- ĉ = min k~y − Φ~ck22 + λk~ck1 , (6)
c 2
~
surements, yk = fi (~xk ) at positions {~xk } with k =
1, . . . , m. To do this, we will express the functions fi as where k.kp denotes the Lp vector norm and λ is a Lan-
linear superpositions of arbitrary, non-linear basis func- grange multiplier that penalizes a large L1 norm of ĉ.
tions φj (~x): The Lp norm of an n-dimensional vector ~x is defined as
n " N
#1/p
X X
fi (~x) = cj φj (~x), (3) k~xkp = p
|xi | (7)
j=1 i=1

where cj are the expansion coefficients and n is the num- for p > 1, where N is the number of components of the
ber of basis functions used in the expansion. The mea- vector ~x.
surements yk = fi (~xk ) impose the following constraints We now demonstrate how the sparse signal reconstruc-
on the expansion coefficients: tion method compares to a standard least squares fit.
We first construct a sparse signal (with sparsity s) in a
n
X given basis. We then sample the signal randomly and at-
fi (~xk ) = yk = cj φj (~xk ) (4)
tempt to recover the signal using the regularized L1 and
j=1
least-squares reconstruction methods. For this example,
for each k = 1, . . . , m. The constraints we choose the discrete cosine basis. For a signal size of
Pnin Eq. 4 can also 100 values, we have a complete and orthonormal basis of
be expressed as a matrix equation: j=1 Φij cj = yi for
i = 1, . . . , m or 100 functions φn (i) (n = 0, . . . , 99) each with 100 values
     (i = 0, . . . , 99):
φ1 (~x1 ) . . . φn (~x1 ) c1 y1    
 φ1 (~x2 ) . . . φn (~x2 )   c2   y2  π 1
φn (i) = F (n) cos i+ n , (8)

 ..   ..  =  ..  ,
   
(5) 100 2
 .  .   . 
φ1 (~xm ) . . . φn (~xm ) cn ym where F (n) is a normalization factor

where Φij = φj (~xi ). In most cases of ODE reconstruc-  √1 for n = 0
100
tion, the number of rows of Φ, i.e. the number of mea- F (n) = q 2 (9)

100 for n = 1, . . . , 99.
surements m, is smaller than the number of columns of
Φ, which is equal to the number of basis functions n used
Note that an orthonormal basis is not a prerequisite for
to represent the signals fi . Thus, in general, the system
the sparse reconstruction method.
of equations (Eq. 5) is underdetermined with n > m.
Similar to Eq. 3, we can express the signal as a super-
Sparse coding is ideally suited for solving underdeter-
position of basis functions,
mined systems because it seeks to identify the minimum
number of basis functions to represent the signals fi . If 99
X
we identify a basis that can represent a given set of sig- g(i) = cj φj (i) . (10)
nals fi sparsely, an L1 regularized minimization scheme j=0
will be able to find the sparsest representation of the sig-
nals [42]. The signal ~g of sparsity s is generated by randomly select-
ing s of the coefficients cj and assigning them a random
amplitude in the range [−1, 1]. We then evaluate g(i) at
B. Sparse Coding m randomly chosen positions i and attempt to recover
g(i) from the measurements. If m < 100, recovering the
original signal involves solving an underdetermined sys-
In general, the least squares (L2 ) solution of Eq. 5
tem of linear equations.
possesses many nonzero coefficients ci , whereas the min-
Recovering the full signal ~g from a given number of
imal L1 solution of Eq. 5 is sparse and possesses only
measurements proceeds as follows. After carrying out
a few non-zero coefficients. In the case of underdeter-
the m measurements, we can rewrite Eq. 4 as
mined systems with many available basis functions, it has
been shown that a sparse solution obtained via L1 reg- ~y = P~g , (11)
ularization more accurately represents the solution com-
pared to those that are superpositions of many basis func- where ~y is the vector of the measurements of ~g and P is
tions [42]. the projection matrix with m × n entries that are either
 4

0 or 1. Each row has one nonzero element that corre-

sponds to the position of the measurement. For each
random selection of measurements of ~g , we solve the re-
duced equation

~y = Θ~c , (12) (a) (b)

where Θ = P Φ. After solving Eq. 12 for ĉ, we obtain a

reconstruction of the original signal

~grec = Φ ĉ . (13)

Fig. 1 shows examples of L1 and L2 reconstruction

methods of a signal as a function of the fraction of the (c) (d)
signal (M = 0.2 to 1) included in the reconstruction
method. Even when only a small fraction of the sig-
nal is included (down to M = 0.2), the L1 reconstruction
method achieves nearly perfect signal recovery. In con-
trast, the least-squares method only achieves adequate
recovery of the signal for M > 0.9. Moreover, when only
a small fraction of the signal is included, the L2 method (e) (f)
is dominated by the mean of the measured points and
oscillates rapidly about the mean to match each mea-
surement.
In Fig. 2, we measured the recovery error d between FIG. 1: Comparison of the least squares L2 (green dashed
the original (~g ) and recovered (~grec ) signals as a function lines) and L1 (blue solid lines) regularized regression to re-
of the fraction M of the signal included and for several cover a randomly generated, discrete signal (red dots in panel
(a)) obtained P by randomly sampling points i from the func-
sparsities s. We define the recovery error as
tion g(i) = 99 n=0 cn φn (i), where φn (i) is given by Eq. 8 and
~g · ~grec i = 0, . . . , 99 is chosen so that all n frequencies can be re-
d(~g , ~grec ) = 1 − , (14) solved. The function was constructed to be 3 sparse (s = 3)
k~g k2 k~grec k2
with only c0 , c4 , and c15 6= 0. The six panels (a)-(f) show the
where ~g · ~grec denotes the inner product between the two reconstructions for M = 1, 0.9, 0.8, 0.5, 0.2, and 0.1 of the
input signal, respectively, included in the measurements. The
vectors ~g and ~grec . This distance function satisfies 0 ≤
red dots in panels (a)-(f) indicate the fraction of the signal
d ≤ 2, where d ≥ 1 signifies a large difference between ~g that was measured and used for the reconstruction.
and ~grec and d = 0 indicates ~g = ~grec .
For sparsity values s = 1 and 3, the L1 reconstruc-
tion gives small errors (d ∼ 0) for M ≥ 0.2 (Fig. 2). In that identifies a basis in which a signal can be expressed
contrast, the error for the L2 reconstruction method is sparsely. This approach is compelling because it does
nonzero for all M < 1 for all s. For a non-sparse sig- not require significant prior knowledge about the signal
nal (s = 20), the L1 and L2 reconstruction methods give and it allows the basis to be learned even from noisy or
similar errors for M . 0.2. In this case, the measure- incomplete data.
ments truly undersample the signal, and thus providing
Sparse basis learning seeks to find a basis Φ that can
less than 20 measurements is not enough to constrain
represent an input of several signals sparsely. We identify
the 20 nonzero coefficients cj . However, when M & 0.2,
Φ by decomposing the signal matrix Y = (~y1 , ~y2 , . . . , ~ym ),
d from the L1 reconstruction method is less than that
where m is the number of signals, into the basis matrix
from the L2 method and is nearly zero for M & 0.5.
Φ and coefficient matrix C,

C. Sparse Basis Learning Y = ΦC . (15)

Columns ~ci of C are the sparse coefficient vectors that

The L1 reconstruction method described in the previ-
represent the signals ~yi in the basis Φ. Both C and Φ
ous section works well if 1) the signal has a sparse repre-
are unknown and can be determined by minimizing the
sentation in some basis and 2) the basis Φ (or a subset
squared differences between the signals and their repre-
of it) contains functions similar to the basis in which the
sentations in the basis Φ subject to the constraint that
signal is sparse. How do we proceed with signal recon-
the coefficient matrix is sparse [43, 47]:
struction if we do not know a basis in which the signal is
sparse? One method is to use one of the common basis m  
sets, such as wavelets, sines, cosines, or polynomials [44–
X 1
Ĉ, Φ̂ = min k~yi − Φ~ci k22 + λk~ci k1 , (16)
46]. Another method is to employ sparse basis learning C,Φ
i=1
2
 5

(a) (b)
(a)

FIG. 3: (a) 128 × 128 pixel image of a cat. (b) Using the
(b) sparse learning method, we obtained 100 8 × 8 pixel basis
functions for the image in (a).

(a) (b)

(c)

FIG. 4: (a) A random collection of pixels (sampling mask)

that represents 30% of the image in Fig. 3 (a). (b) Sparse
basis learning reconstruction of the image using samples of
the image at the locations of the black pixels in (a).

from an undersampled version using the learned basis

FIG. 2: Comparison of the least-squares L2 (green dashed functions in Fig. 3 (b) and then performing sparse recon-
lines) and L1 (solid blue lines) reconstruction errors d (Eq. 14) struction (Eq. 6) for each 8×8 patch of the undersampled
as a function of the fraction M of the input signal g(i) in Fig. 1 image. For this example, we randomly sampled ≈ 30% of
for three values of the sparsity (a) s = 1, (b) 3, and (c) 20. the original image. In Fig. 4 (a), the black pixels indicate
the random pixels used for the sparse reconstruction of
the undersampled image. We decompose the undersam-
pled image into all possible 8 × 8 patches, using only the
where λ is a Langrange multiplier that determines the
measurements marked by the black pixels in the sampling
sparsity of the coefficient matrix C.
mask in Fig. 4 (a). While the reconstruction of the im-
To illustrate the basis learning method, we show the age in Fig. 4 (b) is somewhat grainy, this reconstruction
results of sparse basis learning on the complex, two- method clearly resembles the original image even when
dimensional image of a cat shown in Fig. 3 (a). To learn it is 70% undersampled.
a sparse basis for this image, we decomposed the original In this work, we show that one may also use incomplete
image (128 × 128 pixels) into all possible 8 × 8 patches, data to learn a sparse basis. For example, the case of a
which totals 14, 641 unique patches. The patches were discrete representation of a two-dimensional system of
then reshaped into one-dimensional signals ~yi each con- ODEs is the same problem as basis learning for image
taining 64 values. We chose 100 basis functions (columns reconstruction (Fig. 3). However, learning the basis from
of Φ) to sparsely represent the input matrix Y . Fig. 3 solutions of the system of ODEs, does not provide full
(b) shows the 100 basis functions that were obtained by sampling of the signal (i.e. the right-hand side of the
solving Eq. 16. The 100 × 64 matrix Φ was reshaped into system of ODEs in Eq. 1), because the dynamics of the
8×8 pixel basis functions before plotting. Note that some system is strongly affected by the fixed point structure
of the basis functions display complicated features, e.g., and the functions are not uniformly sampled.
lines and ripples of different widths and angles, whereas To learn a basis from incomplete data, we decompose
others are more uniform. the signal into patches of a given size and then fill in the
To demonstrate the utility of the sparse basis learn- missing values with random numbers. We convert the
ing method, we seek to recover the image in Fig. 3 (a) padded patches (i.e. original plus random signal) into a
 6

signal matrix Y and learn a basis Φ to sparsely represent (a) (b)

the signal by solving Eq. 16. To recover the signal, we
find a sparse representation ĉ of the unpadded signal (i.e.
without added random values) in the learned basis Φ by
solving Eq. 12, where P is the matrix that selects only
the signal entries that have been measured. When then
obtain the reconstructed patch by taking the product Φĉ.
We repeat this process for each patch to reconstruct the
full domain. For cases in which we obtain different values
for the signal at the same location from different patches,
we average the result.
FIG. 5: (a) The function f1 (x1 ) for the Reynolds ODE model
in one spatial dimension (Eq. 17) for the pathogen load in
the range 0 ≤ x1 ≤ 20. Fixed points are marked by solid
D. Models circles. (b) Close up of f1 (x1 ) near the stable fixed point at
x1 = 0 and unstable fixed point at x1 = 0.31. These two fixed
points are difficult to discern on the scale shown in panel (a).
We test our methods for the reconstruction of systems
Trajectories with initial conditions between x1 = 0 and 0.31
of ODEs using synthetic data, i.e. data generated by will move towards the stable fixed point at x1 = 0, while
numerically solving systems of ODEs, which allows us initial conditions with x1 > 0.31 will move toward the fixed
to test quantitatively the accuracy as a function of the point at x1 = 19.49.
amount of data used in the reconstruction. We present
results from systems of ODEs in one, two, and three
dimensions with increasing complexity in the dynamics. (a) (b)
For an ODE in one dimension (1D), we only need to re-
construct one nonlinear function f1 of one variable x1 . In
two dimensions (2D), we need to reconstruct two func-
tions (f1 and f2 ) of two variables (x1 and x2 ), and in
three dimensions (3D), we need to reconstruct three func-
tions (f1 , f2 , and f3 ) of three variables (x1 , x2 , and x3 )
to reproduce the dynamics of the system. Each of the
systems that we study possesses a different fixed point
structure in phase space. The 1D model has two stable
FIG. 6: (a) Pathogen load x1 as a function of time t in the
fixed points and one unstable fixed point, and thus all range 0 ≤ t ≤ 30 for the Reynolds ODE model in one spatial
trajectories evolve toward one of the stable fixed points. dimension (Eq. 17). (b) Close up of the region 0 ≤ t ≤ 15,
The 2D model has one saddle point and one oscillatory which highlights the behavior of the system near the unstable
fixed point with closed orbits as solutions. The 3D model fixed point at x1 = 0.31. Solutions (red solid lines) with
we study has no stable fixed points and instead possesses initial conditions 0 ≤ x1 ≤ 0.31 are attracted to the stable
chaotic dynamics on a strange attractor. fixed point at x1 = 0, while solutions (blue solid lines) with
a. 1D model For 1D, we study the Reynolds model initial conditions x1 > 0.31 are attracted to the stable fixed
for the immune response to infection [14]: point at x1 = 19.49.
 
dx1 x1 kpm sm x1
= f1 (x1 ) = kpg x1 1 − − , (17) two stable fixed points at x1 = 0 and 19.49 and one un-
dt x∞ µm + kmp x1
stable fixed point, separating the two stable fixed points,
where the pathogen load x1 is unitless, and the other pa- at x1 = 0.31 (Fig. 5). As shown in Fig. 6, solutions to
rameters kpg , kpm , kmp , and sm have units of inverse Eq. 17 with initial conditions 0 ≤ x1 ≤ 0.31 are attracted
hours. The right-hand side of Eq. 17 is the sum of to the stable fixed point at x1 = 0, while solutions with
two terms. The first term enables logistic growth of the initial conditions x1 > 0.31 are attracted to the stable
pathogen load. In the absence of any other terms, any fixed point at x1 = 19.49.
positive initial value will cause x1 to grow logistically to b. 2D model In 2D, we focused on the Lotka-
the steady-state value x∞ . The second term mimics a lo- Volterra system of ODEs that describe predator-prey dy-
cal, non-specific response to an infection, which reduces namics [49]:
the pathogen load. For small values of x1 , the decrease is dx1
proportional to x1 . For larger values of x1 , the decrease = f1 (x1 , x2 ) = α x1 − β x1 x2
dt (18)
caused by the second term is constant. dx2
We employed the parameter values kpg = 0.6, x∞ = = f2 (x1 , x2 ) = −γ x2 + δ x1 x2 ,
dt
20, kpm = 0.6, sm = 0.005, µm = 0.002, and kmp =
0.01, which were used in previous studies of this ODE where x1 and x2 describe the prey and predator popula-
model [14, 48]. In this parameter regime, Eq. 17 exhibits tion sizes, respectively, and are unitless. In this model,
 7

prey have a natural growth rate α. In the absence of 3.0

predators, the prey population x1 would grow exponen-
tially with time. With predators present, the prey popu- 2.5
lation decreases at a rate proportional to the product of
both the predator and prey populations with a propor- 2.0
tionality constant β (with units of inverse time). Without 1.5

x2
predation, the predator population x2 would decrease at
death rate γ. With the presence of prey x1 , the preda- 1.0
tor population grows proportional to the product of the
two population sizes x1 and x2 with a proportionality 0.5
constant δ (with units of inverse time). 0.0
For the Lotka-Volterra system of ODEs, there are two 0.0 0.5 1.0 1.5 2.0 2.5 3.0
fixed points, one at x1 = 0 and x2 = 0 and one at x1 =
γ/δ and x2 = α/β. The stability of the fixed points
x1
is determined by the eigenvalues of the Jacobian matrix
evaluated at the fixed points. The Jacobian of the Lotka- FIG. 7: Solutions of the two-dimensional Lotka-Volterra ODE
Volterra system is given by model (Eq. 18) with α = 0.4, β = 0.4, γ = 0.1, and δ =
  0.2 plotted parametrically (x2 (t) versus x1 (t)) for 15 random
α − βx2 −βx1 initial conditions in the domain 0 ≤ x1 ≤ 0.5 and 0 ≤ x2 ≤ 1.
JLV = . (19)
δx2 −γ + δx1 This model has an unstable fixed point at x1 = 0 and x2 = 0
and an oscillatory fixed point at x1 = 0.5 and x2 = 1.
The eigenvalues of the Jacobian JLV at the origin are
J1 = α, J2 = −γ. Since the model is restricted to posi-
tive parameters, the fixed point at the origin is a saddle and chaotic dynamics. The three dimensionless variables
point. The interpretation is that for small populations x1 , x2 , and x3 describe the intensity of the convective
of predator and prey, the predator population decreases flow, temperature difference between the ascending and
exponentially due to the lack of a food source. While un- descending fluid, and spatial dependence of the temper-
harmed by the predator, the prey population can grow ature profile, respectively.
exponentially, which drives the system away from the The system possesses three fixed points at
zero population state, x1 = 0 and x2 = 0. (x1 , x2 , x3 ) = (0, 0, 0), (−β 1/2 (ρ − 1)1/2 , −β 1/2 (ρ −
The eigenvalues of the Jacobian JLV at the second 1)1/2 , ρ − 1), and (β 1/2 (ρ − 1)1/2 , β 1/2 (ρ − 1)1/2 , ρ − 1).
fixed point x1 = γ/δ and x2 = α/β are purely imagi- The Jacobian of the system is given by
√ √
nary complex conjugates, J1 = −i αγ and J2 = i αγ,
where i2 = −1. The purely imaginary fixed point causes
 
−σ σ 0
trajectories to revolve around it and form closed orbits. JL = ρ − x3 −1 −x1  . (21)
The interpretation of this fixed point is that the predators x2 x1 −β
decrease the number of prey, then the predators begin to
die due to a lack of food, which in turn allows the prey When we evaluate the Jacobian (Eq. 21) at the fixed
population to grow. The growing prey population pro- points, we find that each of the three eigenvalues pos-
vides an abundant food supply for the predator, which sesses two stable and one unstable eigendirection in the
allows the predator to grow faster than the food supply parameter regime σ = 10, ρ = 28 and β = 8/3. With
can sustain. The prey population then decreases and the these parameters, the Lorenz system displays chaotic dy-
cycle repeats. For the results below, we chose the pa- namics with Lyapunov exponents `1 = 0.91, `2 = 0.0,
rameters α = 0.4, β = 0.4, γ = 0.1, and δ = 0.2 for and `3 = −14.57. In Fig. 8, we show the time evolution
the Lotka-Volterra system, which locates the oscillatory of two initial conditions in x1 -x2 -x3 configuration space
fixed point at x1 = 0.5 and x2 = 1.0 (Fig. 7). for this parameter regime.
c. 3D model In 3D, we focused on the Lorenz system
of ODEs [1], which describes fluid motion in a container
that is heated from below and cooled from above: III. RESULTS
dx1
= f1 (x1 , x2 , x3 ) = σ(x2 − x1 ) In this section, we present the results of our methodol-
dt
dx2 ogy for ODE reconstruction of data generated from the
= f2 (x1 , x2 , x3 ) = x1 (ρ − x3 ) − x2 (20) three systems of ODEs described in Sec. II D. For each
dt
system, we measure the accuracy of the reconstruction as
dx3
= f3 (x1 , x2 , x3 ) = x1 x2 − βx3 , a function of the size of the sections used to decompose
dt the signal for basis learning, the sampling time interval
where σ, ρ, β are positive, dimensionless parameters that between time series measurements, and the number of
represent properties of the fluid. In different parame- trajectories. For each model, we make sure that the to-
ter regimes, the fluid can display quiescent, convective, tal integration time is sufficiently large that the system
 8

100
10-1
10-2

d
10-3
10-4 0 20 40 60 80 100 120 140
p

FIG. 9: Error d in the ODE reconstruction for the 1D

Reynolds model (Eq. 17) as a function of the patch size p
FIG. 8: Solutions of the three-dimensional Lorenz ODE model used for basis learning using Nt = 10 trajectories with sam-
(Eq. 20) with σ = 10, ρ = 28, and β = 8/3 plotted pling time interval ∆t = 1.
parametrically in the x1 -x2 -x3 plane for two initial condi-
tions (x1 , x2 , x3 ) = (−7.57, 11.36, 9.51) (red solid line) and
(−12.17, −5.23, −11.60) (blue solid line). For this parameter f1 (x1 ) as a function of the sampling time interval ∆t for
regime, the system has three unstable fixed points, lives on
several numbers of trajectories Nt = 1, 2, 20, 50, and
the Lorenz attractor, and possesses chaotic dynamics.
200. We find that the error decreases with the number
of trajectories used in the reconstruction. For Nt = 1,
the error is large with d > 0.1. For large numbers of
can reach the stable fixed points or sample the chaotic
trajectories (e.g. Nt = 200), the error decreases with
attractor in the case of the Lorenz system.
decreasing ∆t, reaching d ∼ 10−5 for small ∆t. The
fact that the error in the ODE reconstruction increases
with ∆t is consistent with notion that the accuracy of the
A. Reconstruction of ODEs in 1D numerical derivative of each trajectory decreases with in-
creasing sampling interval. In Fig. 11, we show the error
We first focus on the reconstruction of the Reynolds in the reconstruction of f1 (x1 ) as a function of the total
ODE model in 1D (Eq. 17) using time series data. We integration time tend . We find that d decreases strongly
discretized the domain 0 ≤ x1 ≤ 19.5 using 256 points, as tend increases for tend < 20. For tend > 20, d reaches
i = 0, . . . , 255. Because the unstable fixed point at a plateau value below 10−4 , which depends weakly on
x1 = 0.31 is much closer to the stable fixed point at ∆t. For characteristic time scales t > 20, the Reynolds
x1 = 0 than to the stable fixed point at x1 = 19.49, ODE model reaches one of the two stable fixed points,
we sampled more frequently in the region 0 ≤ x1 ≤ 0.6 and therefore d becomes independent of tend .
compared to the region 0.6 < x1 ≤ 19.5. In particular, In Fig. 12, we compare accurate (using Nt = 50 and
we uniformly sampled 128 points from the small domain, ∆t = 0.1) and inaccurate (using Nt = 10 and ∆t = 5) re-
and uniformly sampled the same number of points from constructions of f1 (i1 ) for the 1D Reynolds ODE model.
the large domain. Note that we plot f1 as a function of the scaled vari-
In Fig. 9, we show the error d (Eq. 14) in recovering able i1 . The indexes i1 = 0, . . . , 127 indicate uniformly
the right-hand side of Eq. 17 (f1 (x1 )) as a function of the spaced x1 values in the interval 0 ≤ x1 ≤ 0.6, and
size p of the patches used for basis learning. Each data i1 = 128, . . . , 256 indicate uniformly spaced x1 values in
point in Fig. 9 represents an average over 20 reconstruc- the interval 0.6 < x1 ≤ 19.5.
tions using Nt = 10 trajectories with a sampling time We find that using large ∆t gives rise to inaccurate
interval ∆t = 1. We find that the error d achieves a min- measurements of the time derivative of x1 and, thus of
imum below 10−3 in the patch size range 30 < p < 50. f1 (x1 ). In addition, large ∆t does not allow dense sam-
Basis sizes that are too small do not adequately sam- pling of phase space, especially in regions where the tra-
ple f1 (x1 ), while basis patches that are too large do not jectories evolve rapidly. The inaccurate reconstruction
include enough variability to select a sufficiently diverse in Fig. 12 (b) is even worse than it seems at first glance.
basis set to reconstruct f1 (x1 ). For example, in the ex- The reconstructed function is identically zero over a wide
treme case that the basis patch size is the same size as range of i1 (0 ≤ i1 ≤ 50) where f (i1 ) is not well sampled,
the signal, we are only able to learn the input data itself, since the default output of a failed reconstruction is zero.
which may be missing data. For the remaining studies of It is a coincidence that f1 (i1 ) ∼ 0 in Eq. 17 over the same
the 1D model, we set p = 50 as the basis patch size. range of i1 .
In Fig. 10, we plot the error in the reconstruction of We now numerically solve the reconstructed 1D
 9

100 (a)

10-1

10-2
d

10-3

10-4

10-5
10-1 100 101 (b)
∆t

FIG. 10: Reconstruction error d for the 1D Reynolds model

as a function of the sampling interval ∆t for several different
numbers of trajectories Nt = 1 (circles), 5 (stars), 10 (trian-
gles), 20 (squares), 50 (diamonds), and 200 (pentagons) used
in the reconstruction. The data for each Nt is averaged over
20 independent reconstructions.

100
10-1
10-2 FIG. 12: Reconstructions (solid blue lines) of f1 (i1 ) for the 1D
d

Reynolds ODE model in Eq. 17 using (a) Nt = 50 and ∆t =

10-3 0.1 and (b) Nt = 10 and ∆t = 5. f1 is discretized using 256
10-4 points. The indices i1 = 0, . . . , 127 indicate uniformly spaced
x1 values in the interval 0 ≤ x1 ≤ 0.6, and i1 = 128, . . . , 256
10-5 0 20 40 60 80 100
indicate uniformly spaced x1 values in the interval 0.6 < x1 ≤
19.5. The exact expression for f1 (i1 ) is represented by the
tend green circles.

FIG. 11: Reconstruction error d for the 1D Reynolds ODE

sentation of f1 (x1 ).
model as a function of the total integration time tend used for
the reconstruction for Nt = 20 trajectories at several values
of the sampling time ∆t = 0.05 (circles), 0.1 (stars), and 0.5
(triangles). B. Reconstruction of Systems of ODEs in 2D

We now investigate the reconstruction accuracy of our

Reynolds ODE model for different initial conditions and method for the Lotka-Volterra system of ODEs in 2D.
times comparable to tend and compare these trajectories We find that the results are qualitatively similar to those
to those obtained from the original model (Eq. 17). In for the 1D Reynolds ODE model. We map the numerical
Fig. 13, we compare the trajectories x1 (t) for the accurate derivatives for Nt trajectories with a sampling time inter-
(d ∼ 10−5 ; Fig. 12 (a)) and inaccurate (d ∼ 10−2 ; Fig. 12 val ∆t onto a 128 × 128 grid with 0 ≤ x1 , x2 ≤ 2. Similar
(b)) representations of f1 (x1 ) to the original model for six to the 1D model, we find that the error in the reconstruc-
initial conditions. All of the trajectories for the accurate tion of f1 (x1 , x2 ) and f2 (x1 , x2 ) possesses a minimum as
representation of f1 (x1 ) are nearly indistinguishable from a function of the patch area used for basis learning, where
the trajectories for the original model, whereas we find the location and value at the minimum depends on the
large deviations between the original and reconstructed parameters used for the reconstruction. For example,
trajectories even at short times for the inaccurate repre- for Nt = 200, ∆t = 0.1, and averages over 20 indepen-
 10

100
(a)

10-1

10-2

d
10-3

10-4 -2
10 10-1 100 101
(b) ∆t

FIG. 14: Reconstruction error d for the 2D Lotka-Volterra

model (Eq. 18) as a function of the sampling time interval
∆t for different numbers of trajectories Nt = 5 (circles),
10 (stars), 20 (rightward triangles), 50 (squares), 100 (dia-
monds), 200 (pentagons), and 500 (upward triangles) aver-
aged over 20 independent runs.

indicate uniformly spaced x1 and x2 values in the inter-

val 0 ≤ x1,2 ≤ 2. The parameters for the inaccurate
reconstructions were Nt = 30 trajectories and ∆t = 5
(enabling f1 and f2 to be sampled only over 2% of the
FIG. 13: (a) Comparison of the trajectories x1 (t) for the 1D
Reynolds ODE model using the accurate (solid blue lines) and domain), whereas the parameters for the accurate recon-
inaccurate (dashed red lines) reconstructions of f1 (x1 ) shown structions were Nt = 100 trajectories and ∆t = 0.01 (en-
in Fig. 12 (a) and (b), respectively, for six initial conditions: abling f1 and f2 to be sampled over 68% of the domain).
x1 (0) = 0.1, 0.2, 0.3, 1, 2, 5, and 10. The trajectories obtained These results emphasize that even though the derivatives
from f1 (x1 ) in Eq. 17 are given by the open red circles. (b) are undetermined over nearly one-third of the domain, we
Close up of the trajectories in the domain 0 ≤ x1 ≤ 0.4. can reconstruct the functions f1 and f2 extremely accu-
rately over the full domain.

dent runs, the error reaches a minimum (d ≈ 10−5 ) near Using the reconstructions of f1 and f2 , we solved for
p2min ≈ 100. the trajectories x1 (t) and x2 (t) (for times comparable to
In Fig. 14, we show the reconstruction error d as a tend ) and compared them to the trajectories from the
function of the sampling time interval ∆t for several val- original Lotka-Volterra model (Eq. 18). In Fig. 17 (a)
ues of Nt from 5 to 500 trajectories, for a total time tend and (b), we show parametric plots (x2 (t) versus x1 (t))
that allows several revolutions around the closed orbits, for the inaccurate (Fig. 15) and accurate (Fig. 16) recon-
and for patch size p2min . As in 1D, we find that increas- structions of f1 and f2 , respectively. We solved both the
ing Nt reduces the reconstruction error. For Nt = 5, inaccurate and accurate models with the same four sets
d ∼ 10−1 , while d < 10−3 for Nt = 500. d also decreases of initial conditions. For the inaccurate reconstruction,
with decreasing ∆t, although d reaches a plateau in the most of the trajectories from the reconstructed ODE sys-
small ∆t limit, which depends on the number of trajec- tem do not match the original trajectories. In fact, some
tories included in the reconstruction. of the trajectories spiral outward and do not form closed
In Figs. 15 and 16, we show examples of inaccurate orbits. In contrast, for the accurate reconstruction, the
(d ∼ 10−2 ) and accurate (d ∼ 10−5 ) reconstructions of reconstructed trajectories are very close to those of the
f1 (i1 , i2 ) and f2 (i1 , i2 ). The indexes i1,2 = 0, . . . , 127 original model and all possess closed orbits.
 11

(a) (b) (c) (a) (b) (c)

(d) (e) (f) (d) (e) (f)

FIG. 15: Example of inaccurate reconstructions of f1 (i1 , i2 ) FIG. 16: Example of accurate reconstructions of f1 (i1 , i2 ) and
and f2 (i1 , i2 ) (with errors d = 0.04 and 0.03, respectively) f2 (i1 , i2 ) (with errors d = 4 × 10−5 and 5 × 10−5 , respectively)
for the 2D Lotka-Volterra system of ODEs (18). The indexes for the 2D Lotka-Volterra system of ODEs (18). The indexes
i1,2 = 0, . . . , 127 indicate uniformly spaced x1 and x2 values i1,2 = 0, . . . , 128 indicate uniformly spaced x1 and x2 values
in the interval 0 ≤ x1,2 ≤ 2. Panels (a) and (d) give the in the interval 0 ≤ x1,2 ≤ 2. Panels (a) and (d) give the
exact functions f1 (i1 , i2 ) and f2 (i1 , i2 ), panels (b) and (e) exact functions f1 (i1 , i2 ) and f2 (i1 , i2 ), panels (b) and (e) give
give the reconstructions, and panels (c) and (f) indicate the the reconstructions, and panels (c) and (f) indicate the i1 -i2
i1 -i2 values that were sampled (2% of the 128 × 128 grid). values that were sampled (68% of the 128 × 128 grid). The
The white regions in panel (c) and (f) indicate missing data. white regions in panel (c) and (f) indicate missing data. The
In the top row, the color scale varies from −0.8 to 0.8 (from color scales are the same as in Fig. 15. This reconstruction
dark blue to red). In the bottom row, the color scale varies was obtained using Nt = 100 trajectories, a sampling interval
from −0.2 to 0.6 (from dark blue to red). This reconstruction of ∆t = 0.01, and a basis patch area p2 = 625.
was obtained using Nt = 30 trajectories, a sampling interval
∆t = 5, and a basis patch area p2 = 625.
(a) (b)

C. Reconstruction of systems of ODEs in 3D

For the Lorenz ODE model, we need to recon-

struct three functions of three variables: f1 (x1 , x2 , x3 ),
f2 (x1 , x2 , x3 ), and f3 (x1 , x2 , x3 ). Based on the selected
parameters σ = 10, ρ = 28 and β = 8/3, we chose a
32 × 32 × 32 discretization of the domain −21 ≤ x1 ≤ 21,
−29 ≤ x2 ≤ 29, and −2 ≤ x3 ≤ 50. We employed
patches of size 10 × 10 from each of the 32 slices (along
x3 ) of size 32 × 32 (in the x1 -x2 plane) to perform the
FIG. 17: Parametric plots of the trajectories x1 (t) and x2 (t)
basis learning. (solid lines) for the (a) inaccurate and (b) accurate recon-
In Fig. 18, we plot the reconstruction error d versus the structions of f1 and f2 in Figs. 15 and 16, respectively, for
sampling time interval ∆t for several Nt from 500 to 104 four different initial conditions indicated by the crosses. The
trajectories. As found for the 1D and 2D ODE models, open circles indicate the trajectories from the Lotka-Volterra
the reconstruction error decreases with decreasing ∆t and system of ODEs (Eq. 18).
increasing Nt . d reaches a low-∆t plateau that depends
on the value of Nt . For Nt = 104 , the low-∆t plateau
value for the reconstruction error approaches d ∼ 10−3 . used for the reconstructions (with white regions indicat-
In Fig. 19, we visualize the reconstructed functions f1 , ing missing data). The central regions of the functions
f2 , and f3 for the Lorenz system of ODEs. Panels (a)-(c) are recovered with high accuracy. (The edges of the do-
represent f1 , (d)-(f) represent f2 , and (g)-(i) represent main were not well-sampled, and thus the reconstruction
f3 . The 3D domain is broken into 32 slices (along x3 ) was not as accurate.) These results show that even for
of 32 × 32 grid points in the x2 -x3 plane. Panels (a), chaotic systems in 3D we are able to achieve accurate
(d), and (g) give the original functions f1 , f2 , and f3 in ODE reconstruction. In Fig. 20, we compare trajecto-
the Lorenz system of ODEs (Eq. 20). Panels (b), (e), ries from the reconstructed functions to those from the
and (h) give the reconstructed versions of f1 , f2 , and f3 , original Lorenz system of ODEs for times comparable to
and panels (c), (f), and (i) provide the data that was the inverse of the largest Lyapunov exponent. In this
 12

100 parameter regime we studied, this system possesses an os-

cillatory fixed point with closed orbits. In 3D, we studied
the Lorenz model for convective flows. In the parameter
10-1 regime we considered, the system displays chaotic dy-
namics on a strange attractor.
d

For each model, we measured the error in the recon-

10-2 structed system of ODEs as a function of parameters of
the reconstruction method including the sampling time
interval ∆t, number of trajectories Nt , total time tend of
10-3 -3 10-1
10 10-2 100 the trajectory, and size of the patches used for basis func-
tion learning. In general, the error decreases as more data
∆t is used for the reconstruction. We determined the param-
eter regimes for which we could achieve highly accurate
reconstruction with errors d < 10−3 . We then generated
FIG. 18: Reconstruction error d plotted versus the sampling trajectories from the reconstructed systems of ODEs and
time interval ∆t for the 3D Lorenz model Eq. (20) for sev- compared them to the trajectories of the original models.
eral different numbers of trajectories Nt = 500 (circles), 1000 For the 1D model with two stable fixed points, we were
(stars) 5000 (rightward triangles), and 10000 (squares). Each able to achieve extremely accurate reconstruction and
data point is averaged over 20 independent reconstructions. recapitulation of the trajectories of the original model.
Our reconstruction for the 2D model is also accurate and
is able to achieve closed orbits for most initial condi-
case, we find that some of the the trajectories from the tions. For some of the initial conditions, smaller sam-
reconstructed model closely match those from the origi- pling time intervals and longer trajectories were needed
nal model, while others differ from the trajectories of the to achieve reconstructed solutions with closed orbits. In
original model. Since chaotic systems are extremely sen- future studies, we will investigate methods to add a con-
sitive to initial conditions, we expect that all trajectories straint that imposes the constant of the motion on the
of the reconstructed model will differ from the trajecto- reconstruction method, which will allow us to use larger
ries of the original model at long times. Despite this, the sampling time intervals and shorter trajectories and still
trajectories from the reconstructed model display chaotic achieve closed orbits. For the 3D chaotic Lorenz system,
dynamics with similar Lyapunov exponents to those for we can only match the trajectories of the reconstructed
the Lorenz system of ODEs, and thus we are able to re- and original systems for times that are small compared
cover the controlling dynamics of the original model. to the inverse of the largest Lyapunov exponent. Even
though the trajectories of the reconstructed and origi-
nal systems will diverge, we have shown that the recon-
IV. DISCUSSION
structed and original systems of ODEs possess dynamics
with similar Lyapunov exponents. Now that we have
We developed a new method for reconstructing sets validated this ODE reconstruction method on known de-
of nonlinear ODEs from time series data using machine terministic systems of ODEs and determined the param-
learning methods involving sparse function reconstruc- eter regimes that yield accurate reconstructions, we will
tion and sparse basis learning. Using only information employ this method in future studies to identify new sys-
from the system trajectories, we first learned a sparse tems of ODEs using time series data from experimental
basis, with no a priori knowledge of the underlying func- systems for which there is no currently known system of
tions in the system of ODEs, and then reconstructed ODEs.
the system of ODEs in this basis. A key feature of
our method is its reliance on sparse representations of
the system of ODEs. Our results emphasize that sparse
representations provide more accurate reconstructions of
Acknowledgments
systems of ODEs than least-squares approaches.
We tested our ODE reconstruction method on time se-
ries data obtained from systems of ODEs in 1D, 2D, and This work was partially supported by DARPA (Space
3D. In 1D, we studied the Reynolds model for the im- and Naval Warfare System Center Pacific) under award
mune response to infection. In the parameter regime we number N66001-11-1-4184. These studies also benefited
considered, this system possesses only two stable fixed from the facilities and staff of the Yale University Fac-
points, and thus all initial conditions converge to these ulty of Arts and Sciences High Performance Computing
fixed points in the long-time limit. In 2D, we studied the Center and NSF Grant No. CNS-0821132 that partially
Lotka-Volterra model for predator-prey dynamics. In the funded acquisition of the computational facilities.
 13

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

FIG. 19: Reconstruction of the Lorenz system of ODEs in 3D (Eq. 20) using the sampling time interval ∆t = 10−2 and
Nt = 104 trajectories. Panels (a)-(c) indicate f1 (i1 ), (d)-(f) indicate f2 (i2 ), and (g)-(i) indicate f3 (i3 ), where the indexes i1 ,
i2 , and i3 represent uniformly spaced x1 , x2 , and x3 values on the intervals −21 ≤ x1 ≤ 21, −29 ≤ x2 ≤ 29, and −2 ≤ x3 ≤ 50.
Each of the 32 panels along x3 represents a 32 × 32 discretization of the x1 -x2 domain. The first rows of each grouping of
three (i.e. panels (a), (d), and (g)) give the original functions f1 , f2 , and f3 . The second rows of each grouping of three
(i.e. panels (b), (e), and (h)) give the reconstructed versions of f1 , f2 , and f3 . The third rows of of grouping of three (i.e.
panels (c), (f), and (i)) show the points in the x1 , x2 , and x3 domain that were used for the reconstruction. The white regions
indicate missing data. The color scales range from dark blue to red corresponding to the ranges of −500 ≤ f1 (i1 , i2 , i3 ) ≤ 500,
−659 ≤ f2 (i1 , i2 , i3 ) ≤ 659, and −742 ≤ f3 (i1 , i2 , i3 ) ≤ 614 for the groups of panels (a)-(c), (d)-(f), and (g)-(i), respectively.

FIG. 20: We compare the trajectories x1 (t), x2 (t), and x3 (t)

from the reconstructed (solid lines) and original (empty cir-
cles) functions f1 , f2 , and f3 from the 3D Lorenz system of
ODEs (Eq. 20) with the parameters σ = 10, ρ = 28, and
β = 8/3 in the chaotic regime. We plot x1 , x2 , and x3 para-
metrically for two initial conditions indicated by the crosses.

[1] E. N. Lorenz, “Deterministic nonperiodic flow,” Journal [2] C. O. Weiss and J. Brock, “Evidence for Lorenz-type
of the Atmospheric Sciences 20 (1963) 130.
 14

chaos in a laser,” Physical Review Letters 57 (1986) 2804. [22] B. A. Olshausen and D. J. Field, “Sparse coding with an
[3] K. M. Cuomo, A. V. Oppenheim, and S. H. Strogatz, overcomplete basis set: A strategy employed by V1?,”
“Synchronization of Lorenz-based chaotic circuits with Vision Research 37 (1997) 3311.
applications to communications,” IEEE Transactions on [23] M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An
Circuits and Systems II: Analog and Digital Signal Pro- Algorithm for designing overcomplete dictionaries for
cessing 40 (1993) 626. sparse representation,” IEEE Transactions on Signal
[4] H. Aref, “The development of chaotic advection,” Physics Processing 54 (2006) 4311.
of Fluids 14 (2002) 1315. [24] M. Aharon and M. Elad, “Sparse and redundant
[5] M. E. Csete and J. C. Doyle, “Reverse engineering of modeling of image content using an image-signature-
biological complexity,” Science 295 (2002) 1664. dictionary,” SIAM Journal on Imaging Sciences 1 (2008)
[6] A. F. Villaverde and J. R. Banga, “Reverse engineering 228.
and identification in systems biology: Strategies, per- [25] E. C. Smith and M. S. Lewicki, “Efficient auditory cod-
spectives and challenges,” Journal of the Royal Society ing,” Nature 439 (2006) 978.
Interface 11 (2014) 2013.0505. [26] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng,
[7] A. S. Perelson and P. W. Nelson, “Mathematical analysis “Self-taught learning: Transfer learning from unlabeled
of HIV-1 dynamics in vivo,” SIAM Rev. 41 (1999) 3. data,” Proceedings of the 24th International Conference
[8] D. S. Callaway and A. S. Perelson, “HIV-1 infection and on Machine Learning (ACM) (2007) 759.
low steady state viral loads,” B. Math. Biol. 64 (2002) [27] J. Mairal, J. Ponce, G. Sapiro, A. Zisserman, and F.
29. R. Bach, “Supervised dictionary learning,” Advances in
[9] P. W. Nelson and A. S. Perelson, “Mathematical analysis Neural Information Processing Systems 21 (2009) 1033.
of delay differential equation models of HIV-1 infection,” [28] E. J. Candès, J. Romberg, and T. Tao, “Robust uncer-
Math Biosci. 179 (2002) 73. tainty principles: Exact signal reconstruction from highly
[10] H. Dahari, E. Shudo, R. M. Ribeiro, and A. S. Perelson, incomplete frequency information,” IEEE Transactions
“Modeling complex decay profiles of hepatitis B virus on Information Theory 52 (2006) 489.
during antiviral therapy,” Hepatology 49 (2009) 32. [29] D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable
[11] S. A. Gourley, Y. Kuang, and J. D. Nagy, “Dynamics of recovery of sparse overcomplete representations in the
a delay differential equation model of hepatitis B virus presence of noise,” IEEE Transactions on Information
infection,” Journal of Biological Dynamics 2 (2008) 140. Theory 52 (2006) 6.
[12] B. Hancioglu, D. Swigon, and G. Clermont, “A dynami- [30] E. J. Candès “Compressive sampling,” Proceedings of the
cal model of human immune response to influenza A virus International Congress of Mathematicians 3 (2006) 1433.
infection,” Journal of Theoretical Biology 246 (2007) 70. [31] D. L. Donoho, “Compressed sensing,” IEEE Transac-
[13] A. S. Perelson, “Modeling viral and immune system dy- tions on Information Theory 52 (2006) 1289.
namics,” Nat. Rev. Immunol. 2 (2002) 28. [32] E. J. Candès and J. Romberg, “Quantitative robust
[14] A. Reynolds, J. Rubin, G. Clermont, J. Day, Y. uncertainty principles and optimally sparse decompo-
Vodovotz, and G. B. Ermentrout, “A reduced mathe- sitions,” Foundations of Computational Mathematics 6
matical model of the acute inflammatory response: I. (2006) 227.
Derivation of model and analysis of anti-inflammation.” [33] E. J. Candès, J. Romberg, and T. Tao. “Stable signal
Journal of Theoretical Biology 242 (2006) 220. recovery from incomplete and inaccurate measurements,”
[15] J. Day, J. Rubin, Y. Vodovotz, C. C. Chow, A. Reynolds, Communications on Pure and Applied Mathematics 59
and G. Clermont, “A reduced mathematical model of the (2006) 1207.
acute inflammatory response II. Capturing scenarios of [34] R. G. Baraniuk, “Compressive sensing,” IEEE Signal
repeated endotoxin administration.” Journal of Theoret- Processing Magazine 24 (2007) 118.
ical Biology 242 (2006) 237. [35] E. J. Candès and M. B. Wakin, “An introduction to com-
[16] R. Arditi and L. R. Ginzburg, “Coupling in predator- pressive sampling,” IEEE Signal Processing Magazine 25
prey dynamics: Ratio-dependence,” Journal of Theoret- (2008) 21.
ical Biology 139 (1989) 311. [36] D. L. Donoho and J. Tanner, “Precise undersampling the-
[17] E. Baake, M. Baake, H. G. Bock, and K. M. Briggs, orems,” Proceedings of the IEEE 98 (2010) 913.
“Fitting ordinary differential equations to chaotic data,” [37] M. Lustig, D. L. Donoho, and J. M. Pauly, “Sparse MRI:
Physical Review A 45 (1992) 5524. The application of compressed sensing for rapid MR
[18] S. Marino and E. O. Voit, “An automated procedure imaging,” Magnetic Resonance in Medicine 58 (2007)
for the extraction of metabolic network information from 1182.
time series data,” Journal of Bioinformatics and Com- [38] M. Lustig, D.L. Donoho, J.M. Santos, and J.M. Pauly,
putational Biology 4 (2006) 665. “Compressed sensing MRI,” IEEE Signal Processing
[19] J. Bongard and H. Lipson, “Automated reverse engineer- Magazine 25 (2008) 72.
ing of nonlinear dynamical systems,” Proc. Nat. Acad. [39] J. P. Haldar, D. Hernando, and Z.-P. Liang,
Sci. 104 (2007) 9943. “Compressed-sensing MRI with random encoding,”
[20] M. D. Schmidt and H. Lipson, “Data-mining dynamical IEEE Transactions on Medical Imaging 30 (2011) 893.
systems: Automated symbolic system identification for [40] M. Duarte, M. Davenport, D. Takhar, J. Laska, T. Sun,
exploratory analysis,” (ASME 2008 9th Biennial Confer- K. Kelly, and R. G. Baraniuk, “Single-pixel imaging via
ence on Engineering Systems Design and Analysis. Amer- compressive sampling, IEEE Signal Processing Mag. 25
ican Society of Mechanical Engineers, 2008). (2008) 83.
[21] M. D. Schmidt and H. Lipson, “Distilling free-form nat- [41] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image
ural laws from experimental data,” Science 324 (2009) super-resolution via sparse representation,” IEEE Trans-
81. actions on Image Processing 19 (2010) 2861.
 15

[42] D. L. Donoho, “For most large underdetermined systems 426.

of linear equations the minimal L1 norm solution is also [46] K. Judd and A. Mees, “Embedding as a modeling prob-
the sparsest solution,” Communications on Pure and Ap- lem,” Physica D: Nonlinear Phenomena 120 (1998) 273.
plied Mathematics 59 (2006) 797. [47] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online
[43] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. dictionary learning for sparse coding,” Proceedings of
Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, the 26th Annual International Conference on Machine
V. Dubourg, and J. Vanderplas, “Scikit-learn: Machine Learning (2009) 689.
learning in Python,” Journal of Machine Learning Re- [48] M. Mai, K. Wang, G. Huber, M. Kirby, M. D. Shattuck,
search 12 (2011) 2825. and C. S. O’Hern, “Outcome prediction in mathematical
[44] J. P. Crutchfield and B. S. McNamara, “Equations of models of immune response to infection,” PloS One 10
motion from a data series,” Complex Systems 1 (1987) (2015) e0135861.
417. [49] J. D. Murray, “Mathematical Biology I: An Introduc-
[45] K. Judd and A. Mees, “On selecting models for nonlinear tion,” Interdisciplinary Applied Mathematics 17 (2002).
time series,” Physica D: Nonlinear Phenomena 82 (1995)