Long-Term Time-Series Prediction Using Radial Basis

Function Neural Networks

Alex Alexandridisa, Ioannis Th. Famelisa and Charalambos Tsitourasb
a
Technological Educational Institute of Athens, Department of Electronic Engineering, Ag. Spiridonos, GR12210,
Aigaleo, Greece, [email protected]
b
Technological Educational Institute of Sterea Hellas, Department of Automation, GR34400 Psahna Campus,
Greece

Abstract. This work presents the application of radial basis function (RBF) network models on the challenging problem
of providing accurate long-term prediction for time-series. The non-symmetric variation of the fuzzy means (NSFM)
algorithm is used to determine the number and locations of the hidden node RBF centers, whereas the synaptic weights
are calculated using linear regression. The proposed approach is applied to the well-known Van der Pol oscillator with
the aim of training RBF models so that they can accurately predict the full trajectory of the system, given only its initial
state. A comparison with a Runge-Kutta RBF network, highlights the superiority of the proposed method.
Keywords: Fuzzy Means; Neural Networks; Radial Basis Function; Time-Series
PACS: 07.05.Mh, 02.60.Ed, 02.70.-c

INTRODUCTION
Neural networks (NNs) [1] are a set of powerful mathematical tools that simulate the way that the human brain
deals with information and the procedure of learning. NNs have the ability to identify and learn highly complex and
nonlinear relationships from input-output data only, without the use of first principle equations describing the
system. The literature combining numerical initial value problems and NNs is limited. Tsitouras [2], presented a new
type of NN based on Runge-Kutta methods. Later this technique was used for the derivation of new methods that
were best suited for Astronomical problems [3]. On the other hand, NNs have been widely used for time-series
prediction, i.e. the prediction of the future state of a dynamical system based on previously observed values, with
many applications in sales forecasting [4], weather forecasting [5], etc.
Though many successful attempts have been reported on using NNs for predicting the next value of a time-series
(one-step-ahead prediction), the task of providing long-term prediction (multi-step-ahead prediction) remains rather
challenging. This can be explained by taking into account that the NN is normally trained to receive as input the real
state of the system at each time step; however, when attempting to predict multiple time steps ahead, the future real
state is not yet available. In this case, previous predictions have to be fed back as inputs to the network to obtain
predictions for subsequent time steps, resulting to an accumulation of error. Obviously, in the case of multi-step-
ahead prediction, the accuracy of the NN model becomes more important, as even small errors in the initial steps
may cause predictions to rapidly diverge from the true pattern, making the model unusable for longer time periods.
Radial Basis Function (RBF) networks [6] constitute a special network architecture that presents remarkable
advantages over other NN types including better approximation capabilities, simpler network structures and faster
learning algorithms. In a recent publication [7], the accuracy of RBF network models was found to be further
improved using a non-symmetric fuzzy partition of the input space, resulting to the non-symmetric fuzzy means
(NSFM) algorithm. In this work, the NSFM algorithm is applied on training RBF models suitable for multi-step-
ahead prediction. It is shown that the enhanced approximation capabilities of the NSFM-trained models, help the
method to cope with the increased accuracy requirements imposed by the long-term time-series prediction problem.

RBF NETWORKS AND THE FUZZY MEANS ALGORITHM

An RBF network can be considered as a special three layer neural network, which is linear with respect to the
output parameters after fixing all the radial basis function centers and nonlinearities in the hidden layer. The input

Proceedings of the International Conference on Numerical Analysis and Applied Mathematics 2014 (ICNAAM-2014)
AIP Conf. Proc. 1648, 740003-1–740003-4; doi: 10.1063/1.4912958
© 2015 AIP Publishing LLC 978-0-7354-1287-3/$30.00

740003-1
layer distributes the N input variables to the L nodes of the hidden layer. Each node in the hidden layer is associated
with a center, equal in dimension with the number of input variables. Thus, the hidden layer performs a nonlinear
transformation and maps the input space onto a new higher dimensional space. The activity ȝl(uk) of the lth node is
the Euclidean norm of the difference between the kth input vector and the node center and is given by:

N
Pl ( ȣ k ) u k  uˆ l ¦ (u k ,i  uˆl ,i ) 2 , k = 1, 2, …, K (1)
i 1

where K is the total number of data, u k is the input vector and uˆ l is the center of the lth node. The output function of
the node is a radially symmetric function. This work employs the thin plate spline function, where h(ȝ)=ȝ2log(ȝ).
The final output tˆk for the kth data point is produced by a linear combination of the hidden node responses:

L
tˆk ¦wh l Pl u k , k = 1, 2, …, K (2)
l 1

where wl stands for the synaptic weight of the lth node.The synaptic weights are typically calculated using linear
regression of the hidden layer outputs to the real measured outputs (target values). The regression problem can be
trivially solved using linear least squares in matrix form. As far as the calculation of the hidden node center number
and locations is concerned, conventional training techniques like the k-means algorithm [8], postulate an arbitrary
number of RBF centers and then calculate their locations, the final selection being made through a trial-and-error
procedure. An alternative to this time-consuming approach was given by the fuzzy means (FM) algorithm [6], which
has the ability to calculate in one step the number and locations of the hidden node centers. A variant of the FM
algorithm, namely the NSFM algorithm, was found to provide higher prediction accuracy, accompanied by simpler
network structures [7]. What follows is a brief description of the algorithm; the interested reader is referred to the
original publication.
Like the original FM algorithm, the NSFM algorithm is also based on a fuzzy partition of the input space.
However, in this case the partition is non-symmetric, which implies that each input variable is partitioned into a
different number of fuzzy sets, where N is the number of input dimensions. Combining N one-dimensional fuzzy
sets, one can generate a multi-dimensional fuzzy subspace.
The produced fuzzy subspaces form a grid where each node is candidate for becoming an RBF center. The
objective of the NSFM algorithm is to assemble the RBF network hidden layer by selecting only a small subset of
the fuzzy subspaces. This selection is based on a hyper-ellipse placed around each fuzzy subspace center. The hyper-
ellipse marks a boundary between input vectors that receive non-zero or zero membership degrees to each particular
fuzzy subspace. Having defined the membership function, the algorithm proceeds with finding the subset of fuzzy
subspaces that assign a nonzero multidimensional degree to at least one input training vector. The selection is
accomplished using a non-iterative algorithm which requires only one pass of the input data, thus rendering the
center calculation procedure extremely fast, even in the presence of a large database of input examples. Taking
advantage of the short computational times, the number of fuzzy sets per dimension can be optimized through an
exhaustive search procedure, in case the problem involves a relatively low number of input variables. For problems
with higher dimensionality, evolutionary computation techniques like particle swarm optimization can be applied to
calculate the optimum non-symmetric fuzzy partition [9].

NUMERICAL EXPERIMENTS AND CONCLUSIONS

In this section, the NSFM algorithm is applied to the long-term prediction of the states of a nonlinear dynamical
system, known as the Van der Pol oscillator, described by the following second-order ODE:

y t  1  y2 t
 y t  y t 0 (3)

with y 0 y0 and y 0 u0 . Let y t be denoted by x1 t and y t by x2 t . Assuming that (5) is completely

unknown, the objective is to train two discrete dynamic RBF network models able to predict the change 'x1 and

740003-2
'x2 in the state variables of the system, based solely on input-output data from it. Both models will use as inputs the
previous values of the state variables, in order to predict the current value of change for the respective state:

'xi k RBFi x1 k  1 , x2 k  1 , i = 1, 2 (4)

where RBFi is the nonlinear function implemented by the respective RBF network. In order to generate data for
training the two RBF models, the Van der Pol oscillator was simulated using a fourth-order Runge Kutta method. To
be more specific, 40 different initial states were randomly chosen from a uniform distribution within the limits [-3 3]
for x1 and [-2 2] for x2. The system trajectories when starting from each one of the randomly generated initial states
were then calculated, to collect data with a step equal to 0.1. 100 time-steps were used for each trajectory, summing
to a total of 4000 training examples. The data points were split into a training dataset used for determining the RBF
models’ parameters and a validation set used for model selection, using a 75%-25% ratio. For the particular
problem, the number of input variables was low, allowing to perform an exhaustive search testing all the possible
combinations of networks with a partition ranging from 4 to 50 fuzzy sets for each input variable. It should be noted
that the two RBF models share the same hidden layer centers for each given fuzzy partition, due to the common set
of input variables. The best networks were selected based on the Root Mean Square Error (RMSE) value on the
validation dataset. Table 1 summarizes the results of the training procedure, depicting the fuzzy partition, number of
RBF centers and RMSE value for the best networks found.

TABLE 1. Results of the training procedure for the two models

Parameter ǻx1 model ǻx2 model
Fuzzy sets per dimension [41 47] [49 32]
Number of RBF centers 243 208
RMSE 0.00116 0.00097

In order to test the performance of the two models for multi-step-ahead prediction, a new set of 20 initial states,
different than the ones used for training, was employed. First, the two RBF networks were given a set of initial states
ª¬ x1 0 x2 0 º¼ and the predictions ª¬ xˆ1 1 xˆ2 1 º¼ were calculated. At each discrete time step k, the predictions for
the previous time step were fed back as inputs to the network to calculate the new predictions ¬ª xˆ1 k xˆ2 k º¼ :

xˆi k xˆi k  1  'xi k xˆi k  1  RBFi xˆ1 k  1 , xˆ2 k  1 , i = 1, 2 (5)

Figure 1 depicts the phase diagrams for two of the tested initial states, including the real trajectory, along with
the model predictions. For comparison purposes, a Runge-Kutta RBF neural network (RK-RBF) [10] was also
applied to the same problem. The results for the proposed approach (NSFM-RBF) and the RK-RBF network are
depicted in Table 2, including the RMSE value for each one of the testing trajectories and the mean RMSE value for
all of them.

3 2.5
Real Trajectory Real Trajectory
2.5 2
2
+ Predicted Trajectory + Predicted Trajectory
1.5
1.5
1
1
0.5
0.5
0
0
-0.5
-0.5
-1
-1
-1.5 -1.5

-2 -2
-2.5 -2.5
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
(a) (b)
FIGURE 1. Phase diagrams for two of the tested initial states, including the real trajectory along with the NSFM-RBF model
predictions, (a) Initial state: [2 2], (b) Initial state: [-0.1 0.5]

740003-3
 TABLE 2. RMSE values for NSFM-RBF and RK-RBF when starting from different testing initial states
Initial state [x1 x2] NSFM-RBF RK-RBF
RMSE x1 RMSE x2 RMSE x1 RMSE x2
[0.5 0.5] 0.0004 0.0003 0.0040 0.0030
[-0.5 0.5] 0.0005 0.0004 0.0018 0.0015
[-0.5 -0.5] 0.0006 0.0004 0.0007 0.0007
[0.5 -0.5] 0.0010 0.0007 0.0016 0.0013
[1.0 1.0] 0.0009 0.0006 0.0016 0.0011
[-2.0 -2.0] 0.0033 0.0026 0.0052 0.0041
[2.0 2.0] 0.0083 0.0065 0.0027 0.0021
[2.5 -1.0] 0.0295 0.0207 0.0136 0.0093
[-2.5 1.0] 0.0100 0.0071 0.0194 0.0135
[-0.2 0.3] 0.0007 0.0004 0.0012 0.0007
[0.2 -0.3] 0.0004 0.0003 0.0035 0.0026
[-0.2 -0.3] 0.0014 0.0009 0.0032 0.0022
[0.2 0.3] 0.0003 0.0002 0.0023 0.0015
[0.1 0.0] 0.0008 0.0006 0.0055 0.0049
[-0.1 0.0] 0.0014 0.0010 0.0049 0.0045
[0.0 0.1] 0.0014 0.0010 0.0079 0.0055
[0.0 -0.1] 0.0005 0.0004 0.0077 0.0054
[-0.1 0.5] 0.0003 0.0003 0.0012 0.0010
[-1.0 -0.5] 0.0005 0.0004 0.0018 0.0014
Mean RMSE 0.0033 0.0024 0.0047 0.0035

It can be seen that the NSFM-RBF models manage to track accurately the real trajectories for 100 consecutive
time steps, though they are only supplied with the initial state information. As far as the comparison with the RK-
RBF is concerned, the NSFM-RBF models exhibit lower RMSE values in the vast majority of individual
trajectories, as well as lower mean RMSE values, for both state variables. Finally, it should be noted that the RK-
RBF network training procedure is more complicated compared to the NSFM-RBF method, as it involves an
additional weight optimization technique following standard RBF training.

ACKNOWLEDGMENTS
This research has been co-funded by the European Union (European Social Fund) and Greek national resources
under the framework of the "Archimedes III: Funding of Research Groups in TEI of Sterea Hellas" project of the
“Education & Lifelong Learning” Operational Programme.

REFERENCES
1. S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed. Upper Saddle River, NJ: Prentice Hall International,
1999.
2. Ch. Tsitouras, IEEE T. Neural Networ., 13, 222-228, (2002).
3. Ch. Tsitouras and I. Th. Famelis, New Astron., 17, 469–473, (2012).
4. P. Doganis, A. Alexandridis, P. Patrinos, H. Sarimveis, J. Food Eng., 75, 196–204, (2006).
5. I. Maqsood and A. Abraham, Appl. Soft Comput., 7, 995–1004, (2007).
6. A. Alexandridis, H. Sarimveis and G. Bafas, Neural Networks, 16, 1003-1017, (2003).
7. A. Alexandridis, H. Sarimveis and K. Ninos, Adv. Eng. Softw., 42, 830-837, (2011).
8. C. Darken and J. Moody, “Fast Adaptive k-Means Clustering: Some Empirical Results” in International Joint Conference on
Neural Networks (IJCNN), San Diego, CA, 1990, pp. 233-238
9. A. Alexandridis, E. Chondrodima and H. Sarimveis, IEEE T. Neural Networ. Learn. Syst., 24, 219-230, (2013).
10. Y.J. Wang and C.T. Lin, IEEE T. Neural Networ., 9, 294-307, (1988).

740003-4
AIP Conference Proceedings is copyrighted by AIP Publishing LLC (AIP). Reuse of AIP
content is subject to the terms at: http://scitation.aip.org/termsconditions. For more
information, see http://publishing.aip.org/authors/rights-and-permissions.