Benchmarking time series classification - Functional

data vs machine learning approaches

Florian Pfisterer, Xudong Sun1 , Laura Beggel1 , Fabian Scheipl, Bernd Bischl
Department of Statistics, Ludwig-Maximilians-Universität München
Ludwigstr. 33,
arXiv:1911.07511v2 [stat.ML] 24 Feb 2021

80539, München, Germany

Abstract
Time series classification problems have drawn increasing attention in the ma-
chine learning and statistical community. Closely related is the field of func-
tional data analysis (FDA): it refers to the range of problems that deal with
the analysis of data that is continuously indexed over some domain. While of-
ten employing different methods, both fields strive to answer similar questions,
a common example being classification or regression problems with functional
covariates. We study methods from functional data analysis, such as functional
generalized additive models, as well as functionality to concatenate (functional)
feature extraction or basis representations with traditional machine learning al-
gorithms like support vector machines or classification trees. In order to assess
the methods and implementations, we run a benchmark on a wide variety of
representative (time series) data sets, with in-depth analysis of empirical re-
sults, and strive to provide a reference ranking for which method(s) to use for
non-expert practitioners. Additionally we provide a software framework in R
for functional data analysis for supervised learning, including machine learning
and more linear approaches from statistics. This allows convenient access, and
in connection with the machine-learning toolbox mlr, those methods can now
also be tuned and benchmarked.
Keywords: Functional Data Analysis, Time Series, Classification, Regression

? sourcecode available at https://github.com/mlr-org/mlr

∗ Corresponding author: [email protected]
1 Equal Contribution
1. Introduction

The analysis of functional data is becoming more and more important in many
areas of application such as medicine, economics, or geology (cf. Ullah and
Finch [1], Wang et al. [2]), where this type of data occurs naturally. In industry,
functional data are often a by-product of continuous monitoring of production
processes, yielding great potential for data mining tasks. A common type of
functional data are time series, as time series can often be considered as dis-
cretized functions over time.
Many researchers publish software implementations of their algorithms, there-
fore simplifying the access to already established methods. Even though such a
readily available, broad range of methods to choose from is desirable in general,
it also makes it harder for non-expert users to decide which method to apply
to a problem at hand and to figure out how to optimize their performance.
As a result, there is an increasing demand for automated model selection and
parameter tuning.
Furthermore, the functionality of available pipeline steps ranges from simple
data structures for functional data, to feature extraction methods and packages
offering direct modeling procedures for regression and classification. Users are
again faced with a multiplicity of software implementations to choose from and,
in many instances, combining several implementations may be required. This
can be difficult and time-consuming, since the various implementations utilize a
multiplicity of different workflows which the user needs to become familiar with
and synchronize in order to correctly carry out the desired analysis.
There is a wide variety of packages for functional data analysis in R [3] available
that provide functionality for analyzing functional data. Examples range from
the fda [4] package which includes object types for functional data and allows
for smoothing and simple regression, to, e.g., boosted additive regression models
for functional data in FDboost [5]. For an extensive overview, see the CRAN
task view [6].
Many of those packages are designed to provide algorithmic solutions for one
specific problem, and each of them requires the user to become familiar with
its user interface. Some of the packages, however, such as fda.usc [7] or refund
[8] are not designed for only one specific analysis task, but combine several
approaches. Nevertheless, these packages do not offer unified frameworks or
consistent user interfaces for their various methods, and most of the packages
can still only be applied separately.
A crucial advantage of providing several algorithms in one package with a unified
and principled user interface is that it becomes much easier to compare the
provided methods with the intention to find the best solution for a problem at
hand. But to determine the best alternative, one still has to be able to compare
the methods at their best performance on the considered data, which requires
hyperparameter search and, more preferably, efficient tuning methods.
While the different underlying packages are often difficult and sometimes even
impossible to extend to new methods, custom implementations and extensions
can be easily included in the accompanying software.

2
We want to stress that the focus of this paper does not lie in proposing new al-
gorithms for functional data analysis. Its added value lies in a large comparison
of algorithms while providing a unified and easily accessible interface for com-
bining statistical methods for functional data with the broad range of functions
provided by mlr, most importantly benchmarking and tuning. Additionally, the
often overlooked possibility of extracting non-functional features from functional
data is integrated, which enables the user to apply classical machine learning
algorithms such as support vector machines [9] to functional data problems.
In a benchmark study similar to Bagnall et al. [10] and Fawaz et al. [11], we ex-
plore the performance of implemented methods, and try to answer the following
questions:

1. Can functional data problems be solved with classical machine learning

methods ignoring the functional structure of the data as well as with more
elaborate methods designed for this type of data? Bagnall et al. [10]
measure the performance of some non-functional-data-specific algorithms
such as the rotation forest [12], but this does not yield a complete picture.
2. Guidance on the wide range of available algorithms is often hard to obtain.
We aim to make some recommendations in order to simplify the choice of
learning algorithm.
3. Do statistical methods explicitly tailored to the analysis of functional data
[e.g. FDboost, 5] perform well on classical time series tasks? No bench-
mark results for these methods, which provide interpretable results, are
currently available.
4. Many methods that represent functional data in a non-functional domain
have been proposed and are also often applied in practice. Examples for
this include either hand crafted features [cf. 13], summary statistics [14],
or generally applicable methods such as wavelet decomposition [15].
5. Hyperparameter optimization is a very important step in many machine
learning applications. In our benchmark, we aim to quantify the impact
of hyperparameter optimization for a set of given algorithms on several
data sets.

Contributions. As contributions of this paper, we aim to answer the questions

posed above. Additionally, we provide a toolbox for the analysis of functional
data. It implements several methods for feature extraction and directly model-
ing functional data, including a thorough benchmark of those algorithms. This
toolbox also allows for full or partial replication of the conducted benchmark
comparison.

3
2. Related Work

In the remainder of the paper, we focus on comparing algorithms from the func-
tional data analysis and the machine learning domain. Functional data analysis
traditionally values interpretable results and valid statistical inference over pre-
diction quality. Therefore functional data algorithms are often not compared
with respect to their predictive performance in literature. We aim to close this
gap. On the other hand, machine learning algorithms often do not yield inter-
pretable results. While we consider both aspects to be important, we want to
focus on predictive performance in this paper.

2.1. Feature extraction and classical machine learning methods

In this work, we differentiate between machine learning algorithms that can di-
rectly be applied to functional data, and algorithms intended for scalar features,
which we call classical machine learning methods.
A popular approach when dealing with functional data is to reduce the problem
to a non-functional task by extracting relevant non-functional features [1]. Ap-
plying classical machine learning methods after extracting meaningful features
can then lead to competitive results [cf. 16, e.g.] or at least provide baselines,
which are in general not covered by functional data frameworks. In our frame-
work, such functionality is easily available by combining feature extraction, e.g.,
based on extracting heuristic properties [cf. tsfeatures; 14] or wavelet coeffi-
cients [17, 15] and analyzing these derived scalar features with classical machine
learning tools provided by mlr.
Based on some existing functionality of the listed packages, we adapt different
feature extraction methods. Along with different algorithms already proposed
in literature, we propose two new custom methods, DTWKernel and MultiRes-
Features:

tsfeatures [14] extracts scalar features, such as auto-correlation functions,

entropy and other heuristics from a time series.

fourier transforms data from the time domain into the frequency domain
using the fast fourier transform [18]. Extracted features are either phase
or amplitude coefficients.
bsignal B-Spline representations from package FDboost [5] are used as
feature extractors. Given the knots vector and effective degree of freedom,
we extract the design matrix for the functional data using mboost.
wavelets [15] applies a discrete wavelet transform to time series or func-
tional data, e.g., with Haar or Daubechies wavelets. The extracted features
are wavelet coefficients at several resolution levels.
PCA projects the data on their principal component vectors. Only a
subset of the principal component scores representing a given proportion
of signal variance is retained.

4
 DTWKernel computes the dynamic time warping distances of functional
or time series data to (a set of) reference data. We implement dynamic
time warping (DTW) based feature extraction. This method computes
the dynamic time warping distance of each observed function to a (user-
specified) set of reference curves. The distances of each observation to the
reference curves is then extracted as a vector-valued feature. The reference
curves can either be supplied by the user, e.g., they could be several typical
functions for the respective classes, or they can be obtained from the
training data. In order to compute dynamic time warping distances, we use
a fast dynamic time warping [19] implementation from package rucrdtw
[20].
MultiResFeatures extracts features, such as the mean at different levels
of resolution (zoom-in steps). Inspired by the image pyramid and wavelet
methods, we implement a feature extraction method, multi-resolution fea-
ture extraction where we extract features like mean and variance com-
puted over specified windows of varying widths. Starting from the full
sequence, the sequence is repeatedly divided into smaller pieces, where at
each resolution level, a scalar value is extracted. All extracted features
are concatenated to form the final feature vector.

2.2. Methods for functional data

Without feature extraction, direct functional data modeling (both classification
and regression) methods incorporated in our package span two families: The first
family of semi-parametric approaches includes FGAM [8], FDboost [5], and the
functional generalized linear model [FGLM; 21], which are all structured ad-
ditive models. Those methods use (tensor product) spline basis functions or
functional principal components (FPCs) [22] to represent effects fitted in a gen-
eralized additive model. While FGAM and FGLM use the iterated weighted
least square (IWLS) method to generate maximum likelihood estimates, FD-
boost uses component-wise gradient boosting [23] to optimize the parameters.
Additionally, the estimated parameters can be penalized.
PJ A general formula for
this family of methods is ζ(Y |X = x) = h(x) = j=1 hj (x), where ζ represents
a functional of the conditional response distribution (e.g., an expectation or a
quantile), x is a vector of (functional) covariates and hj (x) are partial additive
effects of subsets of x in basis function representation, cf. Greven and Scheipl
[24] for a general introduction.
The second family of methods are non-parametric methods as introduced in
Ferraty and Vieu [25], e.g., based on (semi-)metrics which quantify local or
global differences or distances across curves. For example, the distance between
two
qR instances could be defined by the L2 distance of two curves d(xi (t), xj (t)) =
(xi (t) − xj (t))2 dt. Kernel functions are used to average over the training
instances and weigh their respective contributions based on the value of their
distance semi-metric to the predicted instance. Functional k-nearest neighbors
algorithms can also be defined based on such semi-metrics. Implementations
can be found in packages fda.usc [7] and classiFunc [26].

5
2.3. Toolboxes for functional data analysis
The package fda [4] contains several object types for functional data and allows
for smoothing and regression for functional data. Analogously, the R-package
fda.usc [7] contains several classification algorithms that can be used with func-
tional data. In Python, scikit-fda [27] offers both representation of and (pre-
)processing methods for functional data, but only a very small set of machine
learning methods for classification or regression problems is implemented at the
time of writing.
As a byproduct of the Time-Series Classification Bake-off [10], a wide variety
of algorithms were implemented and made available. But this implementation
emphasizes the benchmark over providing a data analysis toolbox for users, and
is therefore not easily usable for inexperienced users.

2.4. Benchmarks
The recently published benchmark analysis Time-Series Classification Bake-off
by Bagnall et al. [10] provides an overview of the performance of 18 state-of-
the-art algorithms for time series classification. They re-implement (in Java)
and compare 18 algorithms designed especially for time series classification on
85 benchmark time series data sets from Bagnall et al. [28]. In their analysis,
they also include results from several standard machine learning algorithms.
They note that the rotation forest [12] and random forest [29] are competitive
with their time series classification baseline [1-nearest neighbor with dynamic
time warping distance; 30]. Their results show that ensemble methods such as
collection of transformation ensembles [COTE; 31] perform best, but for the
price of considerable runtime.
Deep learning methods applied to time series classification tasks have also shown
competitive prediction power. For example, [11] provide a comprehensive review
of state-of-the-art methods. The authors compared both generative models and
discriminative models, including fully connected neural networks, convolutional
neural networks, auto-encoders and echo state networks, whereas only discrimi-
native end-to-end approaches were incorporated in the benchmark study.
The benchmark study conducted in this work does not aim to replicate or com-
pete with earlier studies like [10], but instead tries to extend their results.

3. Functional Data

In contrast to non-functional data analysis, where the measurement of a single

observation is a vector of scalar components whose entries represent values of the
separate multidimensional features, functional data analysis treats and analyses
the features themselves as functions over their domain. By learning to represent
the underlying function, the carried out analysis is not just restricted to the
measured discrete values but it is possible to sample from (and analyze) the
entire domain space.
In this work, we focus on pairs of features and corresponding labels (x, y) for
supervised learning. In contrast to non-functional data analysis, where the

6
measurement of a single observation is a vector of scalar components, functional
features are function-valued over their domain. The features x = (x1 , ..., xp )
can thus also be a function, i.e., xj = gj (t), g : T → R. In practice, functional
data comes in the form of observed values gj (t), t ∈ {1, ..., L}, where each t
corresponds to a discrete point on the continuum. Those observed values stem
from an underlying function f evaluated over a set of points. A frequent type
of functional data is time series data, i.e., measurements of a process measured
at discrete time-points.
For example, in some electrical engineering applications, signals are obtained
over time at a certain sampling rate, but other domains are possible as well.
Spectroscopic data, for example, are functional data recorded over certain parts
of the electromagnetic spectrum. One such example is depicted in Figure 1. It
shows spectroscopy data of fossil fuels [32] where the measured signal represents
reflected energies in the ultraviolet-visible (UV-VIS) and the near infrared spec-
trum (NIR). In the plot, different colors correspond to different instances. This
is a typical example of a scalar-on-function regression problem, where the inputs
are a collection of spectroscopic curves for a fuel, and the prediction target is
the heating value of the fossil fuel.
In Figure 2, we display two functional classification scenarios. The goal in
those scenarios is to distinguish the class type of the curve, which can also
be understood as a function-on-scalar problem. Figure 2a shows the vertical
position of an actor’s hand while either drawing a toy-gun and aiming at a
target, or just imitating the motion with the blank hand. This position is
measured over time. The two different types of classes of the curves can be
distinguished by the color scheme.
Figure 2b shows a data set built for distinguishing images of beetles from images
of flies based on their outlines. While following the outline, the distance to the
center of the object is measured which is then used for classification purposes.
The latter data sets are available from [28].
The interested reader is referred to Ramsay [21] and Kokoszka and Reimherr
[35] for more in-depth introductions to this topic.

4. Functional Data Analysis with mlrFDA

Along with the benchmark, we implement the software mlrFDA, which extends
the popular machine learning framework mlr for the analysis of functional data.
As the implemented functionality is an extension of the mlr package, all of the
functionality available in mlr transfers to the newly added methods for func-
tional data analysis. We include a brief overview of the implemented function-
ality in Appendix A.1. A more in-detail overview and tutorial on mlr can be
found in the mlr tutorial [36].
mlr provides a unified framework for machine learning methods in R, currently
supporting tasks from 4 main problem types: (multilabel-)classification, regres-
sion, cluster analysis, and survival analysis. For each problem type, many algo-
rithms (called learners) are integrated. This yields an extensive set of modeling
options with a unified, simple interface. Moreover, advanced techniques such

7
 Heat emission for different fossil fuels

0
0.5

0.0
Heat [mJ]
UVVIS Energy

−1

NIR Energy
17.5
20.0
22.5
25.0

−0.5

−2

−1.0

400 600 800 1000 1500 2000 2500

Wavelength [nm] Wavelength [nm]

Figure 1: Scalar-on-function regression: Spectral data for fossil fuels [32]

as hyperparameter tuning, preprocessing and feature selection are also part of

the package. An additional focus lies on extensibility, allowing the user to inte-
grate their own algorithms, performance measures and preprocessing methods.
As mlrMBO seamlessly integrates into the new software, many different tun-
ing procedures can be readily adapted by the user. Tuning of hyperparameters
is usually not integrated in software packages for functional data analysis and
thus would require the user to write additional, non-trivial code that handles
(nested) resampling, evaluation and optimization methods.
mlrFDA contains several functional data algorithms from several R packages,
e.g., fda.usc, refund or FDboost. The algorithms’ functionality, however, re-
mains unchanged, only their user interface is standardized for use with mlr. For
detailed insights into the respective algorithms, full documentation is available
in the respective packages.
Since our toolbox is built on mlr’s extensible class system, our framework is eas-
ily extensible to other methods that have not yet been integrated, and the user
can include his or her own methods which do not necessarily need to be available
as a packaged implementation. Additionally, mlrFDA inherits mlr’s function-
ality for performance evaluation and benchmarking, along with extensive and
advanced (hyperparameter) tuning. This makes our platform very attractive for
evaluating which algorithm fits best to a problem at hand, and even allows for
large benchmark studies.

5. Benchmark Experiment

In order to enable a comparison of the different approaches, an extensive bench-

mark study is conducted. This paper does not aim to replicate or reproduce

8
 2

1
Hand position in X−axis

Distance to center
1

Class Class
0
Gun−Draw Beetle
0
Point Fly

−1 −1

−2
−2

0 50 100 150 0 100 200 300 400 500

Time Outline

(a) Gunpoint (b) BeetleFly

Figure 2: Excerpts from two time series classification data sets. (a): Gunpoint data [33], (b):
BeetleFly Data [34].

results obtained by Bagnall et al. [10] or Fawaz et al. [11]. Instead we focus on
providing a benchmark complementary to previous benchmarks. This is done
because i) the experiments require large amounts of computational resources,
and ii) the added value of an exact replication of the experiments (with open
source code) is comparatively small. Nonetheless, we aim for results that can
be compared, and thus extend the results obtained by Bagnall et al. [10] by
staying close to their setup. The experiments were carried out on a high per-
formance computing cluster, supported by the Leibniz Rechenzentrum Munich.
Individual runs were allowed up to 2.2 GB of RAM and 4 hours run-time for
each evaluation. We want to stress that this benchmark compares implemen-
tations, which does not always necessarily correspond to the performance of
the corresponding theoretical algorithm. Additionally, methods for functional
data analysis are traditionally more focused on valid statistical inference and
interpretable results, which does not necessarily coincide with high predictive
performance.

5.1. Benchmark Setup

A benchmark experiment is defined by four important characteristics: The data
sets algorithms are tested on, the algorithms to be evaluated, the measures
used for evaluating predictive performance, and a resampling strategy used for
generating train and test splits of the data. A comprehensive overview of the
conducted benchmark setup can be obtained from Table 1.
These characteristics are briefly described subsequently before providing and
discussing the results. We use a subset of 51 data sets from the popular UCR
archive [28] in order to enable a comparison of results in [10] with the additional
methods described in this paper. The data sets stem from various application
types such as ECG measurements, sensor data, or image outlines, therefore

9
 Data sets 51 Data sets, see table B.7
Algorithms Function (Package)
Machine Learning: - glmnet (glmnet)
- rpart (rpart)
- ksvm? (kernlab)
- ranger? (ranger)
- xgboost? (xgboost)

Functional Data - classif.knn(fda.usc)

- classif.glm (fda.usc)
- classif.np (fda.usc)
- classif.kernel(fda.usc)
- FDboost (FDboost)
- fgam (refund)
- knn with dtw (classiFunc)

Feature Extraction + ML - feature extraction: see table A.6

- in combination with ML algorithms marked with a ? .
Measures mean misclassification error, training time
Resampling 20-fold stratified sub-sampling;
class balances and train/test set size as in [10].
Tuning 100 iterations of Bayesian optimization (3-fold inner CV).
Corresponding hyperparameter-ranges can be obtained
from tables 3 and 5.

Table 1: Benchmark experiment setup

having varying training set sizes or measurement lengths. For more detailed
information about the data sets, see Bagnall et al. [28].
We selected data using the following criteria: In order to reduce the compu-
tational resources we did i) not run data sets that have multiple versions, ii)
exclude data sets with less then 3 examples in each class iii) remove data sets
with more than 10000 instances or time series longer than 750 measurements.
As some of the classifiers only work with multi-class targets via 1-vs-all clas-
sification, we iv) additionally excluded data sets with more then 40 classes.
In essence, we benchmark small and medium sized data sets with a moderate
amount of different classes.
We add 7 new algorithms and 6 feature extraction methods which can be com-
bined with arbitrary machine learning methods for scalar features (c.f. Table 1).
Additionally we test 5 classical machine learning methods, in order to obtain
a broader perspective on expected performance if the functional nature of the
data is ignored. As we benchmark default settings as well as tuned algorithms,
in total 80 different algorithms are evaluated across all data sets. When com-
bining feature extraction and machine learning methods, we fuse the learning
algorithm and the preprocessing, thus treating them as a pipeline where data
is internally transformed before applying the learner. This allows us to jointly
tune the hyperparameters of learning algorithm and preprocessing method. The
respective defaults and parameter ranges can be obtained from Table 3 (feature
extractors) and Table 5 (learning algorithms). More detailed description of the

10
hyperparameters can be obtained from the respective packages documentation.
In order to generate train/test splits, and thus obtain an unbiased estimate of
the algorithm’s performance, we use stratified sub-sampling. We use 20 different
train/test splits for each data set in order to reduce variance and report the
average. For tuned models, we use use nested cross-validation [37] to ensure
unbiased estimates, where the outer loop is again subsampling with 20 splits,
and the inner resampling for tuning is a 3-fold (stratified) cross-validation. All
compared 80 algorithms are presented exactly the same index sets for the 20
train-test outer subsampling splits.
Mean misclassification error (MMCE) is chosen as a measure of predictive per-
formance in order to stay consistent with Bagnall et al. [10]. Other measures,
such as area under the curve (AUC) require predicted probabilities and do not
trivially extend to multi-class settings.
While Bagnall et al. [10] tune all algorithms across a carefully handcrafted grid,
we use Bayesian optimization [38]. In order to stay comparable, we analogously
fix the amount of tuning iterations to 100.
We use mlrMBO [39] in order to perform Bayesian optimization of the hyperpa-
rameters of the respective algorithm. Additionally, in order to scale the method
to a larger amount of data sets and machines, the R-package batchtools (Bischl
et al. [40], Lang et al. [41]) is used. This enables running benchmark experiments
on high-performance clusters. For the benchmark experiment, a job is defined
as re-sampling of a single algorithm (or tuning thereof) on a single version of a
data set. This allows for parallelization to an arbitrary number of CPU’s, while
at the same time guaranteeing reproducibility. The code for the benchmark is
available from https://github.com/compstat-lmu/2019_fda_benchmark for
reproducibility.

5.2. Results
This Section tries to answer the questions posed in section 1. We evaluate i)
various machine learning algorithms in combination with feature extraction, ii)
classical time series classification approaches, iii) the effect of tuning hyperpa-
rameters for several methods, and iv) try to give recommendations with respect
to which algorithm(s) to choose for new classification problems.
Algorithms evaluated in this benchmark have been divided into three groups:
Algorithms specifically tailored to functional data, classical machine learning
algorithms without feature extraction and classical machine learning algorithms
in combination with feature extraction.

5.2.1. Algorithms for functional data

Performances of algorithms specifically tailored to functional data analysis can
be obtained from Figure 3. The k-nearest neighbors algorithm from package
classiFunc [26] in combination with dynamic time warping [19] distance seems
to perform best across data sets. It is also considered a strong baseline in Bagnall
et al. [10].

11
 1.00 ●
●
●
●
●

0.75
Accuracy

0.50

●
0.25

0.00

classiFunc.knn.dtw fdausc.glm fdausc.kernel fdausc.knn fdausc.np FDboost fgam

Learner

Figure 3: Performances for functional data analysis algorithms in default settings (untuned)
across all 51 data sets.

5.2.2. Machine Learning algorithms with feature extraction

Performances of different machine learning algorithms in combination with fea-
ture extraction with and without tuning can be obtained from Figure 4.

bsignal dtwkernel fourier fpca multires wavelet

1.00

Tuning
0.75
default
mbo
Accuracy

Learner
0.50
glmnet
● ksvm
●
●
● ● ●
● ranger
● ●
● ●
● ●
● ●
● ● ● rpart
0.25 ●
● ●
●
● ● ●
●
●
● ● xgboost
● ● ●
● ● ●
●
● ●
●
● ●
●
● ●
● ●
●
● ●
● ●
●
● ● ● ●
●
0.00 ● ●
n t
m.tu m
rang ranged
er.tu er
r d
xgb xgb par t
t.tun t
ed
n t
m.tu m
rang ranged
er.tu er
r d
xgb xgb par t
t.tun t
ed
n t
m.tu m
rang ranged
er.tu er
r d
xgb xgb par t
t.tun t
ed
n t
m.tu m
rang ranged
er.tu er
r d
xgb xgb par t
t.tun t
ed
n t
m.tu m
rang ranged
er.tu er
ned
xgb xgb par t
t.tun t
ed
n t
m.tu m
rang ranged
er.tu er
ned
xgb xgb par t
t.tun t
ed
ksv ksve

oos oos

ksv ksve

oos oos

ksv ksve

oos oos

ksv ksve

oos oos

ksv ksve

oos oos

ksv ksve

oos oos
ne

ne

ne

ne
n

n
glm

glm

glm

glm

glm

glm
r

Figure 4: Results for feature extraction-based machine learning algorithms with default and
tuned (MBO) hyperparameters across 51 data sets. Hyperparameters are tuned jointly for
learner and feature extraction method.

We conclude that feature extraction using splines (bsignal) and wavelets as

well as extracting dynamic time warping distances works well when combined

12
with conventional machine learning algorithms, even at their default hyper-
parameters. Among the learners, random forests, especially in combination with
bsignal show quite advantageous performance. In addition, we find an obvious
improvement from hyper-parameter tuning for the Fourier feature extraction.
In terms of learners, random forest and gradient boosted tree learners (xgboost)
perform better than support vector machines.

5.2.3. Machine Learning algorithms without feature extraction

Additionally, we aim to provide some insight with regards to the performance of
machine learning algorithms that ignore the functional nature of our data. Fig-
ure 5 provides an overview over the performance of different machine learning
algorithms that are often used together with traditional tabular data. Perfor-
mances in this figure are obtained from algorithms directly applied to the func-
tional data without any additional feature extraction. The widely used gradient
boosting (xgboost) and random forest (ranger) implementations seem to work
reasonably well for functional data even without additional feature extraction.

1.00

Tuning
0.75 default
tuned
Accuracy

Learner
glmnet
0.50
ksvm
ranger
rpart
● xgboost

0.25
●
t

t
ne

os
ar
ge
v

rp
ks

bo
m

n
ra
gl

xg

Learner

Figure 5: Performance of non-functional machine learning algorithms across 51 data sets

applied directly to functional data with and without tuning.

5.2.4. The effect of tuning hyperparameters

From our experiments, we conclude, that tuning hyperparameters of machine
learning algorithms in general has a non-negligible effect on the performance.
Using Bayesian optimization in order to tune algorithm hyperparameters on
average yielded an absolute increase in accuracy of 5.4% across data sets and
learners.
Figure 6 displays the aggregated time over all data sets, taking into account
the time required for hyperparameter tuning. All experiments have been run

13
on equivalent hardware on high-performance computing infrastructure. Due to
fluctuations in server load, this does not allow for an exact comparison with
respect to computation time, but we hope to achieve comparable results as
we repeatedly evaluate on sub-samples. Note that we restrict the tuning to 3
algorithms where tuning traditionally leads to higher performances.2
Running time (sec) (log)

104

102
default
mbo

100

10−2
nn

sc lm

au el

fd knn

FD np

t
am

t
vm

er

xg rt

t
os

ne

os
a
fd ern

ng
au .g
.k

rp
bo

ks

bo
fg
sc

m
.
nc

sc

sc

ra
gl
.k

au
au
Fu
fd
si
as

fd
cl

Learner

Figure 6: Comparison of running time for the different learner classes with default and tuned
hyperparameters across 51 data sets. A log transformation on the running time in seconds is
applied, and the mean running time is visualized for each stratification as a horizontal line
within the violin plot.

5.2.5. Top 10 Algorithms and recommendations

Table 2 showcases the top 10 algorithms from the benchmark in terms of average
rank in predictive accuracy across data sets. With this list, we aim to provide
some initial understanding of the performance of different algorithms and feature
extraction methods. Note that this list by no means reflects performance on
future data sets, but might serve as an indicator, of which algorithms one might
want to try first given computational constraints.
We observe that wavelet extraction in combination with either ranger or xgboost
seems to be very strong. They obtain an average rank of 12.90 and 14.45 (out of
80) respectively. Dynamic time warping distances for k-nearest neighbors indeed
seems to be a strong baseline, even without tuning. Another strong feature
extraction method seems to be the extraction of B-spline features. Using the 10
algorithms above allows us to obtain an accuracy within 5% of the maximum
on 49 of the 51 data sets.

2 Additionally, we find significantly improved performance for tuned FDboost in Figure 3

14
Table 2: Top 10 algorithms by average rank across all data sets. Percent Accuracy describes
the fraction of the maximal accuracy reached for each task.

Algorithm Setting Accuracy % Average Rank

ranger wavelet tuned 0.92 12.90
xgboost wavelet tuned 0.92 14.45
ranger bsignal tuned 0.90 15.02
knn dtw tuned 0.92 15.22
ranger none default 0.90 15.59
ranger bsignal default 0.89 15.71
ranger wavelet default 0.90 16.33
knn dtw default 0.92 16.43
xgboost bsignal tuned 0.90 17.57
ranger none tuned 0.89 18.49

If the only criterion for model selection is predictive performance, (tuned) ma-
chine learning models in combination with feature extraction is a competitive
baseline. This class of methods achieves within 95% of the optimal performance
on 47 out of 51 data sets, while they include the best performing classifier in 35
cases.

5.2.6. Comparison to classical time series classification

Even though the main purpose of this paper is not a direct comparison with
the results from [10], we can use our results to show that applying functional
data approaches and classical machine learning approaches together with feature
extraction can still improve classification accuracy compared to current state-
of-the-art time series classification methods.
In the experiments we conducted, the methods described in this paper improved
accuracy on 9 out of the 51 data sets which is displayed in Figure 7. The 9 data
sets and the corresponding best learner are displayed in Table 4. For each data
set, only the best reached accuracy for both sets of algorithms is displayed.
Additionally, we evaluate how our learners rank in comparison to the individual
bake-off algorithms from [10]. The algorithm which performs best on a data set
obtains the rank 1. The mean rank of the individual learners over all 49 data
sets (we take the intersection of the data sets from our benchmark and the ones
from [10]). The average sorted ranks for the top 50% algorithms are displayed
in Figure 8. We observe that the ensemble methods get the top ranks, which is
no surprise, as for instance the COTE algorithm [42] internally combines several
classifiers from 4 different time series domains.
However, compared to the classical time series algorithms from [10] with the
ensemble methods removed, our functional data algorithms obtain an overall
good rank in accuracy performance, interleaved with the algorithms from [10].
Note that the benchmarks are not exactly comparable due to minor differences
in the benchmark setup, and we instead only include their reported results.

15
 id type values def. trafo
bsignal
bsignal.knots int {3,...,500} 10 -
bsignal.df int {1,...,10} 3 -
multires
res.level int {2,...,5} - -
shift num [0.01,1] - -
pca
rank. int {1,...,30} - -
wavelets
filter chr d4,d8,d20,la8,la20,bl14,bl20,c6,c24 - -
boundary chr periodic,reflection - -
fourier
trafo.coeff chr phase,amplitude - -
dtwkernel
ref.method chr random,all random -
n.refs num [0,1] - -
dtwwindow num [0,1] - -

Table 3: Parameter spaces and default settings for feature extraction methods.

6. Summary and Outlook

In this work, we provide a benchmark along with a software implementation

that integrates the functionality of a diverse set of R-packages into a single user
interface and API. Both contributions come with a multiplicity of benefits:

• The user is not required to learn and deal with the vast complexity of the
different interfaces the underlying packages expose.
• All of the existing functionality (e.g., preprocessing, resampling, perfor-
mance measures, tuning, parallelization) of the mlr ecosystem can now be
used in conjunction with already existing algorithms for functional data.
• We expose functionality that allows us to work with functional data using
traditional machine learning methods via feature extraction methods.
• Integration of additional preprocessing methods or models is (fairly) trivial
and automatically benefits from the full mlr ecosystem.

In order to obtain a broader overview of the performance of the integrated

methods, we perform a large benchmark study. This allows users to get an initial
overview of potential performances of the different algorithms. Specifically,
• We open up new perspectives for time series classification tasks by incor-
porating methods from functional data analysis, as well as feature trans-
formations combined with conventional machine learning models.

16
 Name Algorithm Setting Accuracy
Beef xgboost wavelet tuned 0.83
ChlorineConcentration ksvm none tuned 0.91
DistalPhalanxOutlineAgeGroup ranger none default 0.83
DistalPhalanxOutlineCorrect ranger dtwkernel default 0.83
DistalPhalanxTW ranger bsignal default 0.76
Earthquakes FDboost none default 0.80
Ham xgboost wavelet tuned 0.84
InsectWingbeatSound ranger wavelet default 0.65
SonyAIBORobotSurface1 ksvm wavelet default 0.94

Table 4: Data sets together with corresponding mlrFDA learner and accuracy for which our
learners were able to improve accuracy in the conducted experiments.

• Based on the large scale benchmark, we conclude that many learners

have competitive performance (Figure 7) and additionally offer the inter-
pretability of many functional data analysis methods. Our toolbox serves
as a strong complement and alternative to other time series classification
software.

• The presented benchmark study uses state-of-the-art Bayesian optimiza-

tion for hyperparameter optimization, which results in significant improve-
ments over models that are not tuned. This kind of hyperparameter tuning
is easy to do with mlrFDA. Tuning, albeit heavily influencing performance
is often not investigated. Our benchmark closes this gap in existing liter-
ature.
• We find that extracting vector valued features and feeding them to a con-
ventional machine learning model can often form competitive learners.
• The pareto-optimal set in terms of performance on each data set contains
23 different algorithm−feature-extraction combinations. Our toolbox i)
offers the same API for all methods and ii) allows to automatically search
over this space, and thus allows users to obtain optimal models without
knowing all underlying methods.
Concerning the questions we proposed at the beginning of the paper, we draw
the following conclusions:

• Tuning only a subset of the presented learners and feature extractions, i.e.,
the methods listed in Table 2, is sufficient to achieve good performances
on almost all data sets in our benchmark.
• A simple random forest without any preprocessing can also be a reasonable
baseline for time series data. It achieves an average rank of 15.59 (top 4)
in our benchmark.

17
 parameter type values default trafo
ksvm
C num [-15,15] - 2^x
sigma num [-15,10] - 2^x

ranger
mtry.power num [0,1] - px
min.node.size num [0,0.99] - 2^(log2(n) ∗ x)
sample.fraction num [0.1,1] - -

xgboost
nrounds int {1,...,5000} 100 -
eta num [-10,0] - 2^x
subsample num [0.1,1] - -
booster chr gbtree,gblinear - -
max depth int {1,...,15} - -
min child weight num [0,7] - 2^x
colsample bytree num [0,1] - -
colsample bylevel num [0,1] - -
lambda num [-10,10] - 2^x
alpha num [-10,10] - 2^x

FDboost
mstop int {1,...,5000} 100 -
nu num [0,1] 0.01 -
df num [1,5] 4 -
knots int {5,...,100} 10 -
degree int {1,...,4} 3 -

Table 5: Parameter spaces and defaults used for tuning machine learning and functional data
algorithms. In case no default is provided, package defaults are used. Additional information
can be found in the respective packages documentation.
 1.0 ●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
● ●

0.9 ●

● ●
●

● ●
● ●
● ● ●
●

●
0.8
mlrFDA ACC
● ●
● ●

●
●

● ●
●

●
●

0.7

●
●

0.6

0.5

0.5 0.6 0.7 0.8 0.9 1.0

Bakeoff Learner ACC

Figure 7: Comparing accuracy between our mlrFDA learners and the classical time series
classification algorithms in [10]. For each data set, only the best accuracy for each of the two
benchmarks is shown. We observe that for 9 of the evaluated data sets the classification per-
formance can directly be improved solely by applying our mlrFDA learners, while we perform
on par with the classical time series classification algorithms (when rounding to 3 decimal
digits) on two data sets.

• Most algorithms for functional data (e.g., FDboost) do not perform well in
our benchmark study. As those algorithms are fully interpretable and offer
statistically valid coefficients, they can still be useful in some applications,
and should thus not be ruled out.
• Feature extraction techniques, such as b-spline representations (bsignal )
and wavelet extraction work well in conjunction with machine learning
techniques for vector valued features such as xgboost and random forest.
• Tuning leads to an average reduction in absolute MMCE of 3.59% (ranger),
5.69% (xgboost), 7.78% (ksvm) (across feature extraction techniques) and
11% (FDboost). This holds for all feature extraction techniques, where
improvements range from 1.12% multires to 20.3% fourier.

In future work we will continue to expand the available toolbox along with a
benchmark of new methods, and provide the R community a wider range of
methods that can be used for the analysis of functional data. This includes
not only integrating many already available packages, and as a result to en-
able preprocessing operations such as smoothing (e.g., fda [4]) and alignment
(e.g., fdasrvf [43] or tidyfun [44]), but also to explore and integrate advanced
imputation methods for functional data. Further work will also extend the cur-
rent implementation to support data that is measured on unequal or irregular

19
grids. Additionally, we aim to implement some of the current state-of-the art
machine learning models from the time series classification bake-off [10], such as
the Collective of Transformation-Based Ensembles (COTE) [31]. This enables
researchers to use and compare with current state-of-the-art methods.

Acknowledgements
This work has been funded by the German Federal Ministry of Education and
Research (BMBF) under Grant No. 01IS18036A. The authors of this work take
full responsibilities for its content.

20
References

[1] S. Ullah, C. F. Finch, Applications of functional data analysis: A systematic

review, BMC Medical Research Methodology 13 (2013) 43. URL: https:
//doi.org/10.1186/1471-2288-13-43. doi:10.1186/1471-2288-13-43.
[2] J.-L. Wang, J.-M. Chiou, H.-G. Müller, Functional data analysis,
Annual Review of Statistics and Its Application 3 (2016) 257–295.
URL: https://doi.org/10.1146/annurev-statistics-041715-033624.
doi:10.1146/annurev-statistics-041715-033624.
[3] R Core Team, R: A Language and Environment for Statistical Comput-
ing, R Foundation for Statistical Computing, Vienna, Austria, 2014. URL:
http://www.R-project.org/.
[4] J. O. Ramsay, H. Wickham, S. Graves, G. Hooker, fda: Functional
Data Analysis, 2018. URL: https://CRAN.R-project.org/package=fda,
r package version 2.4.8.
[5] S. Brockhaus, D. Ruegamer, FDboost: Boosting Functional Regression
Models, 2018.
[6] F. Scheipl, Cran task view - functional data analysis, 2018. URL: https:
//cran.r-project.org/web/views/FunctionalData.html.
[7] M. Febrero-Bande, M. Oviedo de la Fuente, Statistical computing in func-
tional data analysis: The r package fda.usc, Journal of Statistical Software
51 (2012) 1–28. URL: http://www.jstatsoft.org/v51/i04/.
[8] J. Goldsmith, F. Scheipl, L. Huang, J. Wrobel, J. Gellar, J. Harezlak,
M. W. McLean, B. Swihart, L. Xiao, C. Crainiceanu, P. T. Reiss, refund:
Regression with Functional Data, 2018. URL: https://CRAN.R-project.
org/package=refund, r package version 0.1-17.
[9] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn.
20 (1995) 273–297. URL: https://doi.org/10.1023/A:1022627411411.
doi:10.1023/A:1022627411411.
[10] A. Bagnall, J. Lines, A. Bostrom, J. Large, E. Keogh, The great time
series classification bake off: a review and experimental evaluation of recent
algorithmic advances, Data Mining and Knowledge Discovery 31 (2017)
606–660.
[11] H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, P.-A. Muller, Deep
learning for time series classification: a review, Data Mining and Knowledge
Discovery (2019) 1–47.
[12] J. J. Rodriguez, L. I. Kuncheva, C. J. Alonso, Rotation forest: A new
classifier ensemble method, IEEE Transactions on Pattern Analysis and
Machine Intelligence 28 (2006) 1619–1630. doi:10.1109/TPAMI.2006.211.

21
[13] C. Stachl, M. Bühner, Show me how you drive and i’ll tell you who you
are. recognizing gender using automotive driving parameters, Procedia
Manufacturing 3 (2015) 5587 – 5594. URL: http://www.sciencedirect.
com/science/article/pii/S2351978915007441. doi:https://doi.org/
10.1016/j.promfg.2015.07.743, 6th International Conference on Ap-
plied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Con-
ferences, AHFE 2015.
[14] R. Hyndman, E. Wang, Y. Kang, T. Talagala, Y. Yang, tsfeatures:
Time Series Feature Extraction, 2018. URL: https://github.com/
robjhyndman/tsfeatures/, r package version 0.1.

[15] E. Aldrich, wavelets: A package of functions for computing wavelet fil-

ters, wavelet transforms and multiresolution analyses, 2013. URL: https:
//CRAN.R-project.org/package=wavelets, r package version 0.3-0.
[16] J. Goldsmith, F. Scheipl, Estimator selection and combination in scalar-on-
function regression, Computational Statistics & Data Analysis 70 (2014)
362–372.
[17] S. Mallat, A theory for multiresolution signal decomposition: The wavelet
representation, IEEE Trans. Pattern Anal. Mach. Intell. 11 (1989) 674–693.
[18] E. O. Brigham, R. E. Morrow, The fast fourier transform, IEEE Spectrum
4 (1967) 63–70. doi:10.1109/MSPEC.1967.5217220.
[19] T. Rakthanmanon, B. Campana, A. Mueen, G. Batista, B. Westover,
Q. Zhu, J. Zakaria, E. Keogh, Searching and mining trillions of time se-
ries subsequences under dynamic time warping, in: Proceedings of the
18th ACM SIGKDD international conference on Knowledge discovery and
data mining, ACM, 2012, pp. 262–270. URL: http://doi.org/10.1145/
2339530.2339576. doi:10.1145/2339530.2339576.
[20] P. Boersch-Supan, rucrdtw: Fast time series subsequence search in r, The
Journal of Open Source Software 1 (2016) 1–2. URL: http://doi.org/10.
21105/joss.00100. doi:10.21105/joss.00100.

[21] J. Ramsay, Functional data analysis, Wiley Online Library, 2006.

[22] A. Srivastava, E. P. Klassen, Functional and Shape Data Analysis, Springer,
2016.
[23] T. Hothorn, P. Bühlmann, T. Kneib, M. Schmid, B. Hofner, Model-based
boosting 2.0, Journal of Machine Learning Research 11 (2010) 2109–2113.

[24] S. Greven, F. Scheipl, A general framework for functional regression mod-

elling, Statistical Modelling 17 (2017) 1–35.
[25] F. Ferraty, P. Vieu, Nonparametric functional data analysis: theory and
practice, Springer Science & Business Media, 2006.

22
[26] T. Maierhofer, F. Pfisterer, classiFunc: Classification of Functional Data,
2018. URL: https://CRAN.R-project.org/package=classiFunc, r pack-
age version 0.1.1.
[27] Grupo de Aprendizaje Automatico - Universidad Autonoma de Madrid,
scikit-fda: Functional Data Analysis in Python, 2019. URL: https://fda.
readthedocs.io.
[28] A. Bagnall, J. Lines, W. Wickers, E. Keogh, The UEA & UCR time series
classification repository, 2017. www.timeseriesclassification.com.
[29] L. Breiman, Random forests, Mach. Learn. 45 (2001) 5–32. URL: https:
//doi.org/10.1023/A:1010933404324. doi:10.1023/A:1010933404324.
[30] P. Tormene, T. Giorgino, S. Quaglini, M. Stefanelli, Matching incomplete
time series with dynamic time warping: An algorithm and an application
to post-stroke rehabilitation, Artificial Intelligence in Medicine 45 (2008)
11–34. doi:10.1016/j.artmed.2008.11.007.

[31] A. Bagnall, J. Lines, J. Hills, A. Bostrom, Time-series classification with

cote: The collective of transformation-based ensembles, IEEE Transactions
on Knowledge and Data Engineering 27 (2015) 2522–2535. doi:10.1109/
TKDE.2015.2416723.
[32] K. Fuchs, F. Scheipl, S. Greven, Penalized scalar-on-functions regression
with interaction term, Computational Statistics & Data Analysis 81 (2015)
38–51. doi:10.1016/j.csda.2014.07.001.
[33] C. A. Ratanamahatana, E. J. Keogh, Three myths about dynamic time
warping data mining., in: H. Kargupta, J. Srivastava, C. Kamath, A. Good-
man (Eds.), SDM, SIAM, 2005, pp. 506–510.

[34] J. Hills, J. Lines, E. Baranauskas, J. Mapp, A. Bagnall, Clas-

sification of time series by shapelet transformation, Data Min.
Knowl. Discov. 28 (2014) 851–881. URL: http://dx.doi.org/10.1007/
s10618-013-0322-1. doi:10.1007/s10618-013-0322-1.
[35] P. Kokoszka, M. Reimherr, Introduction to functional data analysis, CRC
Press, 2017.
[36] J. Schiffner, B. Bischl, M. Lang, J. Richter, Z. M. Jones, P. Probst, F. Pfis-
terer, M. Gallo, D. Kirchhoff, T. Kühn, et al., mlr tutorial, arXiv preprint
arXiv:1609.06146 (2016).

[37] B. Bischl, O. Mersmann, H. Trautmann, C. Weihs, Resampling methods

for meta-model validation with recommendations for evolutionary com-
putation, Evolutionary Computation 20 (2012) 249–275. URL: https:
//doi.org/10.1162/EVCO.a.00069. doi:10.1162/EVCO.a.00069, pMID:
22339368.

23
[38] J. Snoek, H. Larochelle, R. P. Adams, Practical bayesian opti-
mization of machine learning algorithms, in: F. Pereira, C. J. C.
Burges, L. Bottou, K. Q. Weinberger (Eds.), Advances in Neu-
ral Information Processing Systems 25, Curran Associates, Inc.,
2012, pp. 2951–2959. URL: http://papers.nips.cc/paper/
4522-practical-bayesian-optimization-of-machine-learning-algorithms.
pdf.
[39] B. Bischl, J. Richter, J. Bossek, D. Horn, J. Thomas, M. Lang, mlrMBO:
A Modular Framework for Model-Based Optimization of Expensive Black-
Box Functions, 2017. URL: http://arxiv.org/abs/1703.03373.

[40] B. Bischl, M. Lang, O. Mersmann, J. Rahnenführer, C. Weihs, BatchJobs

and BatchExperiments: Abstraction mechanisms for using R in batch en-
vironments, Journal of Statistical Software 64 (2015) 1–25. URL: http:
//www.jstatsoft.org/v64/i11/.
[41] M. Lang, B. Bischl, D. Surmann, batchtools: Tools for r to work on batch
systems, The Journal of Open Source Software 2 (2017). URL: https:
//doi.org/10.21105/joss.00135. doi:10.21105/joss.00135.
[42] A. Bagnall, J. Lines, J. Hills, A. Bostrom, Time-series classification with
cote: the collective of transformation-based ensembles, IEEE Transactions
on Knowledge and Data Engineering 27 (2015) 2522–2535.

[43] J. D. Tucker, fdasrvf: Elastic Functional Data Analysis, 2016. URL: https:
//CRAN.R-project.org/package=fdasrvf, r package version 1.6.0.
[44] F. Scheipl, J. Goldsmith, tidyfun, https://github.com/fabian-s/
tidyfun, 2019.

[45] L. Jin, Q. Niu, Y. Jiang, H. Xian, Y. Qin, M. Xu, Driver sleepiness detection
system based on eye movements variables, Advances in Mechanical Engi-
neering 5 (2013) 648431. URL: https://doi.org/10.1155/2013/648431.
doi:10.1155/2013/648431.
[46] M. Murugappan, M. Rizon, R. Nagarajan, S. Yaacob, Eeg feature ex-
traction for classifying emotions using fcm and fkm, in: Proceedings
of the 7th WSEAS International Conference on Applied Computer and
Applied Computational Science, ACACOS’08, World Scientific and En-
gineering Academy and Society (WSEAS), Stevens Point, Wisconsin,
USA, 2008, pp. 299–304. URL: http://dl.acm.org/citation.cfm?id=
1415743.1415793.

[47] S. Soltani, On the use of the wavelet decomposition for time se-
ries prediction, Neurocomputing 48 (2002) 267 – 277. URL: http:
//www.sciencedirect.com/science/article/pii/S0925231201006488.
doi:https://doi.org/10.1016/S0925-2312(01)00648-8.

24
Appendix A. API overview

For the interested reader, we introduce a brief overview of the API and func-
tionality.

Appendix A.1. Representing functional data in mlrFDA

A sketch of the data structure we use to represent functional data can be found
in the right part of Figure A.9. We assume a data set consists of data for N
observational units, organized in rows of features, where one row contains all
observed features for one observational unit, i.e., each row typically contains
several functional and/or scalar covariates. For a classical, non-functional data
set, the P features are single columns (as depicted in the left part of Figure
A.9). A functional data set, on the other hand, consists of single-column scalar
features as well as functional features of different length for each functional
covariate, each represented by multiple adjacent and connected columns. Each
of these columns contains the evaluations of the functional feature at a certain
argument value for all observational units (right part of Figure A.9).
As an example which will be used throughout the remainder of this paper, we use
the fuelSubset data set from package FDboost, see also Figure 1. It contains
a numeric target variable heatan, the fuel’s heating value, a scalar feature h20,
the fuel’s water content, and two functional features NIR and UVVIS, measured
at 231 and 129 wavelengths, respectively. To start with a clean sheet, we create
a data.frame containing all features as separate columns.

R> library(mlr)
R> library(FDboost)
R> df = data.frame(fuelSubset[c("heatan", "h2o", "UVVIS", "NIR")])

The first step when setting up an experiment in any analysis is to make the data
accessible for the specific algorithms that will be applied. In mlr, the data it-
self, and additional information, such as which column corresponds to the target
variable is stored as a Task, requiring the input data to be of type data.frame.

The list of column positions of the functional features is then passed as argu-
ment fd.features to makeFunctionalData(), which returns an object of type
data.frame in which the columns corresponding to each functional feature are
combined into matrix columns. 3

R> fd.features = list("UVVIS" = 3:136, "NIR" = 137:367)

R> fdf = makeFunctionalData(df, fd.features = fd.features)
R> str(fdf)
’data.frame’: 129 obs. of 4 variables:

3 As an alternative, a list of the column names containing the functional features is also

valid as argument to fd.features, which is especially useful if columns are already labeled.

25
 \$ heatan: num 26.8 27.5 23.8 18.2 17.5 ...
\$ h2o : num 2.3 3 2 1.85 2.39 ...
\$ UVVIS : num [1:129, 1:134] 0.145 -1.584 -0.814 -1.311 -1.373 ...
..- attr(*, "dimnames")=List of 2
.. ..\$ : NULL
.. ..\$ : chr "UVVIS.1" "UVVIS.2" "UVVIS.3" "UVVIS.4" ...
\$ NIR : num [1:129, 1:231] 0.2818 0.2916 -0.0042 -0.034 -0.1804 ...
..- attr(*, "dimnames")=List of 2
.. ..\$ : NULL
.. ..\$ : chr "NIR.1" "NIR.2" "NIR.3" "NIR.4" ...
We additionally specify the name "fuelsubset" and the target variable "heatan".
The structure of the functional Task object is rather similar to the non-functional
Task, with the additional information functionals, which states how many
functional features are present in the underlying data.
R> tsk1 = makeRegrTask("fuelsubset", data = fdf, target = "heatan")
R> print(tsk1)
Supervised task: fuelsubset
Type: regr
Target: heatan
Observations: 129
Features:
numerics factors ordered functionals
1 0 0 2
Missings: FALSE
Has weights: FALSE
Has blocking: FALSE
Has coordinates: FALSE
After defining the task, a learner is created by calling makeLearner. This
contains the algorithm that will be fitted on the data in order to obtain a
model. Currently, mlrFDA supports both functional regression and functional
classification. A list of supported learners can be found in Table 1

Appendix A.2. Machine Learning and Feature Extraction

Classical machine learning algorithms do not take into account the character-
istics of functional data and treat the input data as vector valued features.
Without additional preprocessing, this typically yields poor performance on, as
the models cannot exploit the lower intrinsic dimensionality of the functional
covariates nor the fact that they represent observations over a continuum.
In mlrFDA, classical algorithms can be applied to functional data, however, a
warning message will be displayed. In our example, we train a partitioning tree
on the functional data, while ignoring the functional structure.
R> rpart.lrn = makeLearner("regr.rpart")
R> m = train(learner = rpart.lrn, task = tsk1)
Functional features have been converted to numerics

26
 Name Function Package
Discrete Wavelet Transform extractFDAWavelets() wavelets
Fast Fourier Transform extractFDAFourier() stats
Principal Component Analysis extractFDAPCA() stats
B-Spline Features extractFDABsignal() FDboost
Multi-Resolution Feature Extraction extractFDAMultiResFeatures() -
Time Series Features extractFDATsfeatures() tsfeatures
Dynamic Time-Warping Kernel extractFDADTWKernel() rucrdtw

Table A.6: Feature extraction methods currently implemented in mlrFDA and underlying
packages

For conventional learning algorithms to work well on functional data, informa-

tive scalar features need to be extracted from the functional features.
Feature extraction is applied in practice for a multiplicity of reasons, as it often
not only reduces the dimensionality of the resulting problem, but also allows
researchers to make use of domain knowledge, for example by hand-crafting
features from measurements of continuous processes. Examples for this include
deriving features that allow for sleepiness detection [45], or by extracting features
from electro-cardiogram data in order to detect emotions [46]. The resulting
features often have a much lower dimensionality, which often improves fitted
models. Other preprocessing methods for functional or time series data include
extracting general purpose features such as wavelet coefficients [15, 47], principal
component scores or Fourier coefficients. The resulting scalar features can then
be used with different machine learning methods such as k-nearest neighbors.
In the following section, we showcase the feature extraction procedure using
general purpose features as an example. We want to emphasize that it is also
easily possible to write custom feature extraction methods using the makeFea-
tureExtractionMethod function.
In our example, we extract the Fourier coefficients from the functional feature
UVVIS, and principal component scores from the second functional feature NIR
in order to transform the original task with functional data into a conventional
task.
R> feat.methods = list("UVVIS" = extractFDAFourier(),
"NIR" = extractFDAPCA())
R> extracted = extractFDAFeatures(tsk, feat.methods = feat.methods)
R> extracted

$task
Supervised task: fuelsubset
Type: regr
Target: heatan
Observations: 129
Features:
numerics factors ordered functionals
137 0 0 0
Missings: FALSE

27
 Has weights: FALSE
Has blocking: FALSE
Has coordinates: FALSE

$desc
Extraction of features from functional data:
Target: heatan
Functional Features: 2; Extracted features: 2

As an alternative, the feature extraction can be applied in a wrapper method

makeExtractFDAFeatsWrapper(). In general, a wrapper combines a learner
method with another method, thereby creating a new learner that can be han-
dled like any other learner. In our case, a classical machine learning method
is combined with the data preprocessing step of feature transformation from
functional to non-functional data.

R> wrapped.lrn = makeExtractFDAFeatsWrapper("regr.rpart",

feat.methods = feat.methods)

This is suitable for honest cross-validation of data-adaptive feature extraction

methods like principal components. We can now cross-validate the learner cre-
ated above using mlr’s resample function with 10-fold cross-validation.

R> res = resample(learner = wrapped.lrn, task = tsk1,

resampling = cv10)

In the same way, we can train and predict on data, or benchmark multiple
learners across multiple data sets. Additionally, we can apply a tuneWrapper to
our learner in order to automatically tune hyperparameters of the learner and
the preprocessing method during cross-validation.

Appendix B. Data sets used in the Benchmark

Table B.7 contains all data sets used in the benchmark along with additional
data properties.

Appendix C. Failed and missing experiments

Experiments for some algorithm / data set combinations failed due to implemen-
tation details or algorithm properties. In order to increase transparency, failed
algorithms are listed here, and if available reasons for failure are provided.
At the time of the benchmark, the implementation in the tsfeatures package
was not stable enough to be included in the benchmark.

• classif.fgam
Data sets: BeetleFly, BirdChicken, Coffee, Computers, DistalPhalanx-
OutlineCorrect, Earthquakes, ECG200, ECGFiveDays, ElectricDeviceOn,

28
 GunPoint, Ham, Herring, ItalyPowerDemand, Lightning2, MoteStrain,
ShapeletSim, SonyAIBORobotSurface1, Strawberry, ToeSegmentation1,
TwoLeadECG, Wafer, Wine, Yoga
Reason: Too few instances in some classes, such that p > n.
• classif.fdausc.kernel and .np Data sets: ElectricDeviceOn, Shapelet-
Sim
• classif.fdausc.knn
Data sets: DistalPhalanxTW, EpilepsyX

29
 Euclidean_1NN ●

xgboost_wavelet_default ●

glmnet_bsignal_default ●

ksvm_multires_tuned ●

ksvm_none_tuned ●

xgboost_multires_default ●

ACF ●

xgboost_none_default ●

DDTW_R1_1NN ●

ranger_dtwkernel_default ●

xgboost_bsignal_default ●

ranger_fourier_tuned ●

ranger_fpca_tuned ●

xgboost_multires_tuned ●

ksvm_bsignal_tuned ●

xgboost_fourier_tuned ●

MLP ●

xgboost_dtwkernel_tuned ●

PS ●

ranger_multires_tuned ●

ranger_dtwkernel_tuned ●

BoP ●

ranger_multires_default ●

DTW_R1_1NN ●

DDTW_Rn_1NN ●

WDDTW_1NN ●

SVMQ ●

ranger_none_tuned ●
Algorithm

SAXVSM ●

xgboost_none_tuned ●

RandF ●

LCSS_1NN ●

xgboost_bsignal_tuned ●

ranger_bsignal_default ●

ranger_wavelet_default ●

ranger_bsignal_tuned ●

ranger_none_default ●

DD_DTW ●

knn_dtw_default ●

ERP_1NN ●

DTW_Rn_1NN ●

WDTW_1NN ●

DTD_C ●

TWE_1NN ●

CID_DTW ●

knn_dtw_tuned ●

LPS ●

xgboost_wavelet_tuned ●

ranger_wavelet_tuned ●

MSM_1NN ●

RotF ●

TSBF ●

LS ●

DTW_F ●

TSF ●

EE ●

BOSS ●

ST ●

Flat.COTE ●

HIVE.COTE ●

10 20 30 40 50
Average_Rank

Figure 8: Comparing sorted average performance ranks between our mlrFDA learners (algo-
rithm names in lower case) and the classical time series classification algorithms (algorithm
names in capital) in [10], The mean rank of each individual learner over all 49 data sets is
displayed. Only the first half of all algorithms being compared are displayed here. We observe
that the Ensemble Methods like HIVE.COTE, FLAT.COTE, ST, BOSS, EE occupy the top
tier, while the rest of the rank space are interleaved by our mlrFDA algorithms and algorithms
from [10]

Figure A.9: Schematic comparison of non-functional and functional data representation in

mlrFDA. The green feature is a functional feature spanning multiple columns.
Name Obs. Classes Length Type Split
Adiac 781 37 176 IMAGE 0.50
ArrowHead 211 3 251 IMAGE 0.17
Beef 60 5 470 SPECTRO 0.50
BeetleFly 40 2 512 IMAGE 0.50
BirdChicken 40 2 512 IMAGE 0.50
Car 120 4 577 SENSOR 0.50
CBF 930 3 128 SIMULATED 0.03
ChlorineConcentration 4307 3 166 SIMULATED 0.11
Coffee 56 2 286 SPECTRO 0.50
Computers 500 2 720 DEVICE 0.50
CricketX 780 12 300 MOTION 0.50
DistalPhalanxOutlineAgeGroup 539 3 80 IMAGE 0.74
DistalPhalanxOutlineCorrect 876 2 80 IMAGE 0.68
DistalPhalanxTW 539 6 80 IMAGE 0.74
Earthquakes 461 2 512 SENSOR 0.70
ECG200 200 2 96 ECG 0.50
ECGFiveDays 884 2 136 ECG 0.03
ElectricDeviceOn 1008 2 360 DEVICE 0.63
EpilepsyX 275 4 208 HAR 0.61
FaceAll 2250 14 131 IMAGE 0.25
FacesUCR 2250 14 131 IMAGE 0.09
Fish 350 7 463 IMAGE 0.50
GunPoint 200 2 150 MOTION 0.25
Ham 214 2 431 SPECTRO 0.51
Herring 128 2 512 IMAGE 0.50
InsectWingbeatSound 2200 11 256 SENSOR 0.10
ItalyPowerDemand 1096 2 24 SENSOR 0.06
LargeKitchenAppliances 750 3 720 DEVICE 0.50
Lightning2 121 2 637 SENSOR 0.50
Lightning7 143 7 319 SENSOR 0.49
Meat 120 3 448 SPECTRO 0.50
MedicalImages 1141 10 99 IMAGE 0.33
MoteStrain 1272 2 84 SENSOR 0.02
OSULeaf 442 6 427 IMAGE 0.45
Plane 210 7 144 SENSOR 0.50
RefrigerationDevices 750 3 720 DEVICE 0.50
ScreenType 750 3 720 DEVICE 0.50
ShapeletSim 200 2 500 SIMULATED 0.10
SmallKitchenAppliances 750 3 720 DEVICE 0.50
SonyAIBORobotSurface1 621 2 70 SENSOR 0.03
Strawberry 983 2 235 SPECTRO 0.62
SwedishLeaf 1125 15 128 IMAGE 0.44
SyntheticControl 600 6 60 SIMULATED 0.50
ToeSegmentation1 268 2 277 MOTION 0.15
Trace 200 4 275 SENSOR 0.50
TwoLeadECG 1162 2 82 ECG 0.02
TwoPatterns 5000 4 128 SIMULATED 0.20
UWaveGestureLibraryX 4478 8 315 MOTION 0.20
Wafer 7164 2 152 SENSOR 0.14
Wine 111 2 234 SPECTRO 0.51
Yoga 3300 2 426 IMAGE 0.09

Table B.7: Data sets from the UCI Archive used in the benchmark.