An Intuitive Tutorial to Gaussian Processes

Regression
arXiv:2009.10862v3 [stat.ML] 2 Feb 2021

Jie Wang
Ingenuity Labs Research Institute
[email protected]

February 3, 2021

Offroad Robotics
c/o Ingenuity Labs Research Institute
Queen’s University
Kingston, ON K7L 3N6 Canada
 Abstract

This tutorial aims to provide an intuitive understanding of the Gaussian

processes regression. Gaussian processes regression (GPR) models have been
widely used in machine learning applications because of their representation
flexibility and inherently uncertainty measures over predictions. The basic
concepts that a Gaussian process is built on, including multivariate normal
distribution, kernels, non-parametric models, joint and conditional probability
were explained first. Next, the GPR was described concisely together with
an implementation of a standard GPR algorithm. Beyond the standard GPR,
packages to implement state-of-the-art Gaussian processes algorithms were
reviewed. This tutorial was written in an accessible way to make sure readers
without a machine learning background can obtain a good understanding of
the GPR basics.
Contents
1 Introduction 1

2 Mathematical Basics 1
2.1 Gaussian Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2.2 Multivariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . 4
2.3 Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4 Non-parametric Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Gaussian Processes 9

4 Illustrative example 11
4.1 Hyperparameters Optimization . . . . . . . . . . . . . . . . . . . . . . 12
4.2 Gaussian processes packages . . . . . . . . . . . . . . . . . . . . . . . 14

5 Summary and Discussion 15

A Appendix 16
An Intuitive Tutorial to Gaussian Processes Regression 1

1 Introduction
The Gaussian processes model is a probabilistic supervised machine learning frame-
work that has been widely used for regression and classification tasks. A Gaus-
sian processes regression (GPR) model can make predictions incorporating prior
knowledge (kernels) and provide uncertainty measures over predictions [11]. Gaus-
sian processes model is a supervised learning method developed by computer sci-
ence and statistics communities. Researchers with engineering backgrounds often
find it is difficult to gain a clear understanding of it. To understand GPR, even only
the basics needs to have knowledge of multivariate normal distribution, kernels,
non-parametric model, and joint and conditional probability.
In this tutorial, we present a concise and accessible explanation of GPR. We first
review the mathematical concepts that GPR models are built on to make sure read-
ers have enough basic knowledge. In order to provide an intuitive understanding
of GPR, plots are actively used. The codes developed to generate plots are pro-
vided at https://github.com/jwangjie/Gaussian-Processes-Regression-Tutorial.

2 Mathematical Basics
This section reviews the basic concepts needed to understand GPR. We start with
the Gaussian (normal) distribution, then explain theories of multivariate normal
distribution (MVN), kernels, non-parametric model, and joint and conditional prob-
ability. In regression, given some observed data points, we want to fit a function to
represent these data points, then use the function to make predictions at new data
points. For a given set of observed data points shown in Fig. 1(a), there are infinite
numbers of possible functions that fit these data points. In Fig. 1(b), we show five
sample functions that fit the data points. In GPR, the Gaussian processes conduct
regression by defining a distribution over these infinite number of functions [8].

2.1 Gaussian Distribution

A random variable X is Gaussian or normally distributed with mean µ and vari-
ance σ2 if its probability density function (PDF) is [10]
!
1 ( x − µ )2
PX ( x ) = √ exp − .
2πσ 2σ2
An Intuitive Tutorial to Gaussian Processes Regression 2

10.0 10.0
7.5 7.5
5.0 5.0
2.5 2.5
0.0 0.0
2.5 2.5
5.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
(a) Data point observations (b) Five possible regression functions by GPR

Figure 1: A regression example: (a) The observed data points, (b) Five sample
functions that fit the observed data points.

Here, X represent random variables and x is the real argument. The normal distri-
bution of X is usually represented by PX ( x ) ∼ N (µ, σ2 ). The PDF of a uni-variate
normal (or Gaussian) distribution was plotted in Fig. 2. We randomly generated
1000 points from a uni-variate normal distribution and plotted them on the x axis.

0.40
0.35
0.30
0.25
PX(x)

0.20
0.15
0.10
0.05
0.00
4 2 0 2 4
x
Figure 2: One thousand normal distributed data points were plotted as red vertical
bars on the x axis. The PDF of these data points was plotted as a two-dimensional
bell curve.

These random generated data points can be expressed as a vector x1 = [ x11 , x12 , . . . , x1n ].
By plotting the vector x1 on a new Y axis at Y = 0, we projected points [ x11 , x12 , . . . , x1n ]
into another space shown in Fig. 3. We did nothing but vertically plot points of
the vector x1 in a new Y, x coordinates space. We can plot another independent
An Intuitive Tutorial to Gaussian Processes Regression 3

Gaussian vector x2 = [ x21 , x22 , . . . , x2n ] in the same coordinates at Y = 1 shown in

Fig. 3. Keep in mind that either x1 or x2 is a uni-variate normal distribution shown
in Fig. 2.

3
2
1
0
x

1
2
3
0.0 0.2 0.4 0.6 0.8 1.0
Y
Figure 3: Two independent uni-variate Gaussian vector points were plotted verti-
cally in the Y, x coordinates space.

Next, we randomly selected 10 points in vector x1 and x2 respectively and con-

nected these 10 points in order by lines as shown in Fig. 4(a). These connected
lines look like linear functions spanning within the [0, 1] domain. We can use these
functions to make predictions for regression tasks if the new data points are on (or
close enough to) these linear lines. However, in most cases, the assumption that
new data points are always on the connected linear functions is not held. If we
plot more random generated uni-variate Gaussian vectors, for example, 20 vectors
x1 , x2 , . . . , x20 in [0, 1], and connect 10 random selected sample points of each vec-
tor as lines, we get 10 lines that look more like functions within [0, 1] shown in Fig.
4(b). We still cannot use these lines to make predictions for regression tasks be-
cause they are too noisy. These functions must be smoother, meaning input points
that are close to each other should have similar output values. The“functions” gen-
erated by connecting independent Gaussian vector points are not smooth enough
for regression tasks, we need these independent Gaussian correlated to each other
as a joint Gaussian distribution. The joint Gaussian distribution is described by the
multivariate normal distribution theory.
An Intuitive Tutorial to Gaussian Processes Regression 4

3
2
2
1
1
0
0

x
x

1
1
2
2
3
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Y Y
(a) Two Gaussian vectors (b) Twenty Gaussian vectors

Figure 4: Connecting points of independent Gaussian vectors by lines: (a) Ten

random selected points in two vector x1 and x2 , (b) Ten random selected points in
twenty vectors x1 , x2 , . . . , x20 .

2.2 Multivariate Normal Distribution

It’s common that a system is described by more than one feature variables ( x1 , x2 , . . . , x D )
that are correlated to each other. If we want to model these variables all together
as one Gaussian model, we need to use a multivariate Gaussian/normal (MVN)
[10] distribution model. The PDF of an MVN with D dimension is defined as [10]
 
1 1 T −1
N ( x |µ, Σ) = exp − ( x − µ) Σ ( x − µ) ,
(2π ) D/2 |Σ|1/2 2

where D is the number of the dimension, x represents the variable, µ = E[ x ] ∈

RD is the mean vector, and Σ = cov[ x ] is the D × D covariance matrix. The Σ
is a symmetric matrix that stores the pairwise covariance of all jointly modeled
random variables with Σij = cov(yi , y j ) as its (i, j) element.
We use a bi-variate normal (BVN) distribution as a simplified example to un-
derstand the MVN theory. A BVN distribution can be visualized as a three-dimensional
(3-d) bell curve with heights represent the probability density shown in Fig. 5a.
The projections of the 3-d bell curve on the x1 , x2 plane are ellipse contours plotted
in Fig. 5a and 5b. The shape of ellipses shows the correlations between x1 and x2
points, i.e. how much one variable of x1 is related to another variable of x2 . The
P( x1 , x2 ) is the joint probability
" # of x1 and x2 . For a BVN, the mean vector µ is a
µ1
two-dimensional vector , where µ1 and µ2 are the independent mean of x1
µ2
An Intuitive Tutorial to Gaussian Processes Regression 5

0.20
0.15 4
0.10 3
P(x1, x2)

0.05
0.00 2
1

x2
0

43 1
2
x2 1 0 1 2 3 2
23
3 2 1 0 x1 1 4 2 0 2
x1
(a) 3-d bell curve (b) 2-d ellipse contours

Figure 5: The PDF of a BVN visualization: (a) a 3-d bell curve with height rep-
resents the probability density, (b) ellipse contour projections showing the co-
relationship between x1 and x2 points.

# "
σ11 σ12
and x2 respectively. The covariance matrix is , where the diagonal terms
σ21 σ22
σ11 and σ22 are the independent variance of x1 and x2 respectively. The off-diagonal
terms σ12 and σ21 represent correlations between x1 and x2 . The BVN is expressed
as
" # " # " #!
x1 µ1 σ11 σ12
∼N , ∼ N (µ, Σ) .
x2 µ2 σ21 σ22

It is intuitively understandable that we need the conditional probability rather

than the joint probability for regression tasks. If we cut a slice on the 3-d bell curve
shown in Fig. 5, we get the conditional probability distribution P( x1 | x2 ) shown in
Fig. 6. The conditional distribution is also Gaussian [11].

2.3 Kernels
After reviewing the MVN concept, we want to smooth the functions in Fig. 4(b).
This can be done by defining covariance functions, which reflect our ability to ex-
press prior knowledge of the form of the function we are modelling. In regression,
outputs of the function should be similar when two input are close to each other.
One possible equation format is the dot product A · B = k Akk Bkcosθ, where θ is
An Intuitive Tutorial to Gaussian Processes Regression 6

P (x 1

P (x1 |x = x2 ) = P (x1 |x2 )

, x 2)

x2
x1

Figure 6: The conditional probability distribution P( x1 | x2 ) by cutting a slice on

the PDF 3-d bell curve of a BVN.

the angle between two input vectors. When two input vectors are similar, their dot
product output value is high.
If a function is defined solely in terms of inner products in the input space,
then the function k( x, x 0 ) is a kernel function [11]. The most widely used kernel
or covariance function is the squared exponential (SE) kernel function. The SE
kernel is de-facto default kernel for Gaussian processes [5]. This is because it can
be integrated against most functions that you need to due to its universal property.
And every function in its prior has infinitely many derivatives. It’s also known as
the radial basis function (RBF) kernel or Gaussian kernel function that is defined
as 1
!
( x i − x j )2
cov( xi , x j ) = exp − .
2

In Fig. 4(b), we plotted 20 independent Gaussian vectors by connecting 10 ran-

dom selected sample points of each vector in order by lines. Instead of plotting 20
1 This
is a simplified RBF without hyperparameters for simplicity. A general RBF will be further
explained in section 4.1.
An Intuitive Tutorial to Gaussian Processes Regression 7

independent Gaussian, we generated 10 twenty-variate normal (20-VN) distribu-

tion with an identity covariance function shown in 7(a). It’s the same as 4(b) due to
there is no correlations between points by using an identity as its kernel function.
When a RBF was used as the covariance function, we got smooth lines shown in
7(b).

2.0
2 1.5

1 1.0
0.5
0
0.0
1 0.5
1.0
2
1.5

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

(a) Ten samples of the 20-VN prior with an (b) Ten samples of the 20-VN prior with a RBF
identity kernel kernel

Figure 7: Samples of twenty-variate normal (20-VN) distribution kernelized prior

functions: (a) Ten 20-VN with an identity function covariance, (b) Ten 20-VN with
a RBF covariance.

By adding covariance functions, we obtain smoother lines, and they start to

look like functions. It is natural to consider continuing to increase the dimen-
sion of MVN. Here, dimension refers to the number of multi-variables of MVN.
When the dimension of MVN gets larger, the region of interest will be filled up
with more points. When the dimension becomes infinity, there will be a point to
represent any possible input point. By using MVN with infinity dimensions, we
fit functions with infinity parameters to do regression tasks. We can make pre-
dictions anywhere in the region of interest. In Fig. 8, We plotted 200 samples of
a two hundred-variate normal (200-VN) distribution to get a feeling of functions
with infinity parameters. We call these functions as kernelized prior functions be-
cause there are no observed data points yet. All functions are random generated
by the MVN model incorporating with kernel functions as prior knowledge before
having any observed data points.
An Intuitive Tutorial to Gaussian Processes Regression 8

Figure 8: Two hundred kernelized prior functions of a two hundred-variate nor-

mal distribution.

2.4 Non-parametric Model

This section explains the concept of parametric and nonparametric model [10].
Parametric models assume that the data distribution can be modeled in terms
of a set of finite number of parameters. In regression, given some data points,
we would like to make predictions of the function value y = f ( x ) with a new
specific x. If we assume a linear regression model, y = θ1 + θ2 x, we need to
find the parameters θ1 and θ2 to define the function. In many cases, the linear
model assumption isn’t hold, a polynomial model with more parameters, such as
y = θ1 + θ2 x + θ3 x2 is needed. We use the training dataset D with n observed
points, D = [( xi , yi ) | i = 1, . . . , n] to train the model, i.e. mapping x to y through
basis functions f ( x ). After the training process, we assume all information of the
data are captured by the feature parameters `, thus predictions are independent
to the training dataset D. It can be expressed as P( f ∗ | X∗ , `, D ) = P( f ∗ | X∗ , `), in
which f ∗ are predictions made at unobserved data points X∗ . Thus, when con-
ducting regressions using parametric models, the complexity or flexibility of mod-
els are limited by the number of parameters. If the number of parameters of a
model grows with the size of observed dataset, it’s a non-parametric model. A
non-parametric model does not imply that there are no parameters, but rather in-
finite parameters.
An Intuitive Tutorial to Gaussian Processes Regression 9

3 Gaussian Processes
Before explaining the Gaussian process, let’s do a quick review of the basic con-
cepts we covered. In regression, there is a function f we are trying to model given
observed data points D (training dataset) from the unknown function f. The tradi-
tional nonlinear regression methods typically give one function that is considered
to fit the dataset best. However, there may be more than one function that fit
the observed data points equally well. We saw that when the dimension of MVN
was infinite, we could make predictions at any point with these infinite number
of functions. These functions are MVN because it’s our assumption (prior). More
formally speaking, the prior distribution of these infinite functions is MVN. The
prior distribution represents the expected outputs of f over inputs x without ob-
serving any data. When we start to have observations, instead of infinite number
of functions, we only keep functions that fit the observed data points. Now we
have the posterior, which is the prior updated with observed data. When we have
new observations, we will use the current posterior as prior, and new observed
data points to obtain a new posterior.
Here is a definition of Gaussian processes: A Gaussian processes model is a
probability distribution over possible functions that fit a set of points. Because
we have the probability distribution over all possible functions, we can calculate
the means as the function, and the variances to indicate how confident the predic-
tions are. The key points are summarized as: 1) the function (posteriors) updates
with new observations, 2) a Gaussian process model is a probability distribution
over possible functions, and any finite sample of functions are jointly Gaussian dis-
tributed, 3) the mean function calculated by the posterior distribution of possible
functions is the function used for regression predictions.
It is time to describe the standard Gaussian processes model. All the parameter
definitions follow the classic textbook [11]. Besides the covered basic concepts,
the Appendix A.1 and A.2 of [11] are also recommended to read. The regression
function modeled by a multivariate Gaussian is given as

P(f | X) = N (f | µ, K) ,

where X = [ x1 , . . . , xn ], f = [ f (x1 ), . . . , f (xn )], µ = [m(x1 ), . . . , m(xn )] and Kij =

k (xi , x j ). X are the observed data points, m represents the mean function, and k
represents a positive definite kernel function. With no observation, the mean func-
tion is default to be m(X) = 0 given that the data is often normalized to a zero
mean. The Gaussian processes model is a distribution over functions whose shape
An Intuitive Tutorial to Gaussian Processes Regression 10

(smoothness) is defined by K. If points xi and x j are considered to be similar by

the kernel, function outputs of the two points, f (xi ) and f (x j ), are expected to be
similar. The process of conducting regressions by Gaussian processes model is il-
lustrated by Fig. 9: given the observed data (red points) and a mean function f
(blue line) estimated by these observed data points, we make predictions at new
points X∗ as f(X∗ ).

Figure 9: A illustrative process of conducting regressions by Gaussian processes.

The red points are observed data, blue line represents the mean function estimated
by the observed data points, and predictions will be made at new blue points.

The joint distribution of f and f∗ is expressed as

" # " # " #!
f m (X) K K∗
∼N , ,
f∗ m (X∗ ) KT
∗ K∗∗
 
where K = K (X, X), K∗ = K (X, X∗ ) and K∗∗ = K (X∗ , X∗ ). And m(X), m(X∗ ) =
0.
This is the joint probability distribution equation P(f, f∗ | X, X∗ ) over f and f∗ ,
but regressions need the conditional distribution P(f∗ | f, X, X∗ ) over f∗ only. The
derivation from the joint distribution P(f, f∗ | X, X∗ ) to the conditional P(f∗ | f, X, X∗ )
uses the theorem in Appendix A. The result is
 
T −1
f∗ | f, X, X∗ ∼ N KT∗ K f, K ∗∗ − K ∗ K K ∗ .
In more realistic situations, we don’t have access to true function values but noisy
versions thereof y = f ( x ) + e. Assuming there is an additive independent and
identically distributed (i.i.d.) Gaussian noise with variance σn2 , the prior on the
noisy observations becomes cov(y) = K + σn2 I. The joint distribution of the ob-
served values and the function values at new testing points becomes
! " #!
y K + σn2 I K∗
∼ N 0, .
f∗ KT∗ K∗∗
An Intuitive Tutorial to Gaussian Processes Regression 11

By deriving the conditional distribution, we get the predictive equations for Gaus-
sian processes regression as


f̄∗ | X, y, X∗ ∼ N f̄∗ , cov(f∗ ) ,

where
∆
f̄∗ = E[f̄∗ | X, y, X∗ ]
= KT∗ [K + σn2 I ]−1 y ,
2 −1
cov(f∗ ) = K∗∗ − KT
∗ [K + σn I ] K∗ .

In the variance function cov(f∗ ), it can be noted that the variance does not depend
on the observed output y but only on the inputs X and X∗ . This is a property of the
Gaussian distribution [11].

4 Illustrative example
An implementation of the standard GPR is shown in this section. The example
follows the algorithm in [11] as follows.

L = cholesky(K + σn2 I )
α = L > \ ( L \ y)
f¯∗ = k>
∗α
v = L \ k∗
V[ f ∗ ] = k(x∗ , x∗ ) − v> v.
1 1 n
log p(y | X ) = − y> (K + σn2 I )−1 y − log det(K + σn2 I ) − log 2π
2 2 2

The inputs are X (inputs), y(targets), k(covariance function), σn2 (noise level), and
x∗ (test input). The outputs are f¯∗ (mean), V[ f ∗ ] (variance), and log p(y | X ) (log
marginal likelihood).
An example result was plotted shown in Fig. 10. We conducted regression
within the [-5, 5] region of interest. The observed data points (training dataset)
were generated from a uniform distribution between -5 and 5. Any point within
the given interval [-5, 5] was equal likely to be drawn by a uniform. The functions
were evaluated at evenly spaced points between -5 and 5. The regression func-
tion is composed of mean values estimated by a GPR model. Twenty samples of
posterior mean functions were also plotted within 3µ variances.
An Intuitive Tutorial to Gaussian Processes Regression 12

4 True function
Observed data points
Estimated mean function
3
2
1
0
1
2
3
4 2 0 2 4

Figure 10: A illustrative example of the standard GPR. The observed data points
generated by the blue dotted line (the true function) were plotted as black crosses.
Given the observed/training data points, infinite possible posterior functions were
obtained. We plotted 20 samples of these infinite functions with sorted colors.
The mean function was obtained by the probability distribution of these functions
and plotted as a red solid line. The blue shaded area around the mean function
indicates the 3µ prediction variances.

4.1 Hyperparameters Optimization

We explained the basics of GPR and implemented a simple example. While in
practical, GPR models are more complex than it. Kernel functions play significant
roles in GPR. The choice of kernel functions determines almost all the generaliza-
tion properties of a GP model [4]. There are many covariance functions to choose
or make your own for a Gaussian process depending on your specific problem.
These criteria include if the model is smooth, if it is sparse, if it can change drasti-
cally, and if it needs to be differentiable [5]. More depth information on choosing a
kernel/covariance function for a Gaussian process can be found in [5]. In kernels,
the hyperparameters optimization is essential. Here we will use the most widely
used kernel, RBF, as an example to explain the hyperparameters optimization. The
general RBF function is given by
An Intuitive Tutorial to Gaussian Processes Regression 13

 1 
k (xi , x j ) = σ2f exp − (xi − x j )T (xi − x j ) ,
2l
where σ f and l are hyperparameters [5]. The vertical scale σ f describes how much
vertically the function can span. The horizontal scale l indicates how quickly the
correlation relationship between two points drops as their distance increases. The
effect of l was shown in Fig. 11. A higher l provided a smoother function and a
smaller l gave a wigglier function.

6
12.5
4 10.0
2 7.5
5.0
0
2.5
2 0.0
4 2.5
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
(a) l = small (b) l = medium

2
0.0 0.2 0.4 0.6 0.8 1.0
(c) l = large

Figure 11: The function smoothness affected by the horizontal scale l of hyperpa-
rameters.

The optimized hyperparameters Θ∗ are determined by the log marginal likeli-

hood [11] as
Θ∗ = arg max log p(y | X, Θ) .
Θ
Thus, considering hyperparameters, a more general equation of predictions at the
new testing points is [2]
f̄∗ | X, y, X∗ , Θ ∼ N f̄∗ , cov(f∗ ) .

Note that after learning/tuning the hyperparameters, the predictive variance cov(f∗ )
depends on not only the inputs X and X∗ but also the outputs y [1]. With the op-
timized hyperparameters σ f = 0.0067 and l = 0.0967, the regression result of the
An Intuitive Tutorial to Gaussian Processes Regression 14

observed data points in Fig. 11 was shown in Fig. 12. Here, the hyperparameters
optimization was conducted by the GPy package, which will be introduced in the
next section.

15 Observed data points

Estimated mean function
10
5
0
5
10
0.0 0.2 0.4 0.6 0.8 1.0
Figure 12: Regression result with the optimized hyperparameters σ f and l.

4.2 Gaussian processes packages

This section will review three software packages written in Python for Gaussian
processes implementations. One of the most well-known GP packages is GPy [3].
GPy is a maturely developed package with well-documented explanations since
2012. GPy uses NumPy to perform all its computations. For tasks that do not re-
quire heavy computations, GPy is sufficient and stable to use. However, GPR is
computationally expensive in high dimensional spaces (features more than a few
dozens) because it uses the whole samples/features to do predictions. The more
observations, the more computations are needed for predictions. A package that
includes state-of-the-art algorithms is preferred for efficient implementations of
complex GPR tasks. For GPR tasks that are computationally expensive, GPU ac-
celeration is especially useful. GPflow [3] origins from GPy with much of a similar
interface. GPflow leverages TensorFlow as its computational backend. GPyTorch
[7] is a recently developed package that provides GPU acceleration through Py-
Torch. It also contains many state-of-the-art Gaussian processes algorithms. Simi-
lar to GPflow, GPyTorch provides automatic gradients, thus complex models such
as embedding deep neural networks in GP models can be easier developed.
An Intuitive Tutorial to Gaussian Processes Regression 15

5 Summary and Discussion

A Gaussian process is a probability distribution over possible functions that fit a
set of points [11]. A Gaussian processes regression model provides uncertainty
estimations together with prediction values. The prior knowledge of the form of
functions can be integrated by using kernel functions.
The GPR model in this tutorial is the standard or vanilla Gaussian processes
[6]. There are two main constraints with it: 1) The overall computation complexity
is O( N 3 ), where N is the dimension of the covariance matrix K. 2) The memory
consumption is quadratic. Because of the computation complexity and memory
consumption, the standard Gaussian processes model gets struck quickly. For re-
gression tasks with a big dataset, sparse GP is used to reduce computational com-
plexity [9].

References
[1] Zexun Chen and Bo Wang. How priors of initial hyperparameters affect Gaus-
sian process regression models. Neurocomputing, 275:1702–1710, 2018.

[2] Zhenwen Dai. Computationally efficient GPs. In Gaussian Process Summer

School (GPSS), 2019.

[3] Alexander G De G. Matthews, Mark Van Der Wilk, Tom Nickson, Keisuke Fu-
jii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James
Hensman. GPflow: A Gaussian process library using TensorFlow. The Journal
of Machine Learning Research, 18(1):1299–1304, 2017.

[4] David Duvenaud. The Kernel Cookbook. Available at https://www.cs.

toronto.edu/~duvenaud/cookbook.

[5] David Duvenaud. Automatic model construction with Gaussian processes. PhD
thesis, University of Cambridge, 2014.

[6] Roger Frigola, Fredrik Lindsten, Thomas B Schön, and Carl Edward Ras-
mussen. Bayesian Inference and Learning in Gaussian Process State-Space
Models with Particle MCMC. In Advances in Neural Information Processing Sys-
tems, pages 3156–3164, 2013.
An Intuitive Tutorial to Gaussian Processes Regression 16

[7] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and An-
drew Gordon Wilson. GPyTorch: Blackbox Matrix-Matrix Gaussian Process
Inference with GPU Acceleration. In Advances in Neural Information Processing
Systems, 2018.

[8] Zoubin Ghahramani. A Tutorial on Gaussian Processes (or why I don’t use
SVMs). In Machine Learning Summer School (MLSS), 2011.

[9] Haitao Liu, Yew-Soon Ong, Xiaobo Shen, and Jianfei Cai. When Gaussian
process meets big data: A review of scalable GPs. IEEE Transactions on Neural
Networks and Learning Systems, 2020.

[10] Kevin P Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press,
2012.

[11] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes

for Machine Learning. The MIT Press, 2006.

A Appendix
The Marginal and Conditional Distributions of MVN theorem: suppose X = ( x1 , x2 )
is a joint Gaussian with parameters
" # " # " #
µ1 Σ11 Σ12 Λ 11 Λ 12
µ= , Σ= , Λ = Σ −1 = ,
µ2 Σ21 Σ22 Λ21 Λ22

then the marginals are given by

p( x1 ) = N ( x1 |µ1 , Σ11 ) ,
p( x2 ) = N ( x2 |µ2 , Σ22 ) ,

and the posterior conditional is given by

p ( x 1 | x 2 ) = N ( x 1 | µ 1|2 , Σ 1|2 )
−1
µ1|2 = µ1 + Σ12 Σ22 ( x2 − µ2 )
−1
= µ1 − Λ11 Λ12 ( x2 − µ2 )
= Σ1|2 (Λ11 µ1 − Λ12 ( x2 − µ2 ))
−1 −1
Σ1|2 = Σ11 − Σ12 Σ22 Σ21 = Λ11