Bayesian Optimization

CSC2541 - Topics in Machine Learning

Scalable and Flexible Models of Uncertainty
University of Toronto - Fall 2017
Overview

1. Bayesian Optimization of Machine Learning Algorithms

2. Gaussian Process Optimization in the Bandit Setting
3. Exploiting Structure for Bayesian Optimization
Bayesian Optimization
of Machine Learning Algorithms
J. Snoek, A. Krause, H. Larochelle, and R.P. Adams (2012)
Practical Bayesian Optimization of Machine Learning Algorithms

J. Snoek et al. (2015)

Scalable Bayesian Optimization Using Deep Neural Nets

Presentation by: Franco Lin, Tahmid Mehdi, Jason Li

Motivation

Performance of Machine Learning algorithms are usually dependent on the

choice of hyperparameters

Picking the optimal hyperparameter values are hard

- Ex. grid search, random search, etc.

- Instead could we use a model to select which hyperparameters will be
good next?
Bayes Opt. of Machine Learning Algorithms

- Bayesian Optimization uses all of the information from previous

evaluations and performs some computation to determine the next
point to try

- If our model takes days to train, it would be beneficial to have a well

structured way of selecting the next combination of hyperparameters
to try

- Bayesian Optimization is much better than a person finding a good

combination of hyperparameters
Bayesian Optimization

Intuition:

We want to find the peak of our true function (eg. accuracy as a function of
hyperparameters)

To find this peak, we will fit a Gaussian Process to our observed points and
pick our next best point where we believe the maximum will be.

This next point is determined by an acquisition function - that trades of

exploration and exploitation

Lecture by Nando de Freitas and a Tutorial paper by Brochu et al.

 Bayesian Optimization Tutorial

Brochu et al., 2010, A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement
Learning
 Bayesian Optimization Tutorial

Find the next best point xn that maximizes acquisition function

Brochu et al., 2010, A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement
Learning
 Bayesian Optimization Tutorial

Evaluate ƒ at the new observation xn and update posterior

Update acquisition function from new posterior and find the next best point
Brochu et al., 2010, A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement
Learning
 Acquisition Function Intuition

- We will use the acquisition function

Probability of Improvement (PI) as
an example.

- We want to find the point with the

largest area above our best value

- This corresponds to the maximum

of our acquisition function

Brochu et al., 2010, A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement
Learning
Acquisition Functions
- Guides the optimization by determining which point to observe next
and is easier to optimize to find the next sample point

Probability of Improvement (PI)

Expected Improvement (EI)

GP-Upper/Lower Confidence Bound (GP-UCB/LCB)

The Prior
- Power of Gaussian Process depends on covariance function
- For optimization, we don’t want kernels that produce unrealistically
smooth sample functions
- Automatic Relevance Determination (ARD) Matern 5/2 kernel is a
good choice
Kernel Hyperparameters
Marginalize over hyperparameters and
compute integrated acquisition function

Approximate integral with Monte Carlo

methods
Considerations for Bayes Opt

- Evaluating f may be time-consuming

- Modern optimization methods should take advantage of

multi-core/parallel programming
Expected Improvement per Second

- Evaluating f will take longer in some regions of the parameter space

- We want to pick points that are likely to be good and evaluated quickly
- Let c(x) be the duration time to evaluate f(x)
- Use GP to model ln[c(x)]
- we can compute predicted expected inverse duration which allows us
to obtain the EI per Second as a function of x
 Parallelizing Bayes Opt
- Can we determine which x to evaluate next, while other points are being
evaluated?
- Idea: Utilize tractable properties of GP to get Monte Carlo estimates of
acquisition function under different results from pending function evaluations

Consider the case where N evaluations have completed, with data {xn,yn}n=1N, and J
evaluations are pending {xj}j=1J
Parallelization Example
- We’ve evaluated 3 observations and 2
are pending {x1,x2}
- Fit a model for each possible realization
of {f(x1), f(x2)}
- Calculate acquisition for each model
- Integrate all acquisitions over x
Results
● Branin-Hoo
● Logistic Regression MNIST
● Online LDA
● M3E
● CNN CIFAR-10
Logistic Regression - MNIST
CIFAR-10
● 3-layer conv-net
● Optimized over
○ Number of epochs
○ Learning rate
○ L2-norm constants
● Achieved state of the art
○ 9.5% test error
GP Bayesian Optimization - Pros and Cons
● Advantages
○ Computes the mean and variance

● Disadvantages
○ Function evaluation is cubic on the number of inputs
Scalable Bayesian Optimization Using Deep Neural Networks

● Replace a Gaussian Process with a Bayesian Neural Network

● Use a deterministic neural network with Bayesian linear regression on
the last hidden layer
● More accurately, use Bayesian linear regression with basis functions
○ DNN: Rk -> Rd
○ Bayesian linear regression: Rd -> R
○ k is the dimensionality of the input, and d is the number of hidden
units in the last layer
Bayesian Linear Regression
● Still requires an inversion
● Linear in the number of observations
● Cubic in the basis function dimension or number of hidden units, D
Results
Gaussian Process Optimization
in the Bandit Setting
N. Srinivas, A. Krause, S. Kakade, and M. Seeger (2010)
Gaussian process optimization in the bandit setting: No regret and experimental design

Presentation by: Shadi Zabad, Wei Zhen Teoh, Shuja Khalid

The Bandits are Back!

- We just learned about some exciting new

techniques for optimizing black box functions.
Can we apply them to the classic multi-armed
bandit problem?
- In this case, we’d like to optimize the unknown
reward function.

Credit: D. Tolpin at ECAI 2012

Cost-bounded Optimization

- In the bandit setting, the optimization procedure is cost-sensitive:

There’s a cost incurred each time we evaluate the function.
- The cost is proportional to how far the point is from the point of
maximum reward.
- Therefore, we have to optimize the reward function while
minimizing the cost incurred along the way.
An Infinite Number of Arms

- The multi-armed bandit algorithms and analyses we’ve seen so far

assumed a discrete decision space (e.g. a decision space where
we have K slot machines).
- However, in Gaussian Process optimization, we’d like to consider
continuous decision spaces.
- And in this domain, some of the theoretical analyses that we
derived for discrete decision spaces can’t be extended in a
straightforward manner. Credit: @Astrid, CrossValidated
Multi-armed Bandit Problem: Recap

- The basic setting: We have a decision space that’s

associated with an unknown reward function.
- Discrete examples: Slot machines at a casino, drug trials.
- Continuous examples: Digging for oil or minerals, robot
Credit: Gatis Gribusts
motion planning.
- In this setting, a “policy” is a procedure for exploring
the decision space. An optimal policy is defined as a
procedure which minimizes a cost measure. The most
common cost measure is the “regret”. Credit: Intelligent Motion Lab (Duke U)
A Measure of Regret

- In general terms, regret is defined as “the loss in

reward due to not knowing” the maximum points
beforehand.
- We can formalize this notion with 2 concepts:
- Instantaneous regret (rt): the loss in reward at step t:

rt = f(Dmax) - f(Dt)
- Cumulative regret (RT): the total loss in reward after
T steps:

RT = ∑rt
Minimizing Regret: A Tradeoff

- As we have seen before, we can define policies that

balance exploration and exploitation. Some of the
policies we’ve looked at are:
- Epsilon-greedy
- Thompson sampling
- Upper Confidence Bound (UCB)
- Some of these policies perform better than others
in minimizing the average regret over time.

Average Regret = RT / T
Credit: Russo et al., 2017
Asymptotic Regret

- We can also look at the cumulative or average

regret measure as the number of iterations goes to
infinity.
- An algorithm is said to be no-regret if its asymptotic
cumulative regret rate is sublinear with respect to T sqrt(T) and log(T) are examples
of sublinear regret rates w.r.t. T.
(i.e. the number of iterations)
Why is Asymptotic Regret Important?

- In real world applications, we know neither

instantaneous nor average regret. So, why
are we concerned with characterizing their
asymptotic behavior?
- Answer: Bounds on the average regret tell
us about the convergence rate (i.e. how fast
we approach the maximum point) of the
optimization algorithm.

Credit: N. de Freitas et al., 2012

Regret Bounds in Discrete Decision Spaces

- In the previous lecture, we discussed “In the traditional K-arm bandit

asymptotic regret in discrete decision literature, the regret is often
characterized for a particular
spaces where we have K slot machines or
problem in terms of T, K, and
drug trials. problem dependent constants. In
- We also looked at theorems by Auer et al. the K-arm bandit results of Auer et
that derive an upper bound on the regret al. [2002], this problem dependent
constant is the ‘gap’ between the loss
rate for the UCB algorithm in discrete of the best arm and the second best
settings. arm.”

Dani et al. 2008

Regret Bounds in Continuous Decision Spaces

- Dani et al.** extended Auer et al.’s theoretical results to

continuous decision spaces and proved upper and lower regret
bounds for the UCB algorithm.
- However, their method places restrictions on the types of reward
functions considered, primarily: The functions are defined over
finite-dimensional linear spaces.

** Dani, V., Hayes, T. P., and Kakade, S. M. Stochastic linear optimization under bandit feedback. In COLT, 2008.
Infinite-dimensional Functions

- Srinivas et al. propose to relax some of the restrictions

of Dani et al.’s analysis and extend the results to
random, infinite-dimensional functions.
- Earlier in the semester, we learned about a method for
generating such classes of functions: Gaussian
Processes.
- Idea: Assuming the target reward function is sampled
from a Gaussian Process, try to optimize it using
GP-UCB.
- How to derive regret bounds for those classes of Credit: Duvenaud, The Kernel Cookbook

functions?
Using Information Gain
To Derive Regret Bounds
Information Gain

Credit: Srinivas et al. 2010

Recall Mackay (1992) paper: Information gain can be quantified as change in entropy

In this context:

Information gain = entropy in prior - entropy in posterior after yA sampled

Note:
information gain depends on
= H(f) - H( f | yA) kernel of GP prior and input
space
= I(f; yA), mutual information between f and observed yA

= I(yA; f) = H(yA) - H( yA|f) = (log| 2I + KA|)/2 - (log| 2I |)/2 = (log|I + -2

KA|)/2
 If our goal is just exploration ...

Greedy Experimental Design Algorithm:

Sequentially, find

However the worse point we

select, the more penalty we get
Credit: Srinivas et al. 2010
GP - UCB to the rescue

Explore Exploit

Credit: Srinivas et al. 2010

Maximum information Gain

Definition:

Maximum information gain after T data points sampled,

This term will be used to quantify the regret bound for the algorithm
 Regret Bounds - Finite Domain
Assuming some strictly sublinear T...
(we will verify later that this is
achievable by choice of kernels),
We can find some sublinear function
f(T) bounding above
Theorem 1:

Assumptions: f(T)
- Finite D
- f sample of a GP with mean 0,
- k(x, x’) of GP s.t. k(x,x) (variance) not greater than 1
P(RT curve lies below)
Then, by running GP-UCB for f with is at least 1-

We obtain:
T
Regret Bounds II - General Compact+Convex Space

Theorem 2:

Assumptions:
- D compact and convex in [0,r]d,
- f sample of a GP with mean 0,
- k(x, x’) of GP s.t. k(x,x) (variance) not greater than 1
- k(x,x’) s.t. f fulfills smoothness condition -- discussed next

Then, by running GP-UCB for f with

We obtain:
 Regret Bounds II Continued

Theorem 2 requires f to fulfill:

This holds for stationary kernels k(x,x’) = k(x-x’) which are 4-times differentiable:

Squared Exponential Kernel Matern Kernels with v>2

Credit: Srinivas et al. 2010

Bounding Information Gain

Submodularity
-- F is submodular function

This holds if A constructed by

Greedy Experiment Design Rule
T

⇒
T

Credit: Krause, https://las.inf.ethz.ch/sfo/

Bounding Information Gain Continued

We can bound the term by considering the worst allocation of the T samples under
some relaxed greedy procedure (see appendix section C).

In finite space D, this eventually gives us a bound in terms of the eigenvalues of the
covariance matrix for all |D| points:

The faster the spectrum decays, the slower the growth of the bound
Bounding Information Gain Continued

Credit: Srinivas et al. 2010

Bounding Information Gain Continued

Theorem 5: Assume general compact and convex set D in Rd, kernel k(x,x’)≤1:

1. d- dimensional bayesian linear regression:

2. Squared exponential kernel:
3. Matern kernel (v>1) :

Now recall the bound obtained for GP-UCB in theorem 2: With = +

T

Combining the two theorems we obtain the following (1-δ) upper confidence bound
for the total regret, RT (up to polylog factors):

Credit: Srinivas et al. 2010

Results and Discussion
Experimental Setup

● Synthetic and real sensor network data (traffic and temperature)

used to illustrate the differences
● Gaussian Processes - Upper Confidence Bound (GP-UCB) is
compared with various heuristics:
○ Expected Improvement (EI)
○ Most Probable Improvement (MPI)
○ Naive Methods (only mean or only variance)
Experimental Setup

● Synthetic Data Breakdown:

- Functions were sampled from a GP with a squared exponential kernel
- Sample parameters: σ2= 0.025, T = 1000 iterations, = 0.1

Figure: Functions drawn from a GP with squared exponential kernel (lengthscale=0.2) Credit: Srinivas et al. 2010
Experimental Setup

● Real Sensor Temperature Data Breakdown:

- 46 sensors deployed at Intel Research Center (Berkeley) that acquired
data over 5 days at 1 minute intervals
- ⅔ of the data was used to create the kernel matrix
- Sample parameters: σ2= 0.5, T = 46 iterations, = 0.1
- Remaining ⅓ of data was used for testing
- Results averaged over 2000 runs
Experimental Setup

● Real Sensor Traffic Data Breakdown:

- Data from 357 traffic sensors along highway I-880 in South California
captured data for one month from 6am - 11am
- Purpose was to find the minimum speed to identify the most congested
part of the highway during rush-hour
- ⅔ of the data was used to create the kernel matrix
- Sample parameters: σ2= 4.78, T = 357 iterations, = 0.1
- Remaining ⅓ of data was used for testing
- Results averaged over 900 runs
Results
Credit: Srinivas et al. 2010

- For temperature data (b), GP-UCB and EI perform best

- For synthetic data, GP-UCB and EI perform best and MPI is a close comparable
- GP-UCB performs at least on par with existing approaches which do not include regret bounds
Conclusion
Regret bounds in
Introducing Minimizing
continuous Information Gain
Regret Regret
decision spaces

Proofs/Math. Experimental
GP-UCB Results
Analysis Design

- GP-UCB sample rule is discussed in detail and an algorithm is presented

Credit: Srinivas et al. 2010

Conclusion

- The concepts of Information Gain and Regret Bounds are analyzed and their relations represented in
the following theorems:
● Regret Bounds for Finite Domain
● Regret Bounds for General Compact + Convex Space
● Bounding Information Gain
- Synthetic and Real experimental data used to test the algorithm
- GP-UCB is found to perform at least on par with existing approaches which do not include regret
bounds
- Their results are encouraging as they illustrate exploration and exploitation trade-offs for complex
functions
- The paper uses tools (concept of regret and information gain) to come up with a convergence rate for
the GP-UCB algorithm
Exploiting Structure for
Bayesian Optimization
K. Swersky, J. Snoek, R.P. Adams (2014)
Freeze-Thaw Bayesian Optimization

K. Swersky, J. Snoek, R.P. Adams (2013)

Multi-Task Bayesian Optimization

Presentation by: Shu Jian (Eddie) Du, Romina Abachi, William Saunders
Freeze-Thaw Bayesian Optimization
K. Swersky, J. Snoek, R.P. Adams (2014)

Presentation by: Shu Jian (Eddie) Du, Romina Abachi

Intuition

- Human experts tend to stop model training halfway if the loss curve
looks bad.
- Like Snoek 2012 alluded to, we’d like to leverage partial information
(before a model finishes training) to determine what points to evaluate
next.
Big Idea

- To use partial information, we model training loss curves with a GP.

- Let’s assume loss curves look Exponential; roughly .
- Derive a GP kernel to model this between 2 time steps.
- Input x: A set of hyperparameters
- Output y: The model’s loss at a particular time
Exponential Decay Kernel
Exponential Decay Kernel
Demo

Demo:
https://github.com/esdu/misc/raw/master/csc2541/demo1.pdf

Code:
https://github.com/esdu/misc/blob/master/csc2541/csc2541_ftbo_pres_demo.ipynb
Are we done?

- We could model all N training curves over all T timesteps jointly using a
single GP using the Exp Decay Kernel.
- However, since GP takes cubic time to fit, it would run in
time. (We have N*T data points) this is way too slow!
- Paper proposes a generative model to speed this up.
A more efficient way

- Use a global GP to model the asymptotes of each training curve.

- Each training curve’s GP sample their prior mean function (the
best-guess asymptote) from the global GP
- This assumes each training curve
is drawn from a separate GP

Global GP
Joint distribution
･ ･ ･ ･
N ･ N ･ ･ ･
･ ･ ･ ･
1 N

1 ･ ･
1 ･ ･
1 ･ ･
N*T 1 N*T ･ ･
(At most N*T) 1 (At most N*T) ･ ･ ･
1 ･ ･ ･
1 ･ ･ ･
N N*T (At most N*T)
Marginal likelihood
Posterior distribution
Posterior predictive distribution
Aside: To Derive These...

- Authors repeated used:

- Basic Multivariate Gaussian identities
- Woodbury Matrix Identity (See Wikipedia)
What does it look like?
Demo

Demo:
https://github.com/esdu/misc/raw/master/csc2541/demo2.pdf

Code:
https://github.com/esdu/misc/blob/master/csc2541/csc2541_ftbo_pres_demo.ipynb
Which acquisition function to use?

Expected Improvement:

xbest -- input corresponding to minimum output observed so far

(x) and v(x) -- posterior mean and variance of the probabilistic model
evaluated at x

EI used to determine which hyperparameters to try next (baskets)

Acquisition Function: Entropy Search

Idea: How much information does evaluating a new point give us about the
location of the minimum?

- While EI focuses on finding the minimum value of the function, ES tries

to reduce the uncertainty over the location of the minimum.
- Unlike Expected Improvement, takes into account the possibility that
some point other than the best known will be the best.
Acquisition Function: Entropy Search

Given C points X ⊂ X, probability of x ⊂ having the minimum value is:

Probability of function values at all 1 if x is minimum, 0

candidate points otherwise

Goal: reduce uncertainty over this if we observe y at x.

Acquisition Function: Entropy Search

Pmin -- current estimated distribution over the minimum

Pymin is the updated distribution over the location of the minimum with the added
observation y.
In practice, no simple form, so we use Monte Carlo sampling to estimate Pmin
Which acquisition function to use?

Why not choose the model to run based on EI?

-EI looks at value of function

→ would need more trials to find minimum
-ES maximizes information gain from each trial
→ can make better decisions with fewer trials
Algorithm
Experiments
-Logistic Regression: trained using SGD on MNIST. Hyperparameters: norm
constraint on weights, l_2 regularization penalty, minibatch size, dropout
regularization, learning rate

-Online Latent Dirichlet Allocation (LDA): Trained on 250,000 Wikipedia

docs. Hyperparams: number of topics, 2x Dirichlet distribution prior base
measures, learning rate, decay.

-Probabilistic Matrix Factorization (PMF): Trained on 100,000 MovieLens

ratings. Hyperparameters: rank, learning rate, l_2 regularization penalty
Results
Results
Conclusion & Future Work

- Exploit partial information as training is happening.

- Stop, resume, start new runs dynamically
- Can be extended to other problems where partial observations
reduce uncertainty.
- Relies on the key assumption that training curves follow exponential
decay.
- It would be interesting to use more flexible priors for other
problems
Multi-Task Bayesian Optimization
K. Swersky, J. Snoek, R.P. Adams (2013)

Presentation by: William Saunders

Goal

In Bayesian Optimization, it would be useful to be able to re-use

information from related tasks to reduce sample complexity

● Have data from running bayesian optimization on other similar

problems
● Use a computationally cheaper task to find information about a more
expensive task (ie. small subset of the training data)
● Optimize average performance of a set of related tasks
Multi-Task Gaussian Process Kernel

KK

Kx is a kernel indicating the covariance between inputs

Kt is a matrix indicating the covariance between tasks

Kt is marginalized over using a Monte-Carlo sampling method (slice
sampling), as are other kernels parameter (ie. length scale)
Kt is parameterized by its cholesky decomposition RT*R, where R is upper
diagonal with positive diagonal elements

is the Kronecker Product

Multi-Task Gaussian Process

Blue = target task, Red and Green are related tasks

Multi-Task Gaussian Process

Blue = target task, Red and Green are related tasks

Acquisition Function: Expected Improvement

Choose the point that, in expectation, will have the greatest improvement
over the best known point

Assumes that after querying, either the best known point or the queried
point will be the maximum
Acquisition Function: Entropy Search

● Select set of candidate points based on Expected Improvment

● f is assignment of values to all candidate points
● Evaluate using monte-carlo sampling
○ Pmin = the current estimated distribution over the minimum
○ Pymin = the new distribution over the minimum, given an
observation
○ Both these distributions can be approximated by repeatedly
sampling f and determining the minimum of the sample
○ p(y|f), p(f|x) calculated from gaussian process
Acquisition Function: Information Gain/Cost

Observing a point on a related task can never reveal more information than
sampling the same point on the target task

But, it can be better when information per unit cost is taken into account
Acquisition Function: Taking Cost Into Account

Blue = target task, expensive; Green = related task, cheap

Results - Information from Related Task
Results - Faster Task

Red = Multi Task

Blue = Single Task

Conclusion - Multi-Task Bayesian Optimization

● Information from other tasks can be used to speed up bayesian

optimization
● Entropy search can help to find points which are useful for providing
information about where the minimum is, but are not themselves the
minimum