10-725/36-725: Convex Optimization Fall 2019

Lecture 2: August 28
Lecturer: Ryan Tibshirani Scribes: Shuyang Yang, Xingyu Liu, Bo Lei

Note: LaTeX template courtesy of UC Berkeley EECS dept.

Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
They may be distributed outside this class only with the permission of the Instructor.

2.1 Convex Optimization Problem

A convex optimization problem is of the form:

min f (x)
x∈D

subject to
gi (x) ≤ 0, i = 1, ..., m
hj (x) = 0, j = 1, ..., r
where f and gi are all convex , and hj are affine. Any local minimizer of a convex optimization problem is
a global minimizer.

2.2 Convex Sets

2.2.1 Definitions

Definition 2.1 Convex set: a set C ⊆ Rn is a convex set if for any x, y ∈ C, we have

tx + (1 − t)y ∈ C, f or all 0 ≤ t ≤ 1

Definition 2.2 Convex combination of x1 , ..., xk ∈ Rn : any linear combination

Xk
θ1 x1 + ... + θk xk , with θi ≥ 0, and θi = 1
i=1

Definition 2.3 Convex hull of set C: all convex combinations of elements in C. The convex hull is always
convex.

Definition 2.4 Cone: a set C ⊆ Rn is a cone if for any x ∈ C, we have tx ∈ C for all t ≥ 0

Definition 2.5 Convex cone: a cone that is also convex, i.e.,

x1 , x2 ∈ C =⇒ t1 x1 + t2 x2 ∈ C for all t1 , t2 ≥ 0

2-1
2-2 Lecture 2: August 28

Definition 2.6 Conic combination of x1 , ..., xk ∈ R: any linear combination

θ1 x1 + ... + θk xk , with θi ≥ 0

Definition 2.7 Conic hull of set C: all conic combinations of elements in C.

2.2.2 Examples of convex sets

• Empty set, point, line.
• Norm ball: {x : kxk ≤ r}, for given norm k·k, radius r.
• Hyperplane: {x : aT x = b}, for given a, b.
• Halfspace: {x : aT x ≤ b}.
• Affine space: {x : Ax = b}, for given A, b.
• Polyhedron: {x : Ax ≤ b}, where ≤ is interpreted componentwise. The set {x : Ax ≤ b, Cx = d} is
also a polyhedron.
• Simplex: special case of polyhedra, given by conv{x0 , ..., xk }, where these points are affinely indepen-
dent. The canonical example is the probability simplex,

conv{e1 , ..., en } = {w : w ≥ 0, 1T w = 1}

2.2.3 Examples of convex cones

• Norm cone: {(x, t) : kxk ≤ t}, for given norm k·k. It is called second-order cone under the l2 norm
k·k2 .
• Normal cone: given any set C and point x ∈ C, the normal cone is

NC (x) = {g : g T x ≥ g T y, for all y ∈ C}

This is always a convex cone, regardless of C.

• Positive semidefinite cone:
Sn+ = {X ∈ Sn : X  0}
where X  0 means that X is positive semidefinite (Sn is the set of n × n symmetric matrices).

2.2.4 Key properties of convex sets

• Separating hyperplane theorem: two disjoint convex sets have a separating between hyperplane
them. Formally, if C, D are nonempty convex sets with C ∩ D = ∅, then there exists a, b such that

C ⊆ {x : aT x ≤ b}, D ⊆ {x : aT x ≥ b}

• Supporting hyperplane theorem: a boundary point of a convex set has a supporting hyperplane
passing through it. Formally, if C is a nonempty convex set, and x0 ∈ bd(C), then there exists a such
that
C ⊆ {x : aT x ≤ aT x0 }
Lecture 2: August 28 2-3

2.2.5 Operations preserving convexity

2.2.5.1 Operations

• Intersection: the intersection of convex sets is convex.

• Scaling and translation: if C is convex, then aC + b = {ax + b : x ∈ C} is convex for any a, b.
• Affine images and preimages: if f (x) = Ax + b and C is convex, then f (C) = {f (x) : x ∈ C} is convex,
and if D is convex, then f −1 (D) = {x : f (x) ∈ D} is convex. Compared to scaling and translation,
this operation also has rotation and dimension reduction.
• Perspective images and preimages: the perspective function is P : Rn ×R++ → Rn (where R++ denotes
positive reals),
P (x, z) = x/z
for z > 0. If C ⊆ dom(P ) is convex then so is P (C), and if D is convex then so is P −1 (D).
• Linear-fractional images and preimages: the perspective map composed with an affine function,
Ax + b
f (x) =
cT x + d
is called a linear-fractional function, defined on cT x + d > 0. If C ⊆ dom(f ) is convex then so is f (C),
and if D is convex then so is f −1 (D).

2.2.5.2 Example: linear matrix inequality solution set

Given A1 , ..., Ak , B ∈ Sn , a linear matrix inequality is of the form

x1 A1 + x2 A2 + ... + xk Ak  B
for a variable x ∈ Rk . Let’s prove the set C of points x that satisfy the above inequality is convex.

Approach 1: directly verify that x, y ∈ C ⇒ tx + (1 − t)y ∈ C. This follows by checking that, for any
v, !
Xk
T
v B− (txi + (1 − t)yi )Ai ) v ≥ 0
i=1
Pk
k n
Approach 2: let f : R → S , f (x) = B − i=1 xi Ai . Note that C = f −1 (Sn+ ), affine preimage of convex set.

2.2.5.3 Example: conditional probability set

Let U, V be random variables over {1, ..., n}, {1, ..., m}. Let C ⊆ Rnm be a set of joint distributions for U, V ,
i.e., each p ∈ C defines joint probabilities
pij = P(U = i, V = j)
nm
Let D ⊆ R contain corresponding conditional distributions, i.e., each q ∈ D defines
qij = P(U = i|V = j)
Assume C is convex. Let’s prove that D is convex. Write
 
pij
D = q ∈ Rnm : qij = Pn , for some p ∈ C = f (C)
k=1 pkj
where f is a linear-fractional function, hence D is convex.
2-4 Lecture 2: August 28

2.3 Convex Functions

2.3.1 Definitions

Definition 2.8 Convex function: f: Rn → R such that the domain of function f dom(f ) ⊆ Rn is convex.
f (tx + (1 − t)y) ≤ tf (x) + (1 − t)f (y), f or 0 ≤ t ≤ 1
And all x, y ∈ dom(f )

In other words, the function lies below the line segment joining f (x) and f (y)

Definition 2.9 Concave function: opposite inequality of the definition above, so that
f concave ⇔ −f convex

which is to say, f being concave implies -f being convex.

Important modifiers:

• Strictly Convex: f (tx + (1 − t)y) < tf (x) + (1 − t)f (y), f or x 6= y and 0 < t < 1.
In other words, f is convex and has greater curvature than a linear function.
• Strongly Convex: With parameter m > 0, f (− m 2
2 ||x||2 ) is convex.
In other words, f is at least as convex as a quadratic function.

Note: strongly convex implies strictly convex, which subsequently implies convex. In equation format:
strongly convex ⇒ strictly convex ⇒ convex

2.3.2 Examples of convex and concave functions

• Univariate functions
(1) Exponential function: eax is convex for any a over R
(2) Power function: xa is convex for a ≥ 1 or a ≤ 0 over R+ (nonnegative reals); xa is concave for
0 ≤ a ≤ 1 over R+
(3) Logarithmic function: log(x) is concave over R++
• Affine function: aT x + b is both convex and concave.
1 T
• Quadratic function: 2 x Qx + bT x + c is convex provided that Q ≥ 0 (positive semidefinite)
• Least squares loss: ||y − Ax||22 is always convex (since AT A is always positive semidefinite)
• ||x|| is convex for any norm, for example: lp norms
Xn
||x||p = ( xip )1/p f or p ≥ 1, ||x||∞ = max |xi |
i=1,...n
i=1

as well as operator (spectral) and trace (nuclear) norms

r
X
||X||op = σ1 (X), ||X||tr = σr (X)
i=1

where σ1 (X) ≥ ... ≥ σr (X) ≥ 0 are the singular values of the matrix X.
Lecture 2: August 28 2-5

• Indicator function: if C is convex, then its indicator function

(
0, x ∈ C
IC (x) =
∞, x ∈/C

is convex

• Support function: for any set C (convex or not), its support function

IC∗ (x) = max xT y

y∈C

is convex

• Max function: f (x) = max{x1 , ...xn } is convex.

2.3.3 Key properties of convex functions

• A function is convex if and only if its restriction to any line is convex

• Epigraph characterization: a function f is convex if and only if its epigraph

epi(f ) = (x, t) ∈ dom(f ) × R : f (x) ≤ t

is a convex set.

• Convex sublevel sets: if f is convex, then its sublevel sets

x ∈ dom(f ) : f (x) ≤ t

are convex, for all t ∈ R. The converse is not true.

• First-order characterization: if f is differentiable, then f is convex if and only if dom(f) is convex, and

f (y) ≥ f (x) + ∇f (x)T (y − x)

for all x, y ∈ dom(f ). Therefore for a differentiable convex function ∇f (x) = 0 ⇔ x minimizes f .

• Second-order characterization: if f is twice differentiable, then f is convex if and only if dom(f ) is

convex, and ∇2 f (x) ≥ 0 for all x ∈ dom(f ).

• Jensen’s inequality: if f is convex, and X is a random variable supported on dom(f ), then f (E[X]) ≤
E[f (x)].
Pk T
• Long-sum-exp function: g(x) = log( i=1 eai x+bi ) for fixed ai , bi . This is often called the soft max,
since it smoothly approximates maxi=1,...,k (aTi x + bi ).

2.3.4 Operations preserving convexity

• Nonnegative linear combination: f1 , ...fm convex implies a1 f1 + ... + am fm is also convex for any
a1 , ...am ≥ 0.

• Pointwise maximization: if fs is convex for any s ∈ S, then f (x) = maxs∈S is also convex.
Note: the set S is the number of functions fx , which can be infinite.
2-6 Lecture 2: August 28

• Partial minimization: if g(x, y) is convex in x, y, and C is convex, then f (x) = miny∈C g(x, y) is convex.

• Affine composition: if f is convex, then g(x) = f (Ax + b) is convex.

• General composition: suppose f = hg, where g : Rn → R, h : R → R, f : Rn → R. Then:
(1) f is convex if h is convex and nondecreasing, g is convex
(2) f is convex if h is convex and nonincreasing, g is concave
(3) f is concave if h is concave and nondecreasing, g is concave
(4) f is convex if h is convex and nonincreasing, g is convex
Note: To memorize this, think of the chain rule when n = 1:

f 00 (x) = h00 (g(x))g 0 (x)2 + h0 (g(x))g 00 (x)

• Vector composition: suppose that:

f (x) = h(g(x)) = h(g1 (x), ..., gk (x))

where g : Rn → Rk , h : Rk → R, f : Rn → R. Then:
(1) f is convex if h is convex and nondecreasing in each argument, g is convex
(2) f is convex if h is convex and nonincreasing in each argument, g is concave
(3) f is concave if h is concave and nondecreasing in each argument, g is concave
(4) f is concave if h is concave and nonincreasing in each argument, g is convex