DAY 5

 Warming up

Theorems:

Question:

Understanding Question:
𝑥1 𝑎1
𝑥2 𝑎2
𝑥 ∈ ℝ𝑛 → 𝑥 = ( ⋮ ) and the constant vector 𝑎 ∈ ℝ𝑛 → 𝑎=( ⋮ )
𝑥𝑛 𝑎𝑛

𝑆: = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 = 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ }

Show that 𝑆 is a convex set.

 𝑥1 𝑦1 𝑎1
𝑥2 𝑦2 𝑎2
Solution: Take arbitrary 𝑥, 𝑦 ∈ 𝑆 ⊂ ℝ𝑛 , where 𝑥 = ( ⋮ ) and 𝑦 = ( ⋮ ) . Then for 𝑎 = ( ⋮ ) ,
𝑥𝑛 𝑦𝑛 𝑎𝑛
𝑥1
𝑥 2
𝑎𝑇 𝑥 = 𝑐 ⇒ (𝑎1 , 𝑎2 , … , 𝑎𝑛 ) ( ⋮ ) = 𝑐 , that is, 𝑎1 𝑥1 + 𝑎2 𝑥2 + ⋯ + 𝑎𝑛 𝑥𝑛 = 𝑐
𝑥𝑛

And
𝑦1
𝑦2
𝑎𝑇 𝑦 = 𝑐 ⇒ (𝑎1 , 𝑎2 , … , 𝑎𝑛 ) ( ⋮ ) = 𝑐 , that is, 𝑎1 𝑦1 + 𝑎2 𝑦2 + ⋯ + 𝑎𝑛 𝑦𝑛 = 𝑐
𝑦𝑛

NTS: 𝜆𝑥 + (1 − 𝜆)𝑦 ∈ 𝑆 ⊂ ℝ𝑛 for 0 ≤ 𝜆 ≤ 1. That is, we have to show that 𝑎𝑇 [𝜆𝑥 + (1 − 𝜆)𝑦] = 𝑐.

Here,
𝑥1 𝑦1 𝜆𝑥1 (1 − 𝜆)𝑦1 𝜆𝑥1 + (1 − 𝜆)𝑦1
𝑥2 𝑦2 𝜆𝑥 (1 − 𝜆)𝑦2 𝜆𝑥 + (1 − 𝜆)𝑦2
𝜆𝑥 + (1 − 𝜆)𝑦 = 𝜆 ( ⋮ ) + (1 − 𝜆) ( ⋮ ) = ( 2 ) + ( ) =( 2 )
⋮ ⋮ ⋮
𝑥𝑛 𝑦𝑛 𝜆𝑥𝑛 (1 − 𝜆)𝑦𝑛 𝜆𝑥𝑛 + (1 − 𝜆)𝑦𝑛

Now,

𝑎𝑇 [𝜆𝑥 + (1 − 𝜆)𝑦]
𝜆𝑥1 + (1 − 𝜆)𝑦1 𝜆𝑥1 + (1 − 𝜆)𝑦1
( )
𝜆𝑥 + 1 − 𝜆 𝑦2 𝜆𝑥 + (1 − 𝜆)𝑦2
= 𝑎𝑇 ( 2 ) = (𝑎1 , 𝑎2 , … , 𝑎𝑛 ) ( 2 )
⋮ ⋮
𝜆𝑥𝑛 + (1 − 𝜆)𝑦𝑛 𝜆𝑥𝑛 + (1 − 𝜆)𝑦𝑛
= 𝑎1 [ 𝜆𝑥1 + (1 − 𝜆)𝑦1 ] + 𝑎2 [ 𝜆𝑥2 + (1 − 𝜆)𝑦2 ] + ⋯ + 𝑎𝑛 [ 𝜆𝑥𝑛 + (1 − 𝜆)𝑦𝑛 ]
= 𝜆[𝑎1 𝑥1 + 𝑎2 𝑥2 + ⋯ + 𝑎𝑛 𝑥𝑛 ] + (1 − 𝜆)[𝑎1 𝑦1 + 𝑎2 𝑦2 + ⋯ + 𝑎𝑛 𝑦𝑛 ]
= 𝜆𝑐 + (1 − 𝜆)𝑐

∴ 𝑎𝑇 [𝜆𝑥 + (1 − 𝜆)𝑦] = 𝑐.

Hence,
𝜆𝑥 + (1 − 𝜆)𝑦 ∈ 𝑆 ⊂ ℝ𝑛 for 0 ≤ 𝜆 ≤ 1.

DONE

Observation:

𝑆: = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 = 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ }

𝑆1 : = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 < 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ }

𝑆2 : = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 > 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ }

Questions: Show that 𝑆1 and 𝑆2 are convex, where

𝑆: = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 = 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ } [Proved]

𝑆1 : = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 < 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ }

𝑆2 : = {𝑥 ∈ ℝ𝑛 | 𝑎𝑇 𝑥 > 𝑐, for constant vector 𝑎 ∈ ℝ𝑛 & 𝑐 ∈ ℝ }

Alternative Proof:

Here 𝑥1 , 𝑥 2 ∈ 𝑆 ⊂ ℝ𝑛 . So, 𝑥1 = (𝑥11 , 𝑥21 , . . . , 𝑥𝑛1 ) and 𝑥 2 = (𝑥12 , 𝑥22 , . . . , 𝑥𝑛2 ).

Hence, 𝜆𝑥1 = (𝜆𝑥11 , 𝜆 𝑥21 , . . . , 𝜆𝑥𝑛1 ) and (1 − 𝜆)𝑥 2 = ((1 − 𝜆)𝑥12 , (1 − 𝜆)𝑥22 , . . . , (1 − 𝜆)𝑥𝑛2 ). That is,

𝜆𝑥1 + (1 − 𝜆)𝑥 2 = (𝜆𝑥11 , 𝜆 𝑥21 , . . . , 𝜆𝑥𝑛1 ) + ((1 − 𝜆)𝑥12 , (1 − 𝜆)𝑥22 , . . . , (1 − 𝜆)𝑥𝑛2 )

= (𝜆𝑥11 + (1 − 𝜆)𝑥12 , 𝜆 𝑥21 + (1 − 𝜆)𝑥22 , . . . , 𝜆𝑥𝑖1 + (1 − 𝜆)𝑥𝑖2 , . . . , 𝜆𝑥𝑛1 + (1 − 𝜆)𝑥𝑛2 )

Now, proof that 𝑎1 (𝜆𝑥11 + (1 − 𝜆)𝑥12 ) + 𝑎2 (𝜆 𝑥21 + (1 − 𝜆)𝑥22 )+ . . . +𝑎𝑛 (𝜆𝑥𝑛1 + (1 − 𝜆)𝑥𝑛2 ) = 𝑐 … (1)

Since 𝑥1 = (𝑥11 , 𝑥21 , . . . , 𝑥𝑛1 ) and 𝑥 2 = (𝑥12 , 𝑥22 , . . . , 𝑥𝑛2 ) ∈ 𝑆, then we get

𝑎1 𝑥11 + 𝑎2 𝑥21 + . . . +𝑎𝑛 𝑥𝑛1 = 𝑐 and 𝑎1 𝑥12 + 𝑎2 𝑥22 + . . . +𝑎𝑛 𝑥𝑛2 = 𝑐 … (2)

Now,

L.H.S= 𝑎1 (𝜆𝑥11 + (1 − 𝜆)𝑥12 ) + 𝑎2 (𝜆 𝑥21 + (1 − 𝜆)𝑥22 )+ . . . +𝑎𝑛 (𝜆𝑥𝑛1 + (1 − 𝜆)𝑥𝑛2 )

= 𝑎1 𝜆𝑥11 + 𝑎1 (1 − 𝜆)𝑥12 + 𝑎2 𝜆 𝑥21 + 𝑎2 (1 − 𝜆)𝑥22 + ⋯ + 𝑎𝑛 𝜆𝑥𝑛1 + 𝑎𝑛 (1 − 𝜆)𝑥𝑛2

= 𝜆(𝑎1 𝑥11 + 𝑎2 𝑥21 + . . . +𝑎𝑛 𝑥𝑛1 ) + (1 − 𝜆)𝑎1 𝑥12 + 𝑎2 𝑥22 + . . . +𝑎𝑛 𝑥𝑛2

= 𝜆𝑐 + (1 − 𝜆)𝑐 ; [apply the equation (2)]

=𝑐

Self-study:
Definition 1: Inner Product

In the real 𝑛 −space ℝ𝑛 , with the dot product is an inner product space, an example of a Euclidean
vector space. For 𝑥, 𝑦 ∈ ℝ𝑛 ,
𝑥1 𝑦1
𝑥2 𝑦2
〈𝑥, 𝑦〉 = 〈[ ⋮ ] , [ ⋮ ]〉 : = 𝑥 𝑇 𝑦 = ∑𝑛𝑖=1 𝑥𝑖 𝑦𝑖 = 𝑥1 𝑦1 + 𝑥2 𝑦2 + ⋯ + 𝑥𝑛 𝑦𝑛 , where 𝑥 𝑇 is the transpose of 𝑥.
𝑥𝑛 𝑦𝑛
Definition 2: Norm

Inner product spaces are normed vector spaces for the norm defined by
2
||𝑥|| = √〈𝑥, 𝑥〉, that is, ||𝑥|| = 〈𝑥, 𝑥〉
For 𝑥 ∈ ℝ𝑛 ,
𝑥1 𝑥1
𝑥2 𝑥2
〈𝑥, 𝑥〉 = 〈[ ⋮ ] , [ ⋮ ]〉 : = 𝑥 𝑇 𝑥 = ∑𝑛𝑖=1 𝑥𝑖2 = 𝑥12 + 𝑥22 + ⋯ + 𝑥𝑛2 , where 𝑥 𝑇 is the transpose of 𝑥.
𝑥𝑛 𝑥𝑛
2
If 𝑥 ∈ ℝ2 , then ||𝑥|| = 〈𝑥, 𝑥〉 = 𝑥12 + 𝑥22 . .

Cauchy–Schwarz inequality
For 𝑥, 𝑦 elements of a vector space 𝑉
〈𝑥, 𝑦〉 ≤ ||𝑥|| ∙ ||𝑦||,

with equality if and only if x and y are linearly dependent.

Triangle inequality
For 𝑥, 𝑦 elements of a vector space 𝑉
||𝑥 + 𝑦|| ≤ ||𝑥|| + ||𝑦||.

Examples:
 2
Note that, if 𝑥 = (𝑥1 , 𝑥2 ) ∈ 𝐷, then ||𝑥|| = 〈𝑥, 𝑥〉 = 𝑥12 + 𝑥22 ≤ 𝑟 2 .

Question:
 Homework: Prove the theorem below.

Theorem: If 𝑆 and 𝑇 are any two convex sets in ℝ𝑛 , then for all scalars 𝛼 𝑎𝑛𝑑 𝛽, the set

𝛼𝑆 + 𝛽𝑇 is also a convex set.

Definition: Hyperplane
Question: What is the convex hull containing the point 𝑥 = (𝑥1 , 𝑥2 ) and 𝑦 = (𝑦1 , 𝑦2 )?

Answer: The line segment joining 𝑥 = (𝑥1 , 𝑥2 ) and 𝑦 = (𝑦1 , 𝑦2 ) .

Practice Questions:
Solutions:

1. Definition: 𝐶𝑜 (𝑆) = Convex Hull = Intersection of all convex sets containing 𝑆.

That is, 𝐶𝑜 (𝑆) = ⋂𝑖 𝑆𝑖 ; 𝑖 is the index so that 𝑆𝑖 is convex that contains S.

Since 𝑆 is convex, then there exists an 𝑖 so that 𝑆 = 𝑆𝑖 . Hence,

𝐶𝑜 (𝑆) = ⋂ 𝑆𝑖 = S .
𝑖

2. Given, 𝑆 = {𝑥 ∶ |𝑥| = 1}

If 𝑆 ⊂ ℝ, then 𝑆 = {−1, 1}. Then 𝐶𝑜 (𝑆) = Smallest convex set containing 𝑆 = [−1,1]

3. Given 𝑆 = {(0,0), (2,1), (1,2)}.

𝐶𝑜 (𝑆) = Smallest convex set containing 𝑆 = the triangle with the vertises (0,0), (2,1),
and (1,2)

4. Given 𝐴 = {(𝑥1 , 𝑥2 ): 𝑥12 + 𝑥22 = 1}

𝐶𝑜(𝐴) = {(𝑥1 , 𝑥2 ): 𝑥12 + 𝑥22 ≤ 1}.

 5. Given 𝑆 = {(𝑥, 𝑦): 𝑦 2 ≥ 𝑥} ⊂ ℝ2 . 𝑆 is NOT a convex set, but it is unbounded.

Note that 𝑆 ′ = {(𝑥, 𝑦): 𝑦 2 ≤ 𝑥} ⊂ ℝ2 . 𝑆 is a convex set and it is unbounded

6. Given 𝐴 = {(𝑥, 0) : − 1 ≤ 𝑥 ≤ 1} ∪ {(0, 𝑦) : − 1 ≤ 𝑦 ≤ 1}

Theory is DONE
Understanding: Solution of LP
Please submit the following prove.
Question: Consider the LLP:

Maximize 𝑧 = ∑𝑛𝑗=1 𝑐𝑗 𝑥𝑗
Subject to ∑𝑛𝑗=1 𝑎𝑖𝑗 𝑥𝑗 ≤ 𝑏𝑖 ; 1 ≤ 𝑖 ≤ 𝑚.
𝑥𝑗 ≥ 0 ∀ 𝑗, 1 ≤ 𝑗 ≤ 𝑛.

Prove that the set of all feasible solution to the LLP above is convex.
Note:
Given that 𝑇 = {𝑥 ∈ ℝ𝑛 : ∑𝑛𝑗=1 𝑎𝑖𝑗 𝑥𝑗 (≤, =, ≥)𝑏𝑖 for 1 ≤ 𝑖 ≤ 𝑚 and 𝑥𝑗 ≥ 0 for 1 ≤ 𝑗 ≤ 𝑛}
is non-empty, closed and bounded.
We need to prove that the optimal solution exists, and it attains at some vertex of 𝑇.

A closed and bounded  compact

Theorem: A continuous function defined on a (closed and bounded) compact set has it’s
maximum and minimum values.

…………………………

Given that 𝑇 = {𝑥 ∈ ℝ𝑛 : ∑𝑛𝑗=1 𝑎𝑖𝑗 𝑥𝑗 (≤, =, ≥)𝑏𝑖 for 1 ≤ 𝑖 ≤ 𝑚 and 𝑥𝑗 ≥ 0 for 1 ≤ 𝑗 ≤ 𝑛}

is non-empty, closed and bounded.
We need to prove that the optimal solution exists, and it attains at some vertex of 𝑇.

Proof: Since 𝑇 is closed and bounded  𝑇 is compact.

[Theorem: A continuous function defined on a (closed and bounded) compact set has it’s
maximum and minimum values.]
Now, the function 𝑧 = 𝑐 𝑇 𝑥, 𝑥 ∈ ℝ𝑛 is continuous on the non-empty compact set 𝑇, hence the
optimal solution exists.
We need to prove that the optimal is at some vertex.
[Definition: Consider the function 𝑧 = 𝑐 𝑇 𝑥, 𝑥 ∈ ℝ𝑛 .
1. If 𝑧0 = 𝑐 𝑇 𝑥 0 is the maximum value of 𝑧 = 𝑐 𝑇 𝑥 at 𝑥 0 ∈ ℝ𝑛 , then 𝑧0 = 𝑐 𝑇 𝑥 0 ≥ 𝑐 𝑇 𝑥 = 𝑧, for
all 𝑥 ∈ ℝ𝑛
2. If 𝑧0 = 𝑐 𝑇 𝑥 0 is the minimum value of 𝑧 = 𝑐 𝑇 𝑥 at 𝑥 0 ∈ ℝ𝑛 , then 𝑧0 = 𝑐 𝑇 𝑥 0 ≤ 𝑐 𝑇 𝑥 = 𝑧, for
all 𝑥 ∈ ℝ𝑛 ]

Now the number of vertices of the convex polyhedron 𝑇 is finite. Let 𝑥1 , 𝑥 2 , … , 𝑥 𝑘 be the
vertices of the convex set 𝑇.
Since 𝑥1 , 𝑥 2 , … , 𝑥 𝑘 are the vertices of the convex set 𝑇, then 𝑇 is the convex hull of the set of
points 𝑥1 , 𝑥 2 , … , 𝑥 𝑘 , that is, 𝑇 is the smallest convex set containing the points 𝑥1 , 𝑥 2 , … , 𝑥 𝑘 .
Hence, any 𝑥 ∈ 𝑇 can be written as
𝑘 𝑘
𝑖
𝑥 = ∑ 𝜆𝑖 𝑥 , where 𝜆𝑖 ≥ 0 and ∑ 𝜆𝑖 = 1 ⋯ (1)
𝑖=1 𝑖=1
 (a) If the LLP is a minimizing problem, let 𝑧0 = min{𝑧 = 𝑐 𝑇 𝑥 𝑖 ∶ 𝑖 = 1, 2, … , 𝑘}. Then there exists an
𝑖 , 1 ≤ 𝑖 ≤ 𝑘, such that at the vertex 𝑥 𝑖 we have 𝑧0 = 𝑐 𝑇 𝑥 𝑖 and
𝑧0 = 𝑐 𝑇 𝑥 𝑖 ≤ 𝑐 𝑇 𝑥 = 𝑧 for all 𝑥 ∈ {𝑥1 , 𝑥 2 , … , 𝑥 𝑘 }.
Let 𝑥 = ∑𝑘𝑖=1 𝜆𝑖 𝑥 𝑖 ∈ 𝑇 be arbitrary.
𝑘 𝑘
𝑇 𝑇 𝑖
𝑧 = 𝑐 𝑥 = 𝑐 ∑ 𝜆𝑖 𝑥 ; where 𝜆𝑖 ≥ 0 and ∑ 𝜆𝑖 = 1
𝑖=1 𝑖=1
𝑘 𝑘 𝑘
𝑇 𝑖
= ∑ 𝜆𝑖 (𝑐 𝑥 ) ≥ ∑ 𝜆𝑖 𝑧0 = 𝑧0 ∑ 𝜆𝑖 = 𝑧0 .
𝑖=1 𝑖=1 𝑖=1

⇒𝑧 ≥ 𝑧0 ∀𝑥 ∈ 𝑇

That is, the minimum of 𝑧 = 𝑐 𝑇 𝑥 is at the vertex 𝑥 𝑖 for some 𝑖 , 1 ≤ 𝑖 ≤ 𝑘.

(b) Let 𝑧0 = max{𝑐 𝑇 𝑥 𝑖 ∶ 𝑖 = 1, 2, … , 𝑘}. Then there exists an 𝑖 , 1 ≤ 𝑖 ≤ 𝑘, such that at the vertex
𝑥 𝑖 we have 𝑧0 = 𝑐 𝑇 𝑥 𝑖 and
𝑧0 = 𝑐 𝑇 𝑥 𝑖 ≥ 𝑐 𝑇 𝑥 = 𝑧 for all 𝑥 ∈ {𝑥1 , 𝑥 2 , … , 𝑥 𝑘 }.
Let 𝑥 = ∑𝑘𝑖=1 𝜆𝑖 𝑥 𝑖 ∈ 𝑇 be arbitrary.

𝑘 𝑘 𝑘 𝑘
𝑇 𝑇 𝑖 𝑇 𝑖
𝑧 = 𝑐 𝑥 = 𝑐 ∑ 𝜆𝑖 𝑥 = ∑ 𝜆𝑖 (𝑐 𝑥 ) ≤ ∑ 𝜆𝑖 𝑧0 = 𝑧0 ∑ 𝜆𝑖 = 𝑧0 .
𝑖=1 𝑖=1 𝑖=1 𝑖=1

That is, the maximum of 𝑧 = 𝑐 𝑇 𝑥 is at the vertex 𝑥 𝑖 for some 𝑖 , 1 ≤ 𝑖 ≤ 𝑘.

Hence, the optimal solution exists, and it attains at some vertex of 𝑇.