PUE 4110: Classical Optimization techniques

Lecturer: Prof. Philip Ngare

(Contact: [email protected])

Pan African University

Institute for Science, Technology and Innovations
Ph.D in Electrical Engineering

May 25, 2021

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 1 / 18
Introduction
Optimization is the act of obtaining the best result under given
circumstances.
Suppose the required or the benefit desired in any practical situation
can be expressed as a function of certain decision variables, then
optimization can be defined as the process of finding the conditions
that give the maximum or minimum value of a function.
The optimum seeking methods are also known as mathematical
programming techniques
Mathematical programming techniques are useful in finding the
minimum of a function of several variables under a prescribed set of
constraints.
Stochastic process techniques can be used to analyze problems
described by a set of random variables having known probability
distributions.
Statistical methods enable one to analyze the experimental data and
build empirical models to obtain the most accurate representation of
the physical situation.
Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 2 / 18
Statement of optimization problem
An optimization or a mathematical programming problem can be stated as
follows.
Find  

 x1 
x2
 
X =

 ... 

xn
 

Which minimizes f (X) subject to the constraints

gj (X) ≤ 0, j = 1, 2, ..., m

lj (X) = 0, j = 1, 2, ..., p (1)

where
X is an n-dimensional vector called the design vector
f(X) is the objective function, gj (X), lj (X) are inequality and equality
constraints.
Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 3 / 18
 The problem above is called constrained optimization problem.
A similar problem can be stated without constraints, called
unconstrained optimization problems.
Design Vector
All quantities that are treated as variables in the design process are
called design or decision variables xi , i = 1, 2, ..., n.
Certain quantities that are fixed at the outset are called preassigned
parameters.
design variables are collectively represented as a design vector
X = {x1 , x2 , ..., xn }T
Each point in the n-dimensional design space is called a design point
and represents either a possible or an impossible solution to the
design problem.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 4 / 18
Design Constraints

In many practical problems, the design variables cannot be chosen

arbitrarily; rather, they have to satisfy certain specified functional and
other requirements.
The restrictions that must be satisfied to produce an acceptable
design are collectively called design constraints.
Constraints that represent limitations on the behavior or performance
of the system are termed behavior or functional constraints.
Constraints that represent physical limitations on design variables,
such as availability, fabricability, and transportability, are known as
geometric or side constraints.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 5 / 18
Objective Function

The conventional design procedures aim at finding an acceptable or

adequate design that merely satisfies the functional and other
requirements of the problem
In general, there will be more than one acceptable design, and the
purpose of optimization is to choose the best one of the many
acceptable designs available.
The criterion with respect to which the design is optimized, when
expressed as a function of the design variables, is known as the
criterion or merit or objective function.
Thus the selection of the objective function can be one of the most
important decisions in the whole optimum design process.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 6 / 18
 In some situations, there may be more than one criterion to be
satisfied simultaneously. For example, a gear pair may have to be
designed for minimum weight and maximum efficiency while
transmitting a specified horsepower.
An optimization problem involving multiple objective functions is
known as a multiobjective programming problem.
With multiple objectives there arises a possibility of conflict, and one
simple way to handle the problem is to construct an overall objective
function as a linear combination of the conflicting multiple objective
functions.
Thus if f1 (X ) and f2 (X ) denote two objective functions, construct a
new (overall) objective function for optimization as

f (X ) = α1 f1 (X ) + α2 f2 (X ) (2)

where α1 and α2 are constants whose values indicate the relative

importance of one objective function relative to the other.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 7 / 18
Classification of optimization problems
Optimization problems can be classified in several ways, as described
below.
1 Classification Based on the Existence of Constraints

I Any optimization problem can be classified as constrained or uncon-

strained, depending on whether constraints exist in the problem.
2 Classification Based on the Nature of the Design Variables
I Based on the nature of design variables encountered, optimization
problems can be classified as
F the problem to find values to a set of design parameters that make
some prescribed function of these parameters minimum subject to
certain constraints. Such problems are called parameter or static
optimization problems.
F the objective is to find a set of design parameters, which are all
continuous functions of some other parameter, that minimizes an
objective function subject to a set of constraints. This type of problem,
where each design variable is a function of one or more parameters, is
known as a trajectory or dynamic optimization problem.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 8 / 18
3 Classification Based on the Physical Structure of the Problem
Depending on the physical structure of the problem, optimization
problems can be classified as optimal control and nonoptimal control
problems.
Optimal control (OC) problem
I is a mathematical programming problem involving a number of stages,
where each stage evolves from the preceding stage in a prescribed
manner. It is usually described by two types of variables: the control
(design) and the state variables. The control variables define the system
and govern the evolution of the system from one stage to the next, and
the state variables describe the behavior or status of the system in any
stage. The problem is to find a set of control or design variables such that
the total objective function (also known as the performance index , PI)
over all the stages is minimized subject to a set of constraints on the
control and state variables.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 9 / 18
An OC problem can be stated as follows:
Find X which minimizes
l
X
f (X) = fi (xi , yi ) (3)
i=1

subject to the constraints

qi (xi , yi ) + yi = yi+1 , i = 1, 2, ..., l

gj (xj ) ≤ 0, j = 1, 2, ..., l

hk (yk ) ≤ 0, k = 1, 2, ..., l
where xi is the ith control variable, yi the ith state variable, and fi the
contribution of the ith stage to the total objective function; gj , hk , and
qi are functions of xi , yk , and xi and yi , respectively, and l is the total
number of stages.
Read example 1.2, page 16.
Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 10 / 18
Classification Based on the Nature of the Equations
Involved

Nonlinear Programming Problem

If any of the functions among the objective and constraint functions in Eq.
(1) is nonlinear, the problem is called a nonlinear programming (NLP)
problem.
Geometric programming (GMP) problem
is one in which the objective function and constraints are expressed as
posynomials in X.
Where h(X) is a posynomial with N terms, if
11 12 1n N1 N2 Nn
h(X ) = c1 x1a x2a ...xna + ... + cN x1a x2a ...xna

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 11 / 18
A GMP problem can therefore be stated as, find X which minimizes
N0 n
!
pij
X Y
f (X) = ci xj , ci > 0, xj > 0
i=1 i=1

subject to
 
Nk n
q
X Y
gk (X) = aik  xj ijk  > 0, aik > 0, xj > 0, k = 1, 2, ..., m
i=1 j=1

Where N0 and Nk denote the number of posynomial terms in the objective

and kth connstraint function, respectively.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 12 / 18
Quadratic programming problem

A quadratic programming problem is a nonlinear programming problem

with a quadratic objective function and linear constraints. It is usually
formulated as follows:
n
X n X
X n
F (X) = c + qi xi + Qij xi xj
i=1 i=1 j=1

subject to
n
X
aij xi = bj , j = 1, 2, ..., m
i=1

xi ≥ 0, j = 1, 2, ..., m
where c, qi , Qij and bj are constants.
See example 1.5 page 25.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 13 / 18
Linear programming
If the objective function and all the constraints in Eq. (1) are linear
functions of the design variables, the mathematical programming problem
is called a linear programming (LP) problem.
A linear programming problem is often stated in the following standard
form:  

 x1 
x2
 
X =

 ... 

xn
 

Which minimizes f (X) = ni=1 ci xi subject to the constraints

P

n
X
aij xi = bj , j = 1, 2, ..., m
i=1

xi ≥ 0, i = 1, 2, ..., p
where ci , aij and bj are constants.
Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 14 / 18
Classification based on the permissible values of the design
variables

Depending on the values permitted for the design variables, optimization

problems can be classified as integer and real-valued programming
problems.
If some or all of the design variables x1 , x2 , ..., xn of an optimization
problem are restricted to take on only integer (or discrete) values, the
problem is called an integer programming problem. On the other hand, if
all the design variables are permitted to take any real value, the
optimization problem is called a real-valued programming problem.
Example 1.7 page 28.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 15 / 18
Classification based on deterministic nature of the variables

Based on the deterministic nature of the variables involved, optimization

problems can be classified as deterministic and stochastic programming
problems.
Stochastic Programming Problem. A stochastic programming problem is
an optimization problem in which some or all of the parameters (design
variables and/or preassigned parameters) are probabilistic
(nondeterministic or stochastic).

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 16 / 18
Classification based on the separability of the functions
A function f (X ) is said to be separable if it can be expressed as the sum
of n single-variable functions, f1 (x1 ), f2 (x2 ), ..., fn (xn ), that is,
n
X
f (X) = fi (xi )
i=1

A separable programming problem is one in which the objective function

and the constraints are separable P
and can be expressed in standard form as
Find X which minimizes f (X) = ni=1 fi (xi ) subject to
n
X
gj (X) = gij (xi ) ≤ bj , j = 1, 2, ..., m
i=1

where bj is a constant.
See example 1.9.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 17 / 18
Classification based on the number of objective functions

Depending on the number of objective functions to be minimized,

optimization problems can be classified as single and multiobjective
programming problems.

A multiobjective programming problem can be stated as follows:

Find X which minimizes f1 (X), f2 (X), ..., fk (X)

subject to gj (X) ≤ 0, j = 1, 2, ..., m

where f1 , f2 , ..., fk denote the objective functions to be minimized

simultaneously.

Prof. P. Ngare (PAUSTI) PUE 4110: Classical Optimization techniques May 25, 2021 18 / 18