Digital Image

Processing

Lecture
Dr. Rajib Kumar Jha
Associate Professor
Notes-2021 Depart of Electrical Engineering
Indian Institute of Technology Patna
[email protected]
1
 Lecture-11-13
Image Restoration

2
 Contents
• Introduction
• Image Restoration vs. Image Enhancement
• Image degradation/restoration model
• Image Formation process and degradation model
• Degradation model in continuous functions and its
discrete formulation
• Estimation of degradation model
– Observation
– Experimentation
– Mathematical Modeling
• Restoration Techniques
– Inverse filtering
– Minimum Mean square error (Wiener filtering)
– Constrained Least square filtering
What is Image Restoration
What is Image Restoration
What is Image Restoration
What is Image Restoration
 Enhancement/Restoration
Image Enhancement:
– particularly using a low pass filter or using smoothing masks in
spatial domain noise content of the image (high frequency
component) gets reduced. It is the subset of image restoration.

Image Restoration:
– A process which tries to restore an image which has been
degraded by some knowledge of a degradation function; this
operation is known as image restoration.
– In case of image restoration, image degradation model is very-
very important.
– Approach is: Estimate the model which has degraded the
image, once the degradation model is known; then we have to
apply the inverse process to recover the desired image.
1. input image f (x, y) is degraded by a degradation function H;
2. output of f(x,y)*h(x,y) is added to an additive noise eta (x, y);
3. The final degraded output image is g (x, y).
 Image Restoration
g(x,y) =H[f(x,y)] +𝜑 𝑥, 𝑦

Noise terms 𝜑 𝑥, 𝑦 = 0 So,

g(x,y) =H[f(x,y)]; H is the degradation operator

• Image Restoration is applied to the degraded image

g(x, y) and get restored image 𝑓 ′ (x,y).

• Restored image 𝑓 ′ (x,y) is not the exact image f(x,y) so

we calculate the goodness criteria which says the
restored image is how much close to original image.
Input Image & Degraded Image
 Reason for blurred photograph
The four main causes of blurry photos are:
• Out Of Focus
• The subject moves while the shutter is open
• The camera moves while the shutter is open
• Depth Of Field is too shallow: only a part of the image is
in focus. The background and sometimes the foreground is blurred.
The camera moves while the shutter is open

Depth Of Field is too shallow

 Image Restoration
• One of the very important task in the restoration process is
to estimate the degradation model which has degraded the
input image.
• We will see some techniques to estimate the degradation
model. That is, how to estimate the degradation function H.
• This particular operation that is the conversion from f(x, y)
to g(x, y), this can be represented in spatial domain as

g(x,y)=f(x,y)*h(x,y)+h(x,y)
G(u,v)=F(u,v)H(u,v)+N(u,v)
If linear, position-invariant system
Linear, position-invariant degradation
Properties of the degradation function H
• Linear operator H or Superposition theorem
– H[af1(x,y)+bf2(x,y)]=aH[f1(x,y)]+bH[f2(x,y)]

• Position(space)-invariant system
– H[f(x,y)]=g(x,y)
– H[f(x-a, y-b)]=g(x-a, y-b)

This position invariant property says that the response of H at

any point in the image should solely depend upon the value of
the pixel at that particular point and the response will not
depend upon the position of the point in the image.
Degradation model: Continuous function

1 𝑥=0 𝑦=0
𝛿 𝑥, 𝑦 =
0 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

1 𝑥 = 𝑥0 ; 𝑦 = 𝑦0
𝛿 𝑥 − 𝑥0 , 𝑦 − 𝑦0 =
0 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
+∞
𝑓(𝑥, 𝑦) 𝛿 𝑥 − 𝑥0 , 𝑦 − 𝑦0 𝑑𝑥 𝑑𝑦 = 𝑓(𝑥0 , 𝑦0 )
−∞
 Degradation model: Continuous function

The term f(alpha, beta) is independent of the variables

x and y, so, the same expression can be rewritten in a
slightly different form.
 Degradation model: Continuous function
(1)

1. Equation 1 is called the impulse response of H.

2. The response of H on an impulse function when impulse becomes a point of
light, in optics it is known as point spread function or PSF.
3. If h is position invariant then 2 functions f(x, y) & h(x, y) follow the convolution
operation.
4. We already said that input image f(x, y) is actually convolved by the degradation
process that is h(x, y).
5. If impulse response operator h is known then it is possible to find out the
response of this operator h to any arbitrary input f(alpha, beta)
 Image Restoration
• So, we discussed that in image degradation model in
spatial domain we have assumed the degradation
function H is linear and position invariant.
• It is very important to note that many of the
degradation operations which we encounter in reality
can be approximated by such linear space invariant or
linear position invariant models.
• Once a degradation function is approximated by a
linear position invariant model, then by using linear
system theory we can find out the solution for image
restoration process.
 Discrete Formulation
1−𝐷 →2−𝐷 𝑛 𝑥, 𝑦 = 0

𝑓 𝑥 is the input function and

ℎ 𝑥 is the degradation function.

𝑓 𝑥 : 𝑥 = 0,1, … … … . . 𝐴 − 1
ℎ(𝑥): 𝑥 = 0,1, … … … . . 𝐵 − 1

Make the same dimension or size of f(x) and h(x)

then both becomes periodic of period M.
We make both of them to be of dimension capital M by adding
additional number of 0s. fextended (x).
 Use convolution operation
𝑀−1
𝑔𝑒 𝑥 = 𝑚=0 𝑓𝑒 𝑚 ℎ𝑒 𝑥 − 𝑚 ; 𝑥 = 0, 1, … … . .M-1

Matrix form: g=Hf;

𝑓𝑒 (0) 𝑔𝑒 (0)
𝑓𝑒 (1) 𝑔𝑒 (1)
. .
𝑓= 𝑔=
. .
. .
𝑓𝑒 (𝑀 − 1) 𝑔𝑒 (𝑀 − 1)

Matrix H will be of dimension Capital M by Capital M.

 Discrete Formulation
𝐻 = 𝑀𝑋𝑀;
ℎ𝑒 0 ℎ𝑒 −1 … … … … … … . . ℎ𝑒 (−𝑀 + 1)
ℎ𝑒 1 ℎ𝑒 0 … … … … … … . . . . ℎ𝑒 (−𝑀 + 2)
.
𝐻= .
.
.
ℎ𝑒 𝑀 − 1 ℎ𝑒 𝑀 − 2 … … … … … … . . ℎ𝑒 (0)

Let ℎ𝑒 𝑥 → 𝑃𝑒𝑟𝑖𝑜𝑑𝑖𝑐 𝑤𝑖𝑡ℎ 𝑝𝑒𝑟𝑖𝑜𝑑 𝑀

ℎ𝑒 𝑥 + 𝑀 = ℎ𝑒 𝑥
 Discrete Formulation
𝐻 = 𝑀𝑋𝑀;
ℎ𝑒 0 ℎ𝑒 𝑀 − 1 … … … … … . . … … . . ℎ𝑒 (1)
ℎ𝑒 1 ℎ𝑒 0 … … … … … … … . … … . . . . ℎ𝑒 (2)
.
𝐻= .
.
.
ℎ𝑒 𝑀 − 1 ℎ𝑒 𝑀 − 2 … … … … … … . . ℎ𝑒 (0)
1. Different rows of this matrix are actually generated by rotation
to the right of the previous term. So, here the second row is
actually generated by rotating the first row to the right.
2. Similarly, third row is generated by rotating the second row to
right by 1.
3. This is called Circulant Matrix
Discrete Formulation in 2-D
 Discrete Formulation in 2-D
𝑔 = 𝐻𝑓 + 𝑛 𝑤ℎ𝑒𝑟𝑒 ℎ 𝑖𝑠 𝑀𝑁𝑥𝑀𝑁 &

𝑓 𝑖𝑠 𝑀𝑥𝑁 𝑚𝑎𝑡𝑟𝑖𝑥; Hj=MxM size

𝐻0 𝐻𝑀−1 … … … … … . . … … . . 𝐻1
𝐻1 𝐻0 … … … … … … … . … … . . . . 𝐻2
.
𝐻= .
.
.
𝐻𝑀−1 𝐻𝑀−2 … … … … … … … . . 𝐻0
 Discrete Formulation in 2-D
ℎ𝑒 𝑗, 0 ℎ𝑒 𝑗, 𝑀 − 1 … … … … … . . … … . . ℎ𝑒 (𝑗, 1)
ℎ𝑒 𝑗, 1 ℎ𝑒 𝑗, 0 … … … … … … … . … … . . . . ℎ𝑒 (𝑗, 2)
.
𝐻𝑗 = .
.
.
ℎ𝑒 𝑗, 𝑀 − 1 ℎ𝑒 𝑗, 𝑀 − 2 … … … … … … . . ℎ𝑒 (𝑗, 0)

1. This matrix Hj which is actually a component of the degradation

matrix H is a circulant matrix.
2. Now using this block matrix, the matrix H have already been
subscripted in the form of a Circulant matrix.
3. So, this matrix H in this particular case is what is known as a block
circulant matrix.
• The method of restoring the original image from the degraded
image using the estimated degradation function obtained by
one of this 3 methods is what is called a blind deconvolution.
Estimation of degradation functions
Estimation by Observation
Estimation by Observation
Estimation of degradation functions
 Estimation by Mathematical Modelling
1. Atmospheric disturbance also leads to degradation
of an image.
2. This constant K, gives the nature of the turbulence.

3. If the value of K is large, that means the turbulence

is very strong whereas if the value of K is very low, it
says that the turbulence is a mild turbulence.
Estimation of degradation functions
 How Blurring is Introduced
• Whenever we take the snap of the scene, the shutter of
the camera is open for certain duration of time and
during this period, during which the shutter is open, the
object is not stationary, the object is moving.

• Considering any particular point in the imaging plane, the

light which arrives from the scene does not come from a
single point. But the light we get at a particular point on
the imaging sensor is the aggregation of the reflected
light from various points in the scene.

• fast moving object or a longer exposure time or slow

shutter speed of camera may result in blurring artefacts.
• Atmospheric turbulence also creates blur image.
 Motion Blur

Original image, blurred image and de-blurred image

 Estimation of degradation functions
(model bases)
• Mathematical model for uniform linear motion between the image and the
sensor during image acquisition
– Let x0(t) and y0(t) denote time varying components of motion in the x-
and y-directions
• So, once the object is moving, then the total exposure at any point in the
imaging plane can be obtained by integration operation where the
integration has to be done over the period T during which the shutter
remains open.

g ( x, y)   f  x  x0 (t ), y  y0 (t )dt
T
where g ( x, y) is the blurred image
0

T  j 2 ux0 ( t )  vy0 ( t ) 
G (u, v)  F (u, v)  exp dt where
0

T  j 2 ux0 ( t )  vy0 ( t ) 
H (u, v)   e dt is degradation function
0
 Estimation of degradation functions
(model bases)

• Uniform Linear motion in the x and y direction x0(t)=at/T

and y0(t)=bt/T where the image has been displaced by a
total distance a in the x-direction and b in the y-direction

T
H (u, v)  sin  (ua  vb) e  j ( ua  vb )

 (ua  vb)
 Estimation by Modelling
E

Original image Blurred image

Blurring is introduced assuming uniform linear motion and for

obtaining this particular blurring, we have taken a= 0.1 and b = 0.1
Estimation of degradation functions
(model bases)
 Inverse filtering

G(u,v)=F(u,v)H(u,v)

G(u,v)=F(u,v)H(u,v)+N(u,v)
Inverse filtering
Restoration Inverse Filtering
Inverse filtering
5/ 6
Degradation function  k ( u  M / 2)2 ( v  N / 2)2 
H (u, v)  e

Curtain of noise
Cutting off values of the ratio outside a radius of 40, 70,85.
Restoration by Inverse Filtering on
Motion Blur Image
Inverse Filtering doesn’t give good
Result on Motion Blur image
 Inverse filtering doesn’t give good
result on Motion Blur image
• For any integer value of (ua + vb), the corresponding
component H (u, v) will be equal to 0 and for nearly integer
values of (ua + vb), term H(u, v) is going to be very-very low.

• So, for direct inverse filtering when we go for dividing G (u, v)

by H(u,v), you have the Fourier transformation of the
reconstructed image,

• wherever H(u, v) is very low near about 0, the corresponding

F (u, v) term will be abnormally high and

• When we take the inverse Fourier transform of this that very-

very high value is reflected in the reconstructed image.
What is the way out? Can’t we use the inverse filtering
for restoration of motion blurred image?

Solution: Impulse function (A) point spread function G(u,v)

• If we take impulse function and try to find out the point spread
function i.e, if we employ motion blur to the input impulse.
• So by using the same motion blur function, blur this impulse and
what I get is an impulse response. Which is the point spread
function in this particular case.
 Point Spread Function
• because input was an impulse, so for this impulse F (u,v) is
equal to constant which is say something like A,
• now from these two, I can compute H (u,v) which is given by G
(u, v) divided by this constant term A.

• Reason for impulse function an input: If impulse response operator h

is known then it is possible to find out the response of this operator h
to any arbitrary input 𝑓(𝛼, 𝛽).

• Also we know that impulse response h completely characterises any

system.
Restoration Inverse Filtering (PSF)
 Drawback of Inverse Filtering
• With this inverse filtering approach the major problem is
as we said that we have to consider the (u,v) values for
reconstruction which should be within a limited range.

• But, question is how do we say that up to what extent of

(u,v) value we should go?
• That is again image dependent.

• So, it is not very easy to decide that to what extent of

frequency components we should consider for the
reconstruction of the original image if go for direct
inverse filtering.
Wiener filtering

H (u, v) is the degradation function

Wiener filtering

White noise
Wiener filtering
 Constrained Least Squares Filtering
• Performance of the Wiener filtering depends upon the
correct estimation of the value of K, that is, the accurate
estimation of the power spectrum of both the original
undegraded image and noise.

• CLSF does not make any assumption about original

undegraded image.

• It makes use of only the noise probability density

function. it uses the mean and the variance of the noise
and see that how reconstruction using CLSF approach
makes use of these noise parameters.
 Constrained Least Square Filtering
• We will start from the basic equation 𝑔 = 𝐻𝑓 + 𝑛
• Value of H is very-very sensitive to noise or H is noise
dependent.
• To take care of H, we define an optimality criteria and
using that criteria, reconstruction needs to be done.
• Since H is very sensitive to noise, so for reconstruction,
the optimality criteria that we will use is the image
smoothness.
• We know that, Laplacian operator tries to enhance the
irregularities or discontinuities in the image.
• So, if we can minimize the Laplacian of an image, it will
ensure that the reconstructed image will be smoothed.
 Constrained Least Square Filtering
• So, optimality criteria in this particular case is given
by C

• We will try to minimize this criterion function

subject to the constraint that
2
• 𝑔 − 𝐻𝑓 = 𝑛2 where 𝑓is the reconstructed image.
• So, it is called constrained least square filtering.
Constrained Least Square Filtering
Again, without going into details, the frequency domain
solution of CLSF is given below
 Constrained Least Square Filtering
• Constant term gamma is a scalar quantity, it is to be
adjusted so that the specified constrained that is
2
𝑔 − 𝐻𝑓 = 𝑛2 is maintained.

• P(u, v), it is the Fourier spectrum or the Fourier transform

of the Laplacian mask.

• Fourier transformation of P(x,y) which is given in the form

of a 3x3 mask, we need to pair appropriate number of
zeros so that this P(x,y) also becomes a function of
dimension MxN.
Constrained Least Square Filtering
• Gamma has to be adjusted manually for
obtaining the optimum result.

• it is also possible to automatically estimate the

value of gamma by an iterative approach.
Inverse-Wiener-CLSF (With Noise)
Wiener Vs Constrained LSF
Constrained Least Square Filtering
Automatic value of Gamma by iterative approach
Define a residual vector say r, g is the degraded image
𝑓 is the estimated restored image.

𝑟 = 𝑔 − 𝐻𝑓 1. Define a function say

phi of gamma = r
transpose r, or which
𝐹 u, v → γ and 𝑓 → 𝛾 is the Euclidean
𝑠𝑜 𝑟 → 𝛾 norm of r.
2. It can be shown that
this function is
∅ 𝛾 = 𝑟𝑇 𝑟 = 𝑟 2
monotonically
increasing function
2 2 of gamma.
𝑟 = 𝑛 ∓𝑎
 Constrained Least Square Filtering
• That means whenever gamma increases, Euclidean norm of
r also increases; if gamma decreases, Euclidean norm of r
also decreases.

• And by using this property, it is possible to find out what is

the optimum value of gamma within some specified
accuracy.

• We want to estimate the value of gamma such that the

Euclidean norm of r will be equal to n square plus minus
some constant “a” where “a” is nothing but what is the
specified accuracy factor of reconstruction.
Algorithm Steps for Iterative Selection of Gamma
Noise Parameter Estimation
1. Take the uniform region
sub image from the
degraded image and then
draw histogram.
2. Once probability density
function is known; from
this, we can compute the
standard deviation and the
variance of the noise term.
3. We can go for optimum
reconstruction of the
degraded image.
 Noise Parameter Estimation
2
𝑛

2 1 𝑀−1 𝑁−1 2
𝜎𝑛 = 𝑥=0 𝑦=0 𝑛 𝑥, 𝑦 − 𝑚𝑛
𝑀𝑁

1 𝑀−1 𝑁−1
𝑚𝑛 = 𝑥=0 𝑦=0 𝑛 𝑥, 𝑦
𝑀𝑁

2
𝑛 = 𝑀𝑁 𝜎𝑛 2 − 𝑚𝑛 2
• A common source of
periodic noise in an image is
from electrical interference
during the image capturing
process.

• In the frequency domain this

type of noise can be seen as
discrete spikes.
Periodic Noisy Image
Periodic Noise

Contaminated with periodic noise

1. All these bright dots tell us about

the frequency of the periodic
noise.
2. Use an appropriate band pass
filtering to filter out that region
d=2
Thanks