International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1383

A New Exemplar Based Image Completion using
Belief Propagation
N.Nanthini
#1
, Dr. T.Meyyappan, M.Sc., M.phil. M.B.A.,(M.Tech)., Ph.D
*2
,
#
Department of Computer Science and Engineering ,Alagappa University ,Karaikudi,Tamilnadu ,India
Abstract There is a new exemplar-based framework proposed
that treats the image completion and texture synthesis in a
unique manner. That interactive image completion method has
been suggested based on Belief Propagation (BP) approach.
Blocked area or in other words, the area with loss of information
in the target image is completed with BP combined with texture
synthesis in an interactive way. The target image is decomposed
by BP into levels of Intrinsic Mode Functions (IMF) images,
while the user is allowed to indicate the structural image edge to
recover in the unknown image regions. For each level of IMF
image, first the target image patches along with the user-
specified curves in the unknown region are completed. Then the
remaining target image patches are prioritized to complete
according to the image gradient feature. The target image
patches are completed based on the combination frequency
feature values from BP and the texture synthesis. Finally, the
levels of completed IMF images are compounded into the result
image. Experiments prove that the proposed algorithm is not
only able to recover the damaged structural and texture
information of large scale, but also completely blocked image
structures.

Keywords: Belief Propagation, Seam Carving, pruning, message
scheduling.
I. INTRODUCTION
Modern digital technology has made it possible to
manipulate multi-dimensional signals with systems that range
from simple digital circuits to advanced parallel computers.
The goal of this manipulation can be divided into three
categories,
Image Processing image in ->image out
Image Analysis image in ->measurements
out
Image Understanding image in ->high level
description out
Image processing is the methodology to convert an
image into digital form and perform some operations on it, in
order to get an enhanced image or to extract some useful data
from it. It is a type of signal privilege in which input is an
image, like video frame or photograph and the output might
be either image or characteristics associated with that same.
Usually image processing system includes treating images as
two dimensional signals while applying already set signal
processing methods to them. We will restrict ourselves to two-
dimensional (2D) image processing although most of the
concepts and techniques that are to be described can be
extended easily to three or more dimensions. We begin with
certain basic definitions
An image defined in the "real world" is considered to be a
function of two real variables, for example, a(x, y) with a as
the amplitude (e.g. brightness) of the image at the real
coordinate position (x, y). An image may be considered to
contain sub-images sometimes referred to as regions-of-
interest, ROIs, or simply regions.
This concept reflects the fact that images frequently contain
collections of objects each of which can be the basis for a
region. In a sophisticated image processing system it should
be possible to apply specific image processing operations to
selected regions. Thus one part of an image (region) might be
processed to suppress motion blur while another part might be
processed to improve color rendition. The amplitudes of a
given image will almost always be either real numbers or
integer numbers.
The latter is usually a result of a quantization process that
converts a continuous range (say, between 0 and 100%) to a
discrete number of levels. In certain image-forming processes,
however, the signal may involve photon counting which
implies that the amplitude would be inherently quantized. In
other image forming procedures, such as magnetic resonance
imaging, the direct physical measurement yields a complex
number in the form of a real magnitude and a real phase.
Image completion is one of the key areas in image processing.
It is an important photo-editing task which involves
synthetically filling a hole in the image such that the image
still appears natural. State-of-the-art image completion
methods work by searching for patches in the image that fit
well in the hole region. Our key insight is that the image
patches remain natural under a variety of transformations
(such as scale, rotation and brightness change), and it is
important to exploit this. We propose and investigate the use
of different optimization methods to search for the best
patches and their respective transformations for producing
consistent, improved completions.
International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1384

II. EXISTING SYSTEM

There have been three main approaches so far, for dealing
with the image completion problem. Below are the existing
methods for the image completion process.

Statistical-based methods,
PDE-based methods,
Exemplar-based methods.

A. Statistical - Based Methods:
These methods are mainly used for the case of texture
synthesis. Typically, what these methods do is that, given an
input texture, they try to describe it by extracting some
statistics through the use of compact parametric statistical
models .Then, in order to synthesize a new texture, these
methods typically start with an output image containing pure
noise, and keep perturbing that image until its statistics match
the estimated statistics of the input texture. [10].
Besides the synthesis of still images, parametric statistical
models have been also proposed for the case of image
sequences [12]. However, the main drawback of all methods
that are based on parametric statistical models is that, as
already mentioned, they are applicable only to the problem of
texture synthesis [11], and not to the general problem of
image completion.
But even in the restricted case of texture synthesis, they can
synthesize only textures which are highly stochastic and
usually fail to do so for textures containing structure as well.
Nevertheless, in cases where parametric models are applicable,
they allow greater flexibility with Respect to the modification
of texture properties [15].




(a) (b) (c)
Fig 1 a .Original image b. Image with missing region
c. Completion using image inpainting.
Image inpainting methods, when applied to large or
textured missing regions, very often over smooth the image
and introduce blurring artifacts.
B. PDE - Based Methods:
These methods, on the other hand, try to fill the missing
region of an image through a diffusion process, by smoothly
propagating information from the boundary towards the
interior of the missing region. According to these techniques,
the diffusion process is simulated by solving a partial
differential equation (PDE) [2], which is typically non-linear
and of high order.

The main disadvantage of almost all PDE based methods is
that they are mostly suitable for image inpainting situations.
This term usually refer to the case where the missing part of
the image consists of thin, elongated regions[19],[17].

Furthermore, PDE-based methods implicitly assume that the
content of the missing region is smooth and non-textured [16].
For this reason, when these methods are applied to images
where the missing regions are large and textured, they usually
over smooth the image and introduce blurring artifacts.

C. Exemplar-Based Methods:
These methods try to fill the unknown region simply by
copying content from the observed part of the image [18]. All
exemplar-based techniques for texture synthesis that have
appeared until now, were either pixel-based [4],[5], or patch-
based [6],[9],[10], meaning that the final texture was
synthesized one pixel, or one patch at a time by simply
copying pixels or patches from the observed image
respectively.

Fig 2: Exemplar based method
Recent exemplar-based methods also place emphasis on the
order by which the image synthesis proceeds, usually using a
confidence map for this purpose. There are two main
handicaps of related existing techniques.
Exemplar based methods for texture synthesis has been also
used for the case of video. E.g Schodl et al. [9] are able to
synthesize new video textures simply by rearranging the
recorded frames of an input video, while the texture synthesis
method of Kwatra et al. [2],[3] that has been mentioned above
applies to image sequences as well.
First, the confidence map is computed based on heuristics and
ad hoc principles that may not apply in the general case and
second, once an observed patch has been assigned to a
missing block of pixels[1], that block cannot change its
International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1385

assigned patch thereafter. This last fact reveals the greediness
of these techniques, which may lead to visual inconsistencies.

D. Gradient-Based Filling:

For computing DCT coefficients on image blocks with
unknown pixels, if the unknown pixels are filled with average
color of known pixels[5], the DCT coefficients do not reflect
the texture or the structural information in the block very well.

For example, if the missing region is located at the center of
an image block with progressive color change from left to
right, filling with average color is not a good approximation.
For a smooth image, the gradient at pixels will be
approximately equal to zero. Based on this observation, we
developed a gradient-based filling method to determine the
unknown pixels before computing DCT.In detail, for each
unknown pixel pi;j , letting the discrete gradient at this pixel
be zero will lead to linear equations k linear equations with
k > l which is actually an over determined linear system[23].
Note that, the gradient filling may generate smoother images
at those unknown pixels when the known pixels are highly
textured. This will be overcome by a revision of the
Criminisiets method [1]. We find more than one matched
patches (actually 0:1% of the toal exemplars), among which
the one with the highest SSD score on the known pixels is
finally selected.
Furthermore, using m-dominant DCT coefficients in selection
will also reduce the effect led by the pixels filled in this step.




Fig 3: An example of damaged ellipses: (left) the input image (200 150
pixels), (middle) the result of Criminisis method (in 10.36 sec.), and (right)
our result (in 0.67 sec.).
E. Drawbacks:
The main drawback of all methods that are based on
parametric statistical models is that they are
applicable only to the problem of texture synthesis,
and not to the general problem of image completion
The main disadvantage of almost all PDE based
methods is that they are mostly suitable for image
inpainting situations. This term usually refer to the
case where the missing part of the image consists of
thin, elongated regions.
Most of the exiting methods in literature take a long
time to retouch one image, which is far from
practical using in an interactive image processing and
editing.
It does not give satisfactory result in the regions wish
large unknown areas and highly textured region.
Lose linear structure and composite texture
All these approaches are extremely slow due to the
high computational complexity.
III. PROPOSED SYSTEM
We are presenting an operator called seam carving for
content-aware resizing of images computed as the optimal
paths on a single image and removing from an image.
This operator can be used for image manipulations including:
image retargeting and object removal. The operator can be
easily integrated with various saliency measures, as well as
user input, to guide the resizing process. In addition, we
define a data structure for multi-size images that support
continuous resizing ability in real time we present a
representation of multi-size images that encodes, for an image
of size (mn), an entire range of retargeting sizes from ab to
mn and allow the user to retarget an image continuously in
real time.
A. System Flow Diagram:













1) Seam carving: Seam carving is an image resizing
algorithm that alters the dimensions of an image not by
scaling or cropping, but rather by intelligently removing
pixels from (or adding pixels to) the image that carry little
importance. The importance of a pixel is generally measured
by its contrast when compared with its neighbour pixels, but
other measures may be used. Additionally, it's possible to
define (or auto detect) areas of high importance (faces,
buildings, etc.) in which pixels may not be deleted, and
conversely, areas of zero importance which should be
removed first.
Input Image
(100%)
Tools and Adjust the
Image

Sobel
Prewitt
Laplace
Img Resize
Forward Pass Backward Pass Img
<100%
Pruning
Seamcarving
100% Output Image
Belief Propagation
International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1386

From this information, the algorithm detects seams
(continuous lines of pixels joining opposite edges of the image)
which have the lowest importance, and deletes those. This
shrinks the image by one horizontal or vertical line, depending
on which direction the seam ran. Seam carving can also be run
in reverse by adding interpolated pixels along the lowest
energy seam.
With this technique, it's now possible to reduce or
enlarge pictures by a wide range, while still retaining details.
In normal image scaling, it's often not possible to scale about
factor greater than 2 or less than 0.5, without losing much of
the images quality.
2) BP (Belief Propagation) algorithm: The main
advantage is that we now can hopefully apply belief
propagation (i.e a state-of-the-art optimization method) to our
energy function. The reason is the intolerable computational
cost of BP, caused by the huge number of existing labels.
Motivated by this fact, one other major contribution of this
work is the proposal of a novel MRF optimization scheme,
called Priority-BP that can deal exactly with this type of
problems, and carries two signicant extensions over standard
BP: one of them, called dynamic label pruning, is based on the
key idea of drastically reducing the number of labels.
However, instead of this happening beforehand (which will
almost surely lead to throwing away useful labels), pruning
takes place on the y (i.e while BP is running), with a
(possibly) different number of labels kept for each node. The
important thing to note is that only the beliefs calculated by
BP are used for that purpose. This is exactly what makes the
algorithm generic and applicable to any MRF. Furthermore,
the second extension, called priority-based message
scheduling, makes use of label pruning and allows us to
always send cheap messages between the nodes of the
graphical model.
Moreover, it considerably improves BPs convergence, thus
accelerating completion even further. The signicance of our
contribution also grows due to the fact that (as we shall see)
Priority-BP is a generic algorithm, applicable to any MRF
energy function. This is unlike any prior use of Belief
Propagation and, therefore, our method resolves, for the rst
time, what is currently considered as one of the main
limitations of BP: its inefficiency to handle problems with a
huge number of labels.
In fact, this issue has been a highly active research topic over
the last years. Until now, however, the techniques that have
been proposed were valid only for restricted classes of MRFs.
Not only that, but our priority-based message scheduling
scheme can be used (independently of label pruning) as a
general method for accelerating the convergence of BP.
3) MRF (Markov Random Field): The Markov
Random Field using the segmentation is obtained as a cartoon
image, which is basically a labelling of the input image I.
Hence for each pixel s, the region-type (or pixel class) that the
pixel belongs to is specied by a class label, us , which is
modelled as a discrete random variable taking values in LZ{1,
2,.,L}. The set of these labels uZfus ;s2Sg is a random eld,
called the label process.
Furthermore, the observed image features (colour
and texture) are supposed to be a realization fs js2Sg from
another random eld, which is a function of the label process
u. basically, the image process F represents the manifestation
of the underlying label process.
4) Label pruning: Pruning is mainly done under
Priority-BP algorithm with forward pass and backward pass
by labelling the node which to be scheduled. The actual
message scheduling mechanism as well as label pruning takes
place during the forward pass. The role of the backward pass
is then just to ensure that the other half of the messages gets
transmitted as well.
The main idea of label pruning is that, as we are
visiting the nodes of the MRF during the forward pass, we
dynamically reduce the number of possible labels for each
node by discarding labels that are unlikely to be assigned to
that node. Its use is allowed only in conjunction with our
priority-based message scheduling scheme of visiting most
confident nodes first.
This keeps our method generic and, therefore,
applicable to any energy function. A key observation,
however, relates to the fact that label pruning is a technique
not meant to be used on its own.
5) Message scheduling: We use a specific message
scheduling scheme, whose goal will be twofold. On one hand,
it will make label pruning possible and favor the Circulation
of cheap messages. On the other hand, it will speed up BPs
convergence. This issue of BP message scheduling, although
known to be crucial for the success of BP, it has been largely
overlooked until now.
6) Forward Pass: Each iteration of Priority-BP is
divided into a forward and a backward pass. The actual
message scheduling mechanism as well as label pruning takes
place during the forward pass. This is also where one half of
the messages get transmitted. To this end, all nodes are visited
in order of priority. Each time we visit a node, say p, we mark
it as committed meaning that we must not visit him again
during the current forward pass.
We also prune its labels and then allow him to transmit its
cheap (due to pruning) messages to all of its neighbours
apart from the committed ones. The priorities of all
neighbours that received a new message are then updated and
the process continues with the next uncommitted (i.eunvisited)
node of highest priority until no more uncommitted nodes
exist.
7) Backward Pass: The role of the backward pass is
then just to ensure that the other half of the messages gets
International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1387

transmitted as well. To this end, we do not make use of
priorities but simply visit the nodes in reverse order, with
respect to the order of the forward pass and just transmit the
remaining unsent messages from each node. For this reason,
no label pruning takes place during this pass.
We do update node priorities, though, so that they are
available during the next forward pass. Also, as we shall see, a
nodes priority depends only on the current beliefs at that node.
One big advantage out of this is that keeping the node
priorities up-to-date can be done very efficiently in this case,
since only priorities for nodes with newly received messages
need to be updated.
B. Advantages of Proposed System:
We here propose the image completion process with a handful
of advantages providing an image with visually possible
outcome without any inconsistencies, thereby providing the
necessary image enlarging.
A simple image operator called seam carving is used here that
supports content-aware image resizing for both reduction and
expansion. A seam is an optimal 8-connected path of pixels on
a single image from top to bottom, or left to right, where
optimality is defined by an image energy function. No user
intervention is required by our method, which avoids greedy
patch assignments by maintaining many candidate source
patches. In this way, each missing block of pixels is allowed
to change its assigned patch many times throughout the
execution of the algorithm, and is not enforced to remain tied
to the first label that has been assigned to it during the early
stages of the completion process.
Our formulation applies not only to image completion, but
also to texture synthesis and image inpainting, thus providing
a unified framework for all of these tasks.To this end, a novel
optimization scheme is proposed, the Priority-BP algorithm,
which carries 2 major improvements over standard belief
propagation: dynamic label pruning and priority based
message scheduling.
C. Algorithm Used:
A pseudo code description of priority-BP is contained in the
following algorithm.

Nodes and algorithm 1 Priority-BP
assign priorities to declare them uncommitted
for k =1 to K do {K is the number of iterations}
execute Forward Pass and then Backward Pass
assign to each node p its label xp that maximizes bp()
Forward Pass
for time =1 to N do {N is the number of nodes}
p =uncommitted node of highest priority
apply label pruning to node p
forward Order[time] =p ; p committed = true;
for any uncommitted neighbor q of node p do
send all messages mpq () from node p to node q
update beliefs bq () as well as priority of node q
Backward Pass
for time =N to 1 do
p =forward Order[time]; p committed = false;
for any committed neighbor q of node p do
send all messages mpq() from node p to node q
update beliefs bq() as well as priority of node
IV. EXPRIMENTAL RESULT









Fig 4 Input Image








Fig 5 Masked Image



International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1388










Fig 6 Visiting order during forward pass









Fig 7 Priority-BP result (100%)

V. CONCLUSION AND FUTURE SCOPE
In this paper we have covered the unifying image
completion approach, texture synthesis and image inpainting.
In order to prevent the visually inconsistent results we try to
avoid greedy patch assignments, and instead pose all of these
tasks as a discrete labelling problem with a well defined
global objective function. To solve this issue, we proposed the
novel optimization scheme known as Priority-BP (Belief
Propagation) which carries two vital extensions over standard
BP.
Priority-based message scheduling
Dynamic label pruning
This optimization scheme doesnt rely on any image-specific
prior knowledge and can thus be applied to all kinds of images.
Moreover it is generic (can be applicable to any MRF energy)
and also the experimental results on a wide variety of images
have verified the effectiveness of our method.

One interesting avenue of future work would be to extend our
framework so that it can be used for other types of completion
problems as well. It would be interesting to test our
framework on problems such as video completion or
geometric completion.
Also, in the future, we plan to allow the inclusion of
more refinement terms into our energy function. We believe
that this will be a very elegant and easy-to apply way for
allowing the user to impress either high-level or low-level
constraints on to the completion process.
In this manner, the user could gain a control over the
final result. Furthermore, this would make our method suitable
for problems such as constrained texture synthesis. Finally,
besides image completion, we also plan to test our Priority-BP
algorithm, which is a generic MRF optimization scheme, to
other labeling problems as well, for which the large
cardinality of their state-space causes them to have a very high
computational cost.

REFRENCES
[1]
A. Criminisi, P. Perez, and K. Toyama, Region
filling and object removal by exemplar-based
image inpainting, IEEE Transactions on Image
Processing, vol. 13, pp. 12001212, September 2004.
[2]
M. Bertalmio, G. Sapiro, V. Caselles, and C.
Ballester, Image inpainting, in ACM SIGGRAPH
2000. ACM, 2000, pp. 417424.
[3]
M. Bertalmio, A. L. Bertozzi, and G. Sapiro,
Navier-stokes, fluid dynamics, and image and video
inpainting, in IEEE CVPR 2001, vol. I. IEEE, 2001,
pp. 355362.
[4]
I.Drori, D. Cohen-Or, and H. Yeshurun, Fragment-
based image completion, ACM Trans. Graph., vol.
22, no. 3, pp. 303312, 2003.L.-Y. Wei and M.
Levoy, Fast texture synthesis using tree-structured
vector quantization, in Proceedings of SIGGRAPH
00. New York, NY, USA: ACM Press, 2000, pp.
479488.
[5]
P. Harrison, A non-hierarchical procedure for re-
synthesis of complex textures, in WSCG 2001
Conference proceedings, 2001, pp. 190197.
[6]
J. Jia and C.-K. Tang, Image repairing: Robust
image synthesis by adaptive nd tensor voting, in
Proceedings of Computer Vision and Pattern
Recognition, vol. 1. Los Alamitos, CA, USA: IEEE
Computer Society, 2003, p. 643.
[7]
N. Komodakis and G. Tziritas, Image completion
using global optimization, in IEEE CVPR 2006, vol.
I. IEEE, 2006, pp. 442452.
[8]
J. Sun, L. Yuan, J. Jia, and H.-Y. Shum, Image
completion with structure propagation, ACM
Transactions on Graphics (SIGGRAPH 2005), vol.
24, pp. 861868, 2005.
[9]
J. Portilla and E. P. Simoncelli, A parametric texture
model based on joint statistics of complex
[10]
D. J. Heeger and J. R. Bergen, Pyramid-based
texture analysis/synthesis, in SIGGRAPH, 1995, pp.
229238.
International Journal of Computer Trends and Technology (IJCTT) volume 4 Issue 5May 2013

ISSN: 2231-2803 http://www.ijcttjournal.org Page 1389

[11]
Y. W. S. Soatto, G. Doretto, Dynamic textures, in
Intl. Conf. on Computer Vision, pp. 439446.
[12]
W. Fitzgibbon, Stochastic rigidity: Image
registration for nowhere-static scenes, in ICCV,
2001, pp. 662669.
[13]
M. Szummer and R. W. Picard, temporal texture
modeling, in Proc. of Int. Conference on Image
Processing, vol. 3, 1996, pp. 823826.
[14]
G. Doretto and S. Soatto, Editable dynamic
textures. in CVPR (2), 2003, pp. 137142.
[15]
C. Ballester, M. Bertalmo, V. Caselles, G. Sapiro,
and J. Verdera, Filling-in by joint interpolation of
vector fields and gray levels. IEEE Transactions on
Image Processing, vol. 10, no. 8, pp. 12001211,
2001.
[16]
A. A. Efros and T. K. Leung, Texture synthesis by
non-parametric sampling. in ICCV, 1999.
[17]
T. Chan and J. Shen, Non-texture inpaintings by
curvature-driven diffusions, J. Visual Comm.
Image Rep., vol. 12(4), pp. 436449, 2001.
[18]
Q. Wu and Y. Yu, Feature matching and
deformation for texture synthesis. ACM Trans.
Graph., vol. 23, no. 3, pp. 364367, 2004.
[19]
M. Ashikhmin, Synthesizing natural textures, in
Symposium on Interactive 3D Graphics, 2001, pp.
217226.
[20]
A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless,
and D. H. Salesin, Image analogies, in
SIGGRAPH 2001, Computer Graphics Proceedings.
ACM Press / ACM SIGGRAPH, 2001, pp. 327340.
[21]
A. A. Efros and W. T. Freeman, Image quilting for
texture synthesis and transfer, in SIGGRAPH 2001,
Computer Graphics Proceedings. ACM Press /
ACM SIGGRAPH, 2001, pp. 341346.