Sindh Univ. Res. Jour. (Sci. Ser.) Vol.

48 (4) 741-746 (2016)

SINDH UNIVERSITY RESEARCH JOURNAL (SCIENCE SERIES)

Non Blinded Image Inpainting With Low Rank Non-Negative Matrix Factorization

A. AKBAR, M. SARIM, A. B. SHEIKH, N. M. LASHARI

Federal Urdu University ofArt Science and Technology

Received 6th January 2015 and Revised 10th March 2016

Abstract: In digital image processing object removal from a digital image is termed as inpainting. Image recuperation especially with
high curvature are inpainting challenges. Due to both colour and texture preservation non blinded exemplar based approaches are
famous among other inpainting algorithms. In general these algorithm possess greedy approach in the selection of a target patch, the
selection is based upon dedicated priority defined on every patch at contour of the target region, this may jeopardize searching and
selection of a good exemplar in global space. The whole process is time consuming and computationally expensive. To tackle said
problems a novel algorithm is proposed which reduces the dimension of global search space using neighbour patches information of
target region throughNon-Negative Matrix Factorization (NMF). Due to NMF nature of lower dimension approximation, improvised
efficiency is observed in an experiment without compromising the quality of inpainted image. The inpainted result of the proposed
algorithm is comparable with other inpaintingtechniques.

Keywords: Non-negative matrix factorization, exemplar

1. INTRODUCTION the target region is searched globally and partially

Digital image inpainting aims to recuperate an replicated to the corresponding target region (Ashkmin,
image in such a way an unsuspicious viewer would not 2001) (Crimnsie, 2003) (Bonet, 1997) (Hezy et al.,2003)
be able to detect the indication of object removal and (Leung, 1999) (Freeman, 2001) (Harrison, 2001) (Tang,
reconstruction. 2003) (Xu et al., 2001) while dealing with natural
textured image, these approaches find difficulties for
Image anomalies in digital images lead to extensive filling missing piece.With the fusion of both approaches
research. Existing inpainting methods are bifurcate as (M.Bertalmio, 2003) proposed a method which used both
variational and exemplar approaches. Many of texture and structure propagation but the resultant
variational approaches have utilized Bertalmio et inpainted image produces a blur and this method is
al(Bertalmio 2000) algorithm.Approaches (Chen and capable to inpaint only small target. Synthesis by
Shen, 2001) (Chen, 2001) (Tele, 2004) mainly deal with exemplar was first utilized by Harrison (Harrison,
structural properties and work at pixel level on input 2001) for large object removal, this technique considers
mask image. These approachestreat image as three the texture of the neighborhood pixel along the level
separate channels. For each channel these approaches lines(isophotes) but may suffer as a strong linear
interpolate information outside the mask along edge structure sometimes revoked by the neighbour noise.
isophote(the direction and intensity of pixels on target Freeman (Pasztor, 2000) has proposed non-hierarchical
edge) while preserving boundaries.Variation in color is procedure for re-re-synthesize an image by considering
approximated locally using partial differential close neighbour pixels locally with the help of a user
equations(PDE) in smooth fashion across the mask defined 2D mask having same texture.(Morel, 1998) has
region of image. Despite of computational demand PDEs proposed parallel hierarchical procedure in two levels
are widely used among research communities as PDE’s first by labeling texture map and second by segmenting
satisfies arbitrary values on the boundary of domain
when propagates small structure.
For large object recovery these approaches are
unsuccessful. Non parametric patch based exemplar
approaches are ideal while dealing with large objects as
they are supported by texture synthesis. In two
dimensional textural pattern the most compatible texture
fragment of the source region to the texture fragment of Fig.1: An original image, Mask and Inpainted image
++
Corresponding author, Email:[a.akbar,msarim,a.basit]@fuuast.edu.pk Email:[email protected]
 A. AKBAR et al. 742

the composite texture into sub texture. Criminsi the rows and columns of a given matrix. The matrices of
extended(Harrison, 2001) the combination of both two non-negative factors obtained by NMF are W 
textural and structural approaches in one model and  rd and H   dc as
define priority to each block centralized at the contour of
the target region, priority is defined on the basis of (1)
confidence and data terms. Wong (Wong, 2008)uses
global information obtained from several weighted Where, d is the decomposition factor. The factors W
similarity samples within the image, and these sample and H are obtained by minimizing the following
aggregate to fill the missing part but produces blurring objective function.
effects. Sangeetha et al, Muthu et al.(Sangeetha, 2011)
(Vallilka, 2013) extended (Criminisi, 2003) they , ∑ ∑ , , (1)
introduced additive weights to the priority term instead
of multiplication based priority. (Pirtika. 2014) have 3. PROPOSED INPAINTING USING
extended (Chen and Shen, 2001) by introducing NON-NEGATIVE MATRIX FACTORIZATION.
similarity metric based on an image gradient on the In an image I , the observed known source region
assumption that linear structure propagation to target is  , the unknown area target of an image I is
region on color basis is not sufficient. (Zhang, 2012) has
represented by  and target edge is  . The
extended method by defining priorities on contour on the
basis of color distribution. Xu and Sun (Sun, 2010) algorithm initiate from a patch  p localized at the target
adopted a sparse representation theory to fix the partial boundary having both known and unknown observation,
informative patch. Wangand Zhang (Wang, 2011) have on the assumption that the most reliable information can
given an idea of Weighted Sparse Non-Negative matrix be determined through neighbors of the target patch of an
factorization,first by defining the weight matrix in image, the neighbor region N is defined and searched
which 1 and 0 belongs to the source and target in global space. In search of good exemplar there is a
respectively,second by estimating similar patch using chance that the patch n  N selects itself, for avoiding
i
expectation maximization(EM) algorithm in which the
corresponding missing values are replaced at E-step and this confusion N and  are removed from source
unweighted nmf is applied in M-step. Mao and Saulgive region. Now source becomes
an addition of unweighted sparse constraint for
predicting the missing part as (Choi, 2009).This paper  =     N  (3)
proposed a novel approach to improvise efficiency by
utilizing the information of neighbor patches having all Four connected neighbors patches ni ,i = 1,2,3,4 are
pixels ON (no missing information)in praxis of Non
introduced to the selected target patch  p in clock wise
negative matrix factorization(NMF), refer section2.
Assuming that the most reliable information can be direction with their position z preserved refer (Fig.2).
obtained through neighbors to the target. These neighbor
patches have full structural and textural information, this
will revoke the need of weight matrix and sparse
representation like previous technique (Sun, 2010).
Fundamentals of proposed algorithm are in section 0,
result and comparisons on both synthetic and natural
inpainted images are under section 3.

2. METHOD FOR LOW RANK NON-

NEGATIVE MATRIXFACTORIZATION
Non-Negative Matrix Factorization has widely
used Low rank approximation (LRA) technique for the
dimension reduction of non-negative data matrix(all
elements of matrix are positive) was firstly introduced by
Lee (D.Lee, 2000), Formally given non-negative data
matrix is V = [Vx , y ]   rc , where x in matrix
Vx , y represents the dimension of the data and y is the
Fig.2: The Selected Target Edge Patch And Its Associated
number of sample. Indata matrix  rc . represents Neighbor Patches.
 Non Blinded Image Inpainting With Low Rank... 743

The first patch ni having all pixel ON or patch ni to  p is selected and its position z is
ni  N is used to search the almost identical patch preserved for future utilization the patch ni is also
 qn in the source region, from  qn the patch  q is T
converted to column matrix V . To find the objective
 qn
T T
selected which is at the compliment position z of coefficient H , factorize V with basis matrix
S
is d x 1. To find
T
. The partial part of the selected patch  q is utilized for W R d , Where the dimension of H
the most relevant patch in source space S , the
filling the missing portion of the target patch p and S T
coefficient of source space H and coefficient H
updates the target region. The algorithm continues till all
contain d feature of patch ni are searched using L2
patches in the target region  get updated. The
algorithm procedure is time consuming as the source -norm.
region is searched globally to find the similar patch. The | , | (5)
time issue the dimension of search space is reduced as
defined in next subsection. The patch  qn is localized at pixel qn is almost

A. Dimension Reduction Using NMF

identical to patch ni and assuming that patch  q is
A collection of non-negative patches localized at almost identical to p which is at position z to  qn ,
each pixel in search space  is formally define by
the corresponding pixels of q are propagated(filled
VRSC , refer(Fig. 3)
in) and update the target region  . This routine
continues till the whole target of the image is completely
filled in.
3. COMPARATIVE RESULTS
The proposed algorithm is tested on multiple
synthetic and real images having complex textures and
high curvatures. The visual and quantitative comparison
with othertechniques shows that the proposed algorithm
is capable to handle large object recovery more acurately
and efficiently. All testsare performed on 2.50 GHZ
machine.
A. Time comparison
Due to the dimension reduction nature of NMF .
Fig.3: The search space is created by the collection of all source
patches with RGB channels. The proposed algorithm performs faster than other
traditional techniques.Refer (Fig.4)for time comparison.
Patches of size a x b localized at all the pixels in
the source region  is defined as S . Those patches
having partial information localized at  and their
associated neighbor patches n are removed from S
as these patches are required for fill-front. Each patch in
S form column of two dimensional matrix , for
RGB color image R = 3  a  b and C is the number
of observation in the space S .The search space is
reduced by Non-negative matrix factorization using an
equation 2 as
, ∑ ∑ , , (2) Fig. 4: The above figureshows that the time taken by the proposed
algorithm for the recovery of both synthetic and natural images is
S less than the other techniques.
The obtained factor WRd is a basis matrix while  
H dSC is a coefficient matrix. To find the most relevant B. Quality comparison
Peak Signal to noise ratio is used for estimating an
patch to p in the source region, the first full neighbor image qualitybetween reference and inpaintedimage
A. AKBAR et al. 744

using equation(3).(Fig. 5) shows that the proposed (4)

algorithm produces better quality by alleviating the noise
effect using NMF. Where, , (8)
and MSE is the mean squared error calculated using
formula.
, , ,
∑ , (9)

C. Visual Comparison
Proposed algorithm is tested on various images,the
result shows that proposed algorithm recover images in
more acceptable view to unsuspicious eye while
comparing to other techniques.See (Fig.6) for visual
  comparison.
Fig.5: The above chart shows that the quality of recovered images
by the proposed algorithm is higher than other especially where
there is complex texture.
 Non Blinded Image Inpainting With Low Rank... 745

Original. Mask. Criminisi. Sangeetha. Proposed. 

Fig. 6: Visual comparison of different approaches and the proposed technique.

4. CONCLUSION technique which will recover the target region by

Removal of large objects and curved structures searching multiple images as the source.
efficiently is the main emphasis of this paper.An object REFERENCES:
removal without leaving an evidence of background Ashkmin, M., (2001). Synthesis natural texture. ACM
construction is a challenge, which proposed algorithm Symposium on interactive 3D Graphics, 217-226.
has attained through the neighborhood of the target
region in application of NMF. Our approach is based on Bertalmio, M., (2003). Simultaneous structure and
an exemplar and is capable to propagate both texture and texture image inpainting. In Proceeding of Comp. Vision
structure.The proposed method performs well in contrast and Pattern Recogination.
 

to previous restoration and object removing techniques Bertalmio, M., (2000). Image Inpainting. 27th annual
because of dimension reduction it performs faster than conference on Computer graphics and interactive
earlier algorithm. Currently proposed algorithm is techniques417-424. ACM Press/Addison- Weskey
extending to multi images with the dimension reduction publishing CO.
A. AKBAR et al. 746

Bonet, J. D., (1997). Multiresoution Sampling Procedure Pirtika, H., A. G. Patel, (2014). Improve Robust
for analysis and synthesis of texture images. In Algorithm for examlar based image inpainting.
proceeding of SIGGRAPH, 31, 361-368. International journal of Advance Research in Computer
Application, 101, 23-27.
Chen, T., J.Shen, (2001). Local inpainting models and
TV inpainting. SIAM Journal applied Maths, 1019-1043. Sangeetha, K., (2011). A Novel Exampler based image
inpainting algorithm for natural image scene completion
Chen, T., (2001). Non -Texture diffusion by curvature with improved patch priortizing. Int. Journal of
diffusion. Journal of Visual Communication and Image Computer application, 36, 7-12.
Representation, 436-449.
Saul, L.K.,(2004). Modeling distance in large- scale
Choi, S., (2009). Weighted Non-Negative Matrix Matrix networks by atrix factorization. ACM Internet
Factorization. In ICCASSP, 1541-1544. Measurement 278-287 SIGGCOM.
Criminisi, A., (2003). Object removal by examplar based Shum, Y. Q., (2001). Real Time Texture Synthesis by
inpainting. In proceeding of Computer Vision and patch based Sampling. ACM Transaction on Graphics
Pattern Recogination, 721-728. (TOG), 20, 127-150.
 
Drori, H. Y. I., (2003). Fragment based image Sun, J., (2010). Image Inpainting by Patch Propagation
compeletion. InProceeding of ACM Transition On using Patch Sparsity. International Conference on Image
Graphics SIGGRAPH, 22,303-312. Processing.19, 1153-1165. IEEE Transaction.
Ford, J., (2006). Learning from incomplete rating using Tang, C. K., (2003). Image repairing : Robust image
non-negative matrix factorization. International synthesis by adaptive and tensor voting. In Proceeeding
Conference on Data Mining549-553. SIAM. of Computer Vision and Pattern Recogination. Madison.
Freeman,T., (2001). Image quilting for texture synthesis
and transfer. In proceeding of SIGGRAPH, 341-346. Tele, A., (2004). An Image Inpainting technique based
on the fast marching method. Journal of graphics tool,
Harrison, P., (2001). A Non-hierarcial procedure of 09, 23-24.Vallilka,N., 2013.
re-synthesis of complex texture. In Proceeding of Central
Europe Comp. Graphics and Comp. Vision. Wang, Y. X., (2011). Image Inpainting Via weigthed
Sparse Non-Negative Matrix Factorization. 18th
Lee, D., (2000). Algorithm for n0n-negative matrix International Conference on Image Processing
factorization. NIPS556-562. MIT Press. 3409-3412. IEEE.
Leung, T., (1999). Texture Synthesis by Non Paramatric
Sampling. In Proceeding of Int.l Conf Computer Vision, Wong, O., (2008). A non-local means approach to
1033-1038. examplar based inpainting. Int. Conf. on Image
Processing ICIP2600-2603. IEEE.
Morel, J., (1998). Level LInes based disocclusion. Int.
Conf. on Image Processing. Chicago. Zhang, Q., (2012). Exemplar based image inpainting
Pasztor, E. C., (2000). Learning Low Level Vision. Int. using color distribution analysis. Journal of Information
Journal of Computer Vision, 40, 2547. Science and Engineering, 641-654.