2003 Siggraph Perez
2003 Siggraph Perez
313
object, associated edgels are removed; to add an object, associated S f∗
edgels as well as color values on both sides of each of these edgels f
are incorporated. The new image is then obtained by interpolat-
ing smoothly the colors associated to the new set of edgels. This Ω
amounts to solving a Laplace equation (a Poisson equation with a
null right hand side) with Dirichlet boundary conditions given by ∂Ω
colors around edgels. Editing edgels and associated colors is not v g
always simple. In addition, image details are lost when converting
to and from the contour domain, which might be undesirable. The Figure 1: Guided interpolation notations. Unknown function f
sparse edgel-based representation is indeed incomplete, as opposed interpolates in domain Ω the destination function f ∗ , under guid-
to the related representation based on wavelet extrema [Mallat and ance of vector field v, which might be or not the gradient field of a
Zhong 1992], which are complete but less adapted to manual edit- source function g.
ing.
In [Lewis 2001], spots are removed from fur images by separat- 2 Poisson solution to guided interpolation
ing out the brightness component from details in a selected region
and replacing the brightness by harmonic interpolation (solving a Guided Interpolation In this section, we detail image interpola-
Laplace equation) of the brightness at the selection boundary. tion using a guidance vector field. As it is enough to solve the inter-
polation problem for each color component separately, we consider
only scalar image functions. Figure 1 illustrates the notations: let
In terms of image editing functionalities, two existing techniques
S, a closed subset of R2 , be the image definition domain, and let
achieve seamless cloning as the basic instance of our system does.
c Photoshop
c 7’s Healing Brush [Adobe
c Ω be a closed subset of S with boundary ∂ Ω. Let f ∗ be a known
The first one is Adobe
scalar function defined over S minus the interior of Ω and let f be
2002]. To the best of our knowledge, the technique used by this
an unknown scalar function defined over the interior of Ω. Finally,
tool has not been published. Therefore, we don’t know whether or
let v be a vector field defined over Ω.
not it uses a Poisson solver.
The simplest interpolant f of f ∗ over Ω is the membrane inter-
polant defined as the solution of the minimization problem:
The second technique is the multiresolution image blending pro-
posed in [Burt and Adelson 1983]. The idea is to use a multires-
ZZ
min |∇ f |2 with f |∂ Ω = f ∗ |∂ Ω , (1)
olution representation, namely a Laplacian pyramid, of the images f Ω
of interest. The content of the source image region is mixed, within
each resolution band independently, with its new surrounding in where ∇. = [ ∂∂ x. , ∂∂ y. ] is the gradient operator. The minimizer must
the destination image. The final composite image is then recov- satisfy the associated Euler-Lagrange equation
ered by adding up the different levels of the new composite Lapla-
cian pyramid thus obtained. The technique results in multiresolu- ∆ f = 0 over Ω with f |∂ Ω = f ∗ |∂ Ω , (2)
tion mixing where finest details are averaged very locally around
where ∆. = ∂∂ x2. + ∂∂ y2. is the Laplacian operator. Equation 2 is a
2 2
the boundary of the selection, while lower frequencies are mixed
over much larger distances around these boundaries. This fast tech- Laplace equation with Dirichlet boundary conditions. For image
nique achieves an approximate insertion of the source Laplacian editing applications, this simple method produces an unsatisfactory,
in the destination region (on the first level of the Laplacian pyra- blurred interpolant, and this can be overcome in a variety of ways.
mid) whereas we perform this Laplacian insertion exactly via the One is to use a more complex differential equation as in the “in-
solution of a Poisson equation. More importantly, multiresolution painting” technique of [Bertalmio et al. 2000]. The route proposed
blending incorporates data from distant source and destination pix- here is to modify the problem by introducing further constraints in
els, via the upper levels of the pyramid, within the final composite the form of a guidance field as explained below.
image. This long range mixing, which might be undesirable, does A guidance field is a vector field v used in an extended version
not occur in our technique. In addition, our system offers extended of the minimization problem (1) above:
functionality besides opaque seamless cloning, see Sections 3 and ZZ
4. min |∇ f − v|2 with f |∂ Ω = f ∗ |∂ Ω , (3)
f Ω
Finally, whereas we propose a guided interpolation framework, whose solution is the unique solution of the following Poisson equa-
with the guidance being specified by the user, e.g., in the form of tion with Dirichlet boundary conditions:
a source image in the case of seamless cloning, various interpo-
lation methods have been proposed to fill in image regions auto- ∆ f = divv over Ω, with f |∂ Ω = f ∗ |∂ Ω , (4)
matically using only the knowledge of the boundary conditions.
A first class of such approaches is composed of inpainting tech- where divv = ∂∂ ux + ∂∂ vy is the divergence of v = (u, v). This is the
niques [Ballester et al. 2001; Bertalmio et al. 2000] where PDE- fundamental machinery of Poisson editing of color images: three
based interpolation methods are devised such as to continue the Poisson equations of the form (4) are solved independently in the
isophotes hitting the boundary of the selected region. The PDEs three color channels of the chosen color space. All the results re-
to be solved are more complex than the Poisson equation, and work ported in this paper were obtained in the RGB color space, but sim-
only for bridging fairly narrow gaps in relatively texture-free re- ilar results were obtained in CIE-Lab for instance.
gions. Example-based interpolation methods [Barret and Cheney When the guidance field v is conservative, i.e., it is the gradient
2002; Bornard et al. 2002; Efros and Leung 1999] where the new of some function g, a helpful alternative way of understanding what
image region is synthesized using an arrangement of many small Poisson interpolation does is to define the correction function f˜ on
patches are an interesting alternative to inpainting. These methods Ω such that f = g + f˜. The Poisson equation (4) then becomes the
handle large holes and textured boundaries in a more convincing following Laplace equation with boundary conditions:
way. Moreover, they can also be used to import textures as shown
in [Efros and Freeman 2001; Hertzmann et al. 2001]. ∆ f˜ = 0 over Ω, f˜|∂ Ω = ( f ∗ − g)|∂ Ω . (5)
314
Therefore, inside Ω, the additive correction f˜ is a membrane inter-
polant of the mismatch ( f ∗ − g) between the source and the desti-
nation along the boundary ∂ Ω. This particular instance of guided
interpolation is used for seamless cloning in Section 3.
min ∑
f |Ω hp,qi∩Ω6=0/
( f p − fq − v pq )2 , with f p = f p∗ , for all p ∈ ∂ Ω, (6) for all hp, qi, v pq = g p − gq , (11)
315
source/destination
Figs. 6 and 7.
swapped textures
source destination
316
source/destination seamless cloning mixed seamless cloning
for all hp, qi. As shown in Fig. 9, the content of the selection Ω
gets a flattened appearance, with small grain details washed out,
and the main structure preserved. The extent of this effect depends
obviously on the sparsity of the sieve. The more selective the edge
detector, the sparser the edge map, and the more pronounced the
effect.
Note that this instance of Poisson editing has strong connections
with the contour-domain editing system of Elder and Goldberg [El- Figure 10: Local illumination changes. Applying an appropriate
der and Goldberg 2001]. The difference is that we specify approx- non-linear transformation to the gradient field inside the selection
imately the gradient vectors at edge locations through sparse guid- and then integrating back with a Poisson solver, modifies locally
ance (14), whereas their system relies on an exact specification of the apparent illumination of an image. This is useful to highlight
color values on both sides of each edgel. under-exposed foreground objects or to reduce specular reflections.
317
image inside the selection, such as texture, illumination, or color.
An important common characteristic of all these tools is that there
is no need for precise object delineation, in contrast with the classic
tools that address similar tasks. This is a valuable feature, whether
one is interested in small touch-up operations or in complex photo-
montages.
Although not illustrated in this paper, it is clear that the cloning
facilities described in Section 3 can be combined with the editing
ones introduced in Section 4. It is for instance possible to insert
an object while flattening its texture to make it match the style of a
texture-free destination.
Finally, it is worth noting that the range of editing facilities de-
Figure 11: Local color changes. Left: original image showing rived in this paper from the same generic framework could prob-
selection Ω surrounding loosely an object of interest; center: back- ably be extended further. Appearance changes could for instance
ground decolorization done by setting g to the original color image also deal with the sharpness of objects of interest, thus allowing the
and f ∗ to the luminance of g; right: recoloring the object of interest user to make apparent changes of focus.
by multiplying the RGB channels of the original image by 1.5, 0.5,
and 0.5 respectively to form the source image. Image credits Two landscapes and swimming bear in Fig. 3,
c
flower in Fig.11: from Corel Professional Photos, copyright
2003
Microsoft Research and its licensors, all rights reserved; rainbow in
Fig. 7 courtesy Professor James B. Kaler, University of Illinois.
References
c c
A DOBE
. 2002. Photoshop
7.0 User Guide. Adobe Systems Incorpo-
rated.
BALLESTER , C., B ERTALMIO , M., C ASELLES , V., S APIRO , G., AND
V ERDERA , J. 2001. Filling-in by Joint Interpolation of Vector Fields
Figure 12: Seamless tiling. Setting periodic boundary values on the and Gray Levels. IEEE Trans. Image Processing 10, 8, 1200–1211.
border of a rectangular region before integrating with the Poisson BARRET, A., AND C HENEY, A. 2002. Object-Based Image Editing. ACM
solver yields a tileable image. Transactions on Graphics 21, 3, 777–784.
B ERTALMIO , M., S APIRO , G., C ASELLES , V., AND BALLESTER , C.
2000. Image Inpainting. In Proceedings of ACM SIGGRAPH 2000,
performed by precisely selecting an object and then setting its com- ACM Press / ACM SIGGRAPH, New-York, E. Fiume, Ed., Computer
plement to monochrome. In contrast, Poisson editing frees the user Graphics Proceedings, Annual Conference Series, ACM, 417–424.
from the tedium of precise selection: given a source color image g, B OLZ , J., FARMER , I., G RINPSUN , E., AND S CHR ÖDER , P. 2003. Sparse
(a) the destination image f ∗ is set to be the luminance channel from Matrix Solvers on the GPU: Conjugate Gradients and Multigrid. ACM
g, (b) the user selects a region Ω containing the object, and this Transactions on Graphics. to appear.
may be somewhat bigger than the actual object, and (c) the Pois- B ORNARD , R., L ECAN , E., L ABORELLI , L., AND C HENOT, J.-H. 2002.
son equation (10) is solved in each color channel. An example is Missing Data Correction in Still Images and Image Sequences. In Proc.
presented in Fig. 11. Note that, although the result seems to offer ACM International Conference on Multimedia.
also a precise segmentation of the object for free, this is not actually B URT, P., AND A DELSON , E. 1983. A Multiresolution Spline with Appli-
the case as there is some residual contamination of the destination cation to Image Mosaics. ACM Transactions on Graphics 2, 4, 217–236.
image outside the object. E FROS , A., AND F REEMAN , W. 2001. Image Quilting for Texture Synthe-
Conversely Poisson image editing can be used to modify the sis and Transfer. In Proceedings of ACM SIGGRAPH 2001, ACM Press
color of a loosely selected object. Before solving the Poisson equa- / ACM SIGGRAPH, New-York, E. Fiume, Ed., Computer Graphics Pro-
tion (10), the original image is copied to the destination f ∗ and a ceedings, Annual Conference Series, ACM, 341–346.
version with modified colors is copied to the source g, see Fig. 11. E FROS , A., AND L EUNG , T. 1999. Texture Synthesis by Non-Parametric
Sampling. In Proc. Int. Conf. Computer Vision, 1033–1038.
Seamless tiling When the domain Ω is rectangular, its content E LDER , J., AND G OLDBERG , R. 2001. Image Editing in the Contour
can be made tileable by enforcing periodic boundary conditions Domain. IEEE Trans. Pattern Anal. Machine Intell. 23, 3, 291–296.
with the Poisson solver. The source image g is the original im- FATTAL , R., L ISCHINSKI , D., AND W ERMAN , M. 2002. Gradient Domain
age, and the boundary conditions are derived from the boundary High Dynamic Range Compression. ACM Transactions on Graphics 21,
values of g, such that opposite sides of the rectangular domain cor- 3, 249–256.
respond to identical Dirichlet conditions. In Fig. 12, we have cho- H ERTZMANN , A., JACOBS , C., O LIVER , N., C URLESS , B., AND
∗
sen fnorth ∗
= fsouth = 0.5(gnorth + gsouth ), and similarly for the east S ALESIN , D. 2001. Image Analogies. In Proceedings of SIGGRAPH
and west borders. 2001, ACM Press / ACM SIGGRAPH, New-York, E. Fiume, Ed., Com-
puter Graphics Proceedings, Annual Conference Series, ACM, 327–340.
L AND , E., AND M C C ANN , J. 1971. Ligthness and Retinex Theory. J. Opt.
5 Conclusion Soc. Amer. 61, 1–11.
Using the generic framework of guided interpolation, we have in- L EWIS , J., 2001. Lifting Detail from Darkness. SIGGRAPH 2001 Tech
troduced a variety of tools to edit in a seamless and effortless Sketch.
manner the contents of an image selection. The extent of possi- M ALLAT, S., AND Z HONG , S. 1992. Characterization of Signals from
ble changes ranges from replacement by, or mixing with, another Multi-Scale Edges. IEEE Trans. Pattern Anal. Machine Intell. 14, 710–
source image region, to alterations of some aspects of the original 732.
318