320 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO.

3, MARCH 1995

A Contour-Based Approach to Multisensor

Image Registration
Hui Li, Student Member, IEEE, B. S . Manjunath, Member, IEEE, and S m j i t K. Mitra, Member, IEEE

Abstruct- Image registration is concerned with the estab- such as the dielectric constant [l]. Thus, one of the most
lishment of correspondence between images of the same scene. attractive aspects of multisensor data acquisition is the fact that
One challenging problem in this area is the registration of the sensors are collecting very different types of information
multispectral/multisensor images. In general, such images have
different gray level characteristics, and simple techniques such from the same scene. However, because the information from
as those based on area correlations cannot be applied directly. sensors of different types are inherently different, the problem
On the other hand, contours representing region boundaries of registration is inevitably more complex than registration of
are preserved in most cases. In this paper, we present two images from the same type of sensors. Images from different
contour-based methods which use region boundaries and other channels of the multispectral imaging systems (e.g., Landsat
strong edges as matching primitives. The first contour matching
algorithm is based on the chain-code correlation and other shape thematic mapper (TM), Spot and NOAA advanced very high
similarity criteria such as invariant moments. Closed contours resolution radiometer (AVHRK)) bear unique characteristics
and the salient segments along the open contours are matched determined by the wavelengths at which they are acquired.
separately. This method works well for image pairs in which Substantial differences exist between the images from passive
the contour information is well preserved, such as the optical
images from Landsat and Spot satellites. For the registration of sensors such as Landsat TM and active sensors such as
the optical images with synthetic aperture radar (SAR) images, synthetic aperture radar (SAR). Comparing a Landsat image
we propose an elastic contour matching scheme based on the to a Seasat SAR image, the discrepancies may include cloud
active contour model. Using the contours from the optical image cover in the Landsat image, grainy appearance of the SAR
as the initial condition, accurate contour locations in the SAR image resulting from speckle noise, and radiometric reversal
image are obtained by applying the active contour model. Both
contour matching methods are automatic and computationally in the relative brightness between them [ l ] .
quite efficient. Experimental results with various kinds of image Existing image registration techniques broadly fall into two
data have verified the robustness of our algorithms, which have categories: the area-based and the feature-based methods [2],
outperformed manual registration in terms of root mean square [3]. In the area-based method, a small window of points in the
error at the control points. first image is statistically compared with windows of the same
size in the second image. The measure of match is usually
I. INTRODUCTION the normalized cross-correlation. The centers of the matched

I N many image processing applications it is necessary to

compare multiple images of the same scene acquired by
different sensors, or images taken by the same sensor but
windows are control points which can be used to solve for
the transformation parameters between the two images. One
innovative method in the recent literature is the computational
at different times. These images may have relative trans- vision approach proposed in [4]. In this approach, image
lation, rotation, scale, and other geometric transformations rotation is obtained by taking the difference between the
between them. The goal of image registration is to establish estimated illuminant directions, under the assumption that
the correspondence between two images and determine the the images are taken at about the same time (or under the
geometric transformation that aligns one image with the other. assumption of a stationary source of illumination). Feature
Registering multisensor data enables comparison and fusion points in the images are detected by using a biologically
of information from different sensory modalities, which often motivated model based on the Gabor wavelet decomposition
provide complementary information about the region surveyed. [ 5 ] . The windows surrounding the feature points are matched
For example, optical images from Landsat provide information with each other using an area-based correlation in a hierar-
on chemical composition, vegetation, and biological properties chical manner. However, area-based methods like this are not
of the surface, while radar images from Seasat provide infor- well-adapted to the problem of multisensor image registration
mation on surface roughness and surface electrical properties since the gray-level characteristics of images to be matched
Manuscript received March 21, 1993; revised November 15, 1993. This are quite different. Feature-based methods, which extract and
work was supported by the Lawrence Livermore National Laboratory and by match the common structures from two images, have been
a University of Califomia MICRO grant, with matching funds from ESL Inc.. shown to be more suitable for this task [6]-[9]. One technique
Digital Instruments, Inc., Tektronix, Inc., and Rockwell Intemational Corp.
The associate editor coordinating the review of this paper and approving it uses ancillary data such as high resolution digital elevation
for publication was Prof. Rama Chellappa. maps (DEM) of the area to be registered [6]. Simulated
The authors are with the Center for Information Processing Research, multisensor images with shadowing effects are generated by
Department of Electrical and Computer Engineering, University of Califomia,
Santa Barbara, CA 93106 USA. illuminating the DEM from sensor imaging geometries. The
IEEE Log Number 9408198. actual images are registered with DEM by correlating with
1057-7149/95$04.00 0 1995 IEEE

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 rhe simulated data. A\ ;I result. the multisensor images are Landsat. Spot. and Seasat images. The proposed contour
ioregistered on the coininon grid provided by the DEM. The matching algorithms are robust and accurate if the boundary
~ u t h o r si n [ 61 a l s o suggc\t using ii segmentation scheme for information is preserved through different sensory modalities
.nultisen\or region bountlar!, extraction and the Canny edge and if this information can be successfully extracted. The
(detector for edge detection. The boundary o r edge maps from proposed method\ have outperformed manual registration in
Landsat. Spot and Seasat arc theti matched by a binary cross- terms of root mean square error (RMSE) at the control points
correlation technique. The assumptions of this approach are i n our computer siniulationj.
rhat the image\ to be rcy5tered arc coarsely aligned based This paper is organized as follows: Section I1 discusses
on prior information and that the re\idual rotation error is the basic contour matching procedure; Section 111 reviews the
,mall. The scheme proposed i r i [7]extracts objects from active contour model and discusses the optical-to-SAR image
Landsat and Spot images at diflcicnt scales and matches them registration scheme using this model; Section IV presents the
.ising their structural attributes \uch as ellipticity. thinness, experimental results: and Section V provides thc concluding
m d inclination. I n [ X I c loscd boundarics are extracted by remarks.
.I \egnientation tcchniquct and ujed as matching primitives:
7owever. the good control points on the nonclosed regions are 11. TIII-CO\IOLIKMATC'HING
ALGORITHM
q l e c t c d . In [ Y I 21 contour-based mcthod for registering Spot
m d Seasat images is proposed. A long coastal line is used as A , Corzroirt-E.~trwtioti
.I landmark and the matching is conducted in a coarse-to-fine Fig. I shows the bloch diagram of the proposed basic
l.a5hion u\ing a scale \pace rcprecntation. Edge-and contour-
contour matching scheme. Contour extraction is carried out
based techniques ha\ e also been used in obtaining stereo
i n two steps. First. images I(./..;(/) are convolved with a
(:orrespondence [ IO]-[ I 3 1. but typically they aswine small Laplacian-of-Gaussian (LOG)operator and the edges are de-
t.li\placements to constrain the earch space. The structural tected at the xro-crossing points [2]. The LOG operator i5
dereo matching techniqiie proposed in [ 141 is capable of decomposed into the sum of two separable filters to speed
mitching images with Iarye rotation und translation. However,
up computation 1 161. Simple rero-crossing patterns such a s
!.hismethod is better suitecl for registration of industrial images
rather than thc natural remotely \ensed images from which a
"+ + - -" and '.- - + +" (composed of signs of pixel values
of the tittered image) are detected along both vertical and
\ymbolically reprewired ,triictur;il scene description may not horizontal directions. In the second step. the edge strength
he casy to establish.
at each zero-crossing point is computed by considering the
In this paper. we pre\ent two contour-based image rcg-
slopes of the LOG of the image along both .I' and g directions
stration algorithm,: ;I b;i\ic coritour matching scheme and (denoted by A.,. arid A.,). An edge strength map is defined as
, i n elastic contour matching schcmc for optical-to-SAR i n -
follows:
,ige registration. The propo5eti cherncs are automatic and
iomputationally quite efficient. The basic contour match- !,~{ Jl.?(,+
= >, A.' if ( . I . . g ) is a zero crossing point.
ing scheme uses a two-threshold method to extract well- othew ise.
defined contours and separate them into two groups: open (1)
iontour'\ and clo\cd cotitours. ('hain code correlation and The contours that are retained for further processing satisfy
other shape similarity criteria such a5 moments are used to the following conditions [ 171:
match the closed contoiirs. For the open contours. salient i ) The edge strength at each point along the contour is
\egnients such as comer\ are cietecteci first and then used in greater than 7'1
the matching process. A consistency check is conducted in i i ) At least one point on the contour has an edge strength
the transformation parameter space. This first method works greater than 75
,.vel1 for image pairs in which the contour information is where 'I; and T?are preset thresholds and Ti< 7). 7; is set to
uell preierved. such as the optical images from Landsat preserve the whole contour around the region boundary with-
d Spot satellite\. Because of the coherent nature of the out incurring discontinuities at weak edge points. 7; is chosen
illumination. rxiar image< arc speckled. As a result, the usual large enough to avoid spurious edges. This two-threshold
d g e detectors yield poor results when applied to radar images scheme is implemented by scanning the 7-D edge strength
1c.g.. SAR images) [ I S ] . In this paper, a second method is array. Contour search is initiated whenever one point with a
developed for optical-to-SAR image registration. Rather than value greater than f i is scanned. The search is conducted in
cxtracting contours From optical and SAR images separately both directions of the contour and the neighboring pixels with
and matching them later. a hettet- strategy, as proposed here. values greater than TIare accepted as contour points until no
is to use the optical coiltours a > initial guides for contour neighboring pixel5 are found satisfying this condition. Then
cletection in the SAR images. Assuming that the optical and all the edge strength values along the detected contour are set
Ihe SAR images can be cmarsely aligned. the active contour to zero such that these points will not be visited again. The
tnodel provides an ideal tool for [his task. Another benefit of same search operation continues until the whole edge strength
the proposed elastic matching technique is its ability to handle array has been scanned. The contours are then divided into
deformation between the images t o be matched. We have two categories+losed contours and open contours-and are
Lipplied our algorithms oii three test data w t s : single sensor represented by a chain code. Short contours that cannot be used
image\. 1 ?-band multispectral aei-ial images. and multisensor reliably in the matching process are discarded at this point.

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 322 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 3, MARCH 1995

---- Encoder Matching

2oE
I I 1 O t YC
lranEfo,,,,ed-
IO align with
interpolation
-
--
image A

Fig. I . Schematic diagram for the basic contour matching method.

B . Chain Code Criteria -50

(a)
A digital curve can be represented by an integer sequence
{ a , E {0,1,2,. . . .7}), depending on the relative position
6 6
of the current edge pixel with respect to the previous edge
pixel [ 2 ] , (181. One unit corresponds to an angle of 45'. 4 4
Thus a chain code value of 3, for example, indicates the next
2 2
pixel is on the north-west (135") direction. The standard chain
code representation has certain drawbacks. For example, a line n n

along -22.5' direction is coded as (707070. . .}. To prevent

such wraparound, we convert a length n standard chain code
(ala2 . . . U , } into a modified code { b l b z . . . b,} by a shifting
operation defined recursively by:

{
61 =
b, = q L .q2 is an integer such that (4%- a,) mod 8 = 0
and 1q2 - b,-l I is minimized, i = 2,3, . . . n .

The line along -22.5' direction is then coded as (787878. . .}.

The shifted chain code is further smoothed by a five-point
Gaussian filter {O.l, 0.2,0.4.0.2,0.1}. A comparison of the
(2) Pml 100 200

Fig. 2. Modified chain code representation: (a), (b) A pair of contours

extracted from images shown in Fig. 4(a)-(b); (c), (d) the corresponding
standard chain codes and the modified chain codes for a pair standard chain code representations; (e), (f) the modified chain codes after
of matched contours' is shown in Fig. 2. From here on it is shifting and smoothing.

assumed that the chain code has been shifted and smoothed.
Let contour A be represented by an NA-point chain code the mean-squared-error of two signals. The cosine function
( a , } and let contour B be represented by an NB-point chain ensures D k l 5 1, and Dkl = 1 when there is a perfect match.
code { b 2 } . A correlation measure D k l between two n-point In order to identify the location of best fit between contours
segments, one starting at index k of contour A and the other A and B , an n-point segment from contour A starting at index
one starting at index 1 of contour B , is defined as: IC is slid over contour B. A similarity function CABis then
defined as:
(3)

where M specifies the search range. For a pair of closed

contours A and B , the whole contours can be used for the
matching purpose. Suppose that IC = 0, n = N A , and that
M includes every index of contour B. Then the similarity
function between contours A and B becomes:

1 ,-l
b{+t = b ( l + i ) m o d N f l - -
n
b ( l + ~ ) m o d N> 0
~ _< i < 71. ( 5 ) The rotation between the contours is reflected in the difference
3 =O
between the average values of the corresponding chain code
Here, the modulus operation accommodates the cases in which representations (see Fig. 2(e)-(f)). Since D k l is normalized
the contours are closed. The correlation measure is similar to with respect to the mean value, the similarity function CAB
'
This pair of contours was extracted from the images in Fig. 4 ( a t ( b ) and is invariant to rotation if the quantization effect of the chain
can be found in the contour maps in Fig. 4 ( c H d ) . code is neglected. The derivative of the chain code, obtained

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 hy using the first difference. is alia rotation invariant. How- measure of curvature at the ith point as:
ever. wc found that usin? the mean-subtracted chain code ('/ =
< < 9nI ,
III1lX { l l l i , X ( ~ / 1 , ~ , ~ f , + , I . l U
~ ,..,
fl,+,/-]l}}
~ (IO)
i, inore accurate in locating the poiitions of best matches.
I or the closed contour thc similarity function can be further where o i s the standard deviation of the LoG operator used
iiormali/ed with re\pect to contour length. and becomes scale for the contour extraction. The ith point along the contour is
invariant. In implementation. the chain code of the longer chosen as a salient point if both of the following conditions
closed contour is simply rcsampled by linear interpolation to are satisfied:
have the same number of point\ ah the shorter closed contour. i ) r , 2 T4.and
Subsequently. the similarity criterion i',4[1 is computed based ii) rt 2 for all X. E [ / p . / p ] .~ +
on the resampled chain codes of the same length. Here p is a con\tant that determines the minimum distance
between the salient points and Y; is a threshold specifying the
minimum acceptable curvature. For example. if T4 = 2, then
the salient feature locations correspond to comers where the
'. C ' O I l r / J l r I . h ! f l r ( ' h l l l ~
curve bends by at least 90". The contour segments surrounding
The contour matching process begin\ with the matching the salient points are then used as 1 -D templates in finding the
of closed contours. For cvcry c l o d contour. five shape corresponding matches in the other image.
attributes are computed: the perimeter. the longest and shortest
distance5 from boundary to the centroid, and the first and
D . Esriniutioti c!t 7i~~~ti.sjor-nicrriotr
t'utx"tli.s
kecond invariant moments proposed by Hu 121. These moments
were defined originally t'or 7-D images. For the case of 2- I t is often sufficient to assume that the relationship between
I > contours. the tirst and \econd moments can be defined as a point (S. in one image and its corresponding point
f 01low\: (-?.i-)in the other image can be expressed by a 2-D aftine
transform 141:

j = I
r
where parameter\ ( K . H . 1S. 11') correspond to a scaling
factor, a rotation angle. and translations along the two or-
thogonal directions. respectively. The transform parameters
t 7
{r' [!I,
Xi./., ~

Here , I , , and !/, are the coordinates of each point along the
cxlntour. .r( and g, are the. coordinates of the centroids of the
I?
.r, ) ( ! I ! % ) . (9) are estimated from a set of matched points {
{ [ Ai7,. f; ) }. The problem can be solved through linearization
of the above equations [ A ] . Suppose h- and H can be estimated
based on prior knowledge by fi,l and 0,). and that H = Ho +AH.
1; ) } and

contour. and I / is the length of the contour. Each closed contour Then the following two approximations can be established:
in the tirst image is compared with every closed contours in K, ('os H zz 1'0sH ( , - ( t i 0 sin H,, )AH, and K sir1 H K O sin 0,) +
the second image. The pairs are accepted as candidate matches ( t y 0 ('os H O ) l H . Next, the unknowns (AH. A S . 11') can be
it' the relative differences of theii shape attributes are below solved by the least square5 method. Another method for
wme preset thresholds (c.g.. 2 0 5 ). This step narrows down solving ( I 1 ), as adopted in our implementation, uses variable
the prospective matches to a feu contours with very similar substitutions = ti (.os H and i' = K sill H to obtain the
\hapes. The corresponding pair is theii determined by the chain following new equations:
code based similarity fuiiction ( ' 4 1 3 . Contour ..I in the tirst
image and contour U in the second image are selected as a
matched pair if the following two condition\ are satisfied: Unknowns 11. 1 % . la\. and 11.can be solved in the least
I) i ' i ~2 j i ' 4 1 1 , . where 13' includes all the contours with squares sense based on all the matched points. Scaling factor
similar shapes to contour :l, and K and rotation H can then be determined from I ( and r :
7 ) ( ' l f j > '/:j. whew is a preset threshold which ti = Jm. and H = i i u , t a i i ( i ) for -5 < H < 4.
eliminate\ matches with poor correlation. More general transformation\ such as bivariate polynom&l
In the rare case that niultiple contours get matched to the transformations are more effective if nonlinear distortion exists
\ame contour. the pair with highest value of ( ' . ~ I J Ii s selected. between the image\ I 191. [NI.
The centroids of these matched contour.; are used as control The root mean square error ( R M S E ) between the matched
points from which an initial estimation of the transformation points after the transformation provides a measure of registra-
p"ieters i \ computed to guide the second stage of open tion accuracy and is defined as:
contour matching. lrl

E[(ciS, + (11; + 1S
For matching the open contours. we use salient segments
along the contours rather I han the whole contours as matching
primitives. Salient segments wch as corners can be detected
RAISE =
( ,=I
- .q,)'

troni the chain code reprcsentation. For a contour of length -tI ( I ~ ;- /,A-, + 11' -

with shifted and smoothed c h i n code { U , } . we define a

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 324 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 3. MARCH 1995

where m is the number of matched points. It is important to within 100 meters or 5 pixels. Based on this assumption, the
mention that m must be reasonably large to make the accuracy contours from the optical images can be directly superimposed
measure meaningful (note that RMSE = 0 when m = 2, over SAR images as prior information. The second contour-
whether or not the matched points are correct). matching algorithm proposed in this paper involves the use of
an active contour model [25], [26] for optical-to-SAR image
E . A Consistency Check registration. We first briefly review the idea of the active
contour model and discuss our modification, and then apply it
In the contour matching process some false matches are
to the problem of image registration.
inevitable. Therefore, a global consistency check is necessary
The active contour model is an elastic contour extraction
to ensure correct registration. We exploit the fact that distances
technique that makes use of prior information about the shape
are preserved under a rigid transformation. Our consistency
of the region boundary to obtain a close fit to the data. The
check method - is closely related to the Hough transform
use of prior information helps in robust detection of contours
technique. Let A1A2 denote the distance between points A1
while minimizing the undesirable effects of noise and texture
and Aa. For -__
two sets of m matched points { A , } and {BP},
in the image data. The elastic contour extraction is usually
ratios of A,Aj/B,B, are computed based on all
formulated as an optimization problem involving an energy
possible combinations. The resulting histogram of scale should
functional and an initial contour which specifies the region of
form a cluster corresponding to the true scale difference
interest. The energy functional can be decomposed into three
between the images. The pairs that contribute to the cluster
parts: internal, image, and external energy. The intemal energy
will be accepted as correct matches while the pairs whose
regulates the smoothness and deformation of the contour while
contribution is scattered and away from the peak are declared
the image energy attracts points on the contour toward the true
as mismatches and discarded. The actual consistency check
edge locations in the image. The extemal energy influences the
is done in an iterative fashion. The most likely mismatches
contour deformation through feedback from other sources of
are rejected first, followed by the computation of RMSE
information. In recent years various elastic contour extraction
based on the remaining matched points. If RMSE is too large,
methods have been reported, based on variational calculus
another round of consistency check is carried out. The iteration
[25], dynamic programming [27], and a local descent method
continues until either an acceptable RMSE is achieved or the
(greedy algorithm) 1261.
number of retained matches is less than three, which means no
We review here the greedy approach described in [26] which
matched points can be found. Since RMSE is a good indication
considers only the intemal and the image forces. Let a contour
whether a set of correct matched points has been found, this
be represented by a sequence of vectors: V(S) = (z(s), y(s)),
scheme is capable of eliminating most false matches.
where z(s) and y(s) are the pixel coordinates. The index range
is 1 5 s 5 N, where N is the number of points. Let I ( z , y )
111. OPTICAL-TO-SARIMAGEREGISTRATION be the image function. An energy functional corresponding to
USING AN ACTIVECONTOUR MODEL the contour is defined as:
The presence of speckle noise due to the coherent nature N
of the illumination makes it difficult to detect good edge and E = a(s)Econt +
P ( s ) E c u r v - Y ( S ) E i m a g e (14)
contour information in SAR images [9], [15], [21]. Various Y=l
smoothing techniques [22], [23] can be applied before edge
detection, but this in tum may affect the accuracy of the
contour location. Similar performance limitations are charac-
teristic of statistical edge operators [ 11. Good results based on
the Canny edge detector were reported in 161; however, many
contours corresponding to continuous boundaries were broken.
In order to match such contours with contours from an optical Ecorlt and E,,,, represent the intemal energy of the contour.
image, sophisticated heuristics will be needed to perform edge E,,,,t maintains a uniform distribution of points on the contour
linking for the contours from the SAR image. The LOG edge while E,,,, enforces a smoothness constraint. The image force
detector can produce unbroken contours, but also generates Eimageis the gradient magnitude. a ( s ) , P ( s ) , and y(s) are
false contours because of the textural patterns present in the weighting parameters. Starting with the initial contour
the SAR images. Consequently, for optical-to-SAR image locations, the neighborhood of each point on the contour is
registration the basic contour matching method presented examined to obtain a potential new location for that point.
earlier can only work in limited cases and produces much The contour point is moved to the new location if it results
fewer control points than optical-to-optical image registration. in a lower energy. The iterations continue until a minimum
In contrast to SAR images, optical images have fairly well energy is obtained.
defined edges and hence are easier to work with. Here, we Since edge detectors such as the LOG operator or the Canny
suggest the use of edge information provided by optical images detector usually include a Gaussian smoothing operation to
to better localize the contours in the SAR images. It was determine the scale of the edges to be extracted, it is appro-
shown in [6] and [24] that multisensor data such as Landsat, priate to also incorporate this feature into the image force.
Spot, and Seasat images can be rectified and geocoded to a Considering the speckle noise in the SAR images, we chose
common earth grid and resampled to the same pixel spacing a geometric filter [22] combined with a LOG operator for the

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 image force. The geometric filter works ;IS follows: For any
tliree consecutive pixels dong the horimntal direction, the
e n t e r pixel is decreased b\ 1 if its intensity value is larger than
ils t w o neighbors: wherea... it is increased by I if its intensity
\ d u e is smaller than its ricighbor\. The same operations are
also applied to any three consecutive pixels along the vertical
and two diagonal directions. Multiple iterations of the above
operations are iisually reqtiired. The modified image force is
defined by ;I 2-D convolution as ihllows: it is easy to obtain a set of control points. For the cloaed
contour\ the centroids are used as control points. For the
open contours salient segments \iich as corners are detected i n
n here ( ; ( . I , . !/ j is a 2-D G,iussian function and S'( . I , . y ) is the one contour and are matched with the other contour using
eilge strength map as delined in ( I ). The modified image the similarity function defined in (6). The resulting set of
torce provides a larger bom.1 of attraction. hence enabling control points is hub,ject to ;I consistency check and is used to
convergence to ;I better local niinimuni o f the corresponding compute the least q u a r e error edniates of the trunsformation
energy (unctional. parameters.
For the problem of optical-to-SAR image registration, the
contour5 cktracted from thc optical image can be used as the
it?itial condition tor the contour detection of the SAR image. In
this way the active contour model can be implemented without I v . EX PF-. K I \I I-. \ T.4 I , K t S l i I TS

any human intervention. ,is mentioiied earlier. multisensor In this section. we t i n t provide experimental results for
images can be roughly aligiied ( i n inany cases) within 5 pixels single sensor image registration. multispectral image reg-
based on the knowledge of the sensor platforms. Thus. the istration, and multisensor optical image registration using
contours from the optical image can be directly superimposed the basic contour matching scheme. Although o u r emphasis
o \ w the SAR image. is on multisensor registration. we include a iingle sensor
I f the knowledge of the sctisor platforms is not available and example to illustr;ite the robustness of the algorithni for images
the rectified multisensor data are n o t initially aligned wcll, the without distinct features. We next ihow results ot' optical-to-
optical and SAR image\ h;ive to bc%preregistered manually or SAR (Landsat-to-Seasat and Spot-to-Seasat) image registration
i i G g \ome other automatic algorithms. I n our experiments based on the active contour model. The standard (leviation o f
LI i~ have used either the Ihasic contour' matching algorithm the LOG operator is 3 for 1111 examples. The thresholds for
presented earlier or the I.ourier transform-based technique the two-threshold contour extraction scheme are fine-tuned
pi esented in 12%I to pertorni the prcregistration. The schematic to each type of image. The typical values are TI = 10 and
diagram of the overall irna2e regislration system in which the Ti = 100. when the maximum edge strength i \ normalized
basic contour matching sclieme is adopted as the tirst stage to 255. The parameters for the corner detection art' set to
matching i \ shown i n Fig. 1. In t h i \ case a heavily smoothed T J = 1.8 and p = 2.5. The length of salient segments used
SAR image i \ used instead o f the oriFinal one for the contour in open contour matching is 31. The weighting parameters
ex traction. for the active contour model are constants: rr(.s) = 0.1.
When the optical image has a higher resolution than the ;I(..) = 11.8. and ;,(.q) = 1.2. To evaluate the performance
SAK image. quite often some optical contours have no cor- of our algorithms. we manuall\. \elected distinct lantimarhs
rebponding region boundaric\ i n the SAR image. To discard as control points. .A comparison of transformation parameter4
mistakes caused by the spurious initial contour. ii converged and RMSE obtained by manual regi5tration and contour-based
contour produced by the x.tive contour model algorithm is automatic registration is shown in Table I. The number of
+
accepted o n l y if the averiigc image torcc ( E,'=,II,,,,,,,,.)is manually selected control points and the number of control
above a preset threshold. Tti I \ step helps to ensure that enough points generated b j the proposed algorithms ;ire the same
contour' points lie over or low t o the true edge locations, as t o ensure ii fair comparison of RMSE. Our algorithms are
indicated by the image torn,. At t h i x point. the correspondence reasonably efficienl in terms of computational complexity. The
between contours in the two image\ is hnown (hut not point-to- size o f the images in all examples is 5 17- x 5 12. The contour
point correspondence). From this set o f corresponding contours extraction and contour matching take about YO t o 100 s on

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 326 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4. NO. 3, MARCH 1995

(e)
Fig. 4. (a). (b) Two images from a Mojave Desert sequence; ( c ) . (d) the matched contours with the"+" denoting the centroids of the closed contours and
the salient points along the open contours: (e) mosaic of (a) and (b). (Test data courtesy of JPL, Caltech.)

a Sun Sparc2 workstation. The active contour model used in Single-Sensor Optical Image Registration: Figs. 4(a) and
optical-to-SAR image registration takes about 20 to 40 s. (b) show two images from a Mojave Desert sequence taken

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 with an optical camera. Twenty-four pairs of‘ closed contours matched. Fig. 4(c )-(d) s h o u the matched contour\, with “ f ”
ancl wven pairs of salient points along open contours were denoting the centroidx of closecl contours and the matched

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 328 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 3, MARCH 1995

Fig. 6. (a), (b) Landsat image and a Spot image, respectively; (c), (d) matched contours of (a) and (b); (e), (0 matched contours of the Landsat image
and the 90O-rotated Spot image.

segments of open contours. Fig. 4(e) shows the mosaic of thematic mapper (TM) simulator was used in this experiment.
(a) and (b). The size of each raw image is 1000 x 766 and the
Multispectral Optical Image Registration: A set of 12- images are perfectly registered with each other. Test images
band aerial images of a mountainous area acquired by a (512 X 512) were cut out from different bands of this

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 -* :-
-t--

3'

IO

dn~,i \et after translation5 and roiation. In thi\ way. the successl'ully for the C;ISCS in uhlch at least half of the irnage
grouiid truth of the transfc~rmation par;meters bctween the extents overlap. One example is \hewn in Fig. 5. where
t w images is known. Thc propowd algorithm perlormed ( a ) anti ( b ) correspond to the reroth and eighth bands of'

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 330 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 3, MARCH 1995

I
,

Fig. 8. (a). (b) Landsat image and a Seasat (SAR) image, respectively: (c) contour map of (a); (d) contour map of (b); (e), (f) correspondence between
Landsat and Seasat contour maps in the first stage of the matching process.

the 12-band data set. Six pairs of closed contours and 9 Fig. 5(e). It can be seen from Table I that the estimated
pairs of salient segments have been matched, as shown in transformation parameters are very close to the ground
Fig. 5(c)-(d). The mosaic of the matched images is shown in truth.

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 i

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 332 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 3, MARCH 1995

Fig. 9. (a), (b) Spot image and a Seasat (SAR) image, respectively-in (b), the contours from the Spot image are superimposed as the initial condition
for the active contour algorithm; (c) converged contours; (d) blow-ups of three contours before and after convergence; (e), (0: matching results, with the
"+" marks denoting the centroids of the closed contours and the salient segments of the open contours.

our method again. Twenty-seven matched points have been Fig. 6(e)-(f). Fig. 7(a)-(b) shows the same Landsat image
found in this case and the matched contours are shown in and the scaled-up version of the Spot image. Seven pairs of

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 L1 e( U/ : CONTOUR-RASED APPROACH TO MULTISENSOR [MACE REGISTRATION 333

matched closed contours are identified in Fig. 7(c)-(d). The schemes for image registration. In the first scheme the contours
matched images after transformation are in Fig. 7(e)-(f). are detected separately from two images and are matched at
Multisensor Optical-to-SAR Image Registration: The first a later stage. A chain-code-based matching algorithm and an
example involves a Landsat (optical) image and a Seasat iterative consistency checking scheme have been developed.
(SAR) image. Both have been rectified and resampled to For single-sensor and multisensor optical image registration,
the same pixel spacing as shown in Figs. 8(a)-(b). There is the first algorithm is quite robust and reliable when the
considerable displacement between them, such that the active corresponding contours are available. It can handle images
contour model cannot be directly applied. Fig. X(c)-(d) show with large rotation and translation, and images with scale
the edges extracted from the optical image and the despeckled difference if there exist sufficient numbers of closed regions.
SAR image. It can be seen that the SAR edge map contains A second scheme is developed for optical-to-SAR image
many spurious contours. The result of initial registration using registration. Assuming that multisensor images can be roughly
the basic contour matching scheme is shown in Fig. 8(e)-(f) aligned by rectification and geocoding, an elastic matching
in which nine control points have been found. Fig. 8(g)-(i) scheme based on the active contour model is proposed. The
show the smoothed SAR image (geometric filtering, 8eight motivation here is to tackle the following two problems: (a)
iterations) and the image force for the active contour model, The difficulty in extracting well-defined, unbroken contours
respectively. In the fine tuning stage, a total of 19 contours from a noisy SAR image; and (b) the deformation between
from the Landsat image are transformed and placed over images taken at different times by different sensors (e.g., river
the SAR image, as shown in Fig. 8(i). These contours are boundaries can change significantly after seasonal flooding).
next used as the initial condition for the active contour We found that the image force of the active contour model
algorithm. Due to the difference between the resolutions of had to be computed based on the smoothed SAR image
two sensors, not all initial contours from the optical image in order for contours to converge correctly. The filtering
have corresponding region boundaries in the SAR image. inevitably affects the accuracy of the contour locations. The
In this example, 12 converged contours are accepted after a active contour model also tends to smooth the sharp comers.
thresholding test with respect to the averaged image force. Nevertheless, the registration error incurred by blurred contour
and these contours are shown in Fig. SU). Fig. 8(k)-(l) show locations is compensated to some extent by using the centroids
the final matching result of 25 pairs of closed contours and of closed contours and comer segments as control points,
salient segments, with “+”
denoting centroid positions. since small deviations from the true boundary locations along
The next example is concerned with the registration of Spot the contour may cancel each other. The limitation of this
and Seasat images. These two images have been rectified algorithm lies in the requirement that well-defined, strong
and geocoded to within 5 pixels, and as a result the active contours must be detected from the optical image as initial
contour algorithm is directly applicable. Fig. 9(a) shows the contours.
Spot image and Fig. 9(b) shows the Seasat image with the Previous researchers have noted that the task of automated
contours from the Spot image directly superimposed. The multisensor image registration is very complex [ 6 ] . A com-
1
distances from the initial contours to the correct edge loca- bination of multiple techniques within the framework of an
tions in the SAR image range from l to 5 pixels. Fig. 9(c) artificial intelligence system may be necessary. The proposed
shows the converged contours. A blow-up of three initial and contour-based algorithms can serve as an important building
converged contours in Fig. 9(d) illustrates the active contour block for such a system.
model’s ability to match deformed region boundaries. Twenty-
seven matched points have been obtained and are shown in
ACKNOWLEDGMENT
Fig. 9(e)-(f).
It can be seen from Table I that the estimations of four The authors would like to thank E. Rignot of the Jet
transformation parameters produced by manual and automatic Propulsion Laboratory, Califomia Institute of Technology, and
methods are very close. In each case, the proposed algorithms J. Star of the Geography Department, University of Califomia,
achieve a lower RMSE than manual registration. In general, Santa Barbara, for providing us with the multisensor and
optical-to-SAR image registration, either manual or using multispectral data used in this paper. We also thank the
computerized algorithms, results in a higher RMSE due to the anonymous reviewers for their constructive comments and
noise in the SAR images. We have also tested our algorithms suggestions.
on other data sets, including image pairs that do not cover the
same region. The basic contour-matching algorithm is quite REFERENCES
robust and reliable as long as corresponding contours are
available. The registration scheme would fail if insufficient [ I ] J. Curlander and R. McDonugh, Synthetic, Aperture Radar: Systems and
Signal Processing. New York: Wiley, 1991.
contour information can be extracted. In all cases in which the 121 W. Pratt, Digital Image Processing. New York: Wiley. 1991.
input images did not contain corresponding regions, matched 131 R. Haralick and L. Shapiro, Compurer and Rohor Vision. Volume 2 .
points were not found, as expected. Reading, MA: Addison-Wesley, 1993.
[4] Q. Zheng and R. Chellappa. “A computational vision approach to image
registration.” IEEE Trans. / m a p Processing, pp. 31 1-326. July 1993.
15) B. Manjunath et al., “A robust method for detecting image features
V. DISCUSSION with application to face recognition and motion correspondence,” in
P w c . / n t . Cotif. Punern Recogti. (The Hague, Netherlands), Sept. 1992.
In this paper, we have presented two contour matching pp. 208-212.

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.
 334 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4. NO. 3, MARCH 1995

[6] E Rignot et a l . “Automated multisensor registration Requirements Hui Li (S’90) was born in Xidmen, China, in
and techniques,” Photogrammetric Eng , Remote Sensing, vol 57, pp 1967 He received the B S degree in electncal
1029-1038, Aug 1991 engineering in 1989 from the Fudan University,
[7] A Ventura, A Rampini, and R Schettini, “Image registration by Shanghdi, China, and the M S and Ph D degrees
recognition of corresponding structures,’‘ IEEE Trans Geosc I Remote in electncal and computer engineering from the
Sensrng, vol 28. pp 305-314, May 1990 University of Cdlifomia at Santa Barbara in 1991
[SI A Goshtasby, G Stockman, and C Page, “A region-based approach to and 1993, respectively
digital image registration with subpixel accuracy,” IEEE Trans G e m I After graduation, he joined the Space Systems Di-
Remote Sensing. vol 24, pp 390-399, May 1986 vision, Rockwell International Corp., Downey, CA
191 Y Wu and H Maitre. “A multiresolution approach for registration of His research interests include image enhancement.
a spot image dnd a SAR image,” in Proc lnt Georcr Remote Sensrrig image matching, data fusion, and image analysis.
Svmp , May 1990, pp 635-638
[ 101 J Greenfeld, “An operator-based matching system,” Photogrammet?rc
Eng , Remote Sensing, vol 57, pp 1049-1055, Aug 1991
[ I l l T Schenk, J Li. and C Toth, “Toward an autonomous system tor
orienting digital stereopain,” Photo,yrammerr I i Eng , Remote Sensrng,
vol 57, pp 1057-1064, Aug 1991
1121 S Mason and K Wong, “Imdge . dlignment
. by line triples,” Pho-
togrammetric En,?., Remote Sensing, vol. 58, pp. 1329-1334, Sept. B. S. Manjunath (M’91) received the 5.E degree
1992. in electronics from Bangalore University in 1985,
1131 N. Nasrabadi, “A stereo vision technique using curve-segments and the M.E. degree in systems science and automation
relaxation matching,” IEEE Trans. Pattern Anal. Machine Ir~tell.,vol. from the Indian Institute of Science in January
14, pp. 566572, May 1992. 1987, and the Ph D degree in electncal engineenng
K. Boyer and A. Kak, “Structural stereopsis for 3-D vision,”IEEE Trans. from the University of Southem Califomia (USC)
Pattern Anal. Machine tntell., vol. 10, pp. 144-166, Mar. 1988. in 1991
R. Touzi, A. Lopes, and P. Bousquet, “A statistical and geometrical edge From September 1987 to July 1991, he was a re-
detector for SAR images,” IEEE Trans. Geosci., Remote Sensing, vol. search assistant at the Signal and Image Processing
26, pp. 764-772, Nov. 1988. Institute a1 USC. and in the summer of 1990, he
J. Chen, A. Huertds, and G. Medioni, “Fast convolution with Laplacian- worked at the IBM T. J. Watson Research Center,
of-Gaussian masks,” IEEE Trans. Pattern A n d . Machine Intell., vol. Yorktown Heights, NY He joined the Electrical and Computer Engineering
PAMI-9, pp. 584-590, July 1987.
Department at the University of California, Santa Barbara, as an assistant
J. Canny, “A computational approach to edge detection,” IEEE Trans. professor i n July 1991
Pattern Anal. Machine Intell., vol. PAMI-8, pp. 679698, Nov. 1986.
Dr Mdnjunath was awarded the Bangalore University Gold Medal for the
R. Duda and P. Hart, Pattern Classification and Scene Analysis. Read-
ing, MA: Addison-Wesley. 1973. best grdduating student in electronics engineering in 1985 He wds also a
A. Goshtasby, “Image registration by local approximation methods,” recipient of d National Merit Scholarship from the Govemment of India dunng
Image Vision Comput., vol. 6. pp. 255-261, Nov. 1988. the period 1978-1985 His current research interests include computer vision,
L. Brown, “A survey of image registration techniques,” ACM Computrng medical image analysis, pattern recognition. and neural network5
Surreys, vol. 24, pp. 325-376. Dec. 1992.
A. C. Bovik, “On detecting edges in speckle imagery,” IEEE Trans.
Acoust., Speech. Signal Processing, vol. 36, pp. 1618-1627, Oct. 1988.
T. Grimmins, “Geometric filter for speckle reduction,” Appl. Optics,
vol. 24, pp. 143844, May 1985.
A. Jain, Fundamentals ofDigital Image Processing. Englewood Cliffs.
NJ: Prentice-Hall, 1989.
J. Curlander, R. Kwok, and S. Pang, “A post-processing system for Sanjit K. Mitra (SM’69-F’74) received the B.Sc.
automated rectification and registration of spacebome sar imagery.” Int. (Hon.) degree in physics from Utkal University,
J . Remote Sensing, vol. 8, pp. 621-638, Apr. 1987. Cuttack, India; the M.Sc. (Tech.) degree in ra-
1251 M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour dio physics and electronics in 1956 from Calcutta
models,” in Proc.. 1st Int. Conf Comput. Vision. 1987. University; and the M.S. and Ph.D. degrees in elec-
1261 D. Williams and M. Shah, “A fast algorithm for active contours trical engineering from the University of California,
and curvature estimation.” CVGIP: Image Understanding, vol. 5 5 , pp. Berkeley, in 1960 and 1962, respectively.
14-26, Jan. 1992. He is currently with the Center for Information
1271 A. Amini, S. Tehrani, and T. Weymouth, “Using dynamic programming Processing Research, Department of Electrical and
for minimizing the energy of active contour in the presence of hard Computer Engineering, University of Califomia at
constraints,” in Proc. 2nd Int. Conj. C o t p u t . Vision, 1988. Santa Barbara.
A. Cideciyan et al.. “Registration of high resolution images of the Dr. Mitra in May 1987’ was awarded an Honorary Doctorate of Technology
retina.” in Proc. SPIE, vol. 1652, pp. 310-322, 1992. degree from the Tamper6; University of Technology, Finland.

Authorized licensed use limited to: Malaviya National Institute of Technology Jaipur. Downloaded on February 11,2025 at 07:03:02 UTC from IEEE Xplore. Restrictions apply.