Matching Images Under Unstable Segmentations

Varsha Hedau1, Himanshu Arora2, Narendra Ahuja3

1,3 ECE Department, University of Illinois, Urbana, IL-61801, USA. {vhedau2,n-ahuja}@uiuc.edu2 Objectvideo Inc., Reston, VA-20191, USA. harora@objectvideo.com

Abstract

Region based features are getting popular due to theirhigher descriptive power relative to other features. How-ever, real world images exhibit changes in image segmentscapturing the same scene part taken at different time, un-der different lighting conditions, from different viewpoints,etc. Segmentation algorithms reflect these changes, andthus segmentations exhibit poor repeatability. In this paperwe address the problem of matching regions of similar ob-jects under unstable segmentations. Merging and splittingof regions makes it difficult to find such correspondencesusing one-to-one matching algorithms. We present partialregion matching as a solution to this problem. We assumethat the high contrast, dominant contours of an object arefairly repeatable, and use them to compute partial match-ing cost (PMC) between regions. Region correspondencesare obtained under region adjacency constraints encodedby Region Adjacency Graph (RAG). We integrate PMC ina many-to-one label assignment framework for matchingRAGs, and solve it using belief propagation. We show thatour algorithm can match images of similar objects acrossunstable image segmentations. We also compare the perfor-mance of our algorithm with that of the standard one-to-onematching algorithm on three motion sequences. We con-clude that our partial region matching approach is robustunder segmentation irrepeatabilities.

1. IntroductionThe regions of an object obtained by many segmenta-

tion methods in images taken under slightly different imag-ing conditions (e.g. lighting, viewpoint, etc.) differ signif-icantly. This is because the existing segmentation algo-rithms do not always produce perceptually meaningful re-gions. This results in splitting and mergers of adjacent re-gions, even if the contrast changes very slightly. On theother hand, in conditions where stable regions can be reli-ably obtained, for instance, by using additional informationsuch as, shape cues as in [20], the efficacy of region basedmatching has been successfully demonstrated for high level

vision tasks such as object recognition. Despite their de-scriptive power, regions have not been a very popular choicefor computer vision tasks requiring image matching, partlybecause of the lack of segmentation algorithms which pro-duce repeatable regions.

Most of the existing algorithms for region correspon-dences employ both the absolute region properties e.g.shape, area, mean intensity etc., as well as relational con-straints e.g. adjacency (regions sharing boundary) and hi-erarchy, encoded in graph structures e.g. Region AdjacencyGraphs (RAGs) [17, 20] (encoding adjacency), or segmen-tation trees [1, 2, 13] (encoding hierarchy). The splits andmergers of regions corresponding to the same scene consid-erably affect both (i) the regions’ absolute properties, and(ii) their relationships with other regions in the image. Con-sider for instance two example segmentations of the sameobject, shown in Fig. 1. The corresponding RAGs are over-laid on the top of segmented images (middle column). Hereregion b in the top image is a merger of regions b1 and b2in the bottom image. In the ideal case, b should be matchedto both b1 and b2. However, absolute properties of b1 andb2 differ significantly as compared to their merger b there-fore the unary match cost of b is high with both b1 and b2.Also note that the adjacency relationships between regionschange considerably, e.g., b2 in the bottom image is not ad-jacent to a, but b in the top image, which is the merger con-taining b2, is adjacent to a. This causes irrepeatability in thestructure of (RAGs) of the two images, commonly referredto as structural noise. Matching b with its fragments b1 andb2 is thus difficult by i) any algorithm that outputs a one-to-one matching between regions, or ii) any algorithm that usesonly the unary match cost based on regions’ absolute prop-erties. This paper is aimed at providing a solution for bothof the above problems. It proposes a novel cost measurebetween regions, one that depends on the properties of theportion of overlap between regions after transforming themappropriately and not the properties of entire regions. Wedenote this measure by Partial Match Cost (PMC). Also,the paper integrates this cost measure into a many-to-oneregion matching framework.

Prior Work. One solution to the segmentation instabil-

1

ity problem is allowing access to region segmentations atmultiple scales, e.g., contrast levels, [1, 2, 13]. However,even searching all possible scales of segmentation does notguarantee perfect repeatability. Similar observation is re-ported in [11], wherein the authors use regions obtainedfrom multiple segmentations, at multiple scales, and deter-mine if these regions reliably correspond to the spatial sup-port of natural objects. They conclude that even with largenumber of regions available per image, explicit mergers ofregions are still required to obtain desirable levels of sup-port. Since mergers lead to combinatorial complexity, us-ing regions under such conditions becomes computationallyprohibitive quickly. An approach that would allow partialmatching of regions, so that the split portions could matchthe whole, is therefore desirable.

The problem of structural noise in graphs has been ad-dressed before by relaxing the enforcement of relationalconstraints between regions. For instance, instead of usingthe direct parent-child constraints between regions whilematching segmentation trees, Torsello et al. [16] suggest us-ing transitive closures in trees which connect every node inthe tree to every descendant. Thus any ancestor-descendantpair in one tree can match with any such pair in the other,even if their levels are different. As another example, thetechnique in [17] uses a fixed number of regions with near-est centroids, instead of directly adjacent regions whilematching RAGs. However, even with the relaxed con-straints, the assumption of one-to-one correspondence is of-ten found to be too restrictive for real images.

Another class of approaches perform explicit regionmergers, to deal with problem of high unary match costbetween absolute region properties caused by region merg-ing and splitting [7, 8, 13, 17]. These are guided by someheuristically set thresholds, e.g., those depending on inten-sity contrast between two regions. However, such mergersmay not produce perfect one-to-one correspondences. Ke-selman et al. [9] obtain a model RAG, referred to as LeastCommon Abstraction (LCA), using RAG of exemplar im-ages of a particular object category. Finding LCA requiressearch for isomorphic RAGs among those obtained by allpossible region mergers in the original RAGs. Matchingexplicit splits and mergers of regions as performed by theabove techniques can quickly become computationally pro-hibitive due to their combinatorial nature. The partial re-gion matching approach proposed in this paper can reducethis combinatorial search space drastically. Another class ofprevious approaches is that of inexact graph matching [18],wherein graph edit operations such as node (or edge) substi-tution, deletion and insertions are defined to handle the in-consistencies in graph structure. Inexact graph matching isan n-p complete problem and is computationally intractablefor real images, since the space of all possible edit opera-tions is huge. A number of approximate solutions have been

proposed in literature; however, the convergence in thesecases depends heavily upon initialization.

Overview of our approach. We provide a solution todeal with the irrepeatibility of regions caused by mergingand splitting, and noise in graph structure, while matchingregion adjacency graphs. We define the match score be-tween two regions in a robust fashion, which allows frag-ments of the same region to have low unary match cost withthe original region. We denote this cost as the Partial MatchCost (PMC). Towards computing PMC, we assume that thedominant contours of object are repeatable under differentimaging conditions. In this paper, we consider adjacencyrelationships between regions contained in RAG. Howevernote that PMC is generic enough to be integrated with othertypes of relations, e.g. containment, as observed in segmen-tation hierarchies (trees). To handle the structural noise inRAGs, we formulate the problem of RAG matching as amany-to-one label assignment problem, i.e., allowing formultiple regions from one RAG to match with a single re-gion in the other RAG. We solve this assignment problemusing the loopy belief propagation technique. Our algo-rithm can thus match fragments with their merger region,while respecting the adjacency constraints. In Fig. 1, thirdcolumn, we show example results of our algorithm. Noticethat our algorithm can handle both of the problems due tosegmentation noise, mentioned above. This paper has thefollowing contributions: 1) It uses partial region matchingwhich is a robust similarity measure between regions. Thiseliminates the need for performing a combinatorially largenumber of explicit region mergers, which is otherwise nec-essary to deal with irrepeatibility of regions observed underunstable segmentations. 2) This paper also proposes an ef-fective solution of many-to-one label assignments for theproblem of structural noise in RAGs, which is possible dueto our definition of partial match cost.

The rest of the paper is organized as follows: In Sec. 2,we explain our partial region match cost. In Sec. 3, we ex-plain our label assignment formulation for RAG matching.Experiments and results are described in Sec. 4. Finally, weconclude the paper in Sec. 6.

2. Partial Region MatchingSince the global properties of regions change drastically

among segmentations of the same scene under slightly dif-ferent conditions, (e.g., lighting, viewpoint, etc.), they cannot be used to obtain a reliable match between the segmen-tations. The properties of regions should instead be com-pared in the portions common to the two regions. Considerfor instance the regions b and b1 shown in Fig. 1, corre-sponding to the torso and shirt of the man. Instead of com-paring their global properties, if we compare the propertiesonly in the common physical portion of these regions, i.e.,shirt, we will be able to assign low cost to a match between

Figure 1. Images in the first column show adjacent frames of avideo sequence. Second column shows the region splitting phe-nomenon and the changed RAG structure due to it. We show onlya part of RAG for better visibilty. Third column shows some of thematched regions obtained by our algorithm. Matched regions areshown with the same color. Notice that shirt and pant fragments inthe bottom image are successfully matched to the torso fragmentin the top image.

these regions. We denote the match cost obtained in sucha manner as Partial Match Cost (PMC). As we shall nextexplain, we hypothesize that deduction about such a com-mon portion between the regions in different images can beobtained as a transformation T, by robustly registering theircontours. For two regions, r1 in the first image and r2 inthe second image, and the transformation T between them,we can then define overlapping portions as o1 = r1∩T(r2)and o2 = r2 ∩T

−1(r1), in the domains of the first and sec-ond images, respectively. We define the PMC between r1

and r2 as

PMC(r1, r2) = ||f(o1) − f(o2))|| (1)

where f(r) denotes the properties of region r and || · ||is the euclidian norm. Since Eq. (1) computes the cost ofmatching overlapping portions of registered regions, assum-ing that the transformation between regions has been com-puted correctly, it will always return low cost for matchinga region with one of its fragments (e.g. b with b1 in Fig. 1).Below we explain our methodology for estimating the trans-formation T.

2.1. Region transformationWe assume that dominant contours corresponding to

physical boundaries of different parts of an object are fairlyrepeatable. Due to the presence of a few spurious pixelson the boundaries, the originally closed regions of the ob-ject leak out to form big regions. It is this leakage that ismostly responsible for irrepeatability of regions across im-ages of the same object. We therefore want to exploit theparts of the region boundaries which are undisturbed by theimage noise, to estimate the transformation T between re-gions r1 and r2. Let us denote the boundaries of r1 and r2

by bo1 = {x11, . . . ,x

1k} and bo2 = {x2

1, . . . ,x2m} where

(a) (b)

(c)Figure 2. KDC Registration. Boundaries of regions from the firstimage are shown in blue and the transformed boundaries of corre-sponding fragments in the second image are overlaid on the top indifferent colors. (a) Regions from image pair in Fig. 3(c) are ac-curately registered despite huge scale variations. (b) Regions fromimage pair in Fig. 3(b) are accurately registered under affine trans-formations. (c) The face-neck region is successfully registered toface and neck regions (see Fig. 3(a)).

x1i , i ∈ {1, . . . , k} and x

2j , j ∈ {1, . . . ,m} are boundary

pixels of regions r1 and r2 respectively. Our goal is to alignpieces of boundaries which are common in the two regions.Moreover, in order to successfully align fragments of a re-gion with the whole region, we must do this alignment in amanner such that it is robust to outliers. There has been amuch towards robustly estimating a transformation betweenpoint clouds in such a correspondenceless setting [6,14,19].We adopt robust kernel density correlation (KDC) machin-ery, described in [15], for solving this problem. Accordingto [15], the problem of robustly matching point clouds isformulated as that of matching the respective kernel den-sity (also known as parzen window) estimates [5] of thesamples in each of these point clouds. KDC is used as ameasure of match between these kernel density estimates.The equivalence between kernel density estimators and ro-bust M-estimators was shown in [3], and it is from here thatKDC inherits its robustness to outliers.

Following the formulation in [15], we assume that thepoint sets bo1 and appropriately transformed bo2 are gen-erated from the same underlying pdf. The quality of matchbetween bo1 and T(bo2) is then given by the correlation ofthe corresponding estimates of the underlying pdf, given bythe following equation, as described in [15].

KDC(bo1, T(bo2)) =∑

xi∈bo1

xj∈bo2

K(H−1(xi−T(xj))) (2)

where, K is the kernel function with zero mean, unit area

and identity covariance matrix, and H is a non singularbandwidth matrix. We use Gaussian kernel in our experi-ments. Maximizing KDC measure in Eq. (2) gives us theT between two regions. Given an appropriate initialization,the maximizer of Eq. (2) can be estimated efficiently in aniterative fashion by using variational optimization machin-ery, as shown in [15]. In our implementation, we restrict T

to be an affine transformation. We discuss the initializationin Sec 4.

Fig. 2 shows some example results of our region trans-formation estimation using KDC. Boundary of the region infirst image is shown in blue. The corresponding region ofthe object was broken into fragments in the segmentationof the second image. See segmentation images in secondrow in Fig. 3. We have shown the boundaries of the frag-ment regions overlaid on the top of the original region aftertransforming them using the computed transformation, withdifferent colors. Notice how each fragment from second im-age is robustly registered on the single merger region in thefirst image.

3. Matching Region Adjacency GraphsWe abstract the segmentation of each image as a Region

Adjacency Graph (RAG) constructed in the following fash-ion. For an image I , we define an RAG G = (P,N ) con-sisting of a set of nodes P and a set of edges N ⊂ P × P .Each node p ∈ P represents a region in the image. Anedge (p, q) ∈ N exists if regions p and q share a commonboundary (are adjacent), i.e., p ∼ q. Towards matchingRAGs of two images, G1 = (P,N1) and G2 = (L,N2), wewant a correspondence between their nodes which has a lowpartial match cost, and preserves adjacency relations. Wealso want this correspondence space to include many-to-onematches since a node in one RAG might exist as multiplefragments in the other RAG. We thus define RAG matchingas the minimum unary cost labeling or mapping l : P → Lsatisfying the constraints that ∀(p, q) ∈ N1, we have ei-ther (l(p), l(q)) ∈ N2 or l(p) = l(q). For convenience,we use the notation lp = l(p). Note that the constraints onthe labeling preserve adjacency by allowing adjacent nodesin G1 to take labels corresponding to adjacent nodes in G2

and can produce many-to-one matches by allowing adjacentnodes to take the same label. Following this discussion, wecan write the energy associated with a labeling l as

E(l) =∑

p∈P

Dp(lp) +∑

p,q∈N1

V (lp, lq) (3)

Here, Dp(lp) is the unary cost of matching nodes p ∈ Pand lp ∈ L computed as partial match cost using Eq. (1),i.e., Dp(lp) = PMC(p, lp). The term V (lp, lq) is the cost

of violating the constraints, defined as

V (lp, lq) =

{

0 if lp, lq ∈ N2 or lp = lqK if lp, lq /∈ N2

(4)

Here, K is the constraint violation penalty and is set to ahigh value. Thus the minimizer of E(l) gives an optimumassignment in terms of both the unary cost and adjacencyrelations.

Estimating a global minimum of the energy function inEq. (3) requires a search over the discrete space of all possi-ble labelings. However, efficient iterative techniques basedon message passing algorithms [12] and graph cuts [10]have been proposed in literature to estimate strong localminimum of such an energy function. We use loopy beliefpropagation (LBP), which is a message passing algorithm,to solve the minimization in Eq. (3). In LBP, each node it-eratively receives messages from its neighbors in the graph,and updates its own confidence of having a certain label.Once the messages stop changing beyond a threshold, theiterations are terminated, and each node computes the re-sulting belief over all the labels. The per node label is thenobtained by maximization over these individual beliefs. Ifconvergent, LBP has proved to be a good approximation formany applications [12]. In our experiments, LBP convergesin approximately 50 iterations. For our formulation, we canwrite the message passed by node p to its neighbor q forlabel lq at iteration t as

mtp→q(lq) = min

lp

V (lp, lq) + Dp(lp) +∑

s6=q(s,p)∈N1

mt−1s→p(lp)

After T iterations of message passing, the belief vector atnode q for label lq is computed as

bq(lq) = Dq(lq) +∑

{(p,q)∈N1}

mTp→q(lq) (5)

Finally, the label l∗q that minimizes bq(lq) at node q is se-lected.

4. Experiments and ResultsWe test the performance of the proposed PMC and many-

to-one matching approach with two different experiments:(a) matching images of similar objects taken under differ-ent imaging conditions, and (b) matching images of a mo-tion sequence. For image matching, we present qualitativeresults on image pairs with varying degrees of complexi-ties, validating the ability of our algorithm towards match-ing a region with its fragments, and many-to-one regionmatching (see Sec. 4.1). For matching images of motionsequences, we present both qualitative and quantitative re-sults. On these sequences, we compare the performance of

our algorithm against its one-to-one counterpart which doesnot use PMC (explained in Sec. 4.2).

For all the experiments, image segmentation is per-formed using the mean-shift clustering approach [4]. Foreach image pair, we use the same scale parameters for themean-shift segmentation. However, in order for our as-sumption of having repeatable contours in both images be-ing valid, we pick a scale which preserves dominant con-tours in segmentation output of the two images. As weshall see in the results, there still are numerous examplesof region irrepeatabilites making the region correspondencetask really difficult. This selection of the scale parametercan be completely avoided by taking regions from segmen-tation outputs obtained at multiple scales. Note that we donot separately tune the scale parameters for each image inthe pair to get similar regions. We set the value of constraintviolation cost K in Eq. (4) to 100 and the region propertyfunction f in Eq. (1) is chosen as gray scale absolute inten-sity difference at each pixel. Further, we assume that thetransformation to be estimated using KDC, T, is an affinetransformation. KDC being an iterative approach requires agood initialization to search for the global optimal. For im-age matching experiments, we use multiple initializations,obtained by matching local appearances at corner points onthe image pairs. For matching images of a motion sequence,we initialize the transformation as an identity transforma-tion.

4.1. Matching image pairsWe show the results of our RAG matching algorithm on

a set of image pairs consisting of similar objects, capturedunder different conditions. Each image is represented as anRAG, and PMC is computed between every region pair con-sisting of a region from the first image and a region from thesecond image. Region correspondences are then obtainedby matching RAGs using the algorithm described in Sec 3.Note that our original formulation for label assignment ismany-to-one, however, we re-run the algorithm by revers-ing the order of images to obtain many-to-many correspon-dences. For each set of images in Fig. 3, the original imagesare shown in the top row, the segmentations in the middlerow, and, the matched regions are shown in the same colorsin the bottom row. The nodes which are not matched areshown in black.

Fig. 3(a) shows the matching of face images of the sameperson on different backgrounds. Here, the regions corre-sponding to face, neck and right collar in the left imagefuse together to form a single face-neck-collar merger in theright image. Note that our formulation is able to find cor-respondences between this merger and its individual frag-ments. This shows the robustness of PMC based unarycost and the ability of our RAG matching algorithm towardshandling many-to-one matches. Also note that the regions

neighboring the face-neck-collar portion in the images stillget matched appropriately despite their significantly differ-ent neighborhoods. Thus, our algorithm can also handlestructural noise in RAGs. Fig. 3(b) shows matching resultson images of a graffiti which have complex adjacency struc-ture. Matching these images using any global region matchmeasure or one-to-one RAG matching methodology wouldbe very difficult, due to huge differences in segmentationsin terms of mergers and splits of regions, as well as the adja-cency structure. However, our matching framework is ableto handle this successfully. Note the partial matching forthe regions shown in blue, where the corresponding regionin the second image occupies less than half of the area of theregion in the first image (see the corresponding registrationsin Fig. 2). Also note that despite of instances of structuralnoise in the graphs, we are able to match them correctly.For instance, the regions colored in blue and orange are ad-jacent in the first image, but not in the second, and yet theyare matched correctly.

Fig. 3(c) and (d) show matching results on images of theKremlin and the Statue of Liberty, respectively. Note thatthe Kremlin appears as a single region in one image and isfragmented into multiple regions in the second image. Ouralgorithm is able to match these fragments with the entireregion. Many of these fragments, as shown in Fig. 2(a), aresuccessfully registered to the single Kremlin region, provid-ing a low match cost between these corresponding regions.A similar effect can be seen for the Statue of Liberty imagepair.

Fig. 3(e) and (f) show matching results for rhino andhorse images pairs. Note that the many regions correspond-ing to the object are matched correctly, but background re-gions are not. This is because the background regions andtheir adjacency relationships are different in these images.Also, some regions inside the object are not matched cor-rectly. due to lack of sufficient boundary overlap for reliabletransformation estimation using KDC.

4.2. Matching Motion SequencesWe compare the results of our algorithm, both qualita-

tively and quantitatively, against a one-to-one region match-ing algorithm, which uses absolute region properties forcomputing match cost between regions instead of PMC,implemented in the following manner. The regions aredescribed using their area, centroid, eccentricity, solidity,perimeter, and, mean and variance of intensities. The costof matching two regions is computed as sum-of-squared-differences between these absolute properties. The one-to-one RAG matching algorithm implemented is the sameas the one described in Sec. 3, except that we change theconstraint cost in Eq. (4) so that V (lp, lq) = 0 only whenlp, lq ∈ N2. Thus, there is a high cost for neighboring re-gions taking the same label, i.e., penalizing many-to-one

(a) (b) (c)

(d) (e) (f)Figure 3. Image matching results. Each set of images shows the original images in the top row, the segmented images in the middle row,and matching results in the bottom row. Matched region pairs are shown with the same color in the two images. Images in (a) show resultsfor matching face of a person on different backgrounds. Images in (b) show matching results for Graffiti images which have complexadjacency structure. Note that the instances of partial matching and many-to-many matchings are detected correctly. Images in (c) showsmatching of the Kremlin images under scale variations, all the fragments of the structure are correctly matched to the corresponding mergerregion. Similarly (d), (e), (f) show results on instances of the Statue of Liberty, rhinos and horses. Notice how the different segments onthe object are matched accurately. (This figure is best viewed in color.)

matching.We choose motion sequences for this comparison. Note

that the assumption of one-to-one correspondence is suit-able under this setting. We have considered three differ-ent motion sequences here, namely, the house sequencefrom CMU motion database, the man sequence of a per-son walking on static background, and, the Kwan sequenceof olympic ice-skater Michelle Kwan. Matching is per-formed on successive frames, returning matched regionpairs, which are compared against ground truth regionmatches marked by us. A region pair is termed as an er-ror, if either a) it is a ground truth pair, but not returned bythe algorithm, or, b) it is returned by the algorithm, but nota ground truth match pair. The total region mismatch errorper image is computed as the ratio of the total number of er-roneous matched pairs returned by the algorithm, and the to-

tal number of ground truth region pairs. The pixel mismatcherror for an erroneous pair is computed as the minimum ofthe areas of the two regions in the matched pair. We choosethe minimum area to respect partial matching. The per im-age pixel mismatch error is computed as the ratio of sum ofpixel mismatch errors of each erroneous pair, and the totalimage area. For each sequence, we report the mean per im-age region and pixel mismatch errors, computed as the av-erage of region and pixel mismatch errors of all the framesof that sequence, shown in Fig. 4. Note that the many-to-one matching algorithm has lower errors as compared to theone-to-one matching algorithm, for each of the sequences.As can be seen in Fig. 5, the house sequence has the highestambiguity due to multiple window regions which are verysimilar in appearance. This is not the case with the other se-quences, and hence they have lower region matching errors.

house kwan man0

0.1

0.2

0.3

0.4Region Detection Error

One to oneMany to one

(a) Region Errorhouse kwan man0

0.05

0.1

0.15

0.2Pixel Error

One to oneMany to one

(b) Pixel ErrorFigure 4. Performance comparison. Detection error for ourmethod (many-to-one) is shown in pink and the error for one-to-one matching method is shown in blue. Average errors for threemotion sequences are shown. (a) We have obtained lower regiondetection error for all the three sequences. Performance gain isthe highest for house sequence because of its complex RAG struc-ture, due to which this sequence is more prone to structural noise.(b) Pixel matching error using one-to-one matching is high forthe kwan sequence since it has large background clutter and a num-ber of instances of non-repeating regions in adjacent frames. OurPMC based method gives very low error on this sequence. Mansequence was the least challenging amongst the three and henceboth the methods perform almost equally well on it.

However, the area occupied by these ambiguous regions isvery small, and hence the pixel matching errors are lowerthan the Kwan sequence. The man sequence is the easiestwith not much complex motion, and high repeatability, andhence produces lowest errors.

In Fig. 5 we show example frames of the above motionsequences. Top row shows original images of consecutiveframes in the motion sequences middle row shows matchingresults by our method and the bottom row shows matchingresults returned by the one-to-one method. Notice how ourmethod can match shirt, pant with the torso region in theman sequence. Also hair and face regions are matched byour method but not by one-to-one. In kwan sequence similareffect can be observed. We can match the torso of kwanwith both upper body and lower body whereas one-to-onematches it only with the upper body.

5. Computational ComplexityThe most expensive step towards computing the par-

tial match cost is the computation of robust transformationusing KDC. From Eq. (2), the complexity of computingthe KDC similarity measure between samples from bound-aries of two regions, bo1 = {x1

1, . . . ,x1k} and bo2 =

{x21, . . . ,x

2m} for a given transformation T is O(km),

where k and m are the number of boundary points in therespective boundaries. Let us assume that the KDC con-verges to a stable transformation in T iterations. Thenthe complexity of KDC computations for estimating T isgiven by O(mkT ). Let us denote by nbj

i , the number ofboundary points in th i-th region of j-th image, and by

Nbj =∑

i nbji , the sum of boundary points of all regions

in the jth image. Then, the total complexity of KDC stagecan be computed as

CKDC = O(∑

i1,i2

Tnb1i1nb2

i2) = O(TNb1Nb2) (6)

The loopy belief propagation for many-to-one RAG match-ing has the complexity, CBP = O(N1N2N2t), with num-ber of nodes (regions in first image) N 1 and number of la-bels (regions in second image) N 2 and t iterations. The one-to-one matching approach without PMC has computationalcomplexity of O(N1N2) for computing the match cost be-tween N1 regions in the first image and N 2 regions in thesecond image. The complexity of graph matching stage us-ing label assignment is same as CBP for both the methods.Thus PMC provides increased accuracy at the cost of in-creased computations. However, as demonstrated earlier,even in simplest settings assumption of one-to-one regioncorrespondences is very restrictive. Note that, as comparedto methods which perform explicit region mergers we savesubstantially in terms of computations. For instance, if weconsider all possible mergers of N 1 nodes we get 2N1 nodesand this thus makes the complexity of computing match costexponential O(2N1

2N2

). The cost of graph matching stageis also increased exponentially.

6. ConclusionIn this paper, we propose a solution of partial region

matching for matching a region with its component frag-ments. We further address the problem of structural noise inregion adjacency graphs through our formulation of many-to-one assignment problem. The results of our algorithm formatching images of similar objects with different segmen-tations show that the concept of partial region match costis promising for handling irrepeatability usually observedin the output of image segmentation algorithms. We alsodemonstrate performance enhancement with our many-to-one matching method based on partial match cost, over theone-to-one method which takes absolute properties of theregions into consideration. An immediate possible exten-sion of our work is matching region adjacency graphs con-structed using regions obtained from multiscale segmenta-tion outputs, to ensure better repeatability in terms of imagecontours. Possibility of integrating hierarchical and adja-cency constraints in such region adjacency graphs can alsobe explored.

7. AcknowledgementsThe support of the National Science Foundation under

grant NSF IBN 04-22073 is gratefully acknowledged.—————————————-

(a) House Sequence (b) Kwan Sequence (c) Man SequenceFigure 5. Example frames from three motion sequences. Original images are shown in the top row, matching results obtained by ourmethod is shown in the middle row and one-to-one matching results are shown in the bottom row. Matched regions are shown with thesame color in the left and right images, regions which are not matched are shown in black. (a) House. Note that some of the small regionscorresponding to windows are not matched correctly by one-to-one method while we can match them. (b) Kwan. We can match the lowerbody and upper body of kwan in the right image with the whole body region in the left image whereas one-to-one matching can not achievethis. (c) Man. Notice how the shirt, face, and hair regions are left unmatched by one-to-one method which are matched accurately by ourmethod.

References[1] N. Ahuja and S. Todorovic. Learning the taxonomy and mod-

els of categories present in arbitrary images. In Proc. of IEEEIntl. Conf. on Comp. Vis, 2007.

[2] H. Arora and N. Ahuja. Modelling objects as region layoutsand hierarchies. In Proc. of IEEE Conf. on Comp. Vis. Patt.Rec., pages 927–934, 2006.

[3] H. Chen and P. Meer. Robust computer vision through kerneldensity estimation. In Proc. of 7th Euro. Conf. on Comp. Vis.,pages 236–250, London, UK, 2002. Springer-Verlag.

[4] D. Comaniciu and P. Meer. Mean shift: a robust approachtoward feature space analysis. 24:603–619, 2002.

[5] R. Duda, P. Hart, and D. Stork. Pattern classification. 2ndedR John Wiley Sons.

[6] A. W. Fitzgibbon. Robust registration of 2d and 3d pointsets. Image Vision Comput., 21(13-14):1145–1153, 2003.

[7] C. Gomilla. Graph based object tracking. In Proc. of IEEEIntl. Conf. on Image Proc., 2003.

[8] E. M. J Llados and J. Villanueva. Symbol recognition byerror-tolerant subgraph matching between region adjacencygraphs. In IEEE Trans. Patt. Anal. and Machine Intell., pages1137–1143, 2001.

[9] Y. Keselman and S. J. Dickinson. Generic model abstractionfrom examples. In Sensor Based Intelligent Robots, pages1–24, 2000.

[10] V. Kolmogorov and R. Zabih. What energy functions can beminimized via graph cuts? In ECCV (3), pages 65–81, 2002.

[11] T. Malisiewicz and A. Efros. Improving spatial support forobjects vis multiple segmentations. In Proc. of British Ma-chine Vis. Conf., 2007.

[12] K. P. Murphy, Y. Weiss, and M. I. Jordan. Loopy belief prop-agation for approximate inference: An empirical study. InProc. of Uncertainty in AI’99, pages 467–475, 2009.

[13] M. Pelillo, K. Siddiqi, and S. W. Zucker. Many-to-manymatching of attributed trees using association graphs andgame dynamics. In IWVF, pages 583–593, 2001.

[14] S. Rusinkiewicz and M. Levoy. Efficient variants of the ICPalgorithm. In Proc. of 3rd Intl. Conf. on 3D Digital Imagingand Modeling (3DIM), June 2001.

[15] M. Singh, H. Arora, and N. Ahuja. Robust registrationand tracking using kernel density correlation. In Proc. ofCVPRW’04, page 174, 2004.

[16] A. Torsello and E. R. Hancock. Computing approximate treeedit distance using relaxation labeling. Pattern RecognitionLetters, 24(8):1089–1097, 2003.

[17] P. Tu, T. Saxena, and R. Hartley. Recognizing objects usingcolor-annotated adjacency graphs. In Shape, Contour andGrouping in Computer Vision, pages 246–263, 1999.

[18] C. Wang and K. Abe. Region correspondence by inexactattributed planar graph matching. In Proc. of Intl. Conf. onComp. Vis., pages 440–447, 1995.

[19] F. Wang, B. C. Vemuri, A. Rangarajan, I. M. Schmalfuss,and S. J. Eisenschenk. Simultaneous nonrigid registrationof multiple point sets and atlas construction. In ECCV (3),pages 551–563, 2006.

[20] P. L. Worthington and E. R. Hancock. Region-based objectrecognition using shape-from-shading. In Proc. of 6th Euro.Conf. on Comp. Vis., pages 455–471, 2000.