ELSEVIE.R

Abstract

In this paper, we deal with the problem of how to apply the Hough Transform to the Stereo & Motion matching process . Thepurpose here is two-fold: The first objective is to see how the Hough Transform can improve the robustness of a matchingprocess . The second goal is to see whether it is possible to relax any local constraints used by the matching process . In such a way,a general matching algorithm can be developed for both stereo and motion correspondence . This paper presents a novel methodwhich is applicable to the stereo & motion matching of edge primitives. At the end of this paper, some experimental results withreal images will be shown .

Keywords: Stereo and motion matching ; Hough transform

1 . Introduction

In computer vision, stereo & motion matching isan important topic. This problem can be stated as howto find out the correspondence of image primitivesextracted from two related images which may beeither a pair of stereoscopic images or a pair of timelyconsecutive images. The success of stereo vision anddynamic scene analysis strongly depends on the ro-bustness of stereo & motion matching algorithms . Al-though very extensive investigations on stereo & mo-tion matching (or correspondence) problem havealready been carried out in the past (Ayache andLustman, 1991; Bouthemy, 1990 ; Deriche and Faug-eras, 1990; Grimson, 1985 ; Liu and Huang, 1991 ;Medioni and Nevatia. 1985; Xie, 1993), additionaleffort is still necessary in order to improve the ro-bustness of matching algorithms .

In this paper, we present a Hough-Transform-based

E mail : mmxie(dntu .ac.sg .

November 1994

Pattern Recognition Letters 15 (1994) 1143-1150

Pattern RecognitionLetters

Stereo and motion matching : a Hough-transform inspired method

Ming Xie *CG!TISAS, Nanyang Technological University, Singapore 2263, Singapore

Received 5 August 1993

0167-8655/94/$07 .00 © 1994 Elsevier Science B.V . All rights reservedSSDI0167-8655(94)00046-6

matching algorithm . The advantages of our matchingalgorithm are as follows .

1 . Robustness

We know that a Hough Transform technique is verynoise resistant. Hence, the presented matching algo-rithm should be intrinsically robust .

2. Without the need of any local constraints

A matching algorithm usually has two processingsteps: (a) first to establish the correspondence ofmatch candidates by minimizing an energy functionor maximizing a similarity function ; and (b) secondto refine the results of matches by using a smoothingor interpolation process . In this scheme, the first stepseems to have vital importance . This is why many lo-cal constraints like "epipolar line" or "figural conti-nuity" have to be used in order to reduce matchingambiguities . But for our method, we incorporate a

1144

generalized Hough Transform technique (i .e ., a vot-ing technique) into the matching process . This tech-nique is very robust . Our experimental results showthat this robustness surprisingly allows us to relax ordiscard all the local constraints used at the first stepof matching. Consequently, our algorithm can di-rectly cope with the motion matching problem (where"epipolar line" constraint is not applicable) .

3. Highly parallelism

It is commonly admitted that the Hough Trans-form is a highly parallel operation. On the other hand,since we do not use any local constraints, the min-imization or maximization process at the first stepcan also be carried out in a parallel way . Therefore,our matching algorithm is highly parallel .

This paper is organized as follows . The Hough-Transform-based matching algorithm is presented inthe next section . In Section 3, we give three examplesof experimental results to show the applicability ofour algorithm to both the stereo and motion match-ing problem. Section 4 concludes this paper.

2. Matching algorithm based on Hough transform

In our investigation, we are interested in EdgePrimitives . By applying a grouping process, an edgemap can be segmented into sets of contour chains,closed curves and line segments. These edge primi-tives can commonly be called metachains . Withoutloss of generality, we shall use the term metachain todesignate indifferently contour chains, closed curvesor line segments .

From the point of view of data structure, a meta-chain corresponds to a group of linked contour points .Therefore, given a pair of matched contour points,we can identify a possible correspondence of the twometachains that the two matched contour points be-long to . This implies the obvious fact that a Hough-Transform-like voting technique can be employed toimplement a matching process of metachains whichmakes use of the results of matched contour points .

It is well known that the Hough Transform (seeMaitre, 1986) is a powerful method to relate intrin-sic measures to parameter values by satisfying a

M. Xie /Pattern Recognition Letters 15 (1994) 1143-1150

physical constraint. For example, in the case of fit-ting a line to a set of points, the intrinsic measuresare the coordinates of points; the parameter valuesare the parameters describing a line ; and the physicalconstraint is "a point belongs to a line" . We show herethat this philosophy can be applied to find a solutionto the problem of matching metachains based on theuse of matched contour points .

Consider two related images (I,, I2 ) . We denote C,the set of metachains (e .g. the set of contour chains)obtained from I, and C2 the set of metachains ob-tained from 72 . Suppose that there are NC, meta-chains in C, and NC2 metachains in C2 . If we use thenotation c; to represent the label of a metachain, (C,,C2 ) can be written as :

C,={c,,,jl=t, . . ., NC,} .

C2={c,2,j2=l, . ..,NC2} .

(1)

In order to apply the Hough Transform, we definethe following relations .

1 . The relation R,(P;,, P, 2 ) is "the contour pointPi , in I, matches with the contour point Ps in Iz".

2 . The relation R,(cj ,, c,7 ) is "the metachain c„ inC, matches with the metachain ej2 in C3" .

3 . The relation Rc (Pi , c,) is "the contour point P,belongs to the metachain cj" (i refers to either it ori2, andj refers to either jl orj2 ) .

It is easy to understand that the relations Rn , Rrand RP' are all symmetric. These three relations canonly take a value from the set (TRUE, FALSE), that is :

1 . For the relation R, :

Rc(F I, Pie)

STRUE ifP„ matches with P;2FALSE if Pi , does not match with P,7 .

2 . For the relation R,:_TRUE if c„ matches with c,2 ,

R (c„ , c z )FALSE if c„ does not match with c, 2 .

3. For the relation RI:

Rp(P,, c )_ ITRUE if Pi belongs to c, ,FALSE if P, does not belong to c, .

Suppose that there are NE, contour points in I, andNE, contour points in 1, From the above defined re-lations, we can create the following four relationmatrices :

RP =[Rv(PI,,Pe2)[NE,XNE,,

RC =[R,(c,,,c,2)]NC,XNCS,

RPC, = [R;`(P;, , cp) INE,XNC, ,

RPC2=[R;(Pr2,c12)INE,XNCr-

(2)

In fact, the relation matrix RP is a representationof the results of matched contour points that are ob-tained at the first step of matching . The matrices(RPC,, RPC2 ) may be the output of any edge group-ing algorithm . They represent the relation betweencontour points and metachains . So far, in terms ofHough-Transform terminology, the problem ofmatching metachains can be stated as how to esti-mate the relation matrix RC from the measures RPby satisfying the physical constraints (RPC„ RPCZ ) .To solve it, a voting technique can be employed .

Given a contour point P;, in I,, if we have the fol-lowing condition :

1RP(P;,,Pte) =TRUE ,R6 (R;,, c„)=TRUE ,R~(P,2 , C,,) = TRUE

(3)

then R,(cj ,, c12 ) is probably true . Define H(c,,, c,2 )the number of times that R,(c,,, c, 2 ) occurs to takethe value TRUE . Thus, for each contour point P;, inI,, if the condition (3) is met, we incrementH(c,,, c,2 ) like below:

H(c,,, c,2)=H(c,,, c,7 ) + I .

(4)

After the H(c,,, c,2 ) have been incremented for all thecontour points P,, in I, we evaluate the value ofR, (c„ , c,2 ) by the following rule .

For all cj , in C, do :'If H(c,,, c,2)=max{H(C,,, c,2 ), j2=1, . . ., NC2 ,,

then R,(c„, C,2 ) = TRUE;•

Otherwise, R,(c„, cj2) =FALSE .

Once the relation matrix RChas been obtained, wedirectly know the results of the matched metachains

M. Xie /Pattern Recognition Letters 15 (1994) 1143-1150

1145

between C, and C2 . Furthermore, the voting resultH(c;,, c,2 ) can be considered as a measure of confi-dence of matching for the pairing (c) ,, c, 2 ) . Or, onemay use the normalized value of H(c,,, c,,) bycalculating:

H(cj,, cj2)=H(c,,, c,2 )/ E E H(c,,, c,2 ) .,1 ,2

At this step of matching, we do not make use of anyinformation about the shape description of meta-chains. This may constitute an advantage in the sensethat this matching method could be applicable tomatch deformable metachains .

As can easily be seen from the above voting tech-nique, if c„ and c,, are two different metachains inC„ it is possible that the following situation occurs :

R,(cj,,c,2 )=TRUE and R,(c,, .,c,2)=TRUE .

This implies that the metachains (c;,, c,, .) have acommon matching partner cj2 . In this case, it seemsthat the two metachains should be fused into one me-tachain. We think that the fusion of metachains canbe performed more accurately by working with a se-quence of images . The reason is that a fusion processshould take into account the temporal coherence ofmatching. In our current implementation, we do notdeal with this problem .

Once we have estimated the matrix RC by HoughTransform, we can easily refine the results of matchedcontour points in knowing RC and the relation ma-trices (RPC,, RPCZ ) . In summary, our matching al-gorithm consists of the following three steps .

• Step 1 . To estimate the matrix RP by using anysimilarity criterion proposed in literature and not toinclude any local constraints .

•

Step 2. To estimate the matrix RC by the HoughTransform, in knowing RP and (RPC,, RPC2 ) .

•

Step 3 . To refine the matrix RP in knowing RCand (RPC, . RPCZ ) .

3. Experimental results

As already mentioned, the input to our algorithmis edge data . Before starting the matching process,some preprocessings are necessary which consist ofdetecting the contour points, doing edge thinning andedge linking . In our experiment, we use the Canny

1146

M. Xie /Pattern Recognition Letters 15 (1994) 1 14 3-1150

(0Fig. 1 . Example 1 : Stereo Matching. (a) and (b) are the pair of stereoscopic images ; (c) and (d) are the results related to the matchedcontour chains; (e) and (f) are the results related to the matched closed curves.

M. Xie /pattern Recognition Letters 15 (1994) 1143-1150

(a)

(e)

(b) (c)

(f)

1147

(g)

(h)Fig . 2 . Example 2 : Stereo & Motion Matching. (a) and (b) are the first pair of stereoscopic images ; (c) and (d) are the second pair ofstereoscopic images; (e) and (f) are the results of stereo matching ; (g) and (h) are the results of motion matching .

1148

(a)

(e)

M. Xie /Pattern Recognition Letters 15 (1994) 1143-1150

(b)

bE0u

(c)

CO POLJ

(d)

(g)

(h)Fig. 3 . Example 3 : Stereo & Motion Matching. (a) and (b) are the first pair of stereoscopic images ; (c) and (d) are the second pair ofstereoscopic images ; (e) and (£) are the results of stereo matching ; (g) and (h) are the results of motion matching .

edge detector to extract all the contour points in animage. For edge thinning, we use the algorithm pre-sented in (Hilditch, 1969) . This algorithm preservesthe connectivity of edges after the thinning . Once wehave the thinned edge map, edge linking can easily becarried out by a contour tracing process .The computational time of our matching algo-

rithm linearly depends on the numbers of contourpoints (after thinning process) and metachains (afteredge linking process) on each of the two input im-ages. This is because: (1) the matching in Step I canindependently be done for each contour point sincewe do not use any local constraint, and (2) the HoughTransform in Step 2 is also a parallel process .

In our experiments, a wide range of real images hasbeen used to test the robustness of the presented al-gorithm . Here, we just present three examples . Theimages are all of the size of 512 x 512. The first ex-ample shows the application of our algorithm to stereomatching. The last two examples illustrate the appli-cation of this algorithm to solve both stereo and mo-tion correspondence in a dynamic stereovisionframework. For the first step of matching, we haveused the method presented in (Xie, 1993) .

Example 1 . We have a pair of stereoscopic images (seeFig. I) acquired by a stereovision system mountedon an indoor mobile robot (INRIA Mobile Robot) .For this example, the most interesting primitives are"contour chains" or "closed curves" .

• Fig. Ic and Fig, Id show the results of stereomatching of contour chains : Fig . I c displays the dis-parity field of the matched contour points obtainedat Step 3 of our algorithm. Fig. ld shows the matchedcontour chains .

.Fig. le and Fig. If show the results of stereomatching of closed curves : The disparity field of thecontour points belonging to the matched closed curvesis presented in Fig . Ie. Fig. If shows the matchedclosed curves .

Example 2. The aim of this example is to show theapplicability of our matching algorithm to both thestereo and motion correspondence problem . We havetwo consecutive pairs of stereoscopic images (see Fig .2) as input. These images are acquired by a station-ary stereovision system which is observing a movingblock (with dominating rotation) . In this example,

M. Xie I Pattern Recognition Letters 15 (1994) 1143-1150

1149

we are interested in the matching of contour chains .• Fig. 2e and Fig . 2f show the results of stereo

matching: Fig. 2c shows the disparity field of thematched contour points obtained at Step 3 of our al-gorithm . Fig . 2f displays the matched contour chains .

• Fig. 2g and Fig . 2h show the results of motionmatching: In Fig. 2g, we present the optical flow ofthe contour points which is obtained at Step 3 of ouralgorithm. In Fig . 2h. we illustrate the matched con-tour chains .

Example 3. This example shows the stereo and mo-tion matching in an outdoor environment . These twopairs of stereoscopic images (see Fig. 3) are acquiredby a stereovision system mounted on a moving vehi-cle. Since the outdoor environment is more noisy thanan indoor scene, we have to select more stable imageprimitives to perform feature matching and sceneanalysis . In this case, it seems that closed curves areinteresting primitives for doing matching .

• Fig. 3e and Fig . 3f present the results of stereomatching: Fig. 3e shows the disparity field of thematched contour points obtained at Step 3 of our al-gorithm . Fig . 3f shows the matched closed curves .

• Fig. 3g and Fig . 3h present the results of motionmatching: In Fig . 3g, we show the optical flow of thecontour points which is obtained at Step 3 of our al-gorithm . In Fig . 3h, we illustrate the matched closedcurve .

4. Conclusions

Stereo and Motion matching (or correspondence)is a fundamental problem in computer vision . Now,the robustness of the matching algorithm becomes acentral issue in this domain. This paper has pre-sented one solution of using the Hough Transform toimprove the robustness of matching algorithm . Be-sides the robustness achieved by introducing theHough Transform, another big benefit is that we haveshown that it is no more necessary to use any localconstraints in performing matching . The immediateconsequence is that the developed matching algo-rithm in such a way becomes general and is directlyapplicable to both the stereo and motion matchingproblem. The presented algorithm is now imple-mented in a sequential way . Our future study will be

1150

to develop a parallel matching algorithm based on the

same principle as the one presented in this paper.

References

Ayache, N. and F. Lustman (1991) . Trinocular stereo vision forrobotics. IEEE Trans. Pattern Anal. Machine Intell. 13 (1),73-85 .

Bouthemy, P. (1990) . A maximum likelihood framework fordetermining moving edges . IEEE Trans. Pattern Anal.Machine Intell.11 (5),499-511 .

Deriche, R . and 0 . Faugeras (1990). Tracking line segments.Image and Vision Computing 8 (4),261-270 .

Grimson, W.E.L. (1985) . Computational experiments with afeature based stereo algorithm. IEEE Trans . Pattern Anal.Machine Intell. 7 (1), 17-34 .

M Xie /Pattern Recognition Letters 15 (1994) 1143-1150

Hilditch, C.J . (1969) . Linear skeletons from square cupboards .Machine Intelligence 4, 403-420 .

Liu, Y. and T.S. Huang (1991) . Determining straight linecorrespondences from intensity images. Pattern Recognition24 (6), 489-504 .

Maitre, H. (1986) . Contribution to the prediction of theperformances of the Hough Transform . IEEE Trans . PatternAnai.Machinelntell.8 (5),669-674 .

Medioni. G. and R. Nevatia (1985) . Segment-based stereomatching . Computer Vision, Graphics, and Image Processing31,2-18 .

Xie, M . (1993) . Tracking closed curves by a matching process .Pattern Recognition Lett . 14, 103-110.