AVC89: Polyhedral Object Recognition withSparse Data - Validation of Interpretations

Derrick Holder1 and Hilary Buxton2

!City of London Polytechnic, Department of C.M.S.M.S.,100 Minories, London EC3N 1JY.

2Queen Mary College, Department of Computer Science,Mile End Road, London El 4NS.

simple geometric constraints applied to sparse dataindependently of the coordinate frame of reference, thepossibilities can generally be whittled down to just a few,frequently only one, feasible interpretation. This is donewithout resort to a detailed solution of the surfaceequations. Nevertheless, in sequential form, the algorithmsare not generally fast enough to offer a practical solutionto the problem.

The method of Grimson, Lozano-Perez and others, for thegeneration of feasible interpretations of scenes with sparsedata, has been developed and implemented by the presentauthors on a distributed array processor, the AMT DAP,which operates in SIMD mode. Measurements involvingthe location vectors and the surface normals at m datapoints, considered in pairs, are compared with themaximum and minimum values associated with the nxnpairs faces of a polyhedral object model, in a process thatexploits nxn parallelism.

This paper discusses the subsequent validation of theinterpretations, in which data points have been assignedprovisionally to object model faces.

In many applications, the key task for a robot's visionsystem is to supply the control unit with a quantitative andsymbolic description of its surroundings, telling the robotwhat is where in the scene being viewed1. One approach tothe problem is to enable a robot to identify an object froma set of known objects, and to locate it relative to therobot's sensors.

Murray and Cook1, Grimson and Lozano-Perez2 Faugeras,Ayache and Faverjon3, and others, consider objects in theform of separate, possibly non-convex, polyhedra, forwhich there are accurate geometric models. An object mayhave up to six degrees of freedom relative to the robot'ssensors, which are assumed to be capable of providingthree-dimensional information about the position andorientation of a small set of points on the object.

The general approach to the problem is based on thehypothesis, prediction and verification paradigm that iswidely used in AI. First we generate feasibleinterpretations by means of simple, generally pairwise,geometric comparisons between object models and sensordata, and then we test the interpretations, in detail, forcompatibility with the surface equations of a particularobject model.

Several authors have implemented sequential algorithms,in which measurements involving the location vectors andthe surface normals at m data points, considered in pairs,are compared with the values associated with points innxn pairs of object model faces. It is found that, with

Until recently the best parallel algorithm for the generationof feasible interpretations was one implemented by Flynnand Harris on the Connection Machine at MIT4. Thisachieved parallelism at the expense of processor numberswhich grew exponentially with problem size. However, asimilar degree of parallelism has been achieved by thepresent authors5 on a distributed array processor, the AMTDAP 510, which operates in SIMD mode, with a processorset that is only quadratic in the problem size. This enablesproblems to be handled that would otherwise far outstripthe capacity of the Connection Machine.

We note that these algorithms can equally well be appliedto measurements at points on the edges of a polyhedron,with edge direction replacing surface normal.

Instead of using a small number of discrete measurements,edge matching generally involves the processing of asubstantial volume of grey level data, and the productionof a 2V2D sketch6-7. Nevertheless, this form of input isefficiently provided by the ISOR system8 developed atGEC Hirst Research Centre and currently beingimplemented on the AMT DAP at Queen Mary College.

The purpose of this paper is to discuss parallel algorithmsfor the validation of both face matching and edge matchinginterpretations of visual data. This is required because theinterpretations are based only on simple geometricconstraints, with no guarantee that the object modeldescription is entirely consistent with the data.

THE VALIDATION METHODThe problem is to validate an interpretation in which, inthe case of face matching, sensory data, expressed in termsof position vectors

measured.to within some volume of error relative to thesensor, and outward unit normals

19AVC 1989 doi:10.5244/C.3.4

measured to within some cone of error, have beenprovisionally assigned, for i - 1,2, ...,m, to particular facesof a given polyhedral object model.

The models are described within a database, in terms ofthe Cartesian coordinates of their vertices, together withequations for the faces, of the form

We have to determine R and r0 in such a way that, if jrelates to the object model face to which the data pointwith subscript i has been assigned,

and

i.e.

= n-Rn, = n-

dj + Rn,.r0 =

dj = di-n'i.r0.

where r is the position vector of a point in face j , in alocalised coordinate system, n ; is the outward unit normal,dj is the perpendicular distance from the origin and

An interpretation may be regarded as valid, only if it ispossible to find a rotation and translation of the givenobject model that would result in each data point beingplaced sufficiently accurately on the surface of the object,and, in the case of visual rather than tactile data, withevery data point visible, not being obscured by any part ofthe given object. The possibility of scaling an object modelto match the data is not considered in the algorithmspresented here.

The validation of an interpretation proceeds as follows:-

(i) For every feasible interpretation, establish thelocation and orientation of the given object modelthat is most compatible with the data;

(ii) Confirm that every data point is visible.

(iii) Confirm that every data point lies sufficiently closeto, and within the perimeter of, the object modelface to which it has been assigned;

In the case of edge matching, sensory data expressed interms of position vectors and edge direction vectors

have been provisionally assigned to particular edges. Theperpendicular vector p- from the origin replaces d'it butthe object model database and the validation method areessentially the same.

LOCATION AND ORIENTATION OF THEOBJECT MODELA rigid body rotation and translation may be expressed interms of a 3x3 orthogonal rotation matrix R, and atranslation vector r0 ,

so that

and

Tj -> Rr, + r0,

n, -» Rn;

The orthogonality condition RTR = I imposes 6 non-linearconstraints on the elements of R, and the application of the

method of constrained least squares to the residual vectorsRn, - n,- leads to the condition

R(S+L) = T,

where S and T are the 3x3 matrices whose elements aredefined as follows:-

i=\ i=\

, etc.

and

;, etc.,1=1 «=1

dj -> dj + Rn,.r0.

and L is a symmetric matrix of Lagrangian multipliers.

On the left hand side of the equation the matrix S is alsosymmetric, and it follows that R r T must be symmetric.We thus have three further linear conditions on theelements of R.

It has been demonstrated 9 that the solution of theseequations may be expressed in terms of singular valuedecomposition, with the best result selected from 4possible rotations. However, the Newton-Raphson processreadily lends itself to a parallel implementation. A goodfirst approximation, obtained from the relationship

RS = T,

that applies when the data exactly fit the object model,avoids the likelihood of convergence to a spurioussolution. In practice, the process converges to sufficientaccuracy after just one or two iterations.

The solution for r0 is obtained much more easily, with themethod of least squares applied to the residuals

We note that Faugeras, Ayache and Faverjon, work rathermore compactly with the quaternions (d,- ,n,-) and (dy.n,) toachieve what appears to be an equivalent result, but

20

presumably their algorithm is implemented in sequentialform.

DATA VALIDATIONHaving established the location and orientation of thegiven object model that is most compatible with the data,we may easily determine whether the locations of the datapoints are consistent with the object model face equations,but it remains to be verified that every data point lieswithin the perimeter of the face to which it has beenassigned, and that it is not hidden from view by anotherpart of the object model.

In order to do this we consider the intersections with theedges of a polygon, when a line is drawn from a given datapoint to some external point. There will be an odd numberof intersections if the first point is inside the polygon, andan even number of points if it is outside. For example, theline joining Q to S in Figure 1 has 3 intersections with theedges of the polygon.

AXQ+BYQ<C,

in which case Q is to the left of the line segment, or

Figure 1. A Point within A Non-Convex Polygon

For the purpose of counting intersections, we choose aviewing plane that, for simplicity and efficiency, isorthogonal to one of the coordinate axes, looking towardsthe object, with the external point S at an infinite distanceto the right of the viewing plane (Xs -»<»). We then writethe equation of a line segment, such as that joining Pk toPk+i, in the form

AX+BY

= (Yk+1-Yk),

where

and

If point Q has coordinates (XQJB) in the viewing plane, theline joining Q to S intersects the line segment if, and onlyif, either

and

and

AXQ+BYQ>C,

so that Q is to the right of the line segment, when lookingfrom Pk to Pk+l.

Then, for a given data point to be visible from the positionof the sensor, it must lie in a face that is not directed awayfrom the sensor, and its projection on the viewing planemust not fall within the perimeter of another face that isnearer to the sensor.

These requirements lead to the conditions

Rny.Fj >dj + Rny.r<),

i.e.

Rn;.(r,- - r0) > dh

for all j such that the projection of data point / falls withinthe perimeter of the projection of face j onto the viewingplane.

The same visibility check applies when values of r • arederived from edge matching data, but it then has to beestablished that every data point is sufficiently close to theedge segment to which it has been assigned.

SIMD IMPLEMENTATIONThere are three main directions in which parallelism maybe exploited in the validation processes described above: -

(i) processes applied simultaneously to each data point;

(ii) processes applied simultaneously with regard toevery object model face or edge;

(iii) processes applied simultaneously with regard to allof the edges associated with a given model face.

Parallelism of type (i) occurs throughout, from thesummations required in determining the location andorientation of the object model and checking against faceequations to the final data visibility check, whereas types(ii) and (iii) occur only during the visibility check.

The need for efficient calculation of the 3x3 matricesinvolved in the Newton Raphson process clearly calls forthe exploitation of type (i) parallelism, which also leads tosubstantial gains in the latter stages of validation. Auniformity of approach is considered important, as thisavoids the need for the restructuring of the database thatwould be necessitated by a shift of emphasis to fullyexploit type (ii) or (iii).

However, there is a substantial improvement in overallperformance when a limited restructuring of the objectmodel and data are undertaken at run time, with a view toachieving m xm parallelism during the visibility check.

21

The first task in an SIMD implementation of the validationprocess is to map the model against the data in accordancewith a given interpretation of the form

so that

j = face(i), i=l,2,...,m.

mapped_object(i,) = model(j,),

and to set the unused rows of the mapped object and datamatrices to zero. The initial rotation matrix, the solutionof the Newton Raphson equations and the translationvector for best fit are then computed using standard DAPLibrary subroutines. SHEP and SUM operations are usedto maximise parallelism in setting up the equations for theNewton-Raphson process. The rotation matrix and thetranslation vector are replicated before rotation andtranslation of the object model.

Before proceeding with the validation of individual datapoints, the x,y and z coordinates of the vertices associatedwith given faces, originally stored sequentially in rows, aremoved into the columns of separate DAP matrices, and thecoordinates of the data points are replicated in columnsusing a simple but effective binary algorithm. As aconsequence, when a given row of vertex coordinates isreplicated and related to a matrix of data coordinates, theprocess simultaneously relates every data point to everyobject model face, and m xn parallelism is thus achieved.The organisation of the information within DAP matricesat this stage is illustrated in Figure 2.

model

edgek

replicated

dat

face j

• • mode!

a point i

facej

•-. replicated

• • • * !

Figure 2. The Organisation of Informationwithin DAP Matrices

Transforming into viewing coordinates and initialising aDAP logical matrix inside = .FALSE, we proceed toinvestigate intersections of the line joining each data pointto the external point 5 with successive edges of every face,switching inside between .TRUE, and .FALSE, whenever

an intersection occurs. It is thus rapidly established whichdata points fall inside which faces when these are allprojected onto the viewing plane.

Then, with a fairly obvious nomenclature in DAPFORTRAN:-

in_face = inside(face,).AND.(ABS(delta).LT.tolerance)

and

visible = (.NOT.back_face).AND.ANDCOLS(in_front.OR.(.NOT.inside))

where the real matrix delta and the logical matricesbackjace and in_front are the results from straightforwardparallel calculations, and the function ANDCOLS collatesresults within a given row. Thus the process efficientlydetermines which data points lie sufficiently near to theface to which they have been assigned, and which if anyare not visible from the position of the sensor.

TEST RESULTSThe validation process has been applied successfully to anL-shaped block with 8 faces, viewed as illustrated inFigure 3, and with 8 data points, three of which are in factvisible, at the centre of each face. The total processingtime required to determine the location and orientation ofthe object and to validate the data is about 42 milliseconds.

Figure 3. A View of an L-Shaped Block.

Further tests on the validation process involved threedifferent views of a three-pin electric plug with 27 faces.The first view of the plug, with 14 visible faces, isillustrated in Figure 4. In the second view, lookingupwards with 12 visible faces, the pins were partiallyobscured. The third view, was looking directly down ontothe pins, with 4 faces clearly visible and 5 more at anoblique angle to the sensor.

The data for View 1 consisted of 14 data points at thecentre of each visible face, the one on the visible side ofthe plugbeing obscured by the flange. Three back facepoints were also included in the test data. There were 17data points for View 2, including two in faces of the earth

22

Figure 4. - The Electric Plug (View 1)

pin that were obscured by the neutral pin and one in theunderside of the flange on the far side of the plug, that wastotally obscured by the rest of the plug. There were alsofour back face data points for View 2. For View 3 therewere 12 data points, three in back faces but none otherwiseobscured.

The location and orientation of the plug were determined,in each case, within about 26.5 milliseconds, and the backface and obscured data points were identified by thevalidation process in a further 12 milliseconds. Thisbrought the overall processing time, including thatrequired to generate a feasible interpretation, to about 90milliseconds. A complete breakdown of the validationprocessing times for the electric plug is given in Table 1.

Convert to DAP formatMap object against dataCompute S,T and initial RNewton-Raphson for RRotate objectCompute U and vSolve for r0

Translate objectTotal to locate objectReplicated viewing coordsCheck for inside faceCheck for in face,in front, in back faceTotal to check data

Total to validateinterpretation

processing time(milliseconds)

0.40.9

5.911.14.31.62.31.3

26.52.08.2

2.012.2

40.0

Table I. Processing Times for the Electric Plug

Further tests were then made with simulated errors in thespatial coordinates of the data points, and the surfacenormal directions.

It was found that, whereas coordinate errors of about 0.05inches might simply result in the rejection of the offendingdata points, with the electric plug being about 1.5 inchesacross and viewed from a distance of about 5 inches,errors of the order of 0.25 inches resulted in substantialerrors in r0, leading to the rejection of several valid points.The orientation of the plug, and the run times forvalidation, were not affected by errors in spatialcoordinates.

On the other hand, errors ranging from 0.1 to 0.2 in thedirection cosines of the surface normals led to errors inboth R and r0, with the subsequent rejection of severalvalid points. Again, there was no change in run times,because the errors were not sufficient to provoke furtheriterations of the Newton Raphson process, in computing R.

A possible strategy for dealing with errors such as thesewould be to recompute R and r0 with the suspect pointsremoved and, if necessary, to check them again, againstthe revised location and orientation of the object model.When the position of the plug was recomputed with justfour of the data points from View 1, the maximumdiscrepancy in the elements of R was about 0.02, themajority were much smaller, and the maximumdiscrepancy in the components of r0 was 0.01.

CONCLUDING REMARKSIt has been established in this paper that the hypothesis,prediction verification paradigm that is widely used in A.I.can be applied in SIMD parallel processing mode to theproblem of object recognition, within a very realistic timescale.

Given object model descriptions stored in a database,together with a small number of feasible interpretations, inwhich sensory data points have been provisionallyassigned to the faces of a given object model on the basisof simple geometric comparisons, we have shown how toestablish the location and orientation of the object modelthat is most consistent with the data. We have also shownhow to confirm that every data point is visible and liessufficiently close to the face to which it has been assigned.

It may thus be established which of the interpretations ismost closely consistent with the data, and the location andorientation of the object corresponding to the bestinterpretation is then available for input to a controlsystem.

The overall timescale for interpretation and validation,with obscured data points, back face data and errors in thespatial coordinates of data points correctly identified, isabout 90 milliseconds for the electric that is used as anexemplar.

We have been concerned herein with the static recognitionproblem, with data derived from a snapshot in time, but inprinciple the results might be applied to the dynamicsituation in which previous interpretations can be used to

23

narrow down the search for feasible interpretations, andpossibly speed up the validation process, with priorknowledge of the location and orientation of certainobjects in the scene gained from previous runs of therecognition system.

In this case, failure to arrive at a definite interpretation, atthe outset, is not likely to be a serious problem, for we maydistinguish between exploratory mode, in which we aresimply gathering data with potential feedback to the sensorof requests for further information, and active mode whena valid interpretation is required by the control system.

In the meantime, work is proceeding on the interpretationof edge matching data, and the subsequent validation ofinterpretations, to meet the demands of interfacing with theparallel version of the ISOR system that was mentionedin the introduction to this paper.

REFERENCES

1. Murray D.W. and Cook D.B. "Using the Orientationof Fragmentary 3D Edge Segments for PolyhedralObject Recognition" International Journal ofComputer Vision Vol 2 (1988) pp 147 - 163.

2. Crimson W.E.L. and Lozano-Perez T. "Model-Based Recognition and Localisation from SparseRange or Tactile Data" International Journal ofRobotics Research Vol 3 (1984) pp 3-35.

3. Faugeras O.D., Ayache N. and Faverjon B. "AGeometric Matcher for Recognising and Positioning3-D Rigid Objects" Proc. AJ. Applications ConferenceIEEE (1984) pp 218-224.

4. Flynn A.M. and Harris J.G. "RecognitionAlgorithms for the Connection Machine" Proceedingsof the Ninth International Joint Conference onArtificial Intelligence, Morgan and Kaufman (1985) pp57 - 60.

5. Holder D.R. and Buxton H. "Polyhedral ObjectRecognition with Sparse Data in SIMD ProcessingMode" Proceedings of the Fourth Alvey VisionConference (September 1988) pp 103-110, alsopublished in Image and Vision Computing (February1989).

6. Castelow D.A., Murray D.W., Scott G.L. andBuxton B.F. "Matching Canny Edgels to Compute thePrincipal Components of Optic Flow" Proceedings ofthe Third Alvey Vision Conference (September, 1987)pp 193-200.

7. Murray D.W., Castelow D.A., and Buxton B.F."From an Image Sequence to a Recognised PolyhedralModel" Proceedings of the Third Alvey VisionConference (September, 1987) pp 201-210.

8. Buxton H. and Wysocki J. "The SIMD ParallelImage Sequence Object Recognition (SPISOR)System" (in draft).

9. Buxton B.F. Personal communication (February,1989).

24