a comparative study of nonlinear shape models for digital image processing and pattern recognition

858 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 20, NO. 4, JULY/AUGUST 1990

A Comparative Study of Nonlinear Shape Models for Digital Image Processing and

Pattern Recognition ZI CAI LI, QUAN LIN GU, CHING Y. SUEN, AND TIEN DAI BUI

Abstract -Four nonlinear shape models are presented: polynomial, Coons, perspective, and projective models. Algorithms and some properties of these models are provided. In particular, for a given physical model such as a perspective model, comparisons have been made with other mathematical models. It has been proved that under certain conditions, the perspective models can he replaced by the Coons models. Problems related to substitution and approximation of practical models that facilitate digital image processing are raised and discussed. Experi- mental results on digital images are also presented.

I. INTRODUCTION

INEAR AND BILINEAR MODELS have been de- L veloped for shape transformations in digital images and patterns in [2]. However, nonlinear models are needed to process images and patterns that have undergone an arbitrary distortion. Hence, we will provide the following important models: 1) Polynomial models and the Coons models that can be regarded as basic, mathematical models in shape transformations; 2) Perspective and projective models as physical models corresponding to human vision.

In this paper, we will focus on an interesting aspect that leads to the discovery of substitutions or approximations for a physical transformation by other basic, mathematical models. This study is significant because the physical transformation

T : (5,771 + (X,Y), x = x ( 5 , 7 7 ) , Y = Y(5?77) (1)

is often unknown or unclear. For instance, suppose that a distorted iniage (or pat-

tern) has been obtained when the functions ~(5,771 and y ( 5 , ~ ) in (1) are unknown. Our concern is to find another

Manuscript received October 17, 1989; revised March 16, 1990. This work was supported in part by the Natural Sciences and Engineering Research Council of Canada, by the Fond pour la Formation de Chercheurs et 1'Aide ?i la Recherche of Quebec and by the Minist2re de 1'Enseignement SupCrieur et de la Science (Action Structurante).

Z . C. Li is at Centre de recherche informatique de Montreal, Inc. Montreal, PQ, H3G IN2 and at Department of Computer Science, Concordia University, Montreal, PQ, H3G 1M8, Canada.

Q. Gu was at Jiangsu Institute of Computing Technology, Nanjing, Jiangsu, The People's Republic of China and is now a Visiting Scholar at the Department of Computer Science, Concordia University, Mon- treal, PQ, H3G 1M8, Canada.

C. Y. Suen and T. D. Bui are at Centre for Pattern Recognition and Machine Intelligence, Department of Computer Science, Concordia University, Montreal, PQ, H3G 1M8, Canada.

IEEE Log Number 9036028.

model f to replace or approximate T such that:

1) under f, this distorted image (or pattern) is obtain- able fro? the standard image;

2) under T-', this distorted image (or pattern) can be, at least approximately, normalized back to its origi- nal shape.

Even in the case when the function forms of the physical model T are known but difficult to compute or to find the unknown parameters or constants, the substitution or approximation of T is also important in simplifying the transformation process.

We shall describe two mathematical models: the polynomial and Coons models in Section 11, and two physical models: perspective and projective models in Section 111. Theoretical comparisons of the different models will be made in Section IV. The results of a series of graphical experiments are presented in the last section to verify our theoretical results.

11. BASIC MATHEMATICAL MODELS

We confine the transformation to two dimensions (2-D): T : (5,771 + (X,Y), x = x ( 5 , 7 7 ) , Y = Y(5,77),

( 2) where 5077 is the standard Cartesian coordinate system, and XOY is another Cartesian coordinate system. The transformation (2) converts a standard image (or pattern) in 5077 to another image (or pattern) in XOY. We also suppose that the functions x(5,q) and ~(5,171 in (2) are explicit.

A. Polynomial Models

respect to 5 and 77. We have The functions x ( 5 , q ) and y ( 5 , ~ ) are polynomials with

1) The polynomial models with K-order: T : (5,771 +(X,Y)7 ( 3 )

where K I K J

x = ai51771--1, y = b/5'q1-', (4) J = 0 - 1 = 0 1 = 0 1 = 0

where ai and b: are constants.

0018-9472/90/0700-0858$01.00 01990 IEEE

LI et al.: COMPARATIVE STUDY OF NONLINEAR SHAPE MODELS 859

2) The polynomial models with Bi-K-order defined as (3) and

K x = U i j 5 ' 7 7 j ,

i , j = O

K y = b i j t i v i ,

i , j = O

Fig. 1. Biquadratic transformation.

where a; ; and b j i are constants. where When K = l , we obtain respectively the linear and x = x ( 5 , 7 7 ) = + , ( s , ~ ) ~ ~ + 4 B ( 5 , 7 7 ) x B + 4c(5,77)~c

bilinear models in [2]:

with the constants a,,, b,,; . - , s , r . Suppose that the standard image CR in 5077 is imbed-

ded into a unit square, and that the transformed image S in XOY will be imbedded into a quadrilateral with the boundary curves of polynomials of order K. This transformation can be described by the polynomials of Bi-K-order. The four boundaries of S consist of straight lines for K = 1 and curved lines for K > 1.

There exist the corresponding relations of the reference points between the images S and CR:

PI + P I , F I ~ C R , P I ~ S CR ;S. (8)

Then we can define the basic shape functions 4p(t,77) such that

- T

4c(5,77) = 5 d 2 5 +277 -3)9 40(5,77) = d l - 5)(277 - 2 5 - 4 E ( 5 9 7 7 ) = 45(1- < ) ( I - 77) 7

4 F ( 5 , 7 7 ) =4577(1-77), 4 G ( 5 , 7 7 ) =4577(1-07 4 H ( 5 , 7 7 ) =477(1-5)(1-77). ( 12c)

The expression of ~(5,771 is the same as in (12b), except the coordinates x,, xB, etc. are substituted by yA, y e , etc.

Similarly, other polynomial models such as quadratic, cubic, bicubic models etc., can be derived from the finite element method (see [51).

B. The Coons Models

4p( tP q p ) = 1 , + p ( tQ, v Q ) = 0 when P + Q. (9) We will develop the Coons models of Forrest ill, under

When the shape functions +p(5 ,q) are polynomials, we construct a transformation

T : (5,77) + ( X , Y ) , (loa)

Y = C 4 P , ( 5 ? 7 7 ) Y P , . (lob)

where

x = C 4 P , ( 5 , 7 7 ) X P t 7 I 1

Obviously, the functions x(5, 77) and y ( 5 , ~ ) are polynomials, and the transformation T also satisfies (8).

Based upon the above consideration, we can apply the shape functions in the finite element method with piece- wise K-order Lagrange polynomials [7], [SI to the shape models of image transformations. For example, let K = 2, there are the point-to-point relations (see Fig. 1):

A - & B - E , C - c , D - D 7 ( l l a )

E - E , F - F , G - e , H - f i , ( l l b )

where A , B , C , D are the corner points of aR, and E , F , G, H are the midpoints of a i l , and aCR is the boundary of CR in 5077. Therefore a biquadratic model is defined such as (see Fig. 1)

_ - _ _ _ - _ _

T : (5?77) + ( X , Y > ? (12a)

which an image or pattern can be transformed into arbitrary frames and boundaries. This flexibility will greatly facilitate conversion and recognition of images and patterns, such as the Roman letters and Chinese characters.

Let the unit square be the standard reference element in 5077. Suppose that the boundary curves of the reference element in XOY are known, instead of only the reference points known as in the case of the polynomial models. Denote their mapping relations:

A - A, B ++ B , c - C , D - 0, (13a) AB - ZB, BC - BC, DC - Dc, A D - ZB, (13b)

where the boundary curves AB, etc. can be represented by (see Fig. 2). Then

The arc between A , B : (xl(5), y l ( 5 ) ) , 0 Q 5 < 1

The arc between D , C : ( x2( 25) , y2( 5 )), 0 6 5 Q 1

The arc between A , D : ( x 3 ( q), y3( q)), 0 Q 77 < 1

The arc between B , C : ( x4( 77) , y4( v)), 0 Q 77 < 1

for ~ = 0 , (14a)

for 77'1, (14b)

for 5 = 0 , (14c)

for [=1. (14d)

,860 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 20, NO. 4, JULY/AUGUST 1990

Fig. 2. Coons transformation.

The linear Coons models can be reduced from [ll

When A D and BC are straight lines, we obtain the simple model T,:

x = (~1(r,J2(5,)[ '11111,

Y = ( Y l ( 5 L Y 2 ( 5 n [ ' 7 1 . (16)

When AB and DC are straight lines, we obtain the model T2 in Fig. 2

Below, we present the general Coons models. Let the continuous functions H,(5), Ql(5), H2(77), Q2(77) be monotonous such that

d - H i ( 5 ) < 0 in (O,l), d5

H i ( 0 ) = 1, Hi(l) = 0, i = 1,2, (18)

and

d -ai([) > O in (O,l), d5

Qi(0)=O, Q i ( l ) = l , i=1 ,2 . (19)

Then the general Coons models can be defined as

The expression of y is similar. Evidently, the linear Coons models (15) are the special cases of (20) etc. when

H l ( 0 = H 2 ( 0 = 1- 5, Q l ( 0 = Q&) = 5. (21) Besides, the functions H,(5) and Qi(S ) can be chosen alternatively. For instance, the quadratic Coons models can be reduced from (20) with

Hl(5) = ffz(5) = 1 - t 2 or ( 1 - 5 ) 2 , Ql(5) = @ 2 ( 5 ) = t2? (22)

and the trigonometric Coons models with n-

Hl(5) = H 2 ( 5 ) =cos T5, 77

Ql( 5) = Q2( 5) = sin -5. (23) 2 For simplicity, we consider the case Tl of the straight

lines AD and BC in Fig. 2. Then the functions in the Coons models are reduced to

where d d

-H2(77)<0 and -Q2(77)>0 in (O,l), (25a)

H2(0) = 1, H2(1) = 0, Q2(0) = 0, Q2( l ) = 1. (25b) d77 d77

Then we have the following.

and (25) are one-to-one if Lemma 1: The Coons models with the functions (24)

The points on AB and AB aLe-one-to-one in image conversion, and so those on DC and DC. Any two curved lines in XOY, which are transformed by the Coons models from two different straight lines

5=51, 5 = 5 2 , 5 1 # 5 2 , (26) do not have an intersecting point.

Proofi First, in terms of (241, (25) and Assumption 1) the one-to-one property can be easily proved for the points on a straight line 5 = t1 and on the transformed curve in XOY.

Next, suppose that two points, (tl, ql) and ( E 2 , r / 2 ) where t1 # tZ, are converted into the same point (xl, y l ) in XOY. This implies that an intersecting point ( x l , y l ) has already been found for two Curved lines, transformed

LI et al.: COMPARATIVE STUDY OF NONLINEAR SHAPE MODELS

A 3-DSURFACE To be placed in

861

A PICTURE IN 2 - D perspective or projective t d o m t i o n s

IMAGES IN 2 - D ~~

Fig. 3. Composite transformation T in 2-D

by the Coons model (24) from two straight lines (261, thus contradicting the lemma Assumption 2). This completes the proof of Lemma 1.

Lemma 2: Suppose that a unit square image is converted by a one-to-one transformation into a quadrilateral with two straight lines AD and BC (see TI in Fig. 2). Also the functions x and y can be represented by

x = 77R,(5) + R 2 ( 5 ) , Y = 77U5) + S2(5), (27) where R,(t), R2(5), Sl([), and S2(5) may be arbitrary, continuous functions. Then such a model is one of linear Coons models defined by (16).

Proofi We can rewrite (27) as

Since the one-to-one propertyhglds for the points on XB and AB, as well as those on DC and DC, we obtain

=RA[), x2(5) = R l ( t ) + R A [ ) , (29a)

Yd5) = S2(5), Y2(t> = Sl(5) + M b ) . (29b) This completes the proof of Lemma 2.

111. PHYSICAL MODELS

Two important physical models, namely the perspective and projective models, will be discussed in this section.

Consider the following composite transformations that happen often in commercial advertisements (see Fig. 3): A piece of paper with pictures, Roman letters, or Chinese characters is attached on a three dimensional surface such as a cubic wall, a cylinder or a sphere. When a picture of this is taken by a camera, or viewed by a human as an optical image this composite transformation T is still 2-D:

(30) - = T2-D t 2-D - p2-D + 3-D '3-D t 2-D >

where C3-D+2.D denotes the conversion from the 2-D picture to a 3-D curved plane, and P,, 3-D denotes the perspective or projective transformation from a 3-D image to a 2-D image (see Fig. 3). For the perspective or

PelPpeC'iY. Point

Fig. 4. Perspective transformation.

projective transformation discussed in this paper, we shall use the simpler formulas in the Cartesian coordinates, and the formulas in the homogeneous coordinates can be found in [6].

Suppose that a 3-D image Cl([,q,l) is transformed into a planar image by projection. Let 5775 and XYZ be two Cartesian coordinate systems such that the origin of XYZ is the perspective point, and the coordinates of the origin of 6775 are (xo, yo , 2,) in XYZ. Therefore, the two coordinate systems are related by translation

x = ~ + x , , y = q + y 0 , z = ~ + z , . (31)

A perspective plane T is fixed by three points (xi, yi, zi), i = 1,2,3, which do not lie on a straight line. Therefore, the perspective or projective plane T is denoted by

x y z l x1 Y l 21 1 x2 Y2 2 2 1 x3 Y 3 23 1

Equation (32) can be reduced to

Ax + By + c z + I

where the constants are

A. Perspective Transformations

Any point ((,q, 5) in t775 can be projected to an image point (x,y,z) on T through the perspective line 1 (see Fig. 4):

Y (34) -- --=-

5 + x o T+YO 5 + z o '

where none of the denominators in (34) is zero. The perspective transformation can be defined by (33) and (34) with four reference points (xi, yi, zi), i = 0,1,2,3.


z

t 3-D image -

Fig. 5. Normal n‘ of perspective plane T .

Also (34) yields

Substituting (35) into (33), we obtain

x = - D ( 5 + x o ) / F ,

z = - D ( 5 + z 0 ) / F

Y = - D h + Y o ) / F ,

(36)

with F = A(5 + xo) + B ( q + y o ) + C ( l + zo). We note that the functions x, y , z are fractional linear (not linear!) with respect to 5,q , 5.

Next, since the projective image lies on the inclined plane T in 3-D coordinates (x, y , z), it should be mapped into a 2-D plane for final computer graphics. By using rotations, if one of the three coordinates on z- is constant, the other two coordinates will form a useful 2-D coordinate system.

Denote G = ( A 2 + B 2 + C2)1/2. The normal n’ of T is denoted as ( a , p, y ) from (33) (see Fig. 5 )

cos (Y = COS ( ii, 2) = A / G , COS p = COS (E’, j’) = B / G ,

COS y = COS (n’, Z‘) = C / G , (37)

where 2, j’, z‘ are the axis vectors. Without loss of general- ity, we assume

Denote the projective vector of n‘ on the plane XOY by

ii= (sin 8, cos 8 , O ) , (39) where

B A COS e = sin 8 = ( 40)

( A2 + B2)l12 ’ ( A 2 + B2)l12 ’

Therefore, we can transform the normal Z of T to be parallel to the axis Z by means of combining the following two rotations.

1) The projective plane T is rotated around the Z-axis for a counter-clockwise angle 8. Then n’ will fall on the plane YOZ.

2) This plane T is again rotated around the X-axis for a counter-clockwise angle y.

0-X Fig. 6. Projective transformation.

The two rotations are expressed as

cos8 -sin8 R,= [ o sin8 cos8 0 :j, 1

1 0

0 siny cosy

Therefore, the composite matrix R is

- sin 8

O I cosycos8 - s h y . (42)

sin y sin 6 sin y cos 8 cos y

Let (;I = R ( i), (43)

we can see

X = [ Bx - A y ] / ( A 2 + B 2 ) 1 / 2 . (44a)

j = [ C A + CBy - ( A 2 + B 2 ) z ] / [ G ( A 2 + B2)1’2] ,

(44b)

Z = - D / G . ( 44c)

We note that the value of Z is constant. Then (X, j ) yield the plane coordinates required in 2-D.

B. Projective Transformations

The point ( e ,q , { ) in 575 can also be projected to an image point (x, y , z) on z- (33) through the projective line 1 (Fig. 6).


I

Fig. 7. Picture placed vertically in 3-D.

where none of A , B , and C is zero. Then we obtain B A Y = 77+ YO+ - ( x - 5 - xo) ,

r

image in 3-D

pprpcftive point

Perspective transformation on perspective plane T: y = y * . Fig. 8.

The picture in 3-D remains vertically, but it can be curved arbitrarily along 6 and ;ii.

L

z = 5 + zo + -( x - 5 - x o ) . A

For the perspective transformation, we assume the per- (46) spective plane (Fig. 8)

Therefore, combining (46) and (33) yields 7T: y = y * , (49)

x = { ( B 2 + C2) ( ( + x o ) where y* is a constant. Then the following coordinates (x,z> from (36) form the Cartesian system in 2-D:

- A [ D + B ( v + y o ) + C ( S + z o ) ] } / G , (47a)

T / + z o + c (51)

f d 5 ) + xo

= y * f * ( 5 ) + Yo ’ z = y * f 2 ( 5 ) + Y 0 - c[ D + 4 5 + X O ) + B(77 + Y o ) ] } / G . (47c)

We note that the functions x , y , and z in (47) are all linear with respect to 5, 77 and 5. The rotational simplifi- cation of x, y , z to a 2-D coordinate system is the same as in Section 111-A.

Then a composite transformation T, of the perspective transformation in 2-D is defined by

(52) T,: (5,17) + ( x , z ) .

IV. THEORETICAL ANALYSIS OF MODEL COMPARISONS

For a given physical model (either a perspective or projective model), we shall find a certain mathematical model such as a polynomial or Coons model to substitute or approximate it. This study may simply facilitate the image conversion for that physical model. The substitution or approximation of a real, physical model is significant when the real model is unknown or unclear. For instance, a distorted image or picture may be found, yet the transformation functions x ( 5 , q ) and y ( 5 , ~ ) in (1) are not known. Only some information about the physical background of the image or picture may be known (e.g., where the image comes from, whether or not from the perspective transformation).

Theorem I : Let the perspective plane 7~ be given as (49). Suppose (48) and the following formula hold:

where as is the boundary of the obtained image in XOY. Then the model T, of the perspective transformation can be replaced precisely by the following linear Coons model Tc (see Tl in Fig. 2).

where

A. Substitution o f the Perspective Models (54b) Consider a planar image in a unit square’ R = (0 f 5 f 1,

0 Q TJ f 1). Suppose that this picture is curved as shown in and the continuous functions ( ~ ~ ( 5 1 , y l ( 6 ) ) and Fig. 7, where the coordinates in 3-D are ( x , (5 ) , y2 (5 ) ) denote the curves AB and DC in Fig. 2,

- respectively. $ = f l ( 6 ) , T = f , ( s ) , f = q + c , c isconstant,

Proo~? We rewrite (51) as (48)

where f 1 ( 5 ) and f 2 ( 5 ) are explicit continuous functions. x = R , ( 5 ) , z = 17S1(5) + S , ( 5 ) , ( 5 5 )

864 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 20, NO. 4, JULY/AUCUST 1990

Fig. 9. Picture placed in 3-D, paralleling the axis t.

where

. (56b) Y * Y * ( Z O + c)

f 2 ( & ) + y o 3 S 2 ( S ) = f 2 ( ( ) + y o

Consequently, by noting the function form in (5.51, the desired conclusion of Theorem 1 is true, based on Lemma 2. This completes the proof of Theorem 1.

By applying Theorem 1 we obtain

Corollary I: Let (48) and (49) hold. Suppose that a known image S has been converted by Tp (52) from its standard form. Then this standard form can be recovered precisely from S by the inverse transformation T;’ of the linear Coons model (54).

The conclusions in Theorem 1 and Corollary 1 will be verified by the numerical and graphical experiments in Section V.

Similarly, the standard picture R is placed in 3-D in a

Fig. 10. Perspective transformation on perspective plane T * : a x + B y = r . (56a)

slightly different way from (48) (see Fig. 9)

f = [ + c, where c is a constant,

7 = f i (77) , 4 = f 2 ( 7 7 ) .

The composite model Tp in 2-D is now defined by

T,: (5,77) + ( x , z ) ,

where

5 + x o + c f 2 ( 7 7 ) + 20

fI(7-l) + Y o fd77) + Y o . x = y* , z = y *

Corollary 2: Let (49), (57) hold true. Suppose that a known image S has been converted by Tp (58) from its standard form. Then this standard form can be recovered precisely from S by the inverse transformation T;‘ of the linear Coons model (59).

Now let us consider a vertical, perspective plane in Fig. 10:

T*: a x + p y = r , (60) where a, p, r are constants, and a 2 + p 2 # 0 . When (Y = 0, the simple perspective plane (49) is obtained from (60). The coordinates on T* can be reduced from (36)

r (F+ X o ) 477 + Y o )

r( 4 + 2 0 )

x = - H ’ * 7 Y =

(61a) z = - H ’

where

H = a l + p 7 j + a x 0 + p y o . (61b)

For this case, instead of (41) we can use a rotational matrix defined by

(57)

(58a) 0 1

and perform the rotational transformation to obtain a 2-D Hence we can prove the following theorem in a similar manner.

model Tp (58) can be replaced precisely by the following linear Coons model T, (see T2 in Fig. 2)

system of coordinates

Theorem 2: Let (49), (53) and (57) hold. Then the

T,: (5,77) + ( X , Z ) , (59a)

(63)

where 1

z = dm = constant, (64a)

r [ a7 - p5 + ay0 - P X O ] ?

(2;;) ( 59b)

- r (C+zo) where the continuous functions ( x 3 ( q ) , y 3 ( ~ ) ) and

tively. H ’ ( x 4 ( v ) , y 4 ( q ) ) denote the curves AD and BC, respec- z=------


Then a composite transformation Tp* can be defined by

Tp*: (5,77) + ( J , Z ) . (65)

Next, suppose that the standard image S is curved in

j = f d S ) , 7 7 = f 2 ( t ) , f = a ~ + b , (66)

3-D with the coordinates

where a and b are constants. Therefore, we obtain the Fig. 11. Perspective transformation on cylinder with ellipse. coordinates on T* after the rotation:

r Moreover, for the given spatial coordinates (48), we obtain [.f,(S> - P f d S ) + a y 0 - P x o l , (67a)

r H Z = -( a77 + b + z o ) ,

( 74) (77 + zo + c )

( f l (5) + .o> z = a COSRI(5). where

(67c) We note that the pair (l(t), z ) represent the coordinates

Theorem 4: Let (48) and (70) hold true. Suppose that

H = .fl(O + P f L 5 ) + ax0 + P Y O - By means of Lemma 2, we can obtain after expanding 7i into a plane in 2-D.

Theorem 3: Let (60) and (66) hold true. Also suppose

aa -5 as, (68) asz 5 as, (75)

where as is the boundary of S on T*. Then the model T,* (65) of the perspective transformation can also be replaced precisely by the linear Coons model (54).

Corollary 3: Let (60) and (66) hold true. Suppose that a known image S has been converted by T,* (65) from its standard form. Then this standard form can also be recovered precisely from S by the inverse transformation T;' of the linear Coons model (54).

If the coordinates (66) are changed to

5=f1(77), 77=f2(77), 4 = a S + b , (69)

we can obtain a similar theorem and corollary for the perspective images on T* as in Theorem 2 and Corol- lary 2.

The previous analysis can even be extended to the case of a vertical, perspective curve

+ . :x=acos t , y=bs in t ( 70)

with the constants a, b > 0. In fact, (70) denote an elliptic cylinder (see Fig. 11). We then have from (34)

acost bsint z (71) -=-=- -

5+x, T+YO J + Z O '

Then it follows that

acost. (72) 4-t ZO

When the perspective surface 7i is Fxpanded in a 2-D plane, the composite transformation T, can be defined by

f,: ( 5 , ~ ) + ( l ( t ) , z ) , l ( t ) =cda2s in2 t+bZcos2 t dt.

(73)

where as is the bpndary of S on 7i after the expansion. Then the model T, (73) can also be replaced precisely by the linear Coons models (54).

Corollary 4: Let (48) and (70) hold true_Suppose that a known image S has been converted by T, (73) from its standard form. Then this standard form can also be recovered precisely from S by the inverse transformation T;' of the linear Coons model (54).

The substitution of physical models should still be useful even when the function forms are given by (50) or (74). Since the perspective point (xo ,yo ,zo) may not be easily found, or the integration calculation in (73) is complicated.

Theorems 1-4 summarize the results of model substitution for the vertical plane or cylinder as a perspective surface. To clarify, we draw the applicable conditions of the perspective surface, and the spatial coordinates in 3-D for the model substitution of T, of the perspective transformation by the linear Coons models in Fig. 12(a) and (b). For other perspective planes or curved surfaces, i.e.,

we only provide the eligible locations in 3-D for model substitution in Figs. 12(c)-(f), without detailed discus- sions.

Below we consider approximation of models. Let (49) be the perspective plane, then the coordinates (50) are obtained. Suppose that

We then obtain the following approximations

T: x = x*, or T*: a x + /3z = r , (76)

1771 IYOl. (77)

1 1 = L ( I - ; ) = ~ . - 1 (78) -= ? + Y O Y , [ l + V / Y O l Yo


(C) (f) Fig. 12. Circumstances of spatial and perspective planes for substitution of T, by linear Coons models.

Hence, the fugions (5_0) can be approximated by the polynomial of 5, T j and f :

Y* ( 3 Y o ( :I Y o x ” - ( f + x o ) Y* 1-- , z = - ( J + z o ) 1-- 3

or more roughly by x = - ( f + Y* xo), z = -(H+ Y * 2 0 ) . (80)

Y o Y o

Suppose that the standard image S is placed in 3-D with the following coordinates:

s= a115 + a12577 + a2277 + r ,

r= bl15 + b12577 + b2277 + s,

77 = (P(5,77), (81a)

(81b)

where all , a I 2 ; . -, r , s are constants, and

We obtain

Theorem 5: Let (49), (80, (82) hold true. Then the composite model T, (52) of the perspective transformation can be approximated by the bilinear model with (7).

Based on the approximations (79) or (80), other approximate models can also be found.

B. Substitution of the Projective Models

We note that the functions x, y , z in (47) are linear with respect to t , ~ , f . The analysis of substitution of models is easy. We can see

Theorem 6: Let (49) hold true. Suppose that the standard form S is curved in 3-D with 5,Tj, as polynomials

I (P(5A) I -=K IYol. (82)


of K-order or bi-K-order. Then the composite model T, of the projective model in Section 111-B is also of polynomial models with K-order of bi-K-order.

V. NUMERICAL AND GRAPHICAL EXPERIMENTS

In our experiments, only the composite model Tp of the perspective transformation is chosen as the physical model. Let the perspective plane n- be (49). The region of the standard image within a rectangle S*(O G e* < a, 0 < 77* < b ) is curved in 3-D vertically with the coordinates (48). In order to apply the analysis of substitution in Section IV-A, a dilation

will be first employed to convert S* to a unit square S

We are interested in the images through T, T-’, and T- ‘T, which are performed by the discrete techniques: such as the splitting-shooting method [3], the splitting- integrating method [41, and the combination CSIM [5]. CSIM denotes a combined procedure of the splitting- shooting method for T first, and then the splitting- integrating method for T-’.

We shall carry out two experiments for model substitution. The first experiment is described as follows.

1) Based on the boundary dS of the distorted image S that has been transformed through the composite model T, of the perspective transformation, this image may be converted through T-’T of other shape models from the standard form LR.

In Fig. 13, a rectangular image (2-D) of the gate of a building on the left is wrapped around a cylinder with a quadratic wall (3-D). The projective image on the plane n- is upside down and drawn on the top of the figure. This defines a composite model T, on 2-D with the conversion functions (51). The pattern of a “Gate” is converted through T, and T,- ‘T, by the splitting-shooting method and CSIM, respectively. The normalized pattern (i.e., the one through T-’T) is shown in the right of Fig. 13(a). It is worthy noting that in performing T-’, there is no need to solve the nonlinear equations (51) to obtain (6,771 from ( x , Y ) (see [41, [SI).

Now, let the boundary dS of the top one in Fig. 13(a) be fixed. And suppose that dS is transformed from the boundary aR* of the standard pattern (i.e., the left one). Hence, we find a relation of boundary conversion

( 0 < 5 < 1 , 0 < ~ G l ) .

To satisfy (841, we can choose other shape models alternatively, such as the Coons models T, in Section 11-B, the bi-quadratic model T, (as a polynomial model in Section 11-A), and the harmonic model TH in [51.

The harmonic model is defined by

where x and z are harmonic functions

with the Laplace operator A = d 2 / d 5 * 2 + a2/dq*’. The solutions of (85b) with the Dirichlet condition (84) can be evaluated approximately by numerical methods, e.g., the finite-element methods [71, [SI.

The Coons and harmonic models can convert the images into those with arbitrary boundary or frame. There- fore, the shape models T, and TH can be easily found based on the boundary relation (84). Also the boundary of the top in Fig. 13(a) can be regarded as a quadratic curve, thus yielding a quadratic model T,. As long as the shape models are given, through them the images inside the boundary dS can be transformed, and through their inverse transformations the normalized image inside alR can also be obtained (see Figs. 13).

In Table 1, we list the computed errors of the normalization from the standard form where

A Z = zlq,-q,l, E = ~(61J-@l , ) i ]”z . (86)

In (86) ql and ql are the approximate pixels after normalization, and the standard pixels, respectively. The approximate solutions at, (obtained by CSIM) and Q1, are given by (see [31-151)

Ax = Az = 0 , (85b)

IJ [ CJ

It should be noted that the patterns in Figs. 13(b) are exactly the same as those in Fig. 13(a), refer to Table I, where N , denotes the total number of nonempty pixels. This perfectly verifies the conclusions made in Theorem 1, and also suggests that the linear Coons models (59) can be employed to substitute the composite model T, of the perspective transformation in image conversion provided that the boundary conversion (84) is known.

Next, we carry out the second experiment:

2) Based on the entire image S, which has been transformed through T,, the normalization of this image can be obtained through the inverse transformation T-’ of other shape models.

In Fig. 14(a), the distorted image in the top has been obtained through T,, when the standard pattern is curved in 3-D attached to a cylinder with a cubic wall. The splitting-integrating method is applied for its normalization through TP-‘, shown on the right. Certainly, there might exist some deviation of the normalization from the standard one, and the errors found are listed in Table 11.

Suppose that the boundary conversion:

as 5 dLR* (88)

is given, where dS and d f l * are the boundaries of the distorted and standard patterns, respectively. We shall choose other models, such as the Coons, bicubic and harmonic models to satisfy (88) and to substitute the composite model T,. Therefore, the images inside of dR*


(b) (€0 Fig. 13. Patterns through T and T- 'T in Experiment 1 by splitting-shooting method and CSIM, respectively.

(a) Composite model T, of perspective transformation. (b) Linear Coons model T,. (c) Biquadratic model Te. (d) Harmonic model TH.

TABLE I ERRORS I N NORMALIZATION BY T - ' T OF DIFFERENT SHAPE MODELS

BASED ON BOUNDARY OF TOP OF FIG. 13(a) I N EXPERIMENT 1

can be obtained, and drawn in Figs. 14(b)-(d). The errors of normalization are listed in Table 11.

No pixel differences occur in the normalizations in

A I E E / f l d Figs. 14(a) and (b). Also the quantitative errors between them are as follows

Shape Model

1/2 Perspective model 0 8.803 0.3547 (89) Coons model 0 8.803 0.3547 { z(@y-@:;))2} lJ =0.01, Biquadratic model 0 8.571 0.3453

2 8.845 0.3564 Harmonic model

where @$') and @$) denote the normalized solutions of "Where N , = 616, y* = 150, 8 = 20", x o = 30.5, yo = 50, zo = 20.


(C) (d) Fig. 14. Patterns through T-’ in Experiment 2 by splitting-integrating method. (a) Composite model T,, of perspective

transformation. (b) Linear Coons model T,. (c) Bicubic model Ts. (d) Harmonic model TH.

TABLE 111

BASED ON IMAGE AT TOP OF FIG. 14(a) I N EXPERIMENT 2 ERRORS I N NORMALIZATION BY T OF DIFFERENT MODELS

Shape Model A I E w@la Perspective model 12 9.548 0.3847 Coons model 12 9.548 0.3847 Bicubic model 842 24.58 0.9902 Harmonic model 735 22.91 0.9231

“Where N I = 616, y* = 100, xo = 8, y o = 50, zo = 10.

images through the inverse transformations of T, and T,, respectively. Equation (89) indicates a very small differ- ence, resulting only from errors by the splitting-integrating method. We note that such a perfect coincidence of both normalizations again agrees with Corollary 1. Be- sides, an obvious deviation of the normalization in Figs.

14(c) and (d) implies that their substitutions for this physical case may not be applicable.

To close this section, we have produced the graphs of “water falls” with two-grey-levels in Fig. 15 by Experi- ment 1. Only the pixel errors in the normalization are provided in Table I11 and there exist small relative errors 2.7-4.7% of A Z / N T for the normalization of images through T-’T, where NT is the total number of all possible pixels in the image region, and NI is the total number of nonempty pixels.

In fact, the pictures are photographed by a camera. The camera lens is regarded as the perspective plane T, and supposed to be located as (49). Evidently the coordinates as (48) in 3-D hold true. The boundary of the transformed images is the same, as the top one in Fig. 15(a), through the composite model T, of the perspective


(C)

perspective model and linear Coons model T,. (b) Bilinear model TL. (c) Harmonic model TH. Fig. 15. Image of water falls through T-'T in Experiment 1 by CSIM with 0 = - 10". (a) Composite model T, of

TABLE 111 ERRORS IN NORMALIZATION BY T - ' T OF DIFFERENT MODELS

BASED ON BOUNDARY OF TOP OF FIG. 15(a) IN

EXPERIMENT 1 WITH e = - 10"

Shape Model A I AI/NTa AI/Nl

PersDective and

look similar due to the same configuration. Consequently, a more general substitution of T, by other models rather than T, might still be applicable for some practical applications under certain circumstances such as when the image structure or skeleton is important.

Coons models 90 0.0469 0.0894 Bilinear model 53 0.0276 0.0526 Harmonic model 74 0.0385 0.0735 VI. CONCLUSION

aWhere NT = 1920, NI = 1007.

model, yet the images inside of the boundary are obtained through the perspective, Coon, bilinear, and harmonic models. Since two models, T, and T,, result in the same solutions of image conversion, we draw them on the same figure in Fig. 15(a).

It should be noted that the image through T,T- 'T of other models, such as the bilinear and harmonic models, .

The substitution and approximation of models can lead to an efficient way of simplifying the image conversion in pattern recognition. This study is especially useful when the entire transformation is unknown or too complicated to compute. On the other hand, it should be pointed out that the conclusions hold true only under certain circumstances, as shown by the conditions described in the theorems and corollaries. For instance, even though the precise transformation functions in T, are unknown, some


physical background has to be given, such as through the general forms of spatial coordinates and the properties of the perspective plane in the case of a perspective transformation. Of course, some of these properties can be confirmed once the normalized image has been obtained. The comparative study of shape models is significant for image processing and pattern recognition. Other interesting applications of the models and their substitution (or approximation) can be found in our recent book [5].

Naturally, different conclusions of model substitution and approximation may be drawn under different conditions or for different physical models. Nevertheless, the study described in this paper has already illustrated an efficient way of exploiting a substitution or an approximation.

ACKNOWLEDGMENT

We would like to express our gratitude to the referees for their valuable comments. We also would like to thank Halina Monkiewicz, Terry Czernienko and George Z . Li for the preparation of this paper.

REFERENCES

[l] A. R. Forrest, “On Coons and other methods for the representa- tion of curved surfaces,” Computer Graphics and Image Processing, vol. 1, pp. 341-359, 1972.

Z. C. Li, Y. Y . Tang, T. D. Bui, and C. Y . Suen, “Shape transformation models and their application in pattern recognition,” in press in Int. J . Pattern Recog. Artificial Intell., vol. 4, no. 1, pp. 65-94, Mar. 1990. Z. C. Li, T. D. Bui, C. Y . Suen, and Y. Y . Tang, “Splitting-shooting methods for nonlinear transformations of digitized patterns,” in press in IEEE Trans. Pattern Anal. Machine Intell., in July 1990. Z. C. Li, C. Y. Suen, T. D. Bui, Y. Y . Tang, and Q. L. Gu, “Splitting-integrating methods for normalizing images by inverse transformations,” to appear. Z. C. Li, T. D. Bui, Y. Y . Tang, and C. Y . Suen, Computer Transformation of Digital Images and Patterns. Singapore: World Scientific, 1989. M. A. Penna and R. R. Patterson, Projective Geometry and its Applications to Computer Projective Graphics. Englewood Cliffs, NJ: Prentice-Hall, 1986. G. Strang and G. J. Fix, An Analysis of the Finite Element Method. Englewood Cliffs, NJ: Prentice-Hall, 1972. 0. C. Zienkiewicz, The Finite Element Method, third ed. London: McGraw-Hill. 1977.

Zi Cai Li, photograph and biography not available at time of publication.

Quan lin Gu, photograph and biography not available at time of publication.

Ching Y. Sum, photograph and biography not available at the time of publication.

Tien Dai Bui, photograph and biography not available at time of publication.

a comparative study of nonlinear shape models for digital image processing and pattern recognition

Documents