a real-time video tracking system

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-2, NO. 1, JANUARY 1980

A Real-Time Video Tracking System

ALTON L. GILBERT, MEMBER, IEEE, MICHAEL K. GILES, GERALD M. FLACHS, MEMBER, IEEE,ROBERT B. ROGERS, MEMBER, IEEE, AND YEE HSUN U, MEMBER, IEEE

Abstract-Object identification and tracking applications of patternrecognition at video rates is a problem of wide interest, with previousattempts limited to very simple threshold or correlation (restrictedwindow) methods. New high-speed algorithms together with fastdigital hardware have produced a system for missile and aircraft iden-tification and tracking that possesses a degree of "intelligence" notpreviously implemented in a real-time tracking system. Adaptivestatistical clustering and projection-based classification algorithms areapplied in real time to identify and track objects that change in ap-pearance through complex and nonstationary background/foregroundsituations. Fast estimation and prediction algorithms combine linearand quadratic estimators to provide speed and sensitivity. Weights aredetermined to provide a measure of confi'dence in the data and result-ing decisions. Strategies based on maximizing the probability of main-taining track are developed. This paper emphasizes the theoreticalaspects of the system and discusses the techniques used to achieve real-time implementation.

Index Terms-Image processing, intensity histograms, object iden-tification, optical tracking, projections, tracking system, video datacompression, video processing, video tracking.

INTRODUCTIONI MAGE PROCESSING methods constrained to operate on

sequential images at a high repetition rate are few. Patternrecognition techniques are generally quite complex, requiringa great deal of computation to yield an acceptable classifica-tion. Many problems exist, however, where such a time-consuming technique is unacceptable. Reasonably complexoperations can be performed on wide-band data in real time,yielding solutions to difficult problems in object identificationand tracking.The requirement to replace film as a recording medium to

obtain a real-time location of an object in the field-of-view(FOV) of a long focal length theodolite gave rise to the de-velopment of the real-time videotheodolite (RTV). U.S. ArmyWhite Sands Missile Range began the development of the RTVin 1974, and the system is being deployed at this time. Designphilosophy called for a system capable of discriminatory judg-ment in identifying the object to be tracked with 60 indepen-dent observations/s, capable of locating the center of mass ofthe object projection on the image plane within about 2 per-

Manuscript received September 14, 1978; revised November 19, 1978.This work was supported by the U.S. Army ILIR Program and the U.S.Army Research Office.A. L. Gilbert and M. K. Giles are with the U.S. Army White Sands

Missile Range, White Sands, NM 88002.G. M. Flachs and R. B. Rogers are with the Department of Electrical

Engineering, New Mexico State University, Las Cruces, NM 88003.Y. H. U was with the Department of Electrical Engineering, New

Mexico State University, Las Cruces, NM 88003. He is now withTexas Instruments Incorporated, Dallas, TX 75222.

cent of the FOV in rapidly changing background/foregroundsituations (therefore adaptive), able to generate a predictedobservation angle for the next observation, and required tooutput the angular displacements of the object within theFOV within 20 ms after the observation was made. The sys-tem would be required to acquire objects entering the FOVthat had been prespecified by shape description. In the RTVthese requirements have been met, resulting in a real-time ap-plication of pattern recognition/image processing technology.The RTV is made up of many subsystems, some of which

are generally not of interest to the intended audience of thispaper. These subsystems (see Fig. 1) are as follows:

1)2)3)4)5)6)7)8)9)

10)11)

main optics;optical mount;interface optics and imaging subsystem;control processor;tracker processor;projection processor;video processor;input/output (I/O) processor;test subsystem;archival storage subsystem;communications interface.

The main optics is a high quality cinetheodolite used for ob-taining extremely accurate (rms error - 3 arc-seconds) angulardata on the position of an object in the FOV. It is positionedby the optical mount which responds to azimuthal and eleva-tion drive commands, either manually or from an externalsource. The interface optics and imaging subsystem providesa capability to increase or decrease the imaged object size onthe face of the silicon target vidicon through a 10:1 range,provides electronic rotation to establish a desired objectorientation, performs an autofocus function, and uses a gatedimage intensifier to amplify the image and "freeze" the mo-tion in the FOV. The camera output is statistically decom-posed into background, foreground, target, and plume re-gions by the video processor, with this operation carried onat video rates for up to the full frame. The projection pro-cessor then analyzes the structure of the target regions toverify that the object selected as "target" meets the stored(adaptive) description of the object being tracked. The trackerprocessor determines a position in the FOV and a measuredorientation of the target, and decides what level of confidenceit has in the data and decision. The control processor thengenerates commands to orient the mount, control the inter-face optics, and provide real-time data output. An I/O pro-

0162-8828/80/0100-0047$00.75 © 1980 IEEE

47


Tracking OpticsZ

TV Inter oce Optics MountICamera Optics II

Video+

rTV

I i I i I i I;iZ;ZO 0; 00 A 0 E -E

Control Processor

Tracker Processor

Projection Processor

Video Processor

I0 Processor

RTV Processor1 _ _ .s_l_ .n;Remote

Commo ContEncoder ntc Input

Data Outputideo Tape T

or l lRecorder

L Archival Storage

Fig. 1. RTV tracking system.

cessor allows the algorithms in the system to be changed, in-terfaces with a human operator for tests and operation, andprovides data to and accepts data from the archival storagesubsystem where the live video is combined with status andposition data on a video tape. The test subsystem performsstandard maintenance checks on the system. The communica-tions interface provides the necessary interaction with the ex-ternal world for outputing or receiving data.The video processor, projection processor, tracker processor,

and control processor are four microprogrammable bit-slicemicroprocessors [1], which utilize Texas Instruments' (TIs')new 74S481 Schottky processor, and are used to performthe real-time tracking function.The four tracking processors, in turn, separate the target

image from the background, locate and describe the targetimage shape, establish an intelligent tracking strategy, andgenerate the camera pointing signals to form a fully auto-matic tracking system.Various reports and papers discuss several of the develop-

mental steps and historical aspects of this project [2] - [7].In this paper the video, projection, tracker, and controlprocessors are discussed at some length.

VIDEO PROCESSOR

The video processor receives the digitized video, statisticallyanalyzes the target and background intensity distributions,and decides whether a given pixel is background or target[8]. A real-time adaptive statistical clustering algorithm isused to separate the target image from the background sceneat standard video rates. The scene in the FOV of the TVcamera is digitized to form an n X m matrix representation

P = (pi1) n, m

of the pixel intensities Pij. As the TV camera scans the scene,the video signal is digitized at m equally spaced points acrosseach horizontal scan. During each video field, there are nhorizontal scans which generate an n X m discrete matrixrepresentation at 60 fields/s. A resolution of m = 512 pixelsper standard TV line results in a pixel rate of 96 ns per pixel.Every 96 ns, a pixel intensity is digitized and quantized into

eight bits (256 gray levels), counted into one of six 256-levelhistogram memories, and then converted by a decision mem-ory to a 2-bit code indicating its classification (target, plume,or background). There are many features that can be func-tionally derived from relationships between pixels, e.g., tex-ture, edge, and linearity measures. Throughout the followingdiscussion of the clustering algorithm, pixel intensity is usedto describe the pixel features chosen.The basic assumption of the clustering algorithm is that

the target image has some video intensities not containedin the immediate background. A tracking window is placedabout the target image, as shown in Fig. 2, to sample thebackground intensities immediately adjacent to the targetimage. The background sample should be taken relativelyclose to the target image, and it must be of sufficient size toaccurately characterize the background intensity distributionin the vicinity of the target. The tracking window also servesas a spatial bandpass filter by restricting the target search re-gion to the immediate vicinity of the target. Although onetracking window is satisfactory for tracking missile targetswith plumes, two windows are used to provide additionalreliability and flexibility for independently tracking a targetand plume, or two targets. Having two independent windowsallows each to be optimally configured and provides reliabletracking when either window can track.The tracking window frame is partitioned into a background

region (BR) and a plume region (PR). The region inside theframe is called the target region (TR) as shown in Fig. 2.During each field, the feature histograms are accumulatedfor the three regions of each tracking window.The feature histogram of a region R is an integer-value, in-

teger argument function hR (x). The domain of hR (x) is[O,d], where d corresponds to the dynamic range of theanalog-to-digital converter, and the range of hR (x) is [O, r],where r is the number of pixels contained in the region R;thus, there are r + 1 possible values of hR (x). Since the do-main hR (x) is a subset of the integers, it is convenient todefine hR(x) as a one-dimensional array of integers

h(O), h(l), h(2), * * *, h(d)-Letting xi denote the ith element in the domain of x (e.g.,x25 = 24), and x(j) denote the jth sample in the region R(taken in any order), hR (x) may be generated by the sum

rhR (Xi) = xi,x (j)

j=1

where 6 is the Kronecker delta function

O i*j= { :=j.

A more straightforward definition which corresponds to theactual method used to obtain hR (x) uses Iverson's notation[211 to express hR (x) as a one-dimensional vector of d + 1integers which are set to zero prior to processing the regionR as

h +-(d+ 1)pO.

48

GILBERT et al.: REAL-TIME VIDEO TRACKING SYSTEM

Fig. 2. Tracking window.

As each pixel in the region is processed, one (and only one)element ofH is incremented as

h[x(j)] -h [x(j)] + 1.

When the entire- region has been scanned, h contains the dis-tributions of pixels over intensity and is referred to as thefeature histogram of the region R.

It follows from the above definition that h satisfies theidentity

r=hR(xi) or r-+/h.i =0

Since h is also nonnegative and finite, it can be made to sat-isfy the requirements of a probability assignment functionby the normalization

h-h *. +/h.

Hereafter, all feature histograms are assumed to be normalizedand are used as relative-frequency estimates of the probabilityof occurrence of the pixel values x in the region over whichthe histogram is defined.For the ith field, these feature histograms are accumulated

for the background, plume, and target regions and written

hiR(x): BR(

x

hiR(x): Eh (x)R=Xx

hR (x): E hTR(X) 1

x

after they are normalized to the probability interval [0, 1].These normalized histograms provide an estimate of theprobability of feature x occurring in the background, plume,and target regions on a field-by-field basis. The histogramsare accumulated at video rates using high-speed LSI mem-

ories to realize a multiplexed array of counters, one for eachfeature x.

The next problem in the formulation of a real-time cluster-ing algorithm is to utilize the sampled histograms on a field-by-field basis to obtain learned estimates of the probabilitydensity functions for background, plume, and target points.Knowing the relative sizes of the background in PR, the back-ground in TR, and the plume in TR, allows the computationof estimates for the probability density function for back-ground, plume, and target features. This gives rise to a typeof nonparametric classification similar to mode estimationas discussed by Andrews [91, but with an implementationmethod that allows for real-time realization.

Letting

number of background points in PRtotal number of points in PR

number of background points in TRtotal number of points in TR

number of plume points in TRy == total number of points in TR

and assuming that 1) the BR contains only background points,2) the PR contains background and plume points, and 3) theTR contains background, plume, and target points, one has

hPR (X) = hP(x)hfR(x) = ah4(x) + (1 - ca) hr(x)

hTR (X) = PhB (x) + yhr(x) + (1 - i - T) hT(x).By assuming there are one or more features x where hB(x)

is much larger than hf(x), one has

hpR(x) Epat =

IBhrR(x)- hpR(X)

where c = (1 - a) hf(x) << hf?(x). Now for all features xwhere h,f(x) = 0, one has the solution a = hPR(x)IhPR(x). Forall features x where hip(x)>0, the inequality hpR(x)IhPR(x)>ais valid. Consequently, a good estimate for cx is given by

ax = min {h'R (x)IhPR (X)}x

and this estimate will be exact if there exists one or more fea-tures where hPR(x) 0 0 and hf(x) = 0. Having an estimate ofat and hO(x) allows the calculation of hf(x).

In a similar manner, estimates of,B and y are obtained,

x hfR(X)3min hP(x

*hBR(x)OY

hTR(X)=min '

hp(x)Having field-by-field estimates of the background, plume,

and target density functions (h'(x), h/(x), hf(x)), a linearrecursive estimator and predictor [101 is utilized to establishlearned estimates of the density functions. Letting H(ilj)represent the learned estimate of a density function for theith field using the sampled density functions hi(x) up to thejth field, we have the linear estimator

H(ili)=w H(ili- 1)+(1 - w)hi(x)and linear predictor

H(i + I li) = 2H(fli) - H(i - I ji - 1).The above equations provide a linear recursive method for

compiling learned density functions. The weighting factorcan be used to vary the learning rate. When w = 0, the learn-ing effect is disabled and the measured histograms are usedby the predictor. As w increases toward one, the leamingeffect increases and the measured density functions have a

49


reduced effect. A small w should be used when the back-ground is rapidly changing; however, when the background isrelatively stationary, w can be increased to obtain a morestable estimate of the density functions.The predictor provides several important features for the

tracking problem. First, the predictor provides a better esti-mate of the density functions in a rapidly changing scenewhich may be caused by background change or sunglare prob-lems. Secondly, the predictor allows the camera to have anautomatic gain control to improve the target separation fromthe background.With the learned density functions for the background,

plume, and target features (Hf'(x), HP(x), H1T(x)), a Bayesianclassifier [11] can be used to decide whether a given featurex is a background, plume, or target point. Assuming equala priori probabilities and equal misclassification costs, theclassification rule decides that a given pixel feature x is abackground pixel if

HB(x)>HfI(x) and HB(x)>HiH(x),a target pixel if

HT(x) >>HB(x) and HT(x) >HrP(x),

or a plume pixel if

HP(x)>HB(x) and HfP(x)>HfT(x).The results of this decision rule are stored in a high-speedclassification memory during the vertical retrace period.With the pixel classification stored in the classification mem-ory, the real-time pixel classification is performed by simplyletting the pixel intensity address the classification memorylocation containing the desired classification. This processcan be performed at a very rapid rate with high-speed bipolarmemories.

PROJECTION PROCESSORThe video processor described above separates the target

image from the background and generates a binary picture,where target presence is represented by a "1" and targetabsence by a "0." The target location, orientation, andstructure are characterized by the pattern of 1 entries in thebinary picture matrix, and the target activity is character-ized by a sequence of picture matrices. In the projectionprocessor, these matrices are analyzed field-by-field at 60fields/s using projection-based classification algorithms toextract the structural and activity parameters needed toidentify and track the target.The targets are structurally described and located by using

the theory of projections. A projection in the x-y plane of apicture function f(x,y) along a certain direction w onto astraight line z perpendicular to w is defined by

PW(Z) =f(x,y) dw

as shown in Fig. 3. In general, a projection integrates the in-tensity levels of a picture along parallel lines through the pat-tern, generating a function called the projection. For binary

F -o x

Fig. 3. Projections.

digitized patterns, the projection gives the number of objectpoints along parallel lines; hence, it is a distribution of thetarget points for a given view angle.

It has been shown that for sufficiently large numbers ofprojections a multigray level digitized pattern can be uniquelyreconstructed [12]. This means that structural features of apattern are contained in the projections. The binary inputsimplifies the construction of projections and eliminates in-terference of structural information by intensity variationwithin the target pattern; consequently, fewer projectionsare required to extract the structural information. In fact,any convex, symmetric binary pattern can be reconstructedby only two orthogonal projections, proving that the projec-tions do contain structural information.Much research in the projection area has been devoted to

the reconstruction of binary and multigray level picturesfrom a set of projections, each with a different view angle.In the real-time tracking problem, the horizontal and verticalprojections can be rapidly generated with specialized hard-ware circuits that can be operated at high frame rates. Al-though the vertical and horizontal projections characterizethe target structure and locate the centroid of the targetimage, they do not provide sufficient information to pre-cisely determine the orientation of the target. Consequently,the target is dissected into two equal areas and two orthogonalprojections are generated for each area.To precisely determine the target position and orientation,

the target center-of-area points are computed for the top sec-tion (XCT, YcT) and bottom section (XcB, YcB) of the trackingparallelogram using the projections. Having these points, thetarget center-of-area (Xc, Yc) and its orientation can be easilycomputed (Fig. 4):

XT + XBXc = c c

2

2

yT - yBq=tanI XT X

C C

The top and bottom target center-of-area points are used,rather than the target nose and tail points, since they aremuch easier to locate, and more importantly, they are lesssensitive to noise perturbations.

It is necessary to transform the projection functions into

so


P%(z)

xBXm

Fig. 4. Projection location technique.

a parametric model for structural analysis. Area quantiza-tion offers the advantage of easy implementation and highimmunity to noise. This process transforms a projectionfunction Pw(z) into k rectangles of equal area (Fig. 5), suchthat

Zi+l

Z-

1 Zk+1

Pw(z)dz=Jk7 Pw(z) dz

for i= 1,2,>-,k.

Another important feature of the area quantization modelfor a projection function of an object is that the ratio of linesegments li = Zi+1 Zi and L Zk - Z2,

1-Si= ' for i=2,3, ,k- I

are object size invariant. Consequently, these parameters pro-

vide a measure of structure of the object which is independentof size and location [13]. In general, these parameters changecontinuously since the projections are one-dimensional repre-

sentations of a moving object. Some of the related problemsof these geometrical operations are discussed by Johnston andRosenfeld [14].The structural parameter model has been implemented and

successfully used to recognize a class of basic patterns in a

noisy environment. The pattern class includes triangles,crosses, circles, and rectangles with different rotation angles.These patterns are chosen because a large class of more com-

plex target shapes can be approximated with them.The architecture of the projection processor consists of a

projection accumulation module (PAM) for accumulatingthe projections and a microprogrammable processor forcomputing the structural parameters. The binary targetpicture enters the PAM as a serial stream in synchronization

Zsi Z2 Z3 Z4 Z5 Zk-1 Zk

Fig. 5. Projection parameters.

with the pixel classifier of the video processor. The projec-tions are formed by the PAM as the data are received in realtime. In the vertical retrace interval, the projection processorassumes addressing control of the PAM and computes thestructural parameters before the first active line of the nextfield. This allows the projections to be accumulated in realtime, while the structural parameters are computed duringthe vertical retrace interval.

TRACKER PROCESSORIn the tracking problem, the input environment is restricted

to the image in the FOV of the tracking optics. From this in-formation, the tracking processor extracts the important in-puts, classifies the current tracking situation, and establishesan appropriate tracking strategy to control the tracking opticsfor achieving the goals of the tracking system.The state concept can be used to classify the tracking situa-

tions in terms of state variables as in control theory, or it canbe interpreted as a state in a finite state automaton [15], [16].Some of the advantages of the finite state automaton ap-

proach are as follows.1) A finite state automaton can be easily implemented with

a look-up table in a fast LSI memory.2) A finite state automaton significantly reduces the amount

of information to be processed.3) The tracking algorithm can be easily adjusted to differ-

ent tracking problems by changing the parameters in thelook-up table.4) The finite state automaton can be given many character-

istics displayed by human operators.The purpose of the tracker processor is to establish an intel-

ligent tracking strategy for adverse tracking conditions. Theseconditions often result in losing the target image within or outof the FOV. When the target image is lost within the FOV,the cause can normally be traced back to rapid changes in thebackground scene, rapid changes in the target image due tosun glare problems, or cloud formations that obstruct thetarget image. When the target image is lost by moving out ofthe camera's FOV, the cause is normally the inability of thetracking optics dynamics to follow a rapid motion of thetarget image. It is important to recognize these situationsand to formulate an intelligent tracking strategy to continuetracking while the target image is lost so that the target imagecan be reacquired after the disturbance has passed.To establish an intelligent tracking strategy, the tracker pro-

cessor evaluates the truthfulness and trackability of the track-

-4- Z/ II I

51

Zk + i

I . 1


ing data. The truthfulness of the tracking data relates to theconfidence that the measured tracking data truly define thelocation of the target under track. The trackability of thetarget image relates to the question of whether the targetimage has desirable tracking properties.The inputs to the tracker processor are derived from the

projection representation of the target image by the projec-tion processor. Area quantization is used to transform eachprojection function P(z) into K = 8 equal area intervals asshown in Fig. 5.These inputs are:

1)2)3)4)5)

target size (TSZ);target location (TX, TY);target orientation (TO);target density (TDN);target shape = {(SXi, SY1)Ii = 1, 2, * *, 6}.

Target size is simply the total number of target points. Targetlocation is given by the center-of-area points of the projec-tions. Where Xi and Yi are the parameters of Fig. 5 whenprojected on the x and y axes, respectively,

TX=X5 and TY=Y5.

The target orientation defines the orientation of the targetimage with respect to vertical boresight. Target density isderived from the target length (TL), width (TW), and size(TSZ) by

TDN =TL X TW

TSZ

The target shape is described by the ratio of the lengths ofthe equal area rectangles and the total lengths

SXi = (Xi+ 2 - Xi+ 1)/(X8 - X2)

and

Syi = (yi+ 2 - Yi+ 1)/(Y8 - Y2)

for i= 1, 2, * * *, 6. Observe that the first and last equal area

subintervals are not used in the shape description, since theyare quite sensitive to noise.The tracker processor establishes a confidence weight for

its inputs, computes boresight and zoom correction signals,and controls the position and shape of the target trackingwindow to implement an intelligent tracking strategy. Theoutputs of the tracker processor are as follows.Outputs to Control Processor: 1) Target X displacement

from boresight (DX), 2) target Y displacement from bore-sight (DY), 3) desired change in zoom (DZ), 4) desired changein image rotation (DO), and 5) confidence weight (W).Outputs to Video Processor: 1) tracking window size,

2) tracking window shape, and 3) tracking window position.The outputs to the control processor are used to control

the target location and size for the next frame. The bore-sight correction signals are used to control the azimuth andelevation pointing angles of the telescope. The desired zoom

is used to control the zoom lens, keeping the target visiblewithin the FOV. The desired image rotation controls theimage rotation element to keep the target image vertical. The

confidence weight is used by the control processor much likea Kalman weight to combine the measured and predictedvalues. When the confidence weight is low, the control pro-cessor relies more heavily on the recent trajectory to predictthe location of the target on the next frame.The outputs to the video processor define the size, shape,

and position of the tracking window. These are computedon the basis of the size and shape of the target image and theamount of jitter in the target image location. There is noloss in resolution when the tracking window is made larger;however, the tracking window acts like a bandpass filter andrejects unwanted noise outside the tracking window.A confidence weight is computed from the structural fea-

tures of the target image to measure the truthfulness of theinput data. The basic objective of the confidence weight isto recognize false data caused by rapid changes in the back-ground scene or cloud formations. When these situations aredetected, the confidence weight is reduced and the controlprocessor relies more heavily on the previous tracking datato orientate the tracking optics toward the target image. Thisallows the control processor to continue tracking the targetso that the target image can be reacquired after the perturba-tion passes.The confidence weight measures how well the structural fea-

tures of the located object fit the target image being tracked.A linear recursive filter is used to continually update thestructural features to allow the algorithm to track the de-sired target through different spatial perspectives. Experi-mental studies have indicated that the structural parametersS = {(SXi, SYi)li = 1, 2, * * *, 6} and the target density areimportant features in detecting erratic data. Let TDN(k) and(SXi (k), SYi (k)) for i = 1, 2, ... , 6 represent the measuredtarget density and shape parameters, respectively, for the kthfield, and let (SXi(k),SYi(k)) represent the filtered valuesfor the target shape parameters. The linear filter is definedby

SXi(k + 1) = (k 1 ) SXi (k) + SXi (k)

SYi(k + 1) (k - 1)SYi(k) +SYi(k)

for i = 1, 2, , 6 and a positive integer K. The confidenceweight for the kth field is given by

W(k) = a,i max {l - C(k), O} + a2 min {l, TDN(k)}

where

6 6C(k)= £ SXi(k) - SXi(k) + £ SYi(k) - SYi(k)

i=l i=l

and

O.< 1, t2<1, al+a2=l.

This formulation for the confidence weight has been experi-mentally tested, and it has demonstrated the ability to mea-sure the truthfulness of the tracking data in the tracking en-vironment. The filtered shape parameters are not updated

52


when the confidence weight falls below a given lower thresh-old. This prevents the shape parameters from being updatedincorrectly during periods when the target image is lost orbadly perturbed by the background noise.To formulate an intelligent tracking strategy, the tracking

algorithm needs the ability to respond to a current inputbased upon the sequence of inputs that lead to the currentstate (or situation). A finite state sequential machine pos-sesses such a property because each state represents thecollection of all input sequences that take the machine fromthe initial state to the present state (Nerode's tape equiva-lence [17] ). By defining an equivalence relation R on the tapeset as

XRy if 5(so,X)=5(so,y) Vx, yEE*

the tape set 1* can be partitioned into equivalent classes

[x] =si= {yIxRy VyeZ*}.Consequently, a state represents all input sequences thatproduce a given tracking situation. This interpretation ofinput sequences transforms the development of the track-ing algorithm into a problem of defining a finite state se-quential machine

TA = (S, I, Z, 6, W).

The states of the machine S = {Si, S2, S3, , s,,} define thedifferent tracking situations that must be handled by thetracking algorithm. The inputs to the finite state machineare derived from the image parameters that characterize thesize, shape, and location of the present target image. Theoutput set Z defines a finite set of responses that the track-ing algorithm employs for maintaining track and retaininghigh resolution data. The next state mapping 8: S X I -Sdefines the next state 8(si, ij) = Sk when an input ij is appliedto state si. The output mapping W: S -+ Z is a Moore out-put that defines the proper tracking strategy (response) foreach state.The inputs to the sequential machine are chosen to give the

sequential machine a discrete measure of the trackability ofthe target image. These inputs are:

1)2)3)4)5)6)7)

target size (TSZ);

confidence weight (W);image orientation (Tb);image displacement from boresight (TX, TY);image movement (A TX, ATY);rate of change in target size (ATSZ);rate of change in confidence weight (A W).

The image size is clearly an important measure of the track-ability since the image is difficult to track when the imageis either too small or too large. The confidence weight pro-

vides a measure of whether the image is the target undertrack. Image displacement from boresight gives a measureof whether the image is leaving the FOV. Image movementin the FOV measures the amount of jitter in the target image.The rate of change in the target size and confidence weightallows a prediction of what is likely to happen on subsequent

fields. These inputs are quantized into an 8-bit binary inputvector for the sequential machine.The states of the sequential machine are chosen to define

the major tracking situations. These states are:

1) target acquisition = Sl;2) normal tracking= S2;3) abrupt change in FOV = S3;4) leaving or out of FOV = S4.

S, corresponds to the situation where the tracker is tryingto acquire the target. This situation occurs during initialacquisition and during periods of time when the target imageis lost. S2 corresponds to the situation where the target imageis under track with the normal tracking algorithm and nospecial tracking strategy is required. S3 corresponds to thesituation where the target image undergoes erratic and abruptchanges in its shape or size within the FOV and S4 correspondsto the situation where the target image is leaving or has leftthe FOV of the camera. These states only categorize themajor tracking situations. State parameters are used to furtherrefine the tracking situation by providing a parametric de-scription of the tracking situation.The tracking algorithm views the target structure and ac-

tivity within the tracking window on a field-by-field basis.Based upon these observations, the tracking algorithm mustclassify the present tracking situation and enter the appro-priate state whose outputs best handle the situation. Themotivation of the next state mapping, 8: I X S -+S, is toprovide a tracking strategy to keep the tracker in the normaltracking state. Much experimental effort has been devotedto establish a next state mapping with desirable tracking char-acteristics. A simplified state diagram is given in Fig. 6 whichdefines the essential properties of the state transition behavior.The output set Z defines a finite set of responses that the

tracking algorithm can make to maintain track while retain-ing high resolution data. Several of these responses controlthe tracking window, and they are performed at electronicspeed. However, the zoom lens and tracking optics opera-tions are mechanical in nature and require more time to beperformed.The output set Z contains the following responses:

1) location of tracking window;2) shape and size of tracking window;3) target location (boresight corrections);4) confidence weight;5) desired zoom;6) control shape parameter update procedure.

These output responses give the tracking algorithm the abilityto respond to the tracking situations defined by trackingstates.The motivation for the output mapping, W: I X S -+ Z, is to

associate with each state an intelligent strategy to maintaintrack with high image resolution. The outputs to the imagedecomposition algorithm control the size and position of thetracking window. The outputs to the control processor con-trol the zoom and pointing angle of the tracking optics. Thetracking algorithm must make many compromises between

53


Fig. 6. Next-state mapping.

tracking ability and image resolution. For example, when thetarget image is about to leave the FOV, the zoom can be re-

duced to keep the target in the FOV. Increasing the FOV,however, decreases the target size and image resolution. Thesecompromises have been formulated and solved with standardoptimization techniques. The solutions are precomputed andstored in a look-up table for the tracking algorithm.By implementing the tracking algorithm as a finite state

machine where the outputs are obtained from the look-uptables, it is possible to realize the tracking algorithm with a

standard microprogrammable processor.

CONTROL PROCESSORThe purpose of the control processor is to generate the con-

trol signals that drive the tracking optics. The processor re-

ceives inputs from the tracker processor on the target location(TX, TY), orientation (TO), and length (TL), measured rela-tive to boresight and on a confidence weight (W) that measuresthe reliability of the measured data. Using these inputs, thecontrol processor predicts the desired tracker azimuth andelevation pointing angles (EA (k + 1), eE(k + 1)), the zoom

lens setting (Z(k + 1)) and the image rotation setting (Q(k + 1))

for the next field denoted by (k + 1).The control processor provides the estimated and predicted

values of the variables necessary to orientate the tracking op-

tics toward the target. Using the confidence weight to com-

bine the predicted variables with current measurements, theprocessor provides estimates of current target position vari-ables. These estimates are combined with previous estimatesto predict the target position variables for the next controlinterval.Since the form of the estimator equations is similar for the

azimuth, elevation, zoom, and image rotation variables, thefollowing notation is used in the development:

VE3(k)

em (k)

e(k)

variable to be estimated or predicted (azimuth,elevation, rotation, or zoom);measured values of @(k);estimated value of @(k) using measurementsthrough kth frame;

E1(k+ 1lk) predicted value of O(k + 1) using measure-ments through kth frame and the j index typepredictor;

O(k + I k) predicted value of O(k + 1) using a combina-tion of the set of predictors;

w(k) confidence weight.

Using the confidence weight in a manner similar to that ofa Kalman gain, the estimated value is obtained from the mea-sured and predicted values by

6(k) = (1 - W(k)) 6(klk - 1) + W(k) Em(k)

where the confidence weight is normalized such that 06 W(k) <1. Consequently, the predictors having the functional form

E)(k + I1Ik) = F(O(k), O(k - 1), * ,O(k - r))rely more heavily on the estimated values when the weight islow. This is very important for continuing track when thetarget passes through clouds or noisy backgrounds such asthe mountain/sky interface.A simple filter-predictor that has demonstrated the ability

to track highly maneuverable targets uses a linear convexcombination of a linear-two-point and a quadratic-five-pointpolynomial filter-predictor. This scheme Qis based on thereasoning that the linear-two-point polynomial can betteranticipate a rapid maneuver from the missile, while the quad-ratic-five-point polynomial is used to reduce the varianceof error. The coefficients of the linear convex combinationare chosen to obtain an unbiased minimum variance of errorestimate.In formulating the linear convex combination estimate of

the linear (r1) and quadratic (^q) predictors

e = o101 + at2eq,

it is assumed that the predictors E), and Eq are independent;hence, the minimum variance error estimate [18] is given bythe coefficients

U2Qq)°l=[1g2(@+ 02 (6q)]

a2(lGi)2 - [a2(6 ) + u2(&q)]

and

E) ff(4q) 'i(tk+1 tk) + u2Q(i) bq(tk+1 Itk)@(tk+ I tk) o2 ( + 2 (

is the desired prediction of the chosen parameter.Simulation studies [19] were conducted to compare the

performance of this simple and somewhat primitive filter-predictor with the more robust extended Kalman filter [20].The results of these studies indicated the total performanceof the tracking system with the simple filter-predictor com-pared favorably with the extended Kalman filter. In the lightof the real-time computational constraints, the simple filter-predictor is suitable for more real-time tracking problems.

54


SIMULATION AND HARDWAREA computer simulation of the RTV tracking system, in-

corporating the algorithms used by the four processors de-scribed in this paper, was developed and used to evaluatethe system under realistic tracking conditions. The simula-tion model includes dynamic models for the target trajectoryand the tracking optics and mount in addition to the RTVprocessor algorithms in order to verify the complete track-ing system.The simulation displays the target and plume points as they

are identified by the video processor. These points are super-imposed on a cross hair which locates the boresight of thetracking optics. In addition to the digitized images, thesimulation displays the target and plume tracking windowsand the projections accumulated for each video field. Theseoutputs provide a direct measure of the performance of thefour processors as well as an overall measure of the trackingaccuracy of the system.Two types of input data are available to the simulation-

simulated digitized video fields and actual digitized videofields from video tape recordings of typical tracking sequences.Fig. 7 contains two representative simulation outputs selectedfrom the first 100 fields of simulated digitized video. TheRTV tracker performs well during this simulated tracking se-quence. All processors function properly, and track is main-tained throughout the tracking sequence.

Fig. 8(a) is a halftone graphics display of a field of actualdigitized video which was injected into the RTV simulation.The simulator repeatedly processed the same video fieldsuperimposed on the simulated target trajectory in order totest the static and dynamic responses of the RTV processors.The result after six processing frames, shown in Fig. 8(b),verifies the effectiveness of the processing algorithms usedin the RTV tracker. Since no plume is present, both windowsare tracking the target. Several important conclusions may bededuced from this result: the video processor successfullyidentifies most of the target pixels; the projection data is ac-cumulated and used effectively by the projection and trackerprocessors to obtain the target orientation and the displace-ment of the target from boresight; and both windows aretracking the target and closing down on it to reduce noise.Subsequent to simulation, a hardware model was developed

and is being deployed at the present time. Bench tests of allsubsystems indicate that the projected performance of eachprocessor has been achieved.

CONCLUSIONSThese four subsystems form only a part of the RTV; many

of the other subsystems are of equal complexity. Changescurrently taking place in digital technology coupled withaggressive research in real-time pattern recognition/imageprocessing algorithms are now making possible highly sophisti-cated hardware for recognition and tracking purposes. WhileRTV is an example of this type of a system, work at WSMRand NMSU (and elsewhere) is continuing with the purpose offinding faster and better ways of processing video intelligently.A lot of work remains before a truly versatile video analysis

system exists. RTV is but a step in that direction. Researchdirected toward the solution of this class of problems needs

L

i: ri- i.-i!f.-. ''. .: i' ''

.i

I ,ial ;.%

Fig. 7. Simulation outputs.

. .... ii'. ........................

(b)Fig. 8. (a) Digitized video. (b) Simulation output.

to have two fundamental premises as a starting point. Thefirst is that algorithms should be general, not specificallycreated for a particular scene. Too often pattern recognitionmethods have depended upon extensive "tweaking" of param-eters to yield good results. This approach is not allowablefor real-time processing. Secondly, the algorithms should notrequire extensive data storage, since manipulation of the dataconsumes resources that exceed those available for mostapplications.As is evidenced by RTV, progress can be made; the future

of video processing is bright. The applications in the non-military community possibly exceed the military applicationsin importance. RTV is a forerunner of a large class of intel-ligent video processing machines that will perform a widevariety of tasks of importance to the human family.

AcKNOWLEDGMENTImportant contributions to this project were made by a

number of persons not named herein, including faculty andstudents at universities and staff members at WSMR. Theproject has enjoyed many participants and cosponsors whosecontributions are acknowledged.

55

i::t

.1I'lI... Iii

!;

e4:p

1:-" W

Ai-.1i:R. N;

'j, ilifF;I

JPI,::l.Wt'!i

-.1,A


REFERENCES

[1] P. I. Perez, "A multiprocessor architecture with a master sharedcontrol store," Ph.D. dissertation, New Mexico State Univ.,1978 (available from University Microfilms, Ann Arbor, MI).

[21 G. M. Flachs and A. L. Gilbert, "Automatic boresight correctionin optical tracking," in Proc. 1975 Nat. Aerospace and Elec-tronics Conf., June 1975.

[3] G. M. Flachs, W. E. Thompson, Y. H. U, and A. L. Gilbert, "Areal-time structural tracking algorithm," in Proc. IEEE 1976Nat. Aerospace and Electronics Conf., May 1976.

[4] G. M. Flachs and J. A. Vilela, "A structural model for polygonpatterns," in Proc. IEEE 1976 Nat. Aerospace and ElectronicsConf., May 1976.

[5] G. M. Flachs, W. E. Thompson, J. M. Taylor, W. Cannon, R. B.Rogers, and Y. H. U, "An automatic video tracking system," inProc. 1977Nat. Aerospace and Electronics Conf., May 1977.

[6] R. B. Rogers and G. M. Flachs, "Mathematical modeling andsimulation in a programmed design methodology," in Proc. 1stInt. Conf. on Mathematical Modeling, Aug. 1977.

[71 A. L. Gilbert and M. K. Giles, "Novel concepts in real-timeoptical tracking," in Proc. Army Sci. Conf., June 1978.

[8] N. J. Nilsson, Learning Machines. New York: McGraw-Hill,1965, ch. 4.

[9] H. L. Andrews, Introduction to Mathematical Techniques inPattern Recognition. New York: Wiley-Interscience, 1972,pp.148-150.

(101 B. P. Lathi, Random Signals and Communication Theory. Scran-ton, PA: International Textbook, 1968, pp. 275-279.

[11] I. Selin, Detection Theory. Princeton, NJ: Princeton Univ.Press, 1965, pp. 11-13.

[12] S. K. Chang, "The reconstruction of binary patterns from theirprojections," Commun. ACM, vol. 14, Jan. 1971.

[13] Y. H. U, "Projection theory for real-time vision tracking,"Ph.D. dissertation, New Mexico State Univ., Aug. 1978 (avail-able from University Microfilms, Ann Arbor, MI).

[141 E. G. Johnston and A. Rosenfeld, "Geometrical operations ondigitized pictures," Picture Processing and Psychopictorics,B. S. Lipkin and R. Rosenfeld, Eds. New York: Academic,1970.

[15] R. Romovic and R. B. McGhee, "A finite static approach tothe synthesis of bioengineering control systems," IEEE Trans.Hum. Factors Electron., vol. HFE-7, pp.65-69, June 1966.

[16] E. S. Angel and G. A. Bekey, "Adaptive finite state models ofmanual control systems," IEEE Trans. Man-Mach. Syst., pp.15-20, Mar. 1968.

[171 T. L. Booth, Sequential Machines and Automata Theory. NewYork: Wiley, 1968.

[18] A. Papoulis, Probability Random Variables and Stochastic Pro-cesses. New York: McGraw-Hill, 1965, pp. 385-426.

[191 W. E. Thompson, G. M. Flachs, and T. R. Kiang, "Evaluation offiltering and prediction techniques for real-time video trackingof high performance targets," Trans. 1978 Nat. Aerospace andElectronics Conf., June 1978.

[20] C. B. Chang, R. H. Whitney, and M. Atham, "Application ofadaptive filtering methods to maneuvering trajectory estima-tion," Lincoln Lab. Tech. Note 1975-59, Nov. 1975.

[21] K. E. Iverson, A Programming Language. New York: Wiley,1962.

Alton L. Gilbert (S'68-M'73) was born inElmira, NY, on April 13, 1942. He received theB.S.E.E., M.S.E.E., and Sc.D. degrees, all fromthe New Mexico State University, Las Cruces,in 1970, 1971, and 1973, respectively.From 1961 to 1968 he served with the U.S.

Navy in the Polaris missile program. From1973 to the present he has been a Researcherand Research Manager with the Advanced

I, g g Technology Office of the InstrumentationDirectorate at U.S. Army White Sands Missile

Range, NM As Principle Investigator and Manager of the Real-TimeVideotheodolite program, he has become widely known for his in-terests in video processing methods.

Dr. Gilbert is a member of the Technical Review Committee of theJoint Services Electronics Program, of the Electronics CoordinatingGroup for the U.S. Army, the Information Theory Group of the IEEE,Tau Beta Pi, Eta Kappa Nu, Phi Kappa Phi, and Sigma Xi.

Michael K. Giles was born in Logan, UT, onOctober 24, 1945. He received the B.E.S.and M.S.E.E. degrees from Brigham YoungUniversity, Provo, UT, in 1971, and the Ph.D.degree from the University of Arizona, Tucson,in 1976.He was an Electronics Engineer with the U.S.

Naval Weapons Center, China Lake, CA, from1971 to 1977, where he specialized in the

l W,- | | | |gdesignand evaluation of electrooptical andoptical systems. At China Lake he developed

low-noise photodiode preamplifiers, photoparametric amplifiers, andrapidly tunable dye lasers for Navy applications. Since 1977 he hasbeen with the Instrumentation Directorate, U.S. Army White SandsMissile Range, NM, where he is applying image-processing and pattern-recognition techniques to the development of real-time optical trackingsystems. His research interests also include optical and electroopticalimage and signal processing.

Dr. Giles is a member of the Optical Society of America, the Societyof Photo-Optical Instrumentation Engineers, Tau Beta Pi, and EtaKappa Nu.

Gerald M. Flachs (S'68-M'68) received the-. ~~~~B.S., M.S., and Ph.D. degrees, all from Michigan

State University, East Lansing.He is a Professor of Electrical and Computer

Engineering at New Mexico State University,Las Cruces. His current research interests in-clude image-processing techniques, distributivecomputer systems, and digital system designwith microprogrammable processors.

Robert B. Rogers (M'77) was born in ChinaLake, CA on April 13, 1952. He received theB.S. degree in physics from the New MexicoInstitute of Mining and Technology (NMIMT),Socorro, in 1973, and the M.S. and Ph.D. de-grees in electrical engineering from New MexicoState University (NMSU), Las Cruces, in 1976and 1978, respectively.While at NMIMT, he performed research on

cross-spectral analysis of thunder and lightningchannel reconstruction. He is now at NMSU,

where his primary research area is real-time video filtering with bit-slice microprogrammable processor systems. He is currently with theNew Mexico Solar Energy Institute's Modeling and Analysis Sectionat NMSU.

Dr. Rogers is a member of the American Geophysical Union and theIEEE Computer Society.

Yee Hsun U (S'72-M'79) was born in HongKong, China, on May 30, 1949. He receivedthe B.S.E.E. degree from Purdue University,West Lafayette, IN, in 1972, and the M.S.E.E.and the Ph.D. degrees in electrical engineeringfrom New Mexico State University, Las Cruces,in 1974 and 1978, respectively.From 1976 to 1978, he was a staff engineer

with the Real-Time Video Tracker project atNew Mexico State University. He joined TexasInstruments Incorporated, Dallas, in September

1978 as a member of technical staff of the Corporate Research, Devel-opment & Engineering Division. His research interests are in real-timetracking, vision systems, machine intelligence, and the application ofpattern recognition to industrial automation.

56

a real-time video tracking system

Documents