MAY/JUNE 2004 1094-7167/04/$20.00 © 2004 IEEE 13Published by the IEEE Computer Society

MAY/JUNE 2004 1094-7167/04/$20.00 © 2004 IEEE 13Published by the IEEE Computer SocietyPublished by the IEEE Computer Society

Editor: Richard DoyleJet Propulsion Labrdoyle@jpl.nasa.gov

A I I n S p a c e

altimeter, an inertial-measurement unit, and an algorithmfor combining sensor measurements to estimate horizontalvelocity—the speed across the planet’s surface the landertravels as it descends. Although the sensors are not noveltechnology, the algorithm and flight software that com-bines them are new. This algorithm combines radar, im-

age, and inertial data in a novel way to create a low-cost,robust, and computationally efficient solution to the hori-zontal-velocity-estimation problem.

This article describes the DIMES algorithm, its testing,and its performance during both Mars Exploration Roverlandings.

What led to DIMES

MER’s Entry Descent and Landing system must deliverthe rover and its science payload safely onto the martiansurface. During entry, atmospheric drag slows down thelander far enough that a supersonic parachute can safelydeploy. Just before impact, retro rockets fire to slow it downeven further, at which point airbags inflate. The airbags fullyencompass the lander, cushioning it as it bounces along themartian surface before rolling to rest. If the velocity atimpact is too large, tests have shown that the airbags can ripand tear. The EDL system is designed to slow the landervelocity at impact sufficiently to let the airbags do their job.

In late fall 2000, the EDL design team determined thatwind shear could cause the retro rockets (also known asthe RAD rockets) to fire at an angle that would induce hor-izontal velocity. To eliminate this RAD-induced horizontalvelocity and its adverse effect on airbag performance, theTransverse Impulse Rocket System was added to reorientthe lander system so that the retro rockets would alwaysfire close to vertical. However, this system could not com-pensate for the effect of steady-state winds. This became aproblem when, in the fall of 2001, the MER EDL designteam learned that constant or steady-state winds werecommon on the martian surface.

EDL’s designers soon realized that, given a steady-statehorizontal-velocity measurement, the TIRS system couldalso serve for reorienting the lander system so that the retrorockets fired in a direction to cancel a wind-induced hori-zontal velocity. Because the TIRS system was well alongin development, the difficulty was obtaining a horizontalvelocity measurement. At this late point in EDL develop-ment, adding a traditional Doppler radar-based sensor formeasuring horizontal velocity was not feasible; a radar in-terface to the rover avionics was not in place, and accom-modating a multiple-antenna radar would be difficult.

JPL’s Descent Image Motion Estimation System is the

first machine-vision system for estimating lander

velocity during planetary descent. Composed of sensors

and software, DIMES features a descent imager, a radar

The Mars Exploration RoversDescent Image MotionEstimation System

Yang Cheng, Jay Goguen, Andrew Johnson, Chris Leger, Larry Matthies, Miguel San Martin,and Reg Willson, Jet Propulsion Laboratory

Spirit and Opportunity, the two Mars Exploration Rovers, achievedhistoric landings on the surface of Mars on 4 January and 25 January2004. Everyone who tuned in to the real-time news coverage of theseexciting events heard about the extreme engineering challenges ofdesigning the critical Entry, Descent, and Landing sequences. Every-one who worked on the problem was relieved and amazed at howflawlessly the EDL phases unfolded.

Fortune does indeed favor the well prepared. Engineering suc-cess in deep-space exploration always comes down to anticipatingcontingencies and designing for them. When it came to EDL, someof the challenges were obvious: ensuring the parachute wouldopen robustly, arranging the timing of the various events withgreat precision, and determining how well the airbag technologywould scale up, for example. Some of the challenges were subtler—and one in particular was subtle enough that it wasn’t fully appreci-ated until mission development was well underway.

This was the challenge of martian winds. How to detect and com-pensate for them? In the worst-case scenario, they could tip thevehicle over in the final stages of descent such that the poweredthrust intended to eliminate downward velocity might actuallydrive the platform sideways and down into the surface beyond thesafety envelope of the airbag cushions.

This article tells the story of how this late-understood challengewas addressed successfully—and, as it turned out, critically, for Spirit.

—Richard Doyle

Editor’s Perspective

In late October 2001, Rob Manning, EDLdevelopment manager for MER, bumpedinto Miguel San Martin, MER’s Attitude andControl System lead. San Martin was goingon vacation for the next two weeks, but hestill had MER EDL on his mind. In an in-spired moment, he told Manning to give himtwo images and they would have a solutionto the horizontal-velocity measurement prob-lem. This chance encounter was the begin-ning of DIMES’s development.

At this point, the EDL design team realizedthat, since a rover camera to measure the di-rection to the sun was recently removed fromthe MER design, the MER avionics had anunused camera interface and surplus camera

electronics modules. So, if they could find away to use a camera to measure horizontalvelocity, the impact on MER avionics designwould be minimal. Conceivably, a horizontalvelocity measurement system based onimages could be developed before launch.

On 1 November 2001, Manning emailedAndrew Johnson, a JPL machine vision re-searcher, and Andy Collins, a JPL advanced-imager developer, saying that “MER has aproblem that I think might be solvable with alittle help from you.” This email laid out theconcept for DIMES, a machine-vision systemfor estimating horizontal velocity duringlanding. Over the next two years, this JPLteam designed, implemented, tested, and ver-

ified DIMES. Four reviews, three field tests,and 26 months later, DIMES performed suc-cessfully during Spirit’s landing in GusevCrater on 4 January 2004. As the “MarsExploration Rover Flight Performance” side-bar explains, DIMES contributed to Opportu-nity’s successful landing on MeridianiPlanum three weeks later, on 25 January.

DIMES is the first machine-vision systemto perform onboard analysis of images takenduring planetary descent for landing a pay-load safely on the surface. Many scientistsand engineers were skeptical that DIMES

would work given the difficulty of makingonboard algorithms robust enough to handleall possible anomalies that might occur dur-

14 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Earlier this year, Dimes operated suc-cessfully for the two Mars ExplorationRover landings. Each time, Dimes com-puted a valid velocity that was used bythe EDL system.

MER-A—SpiritOn 4 January 2004, the MER-A or

Spirit spacecraft landed in Gusev Crater,a 165 km diameter crater just south ofthe martian equator. During its des-cent, Spirit took three images of themartian surface at altitudes of 1,983,1,706, and 1,433 meters. These imagesshowed a terrain covered with craters,hollows, and dust devil streaks. DuringEDL, DIMES computed a velocity of (4.1,9.7) m/s at 1,570 m altitude (all veloci-ties are in the local level—north, east—coordinates). After propagation downto RAD fire, this steady-state velocitywas (–1.2,10.7) m/s. Owing to a windgust just before RAD fire, the anglebetween the lander and backshell wassignificant at the time of RAD fire, andwould have caused an additional hori-zontal velocity of (–5.7, 11.7) m/s, mak-ing the combined steady-state andRAD-induced velocity (–6.9, 22.5) m/s.This velocity was significant enough tocause TIRS to fire in its stronger of twomodes, resulting in a reduced bridle cutvelocity of (–11.0, 0) m/s, which causedthe lander to bounce south.

Taken alone, the RAD-induced veloc-ity was less than the threshold to fireTIRS. Had DIMES not been available to measure the steady-statevelocity, the EDL system would not have fired TIRS and thetotal velocity would have been just on the threshold of airbag

performance. Furthermore, the velocity would have been tothe east, toward the rockier terrain surrounding BonnevilleCrater.

Mars Exploration Rover Flight Performance

Figure A. MER-A descent image pairs of Gusev Crater with selected and tracked templates. First pair: (a) first descent image with selected templates shown as red squares and (b) locationof selected templates shown as green squares on second descent image. Second pair: (c) seconddescent image with selected templates shown as red squares and (d) location of selectedtemplates shown as green squares on third descent image.

(a) (b)

(c) (d)

ing planetary landing. The development,verification, and validation activities andDIMES’s successful performance on bothMER-A and MER-B have shown that it ispossible to autonomously process and reactto imagery during landing. This ground-breaking achievement has opened the doorto future image-based safe and precise land-ing systems that will increase science returnthrough landing-hazard avoidance in scien-tifically interesting terrain and pinpointlanding for sample return.

System design constraintsGiven the novelty of using images to

estimate horizontal velocity during descent

and the extremely short development time,there was some concern that DIMES mightproduce an incorrect velocity that couldlead to mission failure. Consequently, wedesigned DIMES to “do no harm”; the DIMES

algorithm will produce no answer ratherthan one that cannot be trusted. DIMES han-dles this internal checking of the velocityestimate by comparing the velocities pro-duced from two image pairs constructedfrom three images taken during descent.

The sequence of events for DIMES is as fol-lows:At 2,000 meters above the surface, theEDL flight software tells DIMES to start takingimages. Each image is available for process-ing 3.75 seconds after image exposure. Given

a nominal descent speed of 75 m/s, DIMES

will take images at altitudes of 2,000, 1,720,and 1,440 m. Because the firing of the RADrockets (RAD fire) occurs around 120 m alti-tude, this leaves only about 17 seconds forvelocity computation. DIMES images whilethe lander is on the parachute and can experi-ence 60 degrees per second instantaneousangular rates and as much a 45-degree rota-tion about nadir and 20˚ off of nadir betweenimages. Wind modeling of the MER landingsites showed that the worst-case steady statehorizontal winds were less than 30 m/s.

Descent imaging of the surface of Marsmust consider numerous imaging effects.First and foremost is the scene’s native con-

MAY/JUNE 2004 www.computer.org/intelligent 15

The DIMES software tracked all of the features correctly butthrew out the second feature of the second image pair becausethe peak width parameter for this feature was larger than the

maximum-allowable peak width parame-ter. Postflight analysis showed that DIMES

did compute the correct velocity.Figure A shows the images taken by the

DIMES camera and the associated featuresthat were tracked.

MER-B—OpportunityOn 25 June 2004, the MER-B or Oppor-

tunity spacecraft landed on MeridianiPlanum, halfway around Mars from Spiritand just north of the martian equator.During its descent, Opportunity took threeimages of the martian surface at altitudesof 1,986, 1,690, and 1,404 meters. Theseimages showed a very bland terrain spot-ted with a few craters. During EDL, DIMES

computed a mostly northern velocity of(8.0, –0.3) m/s at 1,547 m altitude.

After propagation down to RAD fire, thissteady-state velocity was still present (10.4,–2.8) m/s, but here the RAD-induced veloc-ity was (–7.7, 3.9) m/s, which would effec-tively negate the effect of the steady-statewind. Consequently, TIRS did not fire dur-ing the Opportunity landing. After RADfire, the bridle cut velocity was (9.0, –2) m/s,so the lander bounced north, ultimatelyending up in the so-called Eagle Crater.

The DIMES software tracked all of thefeatures correctly but threw out the sec-ond feature of the second image pairbecause the correlation for this featurewas smaller than the minimum-allowablecorrelation. Postflight analysis showed thatDIMES did compute the correct velocity.

Figure B shows the images taken by the DIMES camera andthe associated features that were tracked.

Figure B. MER-B descent image pairs of Meridiani Planum with selected and tracked templates. First pair: (a) first descent image with selected templates shown as red squaresand (b) location of selected templates shown as green squares on second descent image.Second pair: (c) second descent image with selected templates shown as red squares and(d) location of selected templates shown as green squares on third descent image.

(a) (b)

(c) (d)

trast caused by albedo variations (the frac-tion of light that is reflected by a body orsurface) and shading. Shading depends onthe terrain shape (smooth), time of day (2p.m.), the landing site latitude (near theequator), and the season (summer). Theseconditions led to a situation where the sunwas high in the sky, making the parachuteshadow and the associated brighteningaround the shadow likely elements of thescene. Because the parachute shadow moveswith the lander, it cannot be used for mea-suring horizontal velocity.

Some imaging effects are difficult to quan-tify. Dust in the atmosphere can reduce imagecontrast, while large dust particles sticking tothe camera lens can cause circular dots fixed to

the image. Cosmic rays can cause bright pix-els in the image. The lander heat shield couldstill be falling when the images are taken. Allof these effects could confuse an algorithm formeasuring horizontal velocity and must beconsidered when designing the algorithm.

The DIMES camera had to be developedfrom existing MER camera electronics andoptics. We selected the MER navigationcamera optics for the DIMES camera becausethe 45-degree field of view had the best bal-ance between pixel resolution and imageoverlap given the descent dynamics. Modi-fications were made to the optics and elec-tronics to optimize the camera’s perform-ance for the low-contrast martian terrain.The DIMES camera takes images with 5 ms

exposures to minimize the effect of motionblur during imaging. As Figure 1 shows, itis mounted on the radar bracket, which islocated on the corner of the lander that isnominally the lowest point on the landerduring parachute descent.

The DIMES algorithm is implemented inflight software in C on a 20-MHz RAD6000rover processor. Nominally 40 percent of theprocessor over 17 seconds is available forDIMES processing. The DIMES flight softwarecommunicates with other software modulesto obtain measurements: images from theDIMES camera, ground-relative attitude fromintegration of angular rates from a LittonLN200 inertial measurement unit (IMU),and ground-relative altitude obtained from aHoneywell radar altimeter system (RAS).

AlgorithmDIMES takes as input three descent images.

For each descent image, DIMES also requireselements of the lander state at the time ofimage exposure including the surface-rela-tive attitude and altitude. Using the landerstate, DIMES warps each image to the groundplane and then computes horizontal displace-ments between images using image correla-tion. Image correlation is applied to twolocations in the first and second image andtwo locations in the second and third images.This approach produces four image-basedhorizontal-velocity estimates that are com-pared for consistency to each other and theacceleration computed from velocity differ-ences from the IMU.

DIMES uses these velocity consistencychecks along with image correlation met-rics to decide if the computed velocity iscorrect. If DIMES determines that the vel-ocity is correct, it sends a velocity correc-tion to the EDL flight software module forinclusion in the TIRS firing solution. If itdetermines the velocity to be incorrect,DIMES reports that a velocity cannot becomputed and TIRS proceeds without theinput from DIMES.

At the core of the DIMES algorithm is aprocedure for computing horizontal veloc-ity from a single location visible in a pairof images. DIMES applies this procedure totwo locations in two pairs of images toobtain four independent estimates of thehorizontal velocity.

Computing horizontal velocity with two images

Figure 2 shows the operations needed to

16 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Backshell

Parachute

Lander

Inertial measurement unitinside lander

DIMES camera and radaraltimeter on bottom

corner of lander

Figure 1. DIMES imager location.

(b)

(a)

I1qG, AI1

I2qG, AI2

g

Image overlap

I2W2

TI II

Figure 2. Computing horizontal velocity from two images. (a) DIMES uses the attitude(I1qG, I2qG) and altitude (A1 and A2) of the lander to determine the overlap betweenbetween two images (I1, I2). (b) Inside the overlap region, a template (T1) is selected inI1 and this defines a window (W2) in which to search for the template.

compute horizontal velocity from a pair ofimages. First, the DIMES algorithm down-samples the images from the 1,024 × 256hardware binned images provided by theimage acquisition flight software to 256 ×256 images. Next, it warps the images ontothe ground plane, flattening them to re-move radiometric and image frame transfereffects.1 It selects a high-contrast locationin the first image, which it them compares,through image correlation, to pixels over awindow in the second image. The pixelwith highest correlation corresponds to thehorizontal shift between the images, whichcan be converted into a horizontal velocity.

Template selection. After binning, thealgorithm computes the region in the firstimage in which to search for interestingtemplates to track. This region must be inthe overlap between the two images andalso not contain the lander parachute’sshadow. To determine the image overlap,the algorithm computes the projection ofthe corners of the second image into thefirst image using the lander attitude mea-surements and altitude along with the as-sumption that the lander has zero horizon-tal motion between images. To account forperspective projection, a descent cameramodel is used. The projections of the fourcorners define a polygon in the first imageinside which the interesting template loca-tions can be identified.

The zero phase spot is the point on thesurface that is on the line between the sunand the camera. This spot is brighter thanthe surrounding terrain and also containsthe lander shadow. Because the shadow andbrightening can mislead image correlation,the algorithm computes this spot’s locationin the first image from the known sun di-rection and the lander’s attitude. Once itknows the location in the image, the algo-rithm eliminates this region from consider-ation for template selection.

Once it identifies the overlap area in thefirst image, the DIMES algorithm determinesthe location of highest contrast by comput-ing, on a coarse grid of pixels, the HarrisInterest Operator, which is the minimumvariance of a 2D linear approximation ofthe autocorrelation function.2

To reduce computations, this operationoccurs before image flattening and rectifi-cation. Ideally, template selection wouldoccur after these operations, but testing hasshown that template selection before image

processing still results in robust and accu-rate DIMES performance.

The algorithm selects the pixel with thehighest contrast as the template location.Selecting the template also automaticallyselects the window or constrained correla-tion search area in the second image.

Correlation. The algorithm uses image cor-relation to locate the template from the firstimage in the window from the second image(see Figure 3). It places the template at eachpixel location in the window and computesthe pseudo-normalized correlation betweentemplate and window pixels.3,4 The windowpixel with the maximum correlation value(peak) is the template’s most likely locationin the window. The pixel shift between thebest correlation pixel and the window’s cen-ter corresponds to the horizontal motionbetween the images where the DIMES algo-rithm computes the size of the pixels inmeters from the altitude of the first imageand the angular extent of each pixel.

Correlation metrics. Once the correlationfinishes, DIMES checks to make sure theidentified correlation peak is the correctpeak. The Harris Interest Operator calcu-lates the total template contrast metric.This metric serves to detect the imageswith little or no contrast or features. Corre-lation is the metric corresponding to themaximum template and window correla-tion. This metric serves to detect imageswith high noise or high frequency differ-

ences between images. The DIMES algo-rithm measures the correlation peak widthmetric by fitting a biquadratic to correla-tion values around the peak. This metricdetects low-frequency differences betweenthe template and the searched window,which results in inaccurate motion estima-tion, poor correlation, and larger correla-tion errors. The correlation peak ratio is theratio of the highest correlation peak to thesecond-highest correlation peak. This met-ric detects repetitive terrain and multiplefeatures of similar appearance. Figure 4depicts these correlation metrics.

MAY/JUNE 2004 www.computer.org/intelligent 17

Figure 3. Image correlation.

Template

Correlation map

Window

Peak ratioP = R/Rs = 2.5

CorrelationR = 0.97

Figure 4. Correlation performance metrics.

Rs = 0.39

Verr = (0.6, 0.8) m/s

DIMES three-image algorithmThe DIMES algorithm uses two sets of

images to determine the pre-RAD horizon-tal velocity. It requires three images to per-form, with images 1 and 2 used as the firstpair and 2 and 3 used for the second pair.For each pair, the two-image algorithmoperates twice to determine the horizontalvelocity for two templates, resulting in atotal of four velocities.

An interesting advantage of the three-image algorithm is its use of the IMU toperform a check on feature tracking that iscompletely independent of the image data.DIMES computes the differences betweeneach velocity in the first and second imagesets, resulting in four delta velocities. Itcompares each of these delta velocities to

the delta velocity computed from the veloc-ity that the IMU software module has beenpropagating since before entry. If at leastone image delta velocity is consistent withthe IMU delta velocity, DIMES is deemed tohave measured a correct surface relativevelocity. For DIMES to succeed, at least onevelocity from each image set must be cor-rect. Consequently, one template from eachimage set can result in an incorrect correla-tion, without compromising DIMES. Allow-ing for the possibility of template correla-tion failures gives DIMES a robustness tononideal imaging conditions.

If DIMES successfully computes a velocity,it passes an IMU velocity correction to TIRS.This velocity correction is the average of allvalid velocities from the second image set.

DIMES three-image algorithm validitychecking

Built into DIMES are a sequence of checkson data validity, template correlation, andvelocity consistency designed to ensure thatDIMES does not report a false horizontal vel-ocity. Figure 5 shows the overall validitylogic. For DIMES to report a successful veloc-ity computation, all input data (images andstates) must be valid and within bounds, atleast one template from each image set musthave a valid correlation, and a pair of veloci-ties (one from each image set) must be con-sistent with the delta velocity computed fromthe IMU propagation.

Software implementationThe DIMES algorithm was coded for flight

in C under the VxWorks operating system.The DIMES flight software module comprisestwo tasks: the DIMES data task and the DIMES

computation task. DDT’s main responsibilityis to ensure that images are acquired on time,while the DCT implements the DIMES algo-rithm, handles the DIMES command methods(setting and dumping parameters, and dump-ing images), and generates telemetry.

The DDT and DCT interface to a num-ber of other flight software modules. Fig-ure 6 shows the major interfaces used whileDIMES is running. The numbers indicate theordering of events, with the caveat that thefigure shows only a single image acquisi-tion. Steps 5 to 12 repeat three times beforethe velocity correction (step 13) and fin-ished message (step 14) are sent.

Once the lander passes a triggering altitude(2,000 m), the EDL flight software modulestarts the DDT processing. The DDT’s mainresponsibility is to ensure that images andstate measurements are acquired on time andpassed to the DCT.

We implemented the DIMES algorithm in the DCT as an asynchronous dataflowmachine. The algorithm breaks down intooperations that depend on different piecesof input data, so that the DCT will performwhatever operations are possible given thestates and images available so far. Thisapproach eliminates all dependencies on thearrival order of the data, guaranteeing thatthe DCT will make optimal use of availableCPU time. The dataflow machine updateswhenever a piece of data (an image or statemeasurement) arrives from the DDT.

When the DIMES algorithm has com-pleted, a signal goes to the DDT and a sta-tus variable and valid flag are set so that

18 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

valid_data

valid_template_11 valid_template_21

valid_deltaV_11_21

valid_template_11 valid_template_22

valid_deltaV_11_22

valid_template_12 valid_template_21

valid_deltaV_12_21

valid_template_12

valid_template_22

valid_deltaV_12_22

DIMES_VALID

= And

= Or

Figure 5. DIMES validity checking logic.

13–Velocitycorrection

DIMES computationtask (DCT)

EDL-TIRS

3–Run

10–State

12–Image ready

2–run5–acquire8–attitude

1–DIMES start timeEDl-RAD

4–Sun vectorEDL

DIMES data task

(DDT)IMG

6–Image request

7–Timestamp

11–Image ready

IIT 9–Attitude + velocity 14–finished

Figure 6. DIMES tasks and interfaces.

TIRS, which polls DIMES every Real TimeInterrupt (RTI), knows that a velocity cor-rection is available. Figure 7 shows theDIMES algorithm’s dataflow graph. Circlesrepresent input and output data, rectanglesindicate operations, and arrows show theflow of information between inputs, opera-tors, and outputs. Each arrow has a readyflag associated with it, indicating whetherthe data is available (in the case of inputdata) or the processing step has completed(in the case of operators). When all of theinputs to an operation have their readyflags set, the operation can be performed.

Preflight validationValidation of the performance of DIMES

was critical to prove that DIMES would dono harm during EDL. Because we couldnot completely test the entire DIMES flightsystem in a realistic flight-like environ-ment, we broke the validation tests intothree categories:

• Monte Carlo simulation provided veloc-ity accuracy statistics.

• Field testing proved that the camera andalgorithm would produce reasonablevelocity estimates when imaging Mars-like terrain at representative altitudes.

• Flight system testing proved that theflight software worked on the flight sys-tem and that the DIMES velocity answerwas available in time to effect RAD fire.

Monte Carlo performanceWe used Monte Carlo simulation to esti-

mate the performance of DIMES at eachlanding site under realistic EDL dynamicsusing imagery of actual martian landscapes.

To estimate DIMES performance duringlanding, we performed thousands of MonteCarlo runs. During each run, we simulatedsensor measurements and input them intothe DIMES flight software to produce avelocity estimate. We compared this esti-mate to the true horizontal velocity to gen-erate a velocity measurement error. Usingmultiple Monte Carlo runs, we generatedthe velocity error statistics.

Multibody aerodynamic simulations ofthe EDL lander system served to generatemultiple lander trajectories during EDL.These trajectories give the camera’s position,attitude, angular rates, and velocities duringimage exposure. Using this state informa-tion, we then used a high-fidelity model ofthe DIMES camera to generate descent images

by projecting the camera’s field of view ontoan orbital image of the landing site.

We used Mars Observer Camera NarrowAngle images for three reasons:

• The pixel resolution was comparable toDIMES image resolution.

• They were wide enough to contain aDIMES image.

• Many images were covering each MERlanding site.

Figure 8 shows an example of three imagingstates and associated fields of view on a sin-gle MOC-NA image. Finally, we used sensormodels to generate noisy measurements oflander attitude, altitude, and biased horizon-tal velocity from the true EDL trajectories.

Using Monte Carlo simulation, the DIMES

performance at Gusev Crater (Spirit’s land-ing site) was 99 percent valid velocity an-swers and a measurement accuracy of 3.7m/s (3 sigma). At Meridiani Planum (Op-portunity’s landing site), the DIMES perfor-mance was 71 percent valid velocities with3.7 m/s (3 sigma) velocity error. Both errorsare for an average altitude of 1,550 m.

Field testingMonte Carlo simulation is important for

assessing algorithm performance, but it can-not replace taking pictures with a real cam-era at altitude over Mars-like terrain. This isthe purpose of field testing. In the summerand fall of 2002, the DIMES team did a seriesof field tests in the Mojave Desert. Thesetests proved that the DIMES algorithm couldprovide accurate velocity estimates usingreal images taken at altitude and attituderates typical of EDL over terrain that wasrepresentative of the landing sites.

In the DIMES field tests, we collected therequired data to run the algorithm and assessits performance and then, in postprocessing,ran it through the DIMES flight software toestimate velocity. In other words, we per-formed no real-time velocity estimation dur-ing the tests. An engineering model DIMES

camera and IMU were rigidly attached andplaced on a pan-tilt unit. We placed this as-sembly in an inertially stabilized platformattached to the front of a crewed helicopter.A GPS unit served to record position forverifying DIMES velocity measurements. Weused the difference of GPS position and a

MAY/JUNE 2004 www.computer.org/intelligent 19

STATE 1

IMAGE1

STATE2

IMAGE2

STATE3

IMAGE3

process state(1)

bin_image(1)

process state(2)

bin_image(2)

process state(3)

bin_image(3)compute_templates(1)

compute_velocity(1)compute_overlap(2)

compute_templates(2)

compute_dimes_correction()

compute_velocity(2)

compute_overlap(1)

VDIMESVALID

Time (nominal)

Figure 7. Dataflow machine.

GPS registered digital elevation map to sim-ulate altitude measurements. Finally, weobtained an initial estimate of camera atti-tude, which is needed to initialize the gyro

attitude propagation, by taking images oftargets on the ground in known GPS posi-tion. By aligning the 3D ground targets withthe image of the ground targets, we could

determine the camera’s attitude in GPS coor-dinates using standard photogrammetrytechniques. Figure 9 shows the field testequipment and helicopter.

During a typical test, the helicopter flew upto 2 kilometers altitude and then slowly flewdown in a zigzag trajectory until close to theground. For some runs, we activated the pan-tilt unit to induce angular rates up to 60˚/s.Consecutive images were at similar alti-tudes, which is not the case for EDL. How-ever, the zigzag motion generated imagetriples, albeit significantly separated in time,with altitudes and overlap typical of EDL.

We performed field testing at three testsites in the Mojave Desert: the Pisgah LavaFlow, Kelso Sand Dunes, and Ivanpah DryLake Bed. We selected these sites becausethey are free of vegetation, cover a widerange of appearances, and have significantelevation variations. The final field test inearly October 2002 produced the most com-plete set of data for assessing the DIMES algo-rithm’s performance. During this test, weconducted three helicopter flights at differenttimes of day at each test site. When we ranthis data through the DIMES flight software,all image triples that emulated flight-likeconditions resulted in velocity estimates witherrors less than the 5 m/s requirement. Theseruns proved that DIMES worked with flightcamera images independent of the terrainimaged or the time of day.

Flight system testingWe conducted flight system testing to

ensure that the DIMES flight software andcamera operated as expected on the flightsystem. Specifically, flight system testingverified the following:

• Sensor alignments and calibration data• The temporal alignment of data from the

IMU and camera• The accuracy of velocities computed by

the flight software• The performance of the flight software

in fault scenarios • The amount of processing required to

compute the DIMES velocity estimate

Verification of the camera-to-landeralignment—that is, the phasing—for bothspacecraft occurred during assembly testand launch operations at Kennedy SpaceCenter in February through April 2003. Allother testing occurred on a functionallyequivalent avionics and flight software test

20 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Figure 9. Field test equipment, trajectory, and typical image.

Helicopter platform

Pisgah Lava Flow triples

MER EMdescent camera

GPS

LN200 IMU

Gyro-stabilizedgimbal

Pan/tilt unit

Digital elevation maps

Typical DIMES image(Pisgah Lava Flow)

Figure 8. Monte Carlo simulation.

bed called the Cruise and EDL Test Bed inJune through December 2003.

Developed by the Mars Exploration Ro-ver Project, DIMES performed successfullyon the MER Entry Descent and Landingsystem during both rover landings this pastwinter. Future Mars lander missions such asthe Mars Science Laboratory, Phoenix, andMars Sample Return can use DIMES as is.With minor modifications, DIMES can alsoserve for velocity estimation during landingon any planetary or small-body surface,making it useful for proposed space sciencemissions such as the planned Comet Sur-face Sample Return and Lunar SurfaceSample Return. DIMES will also work forautonomous navigation of unmanned aerialvehicles used in surveillance and other de-fense applications.

AcknowledgmentsWe thank Rob Manning and the entire Mars

Exploration Rover project management for sup-porting the development of DIMES. We also thankthe DIMES Review Board for the withering butuseful reviews during DIMES development, andthe DIMES field test team for providing the sys-tem and data that confirmed the performance ofDIMES. In addition, DIMES would not have cometogether so quickly had it not been for the devel-opment of core machine vision components inthe Intelligent Systems and Mars TechnologyPrograms at JPL. The research described in thispublication was carried out at the Jet PropulsionLaboratory, California Institute of Technology,under contract with the National Aeronauticsand Space Administration.

References1. Y. Cheng et al., “Passive Image-Based Hazard

Avoidance for Spacecraft Safe Landing,” Proc.6th Int’l Symp. Artificial Intelligence, Robotics,and Automation in Space (iSAIRAS 01), Euro-pean Space Agency Publications Div., 2001.

2. C. Harris and M. Stevens, “A Combined Cor-ner and Edge Detector,” Proc. 4th Alvey VisionConf., 1988, pp. 147–151.

3. A.E. Johnson and L. Matthies, “Precise Image-Based Motion Estimation for AutonomousSmall Body Exploration.” Proc. 5th Int’l Symp.Artificial Intelligence, Robotics, and Automa-tion in Space (iSAIRAS 99), European SpaceAgency Publications Div., 1999, pp. 627–634.

4. S.I. Roumeliotis,A.E. Johnson, and J.F. Mont-gomery, “Augmenting Inertial Navigation withImage-Based Motion Estimation,” Proc. IEEEInt’l Conf. Robotics and Automation (ICRA2002), IEEE CS Press, 2002, pp. 4326-4333.

MAY/JUNE 2004 www.computer.org/intelligent 21

Yang Cheng, DIMES

Algorithm Developerat the Jet PropulsionLaboratory. Contacthim at ycheng@robotics.jpl.nasa.gov.

Jay Goguen, DIMES

Science Lead at theJet Propulsion Labo-ratory. Contact him at jay.goguen@jpl.nasa.gov.

Andrew Johnson,DIMES Software Leadat the Jet PropulsionLaboratory. Contacthim at aej@jpl.nasa.gov.

Chris Leger,DIMES Flight Soft-ware Developer at theJet Propulsion Labo-ratory. Contact him atcleger@robotics.jpl.nasa.gov.

Larry Mattthies,Supervisor MachineVision Group at theJet Propulsion Labo-ratory. Contact him atlhm@robotics.jpl.nasa.gov.

Miguel San Martin,MER Guidance andControl Project Ele-ment Manager at theJet Propulsion Labora-tory. Contact him atmiguel.sanmartin@jpl.nasa.gov.

Reg Willson, DIMES

Hardware Lead at theJet Propulsion Labo-ratory. Contact him atrwillson@jpl.nasa.gov

Join a community that targets your discipline.

In our Technical Committees, you’re in good company.

www.computer.org/TCsignup/

Looking for a community targeted to your area of expertise?

IEEE Computer Society Technical Committees explore a

variety of computing niches and provide forums for dialogue

among peers. These groups influence our standards development

and offer leading conferences in their fields.

Join aThinkTank