Multisensor obstacle detection and tracking

C. Stillera,* , J. Hippb, C. Rossiga, A. Ewaldb

aRobert Bosch GmbH, FV/SLH, P.O. Box 77 77 77, 31132 Hildesheim, GermanybIBEO Lasertechnik GmbH, Fahrenkro¨n 125, 22179 Hamburg, Germany

Received 3 March 1999; received in revised form 28 August 1999; accepted 14 September 1999

Abstract

This submission is concerned with obstacle detection and tracking for an autonomous, unsupervised vehicle. A multisensor concept isproposed yielding a high level of reliability and security. It includes a variety of different sensor technologies with widely overlapping fieldsof view between the individual sensors. The major sensors for obstacle detection comprise a self-assessing vision sensor directed forwardsand a laser scanner system surveying 3608 around the vehicle. Preliminary results indicate the high reliability of the sensor system.q 2000Elsevier Science B.V. All rights reserved.

Keywords: Multisensor obstacle detection and tracking; Stereo vision; Laser scanner; Driver assistance; Autonomous driving

1. Introduction

Following several decades of intense research in auto-mated vehicle guidance, we are now witnessing marketintroduction of driver assistance functions into standardpassenger cars. Most of these functions are based uponinertial sensors, i.e. sensors measuring the status of thevehicle itself. Anti-lock braking systems (ABS) or vehicledynamic control (VDC) are but few examples.

Recently, much interest has been devoted to environmentsensors, e.g. radar, lidar, sonar or video-based sensors.Information about the status of the environment can beemployed for co-operative driving, thus improving trafficflow, comfort and—at least in the long term—safety offuture traffic. The broad spectrum of applications includesnavigation aids, parking aids as well as adaptive cruisecontrol (ACC).

At the very extreme of the spectrum of automated vehicleguidance functions ranges the complete machine control ofthe vehicle. Much work in this field has been performed byinternational research groups, e.g. [1–4]. These and othergroups have successfully demonstrated the technical feasi-bility of autonomous driving in restricted environments andunder human supervision.

An important issue for the realization of driver assistancesystems is the development of autonomous system conceptsthat yield a maximum level of reliability. As a significant

extension of previous work, the project “AutonomesFahren” (Autonomous Driving) contributes to this field bydevelopment of an autonomous driving system that allowsoperation without the requirement of human supervision.The project consortium comprises industry and SMEs aswell as universities, all located in Lower Saxony, Germany.The major partners are Volkswagen, Bosch, University ofBraunschweig, University of Armed Forces Hamburg,IBEO/Kasprich, and Witt.

As a step towards product introduction, the autonomoussystem and hence its sensorial components described in thispaper are designed for fast mounting into virtually arbitrarypassenger cars. Hence information that is needed for vehicleguidance but is not available in arbitrary standard cars, suchas the vehicles velocity, is not provided to the autonomoussystem. Instead, it has to be gathered by the sensors them-selves.

As mentioned above, the autonomous system can beconsidered as the most complete set of intelligent vehiclefunctions. From this point of view, the reported work contri-butes to the development of innovative sensors, controllersand actuators that allow the realization of intelligent func-tions into modern passenger cars.

All actuators are comprised in a driving robot pushingpedals and turning the steering wheel like a human driver.The passenger cars are driven on a vehicle proving ground.The course has been designed for fast endurance verificationof vehicles. For this reason, the vehicles are exposed tostrong accelerations in all directions through intentionallyextreme road conditions. This imposes challenging

Image and Vision Computing 18 (2000) 389–396

0262-8856/00/$ - see front matterq 2000 Elsevier Science B.V. All rights reserved.PII: S0262-8856(99)00034-7

www.elsevier.com/locate/imavis

* Corresponding author.E-mail address:christoph.stiller@de.bosch.com (C. Stiller).

requirements on both the hardware and the algorithms.Beyond technical work, also non-technical issues areaddressed in the project. These include safety assessment,deployment model, cost effectiveness and mitigation ofresults to driver assistance systems. The interested readeris referred to [5] for a detailed overview on the project.

The focus of this paper is on the sensor concept developedfor the autonomous system and on the realization thereof.The main task of the sensors is twofold, namely lane recog-nition and obstacle detection and tracking. This informationenables the control unit to keep the vehicle on the track andto avoid collision with obstacles, respectively. While theformer task is detailed in [6], two sensors for the lattertask are outlined in this paper.

Reliable detection and tracking of obstacles is a crucialissue for automated vehicle guidance functions. Since thevehicle allows operation in supervised as well as inunsupervised mode a sensor concept is required that offershigh dependability. For this reason, a multisensor concepthas been designed. The sensor outputs are then fed into asensor fusion unit that provides information to thecontroller.

The remainder of this paper is organized as follows. Thefollowing section provides a general system overview anddetails the multisensor concept. A stereo vision sensor and alaser scanner system form the key sensors for obstacledetection. These two sensors are discussed in Sections 3and 4, respectively. Section 5 is concerned with obstacletracking based upon the measurement data of the twosensors. First experimental results for both sensors arereported in Section 6. The paper closes with a summaryand concluding remarks.

2. System overview and multisensor concept

Several technological approaches have been proposed inliterature for environment sensing. Recognition of lanegeometry and the estimation of the ego-position on thelane are frequently performed by infrastructure-basedapproaches. In a well-known approach, sensors detect thefield emitted by permanent magnetic markers or an electricwire buried in the road [3]. Another proposal is the employ-ment of aluminum-coated lane boundaries that can bedetected by radar. Apart from technical reasons, the currentpolitical situation does not encourage the expectation thatsuch infrastructure will be broadly mounted. Furthermore,its worldwide introduction is complicated by a chicken-and-egg kind problem. While the market introduction of sensorsrequires the broad availability of infrastructure, mounting ofsuch infrastructure does not make sense after the sensorsexist in the vehicles.

In order to alleviate these problems, the project consor-tium has decided to develop solutions that do not requireany additional infrastructure. Fig. 1 depicts a block diagramof the sensor and control components. Fig. 2 sketches thesensor components and their fields of view. Redundantinformation about lane geometry and ego-position isachieved by pursuing the following two independentapproaches.

• A combination ofDGPS(Differential Global PositioningSystem) withinertial sensorsand adigital map deter-mines the position and direction of the vehicle as wellas the road geometry. The inertial sensors (INS) areemployed for continuous determination of the ego-position

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396390

Fig. 1. Block diagram of system.

and orientation via integration while the DGPS signalcompensates for the integration drift. Data fusion isperformed in a Kalman filter.

• A stereo vision sensorrecognizes and tracks lane bound-aries. It simultaneously computes the three-dimenstional(3D) geometry of the lane and the ego-position of thevehicle relative to this lane.

In normal operation, each of these two systems indi-vidually yields information about lane geometry and ego-position accurate and reliable enough for vehicle guidance.Fusion of the two sensor outputs is performed in order toachieve reliability in critical situations, e.g. in the event ofcomponent failure. As a positive side effect, sensor fusionalso enhances the accuracy of the measurements, whichenables a smoother vehicle control. Details concerninglane recognition may be found in [6].

Besides recognition of lane boundaries and ego-position,detection and tracking of obstacles that could potentiallycollide with the vehicle is a crucial issue. For the sake ofsecurity this task is addressed by a multitude of differentsensors as depicted in Figs. 1 and 2. The multisensorconcept includes the following sensors:

• A stereo vision sensoris mounted behind the windscreen. It is directed forwards with a field of view of308 horizontally and 238 vertically. It aims at a mid andlong range. This sensor simultaneously fulfills the twotasks of lane recognition and obstacle detection. It isthis suitability to multiple simultaneous tasks thatmakes vision sensors particularly attractive for transferalinto automotive products.

• Laser scanner: two laser scanners are mounted at the leftand right on the front bumper. Each of those sensorscovers a wide viewing angle of 2708 to the left frontand right front, respectively. The sensors overlap in themost important front view in 1808. A third laser scanneris attached to the back of the vehicle surveying the rear1808. In the center 308 field, this sensor scans multiplelines with slightly differing tilt angles, sequentially. Thus

the laser scanner system by itself covers a total look-around. The laser scanners are designed for short, mid,and long range.

• Radar: a front radar sensor with small viewing angle butlong range is mounted on the front bumper. A standard,mass-produced 77 GHz ACC radar is employed.

• Short range radar: several short-range radar sensorscover the vicinity of the vehicle. They do not measureobstacle position but only signal presence or absence ofobjects in their field of view. At present, the sensorsystem does not involve short-range radar sensors.During operation, the 3608surveying sensor system isexpected to detect and track obstacles prior to theirentry into the short range. From this point of view,short-range sensors are not mandatory. However, thesesensors are optionally available in case long-termexperience reveals their need, possibly after start-up.

It is worth noting that the degree of redundancy increaseswith the relevance of the area covered. The rear of thevehicle is surveyed by means of a single sensor, the sidesare each covered by two independent laser scanners andseveral overlapping short range sensors. Finally the, impor-tant, front side of the vehicle is covered by three powerfullong-range sensors, using different wavelengths and signalanalysis principles. Different sensor technologies arecombined in order to reduce the probability of coincidenceof gross errors in the sensor output as much as possible. Thesensor signals are combined by sensor fusion into a jointobstacle map. This serves as input to the subsequent controlunit for vehicle guidance and collision avoidance. Moredetails on sensor fusion and control may be found in [7].By consideration of measurement data and their associatedconfidence and reliability measures, the obstacle mapcomputed by sensor fusion is more precise and reliablethan any of the individual sensor outputs by themselves.

3. Stereo vision

The vast majority of creatures able to navigate throughspace strongly relies on its visual system for this task. Inparticular, it is well known that humans perceive about 90%of the information required for driving visually. This allowsthe conclusion that sufficient information is available in thevisual domain. Furthermore, it can be expected that visionbased driver assistance systems exhibit a fairly transparentbehaviour, e.g. drivers are prepared for a reduced visibilityrange in bad weather conditions. Finally, vision allows theperception of a multitude of different information relevantfor vehicle control. Much of this information has beendesigned for visual perception and is hardly recognizableby other technologies, such as lane markings or traffic signs.Thus it comes as no surprise that much work has beenreported on vision sensors and, in particular, on vision-based obstacle detection and tracking.

Fig. 3 illustrates the obstacle information acquired by the

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396 391

Fig. 2. Sensor overview.

vision sensor. Firstly, the ground plane under the vehicle isdescribed by its normal vector and velocity relative to thecar. Likewise obstacles emerging above the ground planeare described by their 3D position, their size, i.e. height,width and length, and their 3D velocity. It is worth notingthat the sensor acquires not only static but also dynamicinformation for each obstacle.

The performances of any sensor is dependent on environ-mental conditions such as weather or traffic situation. Whileit must be accepted that sensor precision and reliability isaffected under unfavourable conditions, it is consideredcrucial for automovtive applications that each sensor atleast notes its own limited capability and signals this infor-mation to the control unit. This property is referred to asself-assessmentand plays a paramount role in constructionof reliable systems [8].

For the proposed vision system, self-assessment infor-mation, namely confidence and reliability measures, accom-panies every sensor measurement. The covariance matrixKof the estimated parameter vectord is provided as a confi-dence measure in this contribution. It is derived directlyfrom the raw data of the individual sensors and is propa-gated throughout the sensor system [9]. The reliabilityR isrepresented by a goodness-of-fit measure. It uses a distancemeasureq(yud,m), quantifying the fit of the observed dateyto the estimated parametersd and the underlying modelm.The observed datey is considered as a realization of arandom process denoted byY. Finally, the reliabilitymeasure is given by the cumulative probability for thedistance of the data random process exceeding the distanceof the actually observed data.

R� P�q�Yud;m� . q�yud;m�� �1�To be more specific, let us consider the simple example ofmultivariate gaussian observations. Then, natural distancemeasure is provided by the Mahalanobis distance and thereliability measureR reduces to the well-knownx 2-test.

Two main approaches are employed for obstacle detec-tion in literature. The first class of approaches detectsselected two-dimensional (2D) intensity patterns or featuresderived from those patterns in the image sequence. Theselection is commonly chosen ad hoc aiming at identification

of patterns and features that are ‘typical’ for the expectedkind of obstacles. Examples for such patterns and featuresinclude bounding edges [2], dark wheels and shadows belowvehicles [10] as well as symmetry [11].

Fig. 4 illustrates the scheme for the example of the loweredge of the shadow below vehicles. Initially, an area ofinterest is determined. It often exhibits triangular shapeaccounting for the expected width of obstacles thatdecreases with distance. An edge operator is applied tothis area and horizontal edges from light to dark are detectedby appropriate thresholding. The bottom most position of anedge in the image can then be associated to the lower edgeof an obstacle. By applying additional features and plausi-bility checks and by stabilization through temporal filtering,good results are reported for standard situations. However,problems remain for unfavorable conditions, i.e. when nosignificant dark area appears below a vehicle, such as,during sunrise, sun dawn or night. Furthermore, a methodfor obstacle detection based on a model for the visualpattern of obstacles is restricted to a predefined set of‘typical’ obstacles. General visual features for arbitraryobstacles cannot be imposed. This results in an insufficientreliability of such approaches for the purpose of unsuper-vised autonomous driving. For this reason, these methodsare not considered as a solution but only as a potentialadd-on for the purposes of this paper.

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396392

Fig. 4. Obstacle detection using 2D patterns.Fig. 3. Obstacle date acquired by stereo vision.

Fig. 5. Obstacle detection by stereo vision.

A different approach towards obstacle detection employsdisplacement information obtained from a single camera(optical flow) [12] or from a stereo camera (disparity) assketched in Fig. 5. By identification of points in both imagesprojecting the same position of the real world, that realworld position can be reconstructed by triangulation [13].The figure illustrates this technique for the example of asingle pair of corresponding points.

As depicted in Fig. 5, a rectified and calibrated camerapair is assumed, i.e. the retinal planes of the cameras areidentical. In practice, this assumption will be met onlyapproximately. However, intrinsic and extrinsic cameracalibration determines a transformation that associates theimage coordinates of the real camera system uniquely withimage coordinates of a virtual camera pair that perfectlyfulfills the above assumptions. The transformation is projec-tive linear and includes intrinsic calibration and rectifica-tion, Various methods for its computation are known fromliterature [14]. For the sake of notational brevity, we assumein the sequel that calibration and rectification matrices havebeen computed and applied by some suitable method. Inparticular, the imagesg1 and g2 are considered as theimages of the calibrated and rectified left and right camera,respectively.

Let the 3D world coordinate systemx � �x; y; z�T beattached to the left camera as depicted in Fig. 5 and letthe image coordinates of an observed point in the left andright camera be denoted byx1; y1 and x2; y2; respectively.Due to previous calibration and rectification the epipolarconstraint givesy1 � y2: Then the world coordinates ofthe observed point is given as

x

y

z

0BB@1CCA � b

x1 2 x2

x1

y1

1

0BB@1CCA; �2�

whereb denotes the baselength of the stereo camera. Pairsof corresponding points are computed employing aBayesian displacement estimator as outlined in [15,16].Beyond the estimate itself, Bayesian estimators inherentlymodel the a posteriori distributionp(kug1,g2) for givenimages. Thus, in particular, each estimatek � �x1; y1; x2�Tof a point correspondencek � �x1; x1; x2�T is associatedwith an estimate for its covariance matrix

Cov�k� �Z�k 2 k��k 2 k�Tp�kug1; g2�·dk �3�

Furthermore, assuming Gaussian errors, a reliabilitymeasureR is readily defined through ax 2-Test as formu-lated in Eq. (1).

For small errors, the covariance of the point correspon-dence propagates to the world co-ordinates as (see, e.g. [9])

Cov�x� � J�k�·Cov�k�·J�k�T �4�

with the Jacobian matrix

J�k� � 2x2k�

2zx2

x1 2 x20

xx1 2 x2

2yx1 2 x2

zy

x1 2 x2

2zx1 2 x2

0z

x1 2 x2

0BBBBBBB@

1CCCCCCCA �5�

Eqs. (2)–(5) can be used to provide a rough estimate ofthe theoretical range of stereo vision. In our experiments abaselength of 0.3 m and a focal length of 600 pel was used.Assuming that at least displacements of 1 pel can bedetected, one can recognize obstacles up to a theoreticaldistance of 180 m when following Eq. (2). Obviously,there are several artifacts (and also a few possible improve-ments) not yet considered, e.g. lens distortion, which willdeteriorate (improve) system performance. However, theabove calculation shows that the order of magnitude forthe range of such stereo sensors is reasonable for thepurposes in this paper.

Several techniques from statistical signal processing andcomputer vision are applied to extract the desired para-meters from the set of 3D positions. Robust clustering andestimation to gain accuracy and reliability can exploit thelarge number of corresponding points that can be identifiedfor most obstacles. Moreover, temporal prediction andtracking adds to stability of the estimates over time. Thisissue will be detailed in Section 5.

4. Laser scanner

Additionally to the passive vision sensor two activesensor technologies, namely radar and laser scanning areintegrated for obstacle detection. While a commercial77 GHz ACC radar sensor is employed, development of asuitable laser scanner has been performed within the project.

The laser scanner actively emits pulses and measures theincoming reflections of those pulses from the world. Thedistance to the target is then directly proportional to the

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396 393

Fig. 6. LADAR DIGITAL A AF.

time between transmission and reception of a pulse.Emerging from the existing line of products new laser scan-ners with appropriate properties have been developed [17].The result is the LADAR DIGITAL A AF depicted in Fig. 6.The laser pulse is generated by an InGaAs laser diode work-ing in the near infrared. The scanning of the measurementbeam is achieved via a rotating prism. The scan frequency is10 Hz. The ensemble of measurements from each scanforms a 2D instantaneous profile of the real world. Thesingle measurements overlap each other, in order to avoidgaps in the scan within the working range of 250 m. This isan important prerequisite for the detection of slim objects.The laser scanner is eye-safe and fulfills the requirements oflaser class 1.

Measurement control is performed by a 16 bitm-Processor.Simultaneously, distance and angle are measured in order toget a 2D range profile. The angle measurement employs ahigh resolution angle encoder and simple counting achiev-ing an angular beam separation of 0.258. The optical beamhas a divergence of 5 mrad corresponding to 50 cm diameterat 100 m. The range of distances covers the near field begin-ning at the lens up to the long range of 250 m for reflectingand of 100 m for nonreflecting targets. The standard devia-tion of a single pulse range measurement is about 3 cm. Thisprecision in angle and range measurement is crucial forderivation of the lateral and transversal velocity of objects.Furthermore, the high precision of position and dynamicdata of objects allows reliable temporal prediction overshort time intervals.

For the safety of the vehicle and its surroundings thesenew laser scanners are mounted on the vehicle and cover acomplete 3608 field of view arround the vehicle (Fig. 2). Theperformance at various heights is subject of current investi-gations. Two laser scanners are fixed to holders in the heightof the front bumper at the left and right, respectively. Thescanning area is 2708 each in driving direction and to the leftand right of the vehicle, respectively. The shadowed sectionbehind the car is covered by a third laser scanner mounted atthe backside covering 1808 and the middle section 308 withseveral lines. This laser scanner scans the traffic situationbehind the vehicle and provides information that is impor-tant for lane changes and prevention of rear end collisions.This configuration is examined and optimized duringon-road tests.

Each laser scanner has safe electronics. An output signalis switched in case of component failure. The safety outputsof the three laser scanners are ored and connected to thevehicle controller which initiates appropriate action incase of sensor malfunction. The laser scanners incorporatea 32 bit DSP for line data acquisition and evaluation. Themain task of this DSP is object tracking.

5. Estimation and tracking

As outlined in the previous sections, both sensors, the

vision sensor and the laser scanner eventually yield anensemble of measurement positions from the real world.These remain to be clustered to the road surface andobstacles to provide meaningful input to vehicle control.While the vision sensor scans three-dimensionally thelaser scanners survey 2D planner sections of the worldonly. However, several planner sections forming a fan canbe surveyed by the laser scanners as realized for the rearview scanner. Thus, obstacle detection and tracking fromthe intermediate measurements can be treated similarly forthe two sensors.

For the first observation of an obstacle a fast initial clus-tering of measurements is performed based on Euclideandistance of the measurements. Two measurement pointsare assigned to the same obstacle, if their distance isbelow a threshold. This threshold emerges naturally fromthe measurement resolution of each sensor, i.e. from thesampling distance for the laser scanner and from the regionsize employed in disparity estimation for the stereo visionsensor.

Tracking is performed in a Kalman filter that is basedupon a simple model for the dynamic behavior of theobjects. Let d � �x; _x; �x; y; _y; �y; z; _z; �z; h;w; l�T denote thestate vector comprising the desired parameters for a singleobstacle, namely position, speed and acceleration in allthree dimension as well as height, width and length (seeFigs. 3 and 5). A separable piecewise constant white noisejerk model expresses the dynamics as Ref. [18]

d�k 1 1� � Fd�k�1 Gv�k�1 v�k� �6�with the block diagonal transition matrixF and gain vectorG given as

F �

Fx 0 0 0

0 Fy 0 0

0 0 Fz 0

0 0 0 Fh

26666664

37777775;

Fx � Fy � Fz �1 T T2

2

0 1 T

0 0 1

2666437775; Fh � I ;

�7�

G �

Gx

Gy

Gz

0

26666664

37777775; Gx � Gy � Gz �T2

2

T

1

2666437775; �8�

whereT denotes the sampling interval. Finally, the varianceof the scalar jerk noisen (k) is given asTs2

x while the growthnoise covariance is defined as

Cov�v� � T·diag 0;…;0;s2h;s

2h;s

2h

� ��9�

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396394

with s2h denoting growth noise level. This model predicts

the state vector for each obstacle from frame to frame.The measurement equations for the stereo vision sensor

and the laser scanner have been outlined in Sections 3 and 4,respectively. For the example of the stereo vision sensorthey are formulated by linearization of Eq. (2). Thecovariance of the measurement noice vector is determinedthrough Eqs. (3)–(5).

This information completely specifies the Kalman filter.Emerging from the current obstacle map, first a predictionstep is performed in the kalman filter based on Eq. (6), i.e.the state of each obstacle is predicted according to itsdynamic model. Then the measurement data is assigned tothe individual obstacles by a minimum distance operator.Then measurement data is incorporated into the individualstate vectors of the obstacles in the innovation step of theKalman Filter [18].

Although estimation and tracking by the vision sensorsand the laser scanners are performed by the same technique,some implementation issues are individually adapted to thedifferent technologies. While the 2D laser scanners do notallow measurement of position, speed and acceleration invertical direction, some additional features are added to thestate vector of each obstacle in order to stabilize the Kalmanfilter. The state variables include the “characteristic posi-tion” of the segment (which is defined as the center ofgravity of all cluster points) as well as their size. Besides,it includes additional geometrical characteristics, such ascorner points, which serve to stabilize object tracking.

A crucial step for stable obstacle tracking is the assign-ment of the segments to objects. This is performed by clus-tering based on a distance measure between the predictedstate and the observed measurements. The threshold for themaximum distance between a measurement value and thepredicted state of an object forms an a priori chosen para-meter. A segment is assigned to the nearest object with adistance below this threshold. Further details concerning theKalman filter for the laser scanner are given in [3,19].

The dynamic data of all obstacles is transmitted via aCAN Bus interface to the vehicle control unit. This conducts

data fusion of the three laser scanner outputs with the data ofthe other sensors.

6. Experimental results

On-board experiments with a stereo vision sensor and alaser scanner have been performed. Fig. 7 depicts twoexamples of images captured by the left camera of the stereovision sensor mounted in a Bosch experimental vehicle. Thescenes have been taken at a typical German country road.

Obstacles emerging from the road surface are marked andtheir 3D position (light crosses) and size (light boxes, wher-ever stable) are estimated. The viewing distance is up to100 m for CIF resolution (352× 288 pel) at a viewingangle of 308 which is already appropriate for the envisagedapplication. As can be seen in the figure, even smallerobstacles are detected, e.g. the distance of the motorcyclistfollowing the white car in the right standing image isapproximately 70 m. The computational effort required bythe system allows real time operation at video rate on asingle low cost PC processor.

The information from self-assessment of the stereo visionsensor can be used for system control purposes. In general, thegoodness-of-model-fit measure rarely indicates insufficientmeasurements. In most cases, the reasons for such rare insuf-ficiencies are immediately plausible to visual inspection.However, in the extreme case, e.g. when vision is obstructedby fog, the vision sensor can signal ‘complete analysis failure’thus preventing unsafe operation of the system.

The raw data of a scan from the laser scanner is depictedin Fig. 8. The figure illustrates the easy noticeable clustersformed by vehicles and other objects. This is caused by theirnatural separation in distance and by their almost rectangularshape. The separation of segments gets more complicatedwhen obstacles are dense and move with similar speed.Therefore the boundaries of the objects are formed byjumps in range and angle. For all segments the laser scannercomputes the characteristics and measurement date in real-time and propagates this data for object tracking.

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396 395

Fig. 7. Obstacles detected in country road scenes.

7. Summary and conclusions

A multisensor concept has been proposed for autono-mous, unsupervised vehicle guidance within the projectAutonomes Fahren. It accounts for the high level of relia-bility required by that application. The concept incorporatesvarious sensor technologies, including DGPS, radar, visionand lidar. The viewing fields of the individual sensors over-lap widely. The level of redundancy thus achieved isincreasing according to the importance of the surveyedarea. A stereo vision sensor and three laser scanners fromthe core of the sensor system for obstacle detection. Thelaser scanners jointly survey the complete field of 3608around the vehicle. High accuracy is achieved by real-time evaluation of the sensor raw data. Estimation andtracking of obstacles is performed in a Kalman filterwhose measurement input are point position estimatesfrom the vision sensor and the laser scanners, respectively.Preliminary results indicate already a good performance ofthe sensor system. Future work is directed towards systemintegration and experimental evaluation.

Acknowledgements

The project “Autonomes Fahren” is supported by theMinistry of Economics, Technology and Transport of thefederal state of Lower Saxony, Germany.

References

[1] A. Broggi, M. Bertozzi, A. Fascioli, G. Conte, Automatic VehicleGuidance: the Experience of the ARGO Autonomous Vehicle,World Scientific, Singapore, 1999.

[2] E.D. Dickmanns, T. Christians, Relative 3D state estimation forautonomous visual guidance of road vehicles, in: ConferenceIntelligent Autonomous Systems, 1989.

[3] S.E. Shladover, Advanced vehicle control and safety systems, IATSSResearch 21 (2) (1997) 40–48.

[4] C.E. Thorpe, Vision and Navigation—the Carnegie Mellon Navlab,Kluwer Academic, Dordrecht, 1990.

[5] H. Weisser, P. Schulenberg, R. Bergholz, U. Lages, Autonomousdriving on vehicle test tracks: overview, motivation and concept,IEEE Conference on Intelligent Vehicles, 2, 1998, pp. 439–443.

[6] J. Goldbeck, G. Draeger, B. Hu¨rtgen, S. Ernst, F. Wilms, Lane follow-ing combining vision and DGPS, IEEE Conference on IntelligentVehicles, 2, 1998, pp. 445–450.

[7] J.C. Becker, A. Simon, I. So¨hnitz, H. Gollinger, W. Schumacher, Adecentralized path planning and control structure for an autonomousvehicle, IEEE Conference Intelligent Vehicles, 2, 1998, pp. 457–462.

[8] C. Stiller, W. Pochmuller, B. Hurtgen, Stereo vision in driverassistance systems, in: IEEE Conference Intelligent TransportationSystems, No. 3701, November 1997.

[9] R.M. Haralick, Propagating covariance in computer vision, TwelfthIAPR International Conference on Pattern Recognition. ConferenceA: Computer Vision and Image Processing, Jerusalem, Israel, 1, 1994,pp. 493–498.

[10] F. Thomanek, Visuelle Erkennung und Zustandsscha¨tzung vonmehreren Strabenfahrzeugen zur autonomen Fahrzeugfu¨hrung, PhDthesis, University BW Munich, VDI Verlag, Fortschr. Ber., ser. 12,no. 272, 1996.

[11] T. Zielke, M. Brauckmann, W. von Seelen, Intensity and edge-basedsymmetry detection applied to car-following, ECCV, Lecture Notesin Computer Science, 588, 1992, pp. 865–873.

[12] W. Enkelmann, V. Gengenbach, W. Kru¨ger, S. Ro¨ssle, W. Tolle,Obstacle detectin by real-time optical flow evaluation, in: IntelligentVehicles ’94 Symposium, 1994, pp. 99–102.

[13] R.I. Hartley, P. Sturm, Triangulation, Computer Vision and ImageUnderstanding 68 (2) (1997) 146–157.

[14] O. Faugeras, Three Dimensional Computer Vision: a GeometricViewpoint, MIT Press, Cambridge, MA, 1995.

[15] C. Stiller, Motion-estimation for coding of moving video at 8 kbit/swith Gibbs modeled vector field smoothing, SPIE Conference onVisual Communication and Image Processing, 1360, 1990, pp. 468–476.

[16] C. Stiller, Object-based estimation of dense motion fields, IEEETrans. Image Processing 6 (1997) 234–250.

[17] IBEO, Fahrenkro¨n 125, D-22179 Hamburg, Germany: TechnicalDescription of LADAR DIGITAL A AF, http://www.ibeo.de.

[18] Y. Bar-Shalm, X.-R. Li, Estimation and Tracking: Principles, Tech-niques and Software, Artech House, 1995.

[19] A. Kirchner, K. Timm, Vehicle Tracking with a Laser Scanner, in:ISATA Conference, June 1998.

C. Stiller et al. / Image and Vision Computing 18 (2000) 389–396396

Fig. 8. Single scan 2D profile of a highway scene.