active pedestrian protection system, project review - …web.yonsei.ac.kr/hgjung/ho gi jung...
TRANSCRIPT
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
T_AP00966
Active Pedestrian Protection System, Project ReviewActive Pedestrian Protection System, Project Review
2010. 10. 29
Ho Gi Jung
The School of Electronic and Electrical Engineering
Yonsei University, South Korea
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Project OverviewProject Overview
Objectives
To develop active pedestrian protection system (APPS),detecting pedestrian using sensor fusion-based,assessing risk of vehicle-pedestrian collision,and actuating countermeasures to avoid the collision.
Duration
1st Phase (Prototype development): Nov. 2006 – Oct. 2009 (36 months)2nd Phase (Product development): Nov. 2009 – Oct. 2011 (24 months)
Fund
Co-funded by participating companies and government, MKE (Ministry of Knowledge and Economy)For the 1st phase, total budget is 4.7 million USD.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Team OrganizationTeam Organization
MANDO
Project management
System design
Vision-based pedestrian classification
Sensor fusion of NIR vision and range sensor
Sensor fusion of FIR stereo and visible vision
Risk assessment
Actuation
LG Innotek
TOF camera module
TOF camera-based pedestrian recognition
SL
NIR camera module
NIR headlamp
Yonsei Univ.
Radar array-based pedestrian recognition
VisLab
System design
Sensor fusion of NIR vision and range sensor
Ho Gi Jung
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
ScenarioScenario--Driven MethodDriven Method
Almighty classifier is replaced with a combination of strict experts.
• Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, and Ho Gi Jung, “A New Approach to Urban Pedestrian Detection for Automatic Braking,” IEEE Transactions on Intelligent Transportation Systems, Vol. 10, Issue 4, Dec. 2009, pp. 594-605.
• Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, Ho Gi Jung, “Localization and Analysis of Critical Areas in Urban Scenarios”, 2008 IEEE Intelligent Vehicle Symposium, Eindhoven University of Technology, Eindhoven, The Netherlands, June 4-6, 2008, pp. 1074-1079.
• Alberto Broggi, Pietro Cerri, Luca Gatti, Paolo Grisleri, Ho Gi Jung, JunHee Lee, “Scenario-Driven Search for Pedestrians Aimed at Triggering Non-Reversible Systems,” 2009 IEEE Intelligent Vehicle Symposium, 2009 IEEE Intelligent Vehicle Symposium, June 3-5, 2009, Xi’an, Shaanxi, China, pp. 285-291.
False negative False positive
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Critical AreaCritical Area--Centered Pedestrian RecognitionCentered Pedestrian Recognition
The top ranked situation is when a pedestrian popped up from the behind of vehicle parked along a road-side.
We assume that pedestrians in front of ego-vehicle without occlusion would be easily detected by the driver.
Examples of critical area. The second row shows the critical area of situations of the first row.
““Critical areaCritical area””
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Comparison of Four Sensing MethodsComparison of Four Sensing Methods
Four sensing methods were investigated and compared:1) Sensor fusion of NIR vision and range sensor (scanning laser radar)2) TOF (Time Of Flight) camera-based3) Range sensor array-based (mm-wave radar)4) Sensor fusion of FIR stereo and visible vision
Three Criteria
1) Critical area localization performance2) Latency of popping up pedestrian detection3) System cost and possibility of domestic series production
Sensor fusion of NIR vision and range sensor was selected for the next phase.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Sensor Fusion of NIR Vision and Range SensorSensor Fusion of NIR Vision and Range Sensor
• Pietro Cerri, Luca Gatti, Luca Mazzei, Fabio Pigoni, Ho Gi Jung, “Day and Night Pedestrian Detection Using Cascade AdaBoostSystem,” the 13th IEEE International Conference on Intelligent Transportation Systems, Maderia Island, Portugal, 19-22 Sep. 2010, accepted on 2 Jul. 2010.
Vision-based pedestrian classifier
- Haar-feature-based Adaboost
- Ad-hoc-features-based Adaboost
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
(a) General headlamp
Sensor Fusion of NIR Vision and Range SensorSensor Fusion of NIR Vision and Range Sensor
HDRC Camera Module with NIR Sensitivity
NIR Lamp with horizontally wider FOV
(b) Newly developed headlamp
Bi-functional module NIR lamp
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Detected critical area Result of vision-based pedestrian verification
Sensor Fusion of NIR Vision and Range SensorSensor Fusion of NIR Vision and Range Sensor
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
TOF CameraTOF Camera--basedbased
Although prototype was made, this approach was rejected because of several reasons:1) Light power in the near area does not satisfy US FDA eye-safe regulation.2) Required power for the modulating light source is too high.3) Too large size and cooling method.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Radar Sensor ArrayRadar Sensor Array--basedbased
• Seongkeun Park, Jae Pil Hwang, Euntai Kim, Heejin Lee, and Ho Gi Jung, “A neural network approach to target classification for active safety system using microwave radar,” Expert Systems with Applications, Vol. 37, Issue 3, 15 March 2010, pp. 2340-2346.
1) Vehicle corner localization is instable because radar is too sensitive to the vehicle’s pose and contains relatively large variation.
2) Filtering algorithm embedded into the radar prevents rapid detection of popping up pedestrian and it causes significant latency.
Although we developed radar signal-based object classification and tracking, this approach was rejected because it can not address popping up pedestrian scenarios.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Sensor Fusion of FIR Stereo and Visible VisionSensor Fusion of FIR Stereo and Visible Vision
Input stereo images Segmentation result Stereo matching and object detection
HOG-based feature extraction SVM-based classification
Although we recognized this approach was robust to day and night situations, it was rejected because of system price and domestic component sourcing.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Additional Results about Pedestrian ClassifierAdditional Results about Pedestrian Classifier
Sub-window Splitting• In order to extract local characteristics of pedestrian images
36x18 image 18x9 nine overlapped sub-images
• 24 Gabor filters (6 sections in orientation, 4 sections in scale)
• 3 statistical values of filtered result: mean, variance, skewness
• Feature vector dimension: 9x24x3=648
• Ho Gi Jung, Jaihie Kim, “Constructing a pedestrian recognition system with a public open database, without the necessity of re-training: an experimental study,“ Pattern Analysis and Applications, Vol. 13, No.2, May 2010, pp. 223-233.
Gabor Filter Bank (GFB)-based Feature
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Sub-window Splitting
Filtering Extraction with Gabor Filter Bank
Candidate Image
Optimization of SVM Learning Parameters
SVM Learning with Training Data 2
22
1 1 1
0
1maximize ( )2
subject to 0, 0
i jn n n
i i j i ji i j
n
i i ii
W y y e
C y
σα α α
α α
−−
= = =
=
= − ⋅
≥ ≥ =
∑ ∑∑
∑
x x
α
SVM Execution with Cross-Validation Data 2
22
1( )
( | ) ( | )
iNs
i ii
error
f y e
P P non ped P ped non
σα−
−
=
= ⋅
= +
∑s x
x
GA-based Minimization of Perror
With respect to C and σ
• Kernel parameter and regulation parameter are critical
for classification performance.
Parameter Optimization is needed!
• GA is robust to non-linear and discontinuous optimization.
• Gene is composed of two parameters.
• Fitness function is defined by cross-validation.
• When Joachims’ performance estimator is used instead of
cross-validation, all learning database can be used for the
training. Consequently, it leads to improvement of
recognition performance.
Radial Basis Function (RBF)-based SVM and
Genetic Algoritm (GA)-based Optimization
Additional Results about Pedestrian ClassifierAdditional Results about Pedestrian Classifier
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
• DCX pedestrian database
- 5 datasets 3 for training (1 for learning and 2 for cross validation), 2 for final test.
- One dataset: 4,800 pedestrian images (800 persons), 5,000 non-pedestrian images.
• Better performance and lower complexity
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
False Positive Rate
Det
ectio
n R
ate
Comparison of ROCs. The dotted line is ROC of LRF–quadratic SVM shown in Fig. 5(d) of [1] and the solid line is ROC of GFB–RBF SVM.
S. Munder and D. M. Gavrila, “An Experimental Study on Pedestrian Classification”, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 28, No. 11, pp. 1863-1868, Nov. 2006.
Additional Results about Pedestrian ClassifierAdditional Results about Pedestrian Classifier
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Additional Results about Pedestrian ClassifierAdditional Results about Pedestrian Classifier
1) GFB-RBF SVM is competent in pedestrian recognition.2) Performance estimator can replace cross-validation. In this case, dataset used for
cross-validation can be used for learning, and increases resultant performance.3) If sampling method and pre-processing are common, a pedestrian recognition system
constructed with a database can be used for actual application without re-training with the newly acquired database. In other words, publicly open database could be used for a general pedestrian recognition system.
4) A posteriori probability-based post-processing enhances recognition rate while losing a little in the false positive rate.
Four useful facts were found:
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Risk AssessmentRisk Assessment
Considering the possibility of collision avoidance using braking and steering, 3 risk levels were defined.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Risk Levels and Counter MeasuresRisk Levels and Counter Measures
Typical 5-step collision scenario and countermeasures were investigated.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Counter MeasuresCounter Measures
To reduce danger of false actuation, pedestrian protection measures are activated according to a hierarchical strategy.
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Lessons from PCDN StudyLessons from PCDN Study
(a) Without PCDN and AHS• Ho Gi Jung, Byung Moon Kwak, Jeong Soo Shim, Pal Joo Yoon, Jaihie Kim, “Pre-Crash Dipping Nose (PCDN) Needs Pedestrian
Recognition”, IEEE Transactions on Intelligent Transportation Systems, VOL. 9, NO. 4, Dec. 2008, pp. 678-687.
• Feasibility of pre-crash dipping nose (PCDN), developed for vehicle-vehicle collision, was investigated. PCDN showed severity increase in vehicle-pedestrian collision situation.
• Although PCDN was developed for vehicle-vehicle side crashes, there is a possibility that the range sensor for crash detection fail to distinguish a group of pedestrians from a side-faced vehicle. vehicle-vehicle collision countermeasure also needs pedestrian recognition.
(b) With PCDN and AHS
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Experimental ResultsExperimental Results
• For cooperative development, two test vehicles in each site.• SICK LMS 211, NIR sensitive camera.• Active braking by MANDO’s MGH-40 ESCplus via CAN.
Parma University, Italy
MANDO, Korea
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Experimental ResultsExperimental Results
Fig. 10. Some suddenly appearing pedestrians correctly detected (a) in an underground parking, (b) in the rain, (c) behind a misaligned vehicle, (d) behind a wall, and (e) at night, and (f) a suddenly appearing pedestrian detected as a non-dangerous pedestrian (false negative).
10 hours in complex urban scenarios
236km
Various situations were included.
24 true positives
1 false positive
(2×10-6 false positive/frame)
11 false negative
(1 missing, others are alert missing or delayed detection)
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Experimental ResultsExperimental Results
In Italy
In Korea
17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.
Thank You for Your Attention!Thank You for Your Attention!
EE--mail: mail: [email protected]@yonsei.ac.kr
Homepage: http://Homepage: http://web.yonsei.ac.kr/hgjungweb.yonsei.ac.kr/hgjung