introduction to automated driving system toolbox: design ... · vision detector, radar detector,...
TRANSCRIPT
![Page 1: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/1.jpg)
1© 2017 The MathWorks, Inc.
Introduction to Automated Driving System Toolbox:
Design and Verify Perception Systems
Mark Corless
Industry Marketing
Automated Driving Segment Manager
![Page 2: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/2.jpg)
2
Some common questions from automated driving engineers
How can I
visualize vehicle
data?
How can I
detect objects in
images?
How can I
fuse multiple
detections?
vehicle
1011010101010100101001
0101010100100001010101
0010101001010100101010
0101010100101010101001
0101010010100010101010
0010100010100101010101
0100101010101010010101
0100101010010110000101
0100101010101010010010
1010011101010100101010
1
2F
![Page 3: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/3.jpg)
3
Some common questions from automated driving engineers
How can I
visualize vehicle
data?
How can I
detect objects in
images?
How can I
fuse multiple
detections?
vehicle
1011010101010100101001
0101010100100001010101
0010101001010100101010
0101010100101010101001
0101010010100010101010
0010100010100101010101
0100101010101010010101
0100101010010110000101
0100101010101010010010
1010011101010100101010
1
2F
![Page 4: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/4.jpg)
4
Examples of automated driving sensors
Vision-based
object detector
Lane detector
Radar-based
object detector
Lidar
Camera
Inertial
measurement
unit
![Page 5: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/5.jpg)
5
Camera (640 x 480 x 3)
239 239 237 238 241 241 241 242 243 243 243 243 243 243 243 243 243 243 243 243
252 252 251 252 252 253 253 253 255 255 255 255 255 255 255 255 255 255 255 255
254 254 253 253 254 253 255 255 255 255 255 255 255 255 255 255 255 255 255 255
250 251 251 251 251 251 253 251 253 253 255 255 253 255 255 255 255 255 255 255
251 252 251 253 253 253 251 251 253 253 253 253 254 253 253 253 253 253 253 253
252 253 253 254 254 253 253 253 253 253 253 253 253 253 254 253 253 253 253 253
252 253 253 254 254 254 253 253 253 253 253 253 253 253 253 253 251 251 251 251
253 253 253 254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253
253 254 253 254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253
254 254 254 254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253
254 254 254 254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253
254 254 254 254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253
255 255 255 255 255 255 255 255 254 254 254 254 254 254 254 254 254 254 254 254
255 255 255 255 255 255 255 255 254 254 254 254 254 254 254 254 254 254 254 254
253 254 255 255 255 255 254 254 253 253 253 253 254 254 254 254 254 254 254 254
253 255 255 255 255 255 254 254 253 253 253 253 254 254 254 254 254 254 254 254
254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253
254 254 254 254 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253 253
253 253 253 253 253 253 251 251 251 251 251 251 253 253 253 253 253 253 253 253
253 253 253 253 253 253 251 251 251 251 251 251 253 253 253 253 253 253 253 253
253 253 253 253 253 253 251 251 251 251 251 251 253 253 253 253 253 253 253 253
253 253 253 253 253 253 251 251 251 251 251 251 253 253 253 253 253 253 253 253
253 253 253 253 253 253 251 251 251 251 251 251 253 253 253 253 253 253 253 253
253 253 253 253 253 253 251 251 251 251 251 251 253 253 253 253 253 253 253 253
Examples of automated driving sensor data
Vision based
lane detector
Vision DetectorSensorID = 1;
Timestamp = 1461634696379742;
NumDetections = 6;
Detections(1)
TrackID: 0
Classification: 5
Position: [22.61 -0.43 2.24]
Velocity: [-9.86 0 0]
Size: [0 1.75 0]
Detections(2)
TrackID: 1
Classification: 5
Position: [22.8 3.12 2.24]
Velocity: [-9.37 0 0]
Size: [0 1.8 0]
Radar DetectorSensorID = 2;
Timestamp = 1461634696407521;
NumDetections = 23;
Detections(1)
TrackID: 0
TrackStatus: 6
Position: [56.07 17.73 0.34]
Velocity: [-8.50 2.86 0]
Amplitude: 3
Detections(2)
TrackID: 1
TrackStatus: 6
Position: [35.35 19.59 0.34]
Velocity: [-8.02 4.92 0]
Amplitude: 3
Detections(3)
TrackID: 12
TrackStatus: 5
Position: [57.69 3.13 0.34]
Lane DetectorLeft
IsValid: 1
Confidence: 3
BoundaryType: 3
Offset: 1.68
HeadingAngle: 0.002
Curvature: 0.0000228
Right
IsValid: 1
Confidence: 3
Lidar (47197 x 3)
-12.2911 1.4790 -0.5900
-14.8852 1.7755 -0.6475
-18.8020 2.2231 -0.7396
-25.7033 3.0119 -0.9246
-0.0632 0.0815 1.2501
-0.0978 0.0855 1.2561
-0.2814 0.1064 1.2575
-0.3375 0.1129 1.2650
-0.4611 0.1270 1.2572
-0.6184 0.1450 1.2475
-0.8369 0.1699 1.2319
-14.8815 1.8245 -0.6478
-18.8008 2.2849 -0.7403
-25.7134 3.0970 -0.9265
Inertial Measurement UnitTimestamp: 1461634696379742
Velocity: 9.2795
YawRate: 0.0040
![Page 6: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/6.jpg)
6
Visualize sensor data
![Page 7: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/7.jpg)
7
Visualize differences in sensor detections
![Page 8: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/8.jpg)
8
Explore logged vehicle data
▪ Load video data and corresponding mono-camera parameters
>> video = VideoReader('01_city_c2s_fcw_10s.mp4')
>> load('FCWDemoMonoCameraSensor.mat', 'sensor')
▪ Load detection sensor data and corresponding parameters
>> load('01_city_c2s_fcw_10s_sensor.mat', 'vision','lane','radar')
>> load('SensorConfigurationData.mat', 'sensorParams')
▪ Load lidar point cloud data
>> load('01_city_c2s_fcw_10s_Lidar.mat', 'LidarPointCloud')
![Page 9: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/9.jpg)
9
Visualize in image coordinates
%% Specify time to inspect
currentTime = 6.55;
video.CurrentTime = currentTime;
%% Extract video frame
frame = video.readFrame;
%% Plot image coordinates
ax1 = axes(...
'Position',[0.02 0 0.55 1]);
im = imshow(frame,...
'Parent',ax1);
Plot in image coordinates using
“classic” video and image functions likeimshow
![Page 10: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/10.jpg)
10
Visualize in vehicle coordinates
%% Plot in vehicle coordinates
ax2 = axes(...
'Position',[0.6 0.12 0.4 0.85]);
bep = birdsEyePlot(...
'Parent',ax2,...
'Xlimits',[0 45],...
'Ylimits',[-10 10]);
legend('off');
Plot in vehicle
coordinates withbirdsEyePlot
▪ ISO 8855 vehicle axis
coordinate system
– Positive x is forward
– Positive y is left
x
y
![Page 11: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/11.jpg)
11
Visualize expected coverage area (vehicle coordinates)
%% Create coverage area plotter
covPlot = coverageAreaPlotter(bep,...
'FaceColor','blue',...
'EdgeColor','blue');
%% Update coverage area plotter
plotCoverageArea(covPlot,...
[sensorParams(1).X ... % Position x
sensorParams(1).Y],... % Position y
sensorParams(1).Range,...
sensorParams(1).YawAngle,...
sensorParams(1).FoV(1)) % Field of view Plot sensor coverage area with coverageAreaPlotter
![Page 12: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/12.jpg)
12
%% Create detection plotter
detPlot = detectionPlotter(bep, ...
'MarkerEdgeColor','blue',...
'Marker','^');
%% Update detection plotter
n = round(currentTime/0.05);
numDets = vision(n).numObjects;
pos = zeros(numDets,3);
vel = zeros(numDets,3);
labels = repmat({''},numDets,1);
for k = 1:numDets
pos(k,:) = vision(n).object(k).position;
vel(k,:) = vision(n).object(k).velocity;
labels{k} = num2str(...
vision(n).object(k).classification);
end
plotDetection(detPlot,pos,vel,labels);
Visualize detections (vehicle coordinates)
detectionPlotter can be used to visualize
vision detector, radar detector, and
lidar point cloud
Plot vision detections withdetectionPlotter
![Page 13: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/13.jpg)
13
%% Bounding box positions in image coordinates
imBoxes = zeros(numDets,4);
for k = 1:numDets
if vision(n).object(k).classification == 5
vehPosLR = vision(n).object(k).position(1:2)';
imPosLR = vehicleToImage(sensor, vehPosLR);
boxHeight = 1.4 * 1333 / vehPosLR(1);
boxWidth = 1.8 * 1333 / vehPosLR(1);
imBoxes(k,:)=[imPosLR(1) - boxWidth/2, ...
imPosLR(2) - boxHeight, ...
boxWidth, boxHeight];
end
end
%% Draw bounding boxes on image frame
frame = insertObjectAnnotation(frame, ...
'Rectangle', imBoxes, labels,...
'Color','yellow','LineWidth',2);
im.CData = frame;
Visualize detections (image coordinates)
Transform vehicle to
image coordinates with vehicleToImage
Draw boxes withinsertObjectAnnotation
![Page 14: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/14.jpg)
14
%% Create lane detection plotter
lanePlot = laneBoundaryPlotter(bep, ...
'Color','black');
%% Update lane detection plotter
lb = parabolicLaneBoundary([...
lane(n).left.curvature,...
lane(n).left.headingAngle,...
lane(n).left.offset]);
rb = parabolicLaneBoundary([...
lane(n).right.curvature,...
lane(n).right.headingAngle,...
lane(n).right.offset]);
plotLaneBoundary(lanePlot, [lb rb])
Visualize lane boundaries (vehicle coordinates)
Plot lanes in vehicle
coordinates with laneBoundaryPlotter
![Page 15: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/15.jpg)
15
%% Draw in image coordinates
frame = insertLaneBoundary(frame, ...
[lb rb], sensor, (1:100),...
'LineWidth',5);
im.CData = frame;
Visualize lane boundaries (image coordinates)
Plot lanes in image
coordinates with insertLaneBoundary
![Page 16: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/16.jpg)
16
%% Create radar detection plotter
radarPlot = detectionPlotter(bep, ...
'MarkerEdgeColor','red',...
'Marker','o');
%% Update radar detection plotter
numDets = radar(n).numObjects;
pos = zeros(numDets,3);
vel = zeros(numDets,3);
for k = 1:numDets
pos(k,:) = radar(n).object(k).position;
vel(k,:) = radar(n).object(k).velocity;
end
plotDetection(radarPlot,pos,vel);
Visualize radar detections (vehicle coordinates)
Plot radar detections just
like vision detections withdetectionPlotter
![Page 17: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/17.jpg)
17
%% Create lidar detection plotter
lidarPlot = detectionPlotter(bep, ...
'Marker','.',...
'MarkerSize',1.5,...
'MarkerEdgeColor',[0 0.7 0]); % Green
%% Update lidar detection plotter
n = round(video.CurrentTime/0.1);
pos = ...
LidarPointCloud(n).ptCloud.Location(:,1:2);
plotDetection(lidarPlot,pos);
Visualize lidar point cloud (vehicle coordinates)
Plot lidar points just like
vision detections withdetectionPlotter
![Page 18: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/18.jpg)
18
Learn more about visualizing vehicle databy exploring examples in the Automated Driving System Toolbox
▪ Transform between
vehicle and image
coordinates
▪ Plot lidar point cloud▪ Plot object detectors
in vehicle coordinates– Vision & radar detector
– Lane detectors
– Detector coverage areas
![Page 19: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/19.jpg)
19
Some common questions from automated driving engineers
How can I
visualize vehicle
data?
How can I
detect objects in
images?
How can I
fuse multiple
detections?
vehicle
1011010101010100101001
0101010100100001010101
0010101001010100101010
0101010100101010101001
0101010010100010101010
0010100010100101010101
0100101010101010010101
0100101010010110000101
0100101010101010010010
1010011101010100101010
1
2F
![Page 20: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/20.jpg)
20
Classification
Left
Bottom
Width
Height
How can I detect objects in images?
Object
detector
Classification
Left
Bottom
Width
Height
![Page 21: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/21.jpg)
21
Train object detectors based on ground truth
ImagesObject
detector
Train
detector
Ground Truth
Classification
Left
Bottom
Width
Height
Classification
Left
Bottom
Width
Height
![Page 22: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/22.jpg)
22
Train object detectors based on ground truth
Object
detector
Train
detector
Design object detectors with the Computer Vision System Toolbox
Machine
Learning
Aggregate Channel Feature trainACFObjectDetector
Cascade trainCascadeObjectDetector
Deep
Learning
R-CNN(Regions with Convolutional Neural Networks)
trainRCNNObjectDetector
Fast R-CNN trainFastRCNNObjectDetector
Faster R-CNN trainFasterRCNNObjectDetector
Images
Ground Truth
![Page 23: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/23.jpg)
23
How can I create
ground truth?
Specify ground truth to train detector
Object
detector
Train
detector
Ground Truth
Images
![Page 24: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/24.jpg)
24
Specify ground truth to train detector
Ground Truth
Object
detector
Train
detectorGround Truth
Labeler App
Video
![Page 25: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/25.jpg)
25
Manually label ground truth objectswith Ground Truth Labeling App
![Page 26: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/26.jpg)
26
Automate labeling between manually labeled frameswith temporal interpolator
![Page 27: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/27.jpg)
27
Automate labeling based on a manually labeled framewith point tracker
![Page 28: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/28.jpg)
28
Automate initial ground truth of vehicleswith ACF ground truth detector
![Page 29: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/29.jpg)
29
Export labeled regions as MATLAB time table
![Page 30: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/30.jpg)
30
Ground truth labeling to evaluate detectors
VideoObject
detector Evaluate
detections
Ground truth labeling to train detectors
Ground truthGround Truth
Labeler App
Detections
Ground Truth
Object
detector
Train
detectorGround Truth
Labeler App
Video
![Page 31: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/31.jpg)
31
Customize Ground Truth Labeler App
![Page 32: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/32.jpg)
32
Customize Ground Truth Labeler App
Add custom image reader withgroundTruthDataSource
![Page 33: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/33.jpg)
33
Customize Ground Truth Labeler App
Add custom automation algorithmdriving.automation.AutomationAlgorithm
![Page 34: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/34.jpg)
34
Customize Ground Truth Labeler App
Add connection to other tools withdriving.connector.Connector
![Page 35: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/35.jpg)
35
Learn more about detecting objects in imagesby exploring examples in the Automated Driving System Toolbox
▪ Add automation algorithm
for lane tracking
▪ Extend connectivity of
Ground Truth Labeler App▪ Label detections with
Ground Truth Labeler App
![Page 36: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/36.jpg)
36
Learn more about detecting objects in imagesby exploring examples in the Automated Driving System Toolbox
▪ Explore pre-trained
pedestrian detector
▪ Explore lane detector
using coordinate
transforms for mono-
camera sensor model
▪ Train object detector
using deep learning and
machine learning
techniques
![Page 37: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/37.jpg)
37
Some common questions from automated driving engineers
How can I
detect objects in
images?
How can I
fuse multiple
detections?
vehicle
1
2F
How can I
visualize vehicle
data?
1011010101010100101001
0101010100100001010101
0010101001010100101010
0101010100101010101001
0101010010100010101010
0010100010100101010101
0100101010101010010101
0100101010010110000101
0100101010101010010010
1010011101010100101010
![Page 38: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/38.jpg)
38
Example of radar and vision detections of a vehicle
Vision detection
![Page 39: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/39.jpg)
39
Example of radar and vision detections of a vehicle
Radar detection
![Page 40: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/40.jpg)
40
Example of radar and vision detections of a vehicle
Can we fuse detections
to better track the
vehicle?
![Page 41: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/41.jpg)
41
Fuse detections with multi-object tracker
![Page 42: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/42.jpg)
42
Integrate tracker into higher level algorithm
![Page 43: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/43.jpg)
43
Generate C code for algorithm
![Page 44: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/44.jpg)
44
Synthesize scenario to test tracker
![Page 45: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/45.jpg)
45
Test tracker against synthesized data
![Page 46: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/46.jpg)
46
Track multiple object detections
Object
Detections
Multi-Object Tracker
TracksTracking
Filter
Track
Manager
Time
Measurement
Measurement Noise
Time
State
State Covariance
Track ID
Age
Is Confirmed
Is Coasted
• Assigns detections to tracks
• Creates new tracks
• Updates existing tracks
• Removes old tracks
• Predicts and updates state of track
• Supports linear, extended, and
unscented Kalman filters
![Page 47: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/47.jpg)
47
Examples of Kalman Filter (KF) initialization functions
Object
Detections
Multi-Object Tracker
TracksTracking
Filter
Track
Manager
Linear KF(trackingKF)
Extended KF(trackingEKF)
Unscented KF(trackingUKF)
Constant velocity initcvkf initcvekf initcvukf
Constant acceleration initcakf initcaekf initcaukf
Constant turn Not applicable initctekf initctukf
![Page 48: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/48.jpg)
48
Example of configuring multi-object tracker
Object
Detections
Multi-Object Tracker
TracksTracking
Filter
Track
Manager
tracker = multiObjectTracker(...
'FilterInitializationFcn', @initcaekf, ... % Handle to tracking Kalman filter
'AssignmentThreshold', 35,... % Normalized distance from track for assignment
'ConfirmationParameters',[2 ... % Min number of assignments for confirmation
3],... % Min number of updates for confirmation
'NumCoastingUpdates', 5); % Threshold for track deletion
![Page 49: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/49.jpg)
49
Fuse and track multiple detections from different sensors
Radar
Detections
Multi-Object Tracker
TracksKalman
Filter
Track
Manager
Time
Position
Velocity
Vision
Detections
Ob
ject P
acker
Time
Position
VelocityObject Detections
Time
Measurement
Measurement Noise
Time
State
State Covariance
Track ID
Age
Is Confirmed
Is Coasted• Typically unique to application and sensors
• Map sensor readings into measurement matrix
• Specify measurement noise for each sensor
![Page 50: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/50.jpg)
50
Example of packing sensor data into object detection format
Radar
Detections
Time
Position
Velocity
Vision
Detections
Ob
ject P
acker
Time
Position
Velocity
% Example sensor data
radar.Time = 0.049; % (sec)
radar.Position = [10 0.5]; % [x y] (m)
radar.Velocity = [4.4 0.1]; % [x y] (m/sec)
vision.Time = 0.051; % (sec)
vision.Position = [11 0.1]; % [x y] (m)
vision.Velocity = 4.1; % [x] (m/sec)
% Pack to constant acceleration measurement format:
% [positionX; velocityX; positionY; velocityY]
packedDetections(1) = objectDetection(radar.Time,...
[radar.Position(1); radar.Velocity(1);...
radar.Position(2); radar.Velocity(2)],...
'MeasurementNoise', diag([1,1,2,10]));
packedDetections(2) = objectDetection(vision.Time,...
[vision.Position(1); vision.Velocity(1);...
vision.Position(2); 0],...
'MeasurementNoise', diag([1,1,2,10]));
![Page 51: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/51.jpg)
51
Explore demo to learn more about fusing detections
Radar
Detections
Multi-Object Tracker
TracksKalman
Filter
Track
Manager
Vision
Detections
Ob
ject P
acker
Forward Collision Warning
Using Sensor Fusion
product demo illustrates
• Packing sensor data into object
detections
• Initializing Kalman filter
• Configuring multi-object tracker
![Page 52: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/52.jpg)
52
Generate C code withcodegen
Generate C code for algorithmwith MATLAB Coder
![Page 53: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/53.jpg)
53
Generate C-code for algorithmwith MATLAB Coder
%% Create variables that will be used to specify example input arguments
[visionObjects, radarObjects, imu, lanes] = ...
helperReadSensorRecordingsFile('01_city_c2s_fcw_10s_sensor.mat');
egoLane = struct('left', [0 0 0], 'right', [0 0 0]);
time = 0;
positionSelector = [1 0 0 0 0 0; 0 0 0 1 0 0];
velocitySelector = [0 1 0 0 0 0; 0 0 0 0 1 0];
exampleInputs = {visionObjects(1), radarObjects(1), imu(1),...
lanes(1), egoLane, time,...
positionSelector, velocitySelector};
%% Generate code for sensor fusion algorithm: trackingForFCW_kernel
codegen trackingForFCW_kernel -args exampleInputs -config:dll -launchreport
![Page 54: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/54.jpg)
54
Install patch to generate C code from multiObjectTracker
▪ https://www.mathworks.com/support/bugreports/1546972
![Page 55: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/55.jpg)
55
%% Create a new scenario
s = drivingScenario('SampleTime', 0.05);
%% Create road
road(s, [ 0 0; ... % Centers [x,y] (m)
45 0],...
5); % Width (m)
road(s, [35 20; ...
35 -10],...
5);
%% Plot scenario
p1 = uipanel('Position',[0.5 0 0.5 1]);
a1 = axes('Parent',p1);
plot(s,'Parent',a1,...
'Centerline','on','Waypoints','on')
a1.XLim = [0 45];
a1.YLim = [-6 20];
Specify driving scenario and roads
Specify road centers and
width as part of adrivingScenario
![Page 56: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/56.jpg)
56
Add ego vehicle
%% Add ego vehicle
egoCar = vehicle(s);
waypoints = [ 2 -1.25;... % [x y] (m)
28 -1.25;...
30 -1.25;...
36.25 4;...
36.25 6;...
36.25 14];
speed = 13.89; % (m/s) = 50 km/hr
path(egoCar, waypoints, speed);
Specify ego vehicle
path using waypoints and speeds
![Page 57: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/57.jpg)
57
Add ego vehicle
%% Add ego vehicle
egoCar = vehicle(s);
waypoints = [ 2 -1.25;... % [x y] (m)
28 -1.25;...
30 -1.25;...
36.25 4;...
36.25 6;...
36.25 14];
speed = 13.89; % (m/s) = 50 km/hr
path(egoCar, waypoints, speed);
%% Play scenario
while advance(s)
pause(s.SampleTime);
end
Specify ego vehicle
path using waypoints and speeds
![Page 58: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/58.jpg)
58
Add target vehicle and pedestrian actor
%% Add Target vehicle
targetVehicle = vehicle(s);
path(targetVehicle,...
[44 1; -4 1],... % Waypoints (m)
[5 ; 14]); % Speeds (m/s)
%% Add child pedestrian actor
child = actor(s,'Length',0.24,...
'Width',0.45,...
'Height',1.7,...
'Position',[40 -5 0],...
'Yaw',180);
path(child,...
[30 15; 40 15],... % Waypoints (m)
1.39); % Speed (m/s) = 5 km/hr
Specify target vehicle with
varying speed
Specify pedestrian actor size and path
![Page 59: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/59.jpg)
59
View scenario from behind ego vehicle
%% Add chase view (left)
p2 = uipanel('Position',[0 0 0.5 1]);
a2 = axes('Parent',p2);
chasePlot(egoCar,...
'Parent',a2,...
'Centerline','on',...
'ViewHeight',3.5,... % (m)
'ViewLocation',[-8 0]); % [x y] (m)
Add view which
follows vehicle usingchasePlot
![Page 60: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/60.jpg)
60
View scenario from behind ego vehicle
%% Add chase view (left)
p2 = uipanel('Position',[0 0 0.5 1]);
a2 = axes('Parent',p2);
chasePlot(egoCar,...
'Parent',a2,...
'Centerline','on',...
'ViewHeight',3.5,... % (m)
'ViewLocation',[-8 0]); % [x y] (m)
%% Play scenario
restart(s)
while advance(s)
pause(s.SampleTime);
end
Add view which
follows vehicle usingchasePlot
![Page 61: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/61.jpg)
61
Simulate effects of vision detection sensor
Range
measurement
accuracy
degrades with
distance to
object
Angle
measurement
accuracy
consistent
throughout
coverage area
Partially or
completely
occluded
objects are not
detected
Objects in coverage area may not be detected because
they appear above the horizon line
Large range measurement errors may be introduced
for detected objects
Range Effects Occlusion Effects Road Elevation Effects
![Page 62: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/62.jpg)
62
Model vision detection sensor
%% Create vision detection generator
sensor = visionDetectionGenerator(...
'SensorLocation', [0.75*egoCar.Wheelbase 0], ...
'Height', 1.1, ...
'Pitch', 1, ...
'Intrinsics', cameraIntrinsics(...
800,... % Focal length
[320 240],... % Principal point
[480 640]), ... % Image size
'RadialDistortion',[0 0], ...
'TangientialDistortion',[0 0]), ...
'UpdateInterval', s.SampleTime, ...
'BoundingBoxAccuracy', 5, ...
'MaxRange', 150, ...
'ActorProfiles', actorProfiles(s));
Model radar detection
sensor usingradarDetectionGenerator
Coverage area is determined based on cameraIntrinsics
Extrinsic mounting parameters
![Page 63: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/63.jpg)
63
Create birds eye plot to view sensor detections
%% Add sensor birds eye plot (top left)
p3 = uipanel('Position',[0 0.5 0.5 0.5]);
a3 = axes('Parent',p3);
bep = birdsEyePlot('Parent',a3,...
'Xlimits', [0 20],...
'Ylimits', [-10 10]);
legend(a3,'off');
% Create plotters
covPlot = coverageAreaPlotter(bep,...
'FaceColor','blue',...
'EdgeColor','blue');
plotCoverageArea(covPlot,...
sensor.SensorLocation,sensor.MaxRange,...
sensor.Yaw,sensor.FieldOfView(1))
detPlot = detectionPlotter(bep,...
'MarkerEdgeColor','blue',...
'Marker','^');
truthPlot = outlinePlotter(bep);
Plot synthesized
detections in birdsEyePlot
![Page 64: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/64.jpg)
64
Play scenario with sensor models
restart(s)
while advance(s)
% Get detections in ego vehicle coordinates
det = sensor(targetPoses(egoCar),...
s.SimulationTime);
% Update plotters
if isempty(det)
clearData(detPlot)
else % Unpack measurements to position/velocity
pos = cellfun(@(d)d.Measurement(1:2),...
det,'UniformOutput',false);
vel = cellfun(@(d)d.Measurement(4:5),...
det,'UniformOutput',false);
plotDetection(detPlot,...
cell2mat(pos')', cell2mat(vel')');
end
[p, y, l, w, oo, c] = targetOutlines(egoCar);
plotOutline(truthPlot,p,y,l,w,...
'OriginOffset', oo, 'Color', c);
end
![Page 65: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/65.jpg)
65
Learn more about sensor fusionby exploring examples in the Automated Driving System Toolbox
▪ Design
multi-object tracker
based on logged
vehicle data
▪ Synthesize
driving scenario
to test
multi-object tracker
▪ Generate C/C++
code from algorithm
which includes a
multi-object tracker
![Page 66: Introduction to Automated Driving System Toolbox: Design ... · vision detector, radar detector, and lidar point cloud Plot vision detections with detectionPlotter. 13 %% Bounding](https://reader034.vdocuments.site/reader034/viewer/2022042601/5f775878625dce36b026c62d/html5/thumbnails/66.jpg)
66
The Automated Driving System Toolbox helps you…
Visualize vehicle
data
Detect objects in
images
Fuse multiple
detections
▪ Plot sensor detections
▪ Plot coverage areas
▪ Transform between image
and vehicle coordinates
▪ Train deep learning
networks
▪ Label ground truth
▪ Connect to other tools
▪ Design multi-object tracker
▪ Generate C/C++
▪ Synthesize driving scenarios
vehicle
1
2F
1011010101010100101001
0101010100100001010101
0010101001010100101010
0101010100101010101001
0101010010100010101010
0010100010100101010101
0100101010101010010101
0100101010010110000101
0100101010101010010010
1010011101010100101010