automated driving development with matlab and simulink · learn about synthesizing sensor...
TRANSCRIPT
![Page 1: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/1.jpg)
1CONFIDENTIAL |© 2018 The MathWorks, Inc.
What’s new in MATLAB® and Simulink® for
Automated Driving
Percepti
onControl
Planning
Control
Planning
Perception
Mark Corless
Automated Driving Segment Manager
Industry Marketing
2018-05-02
![Page 2: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/2.jpg)
4
How can you use MATLAB and Simulink to develop
automated driving algorithms?
Perception Control
Planning
![Page 3: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/3.jpg)
5
Perception Control
Planning
Examples of how you can use MATLAB and Simulink to develop
automated driving algorithms
Control
Planning
Perception
Deep learning
Path planning
Sensor models &
model predictive control
Sensor fusion
with live data
![Page 4: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/4.jpg)
6
How can you use MATLAB and Simulink to develop
perception algorithms?
Control
Planning
Path planning
Sensor models &
model predictive control
Perception
Deep learning
Sensor fusion
with live data
![Page 5: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/5.jpg)
7
Introduction to Automated
Driving System Toolbox
Automated Driving System Toolbox introduced:
Ground Truth Labeling App to label video data
10:45
![Page 6: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/6.jpg)
8
Automate labeling lanes with Ground Truth Labeler
![Page 7: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/7.jpg)
10
Specify sublabels and attributes in Ground Truth Labeler App
![Page 8: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/8.jpg)
11
Automate labeling pixels with Ground Truth Labeler
![Page 9: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/9.jpg)
12
Learn how to train a deep learning network using this example
▪ Train free space detection network using deep learning
Computer Vision
System ToolboxTM
![Page 10: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/10.jpg)
14
Load and overlay pixel labels
% Load pixel labels
classes = ["Sky"; "Building";...
"Pole"; "Road"; "Pavement"; "Tree";...
"SignSymbol"; "Fence"; "Car";...
"Pedestrian"; "Bicyclist"];
pxds = pixelLabelDatastore(...
labelDir,classes,labelIDs);
% Display labeled image
C = readimage(pxds, 1);
cmap = camvidColorMap;
B = labeloverlay(I,C,'ColorMap',cmap);
imshow(B)
pixelLabelDatastore
manages large collections
of pixel labels
![Page 11: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/11.jpg)
15
Visualize distribution of labeled pixels
% Visualize label count by class
tbl = countEachLabel(pxds)
frequency = tbl.PixelCount / ...
sum(tbl.PixelCount);
bar(1:numel(classes),frequency)
xticks(1:numel(classes))
xticklabels(tbl.Name)
xtickangle(45)
ylabel('Frequency')
Likely to
detect roads
Unlikely to
detect
bicyclist
Labeled pixels in this set are
imbalanced
![Page 12: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/12.jpg)
18
Add weighted layer to compensate for imbalanced data set
% Create weighted layer
pxLayer = pixelClassificationLayer(...
'Name','weightedLabels', 'ClassNames',tbl.Name,...
'ClassWeights',classWeights)
% Replace layer
lgraph = removeLayers(lgraph, 'pixelLabels');
lgraph = addLayers(lgraph, pxLayer);
lgraph = connectLayers(lgraph,...
'softmax', 'weightedLabels');
% Display network structure
plot(lgraph); ylim([0 9.5])
title('Replaced Layers Graph')
Replaced network
layer
![Page 13: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/13.jpg)
21
Train network and view progress
[net, info] = trainNetwork(datasource, lgraph, options);
![Page 14: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/14.jpg)
25
Assess similarity using intersection-over-union (IoU) metric
iou = jaccard(actual,...
expected);
table(classes,iou)
ans =
11×2 table
classes iou
____________ ________
"Sky" 0.92659
"Building" 0.7987
"Pole" 0.16978
"Road" 0.95177
"Pavement" 0.41877
"Tree" 0.43401
"SignSymbol" 0.32509
"Fence" 0.492
"Car" 0.068756
"Pedestrian" 0
"Bicyclist" 0
![Page 15: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/15.jpg)
27
Distribution of labels in data affects intersection-over-union (IoU)
Underrepresented classes such as Pedestrian and Bicyclist are
not segmented as well as classes such as Sky and Road
Distribution of labels in original data set Evaluation metrics of network
![Page 16: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/16.jpg)
29
Detection drivable space using semantic segmentation
![Page 17: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/17.jpg)
30
Learn more about developing deep learning perception algorithms
with these examples
▪ Train free space detection network using deep learning
Computer Vision
System ToolboxTM
▪ Add semantic segmentation
automation algorithm to
Ground Truth Labeler AppAutomated Driving
System ToolboxTM
▪ Generate CUDA® code to
execute directed acyclic graph
network on an NVIDIA GPU
GPU CoderTM
![Page 18: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/18.jpg)
32
How can you use MATLAB and Simulink to develop
perception algorithms?
Control
Planning
Path planning
Sensor models &
model predictive control
Perception
Sensor fusion
with live data
Deep learning
![Page 19: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/19.jpg)
33
Automated Driving System Toolbox introduced:
Multi-object tracker to develop sensor fusion algorithms
Detections
Multi-Object Tracker
TracksTracking
Filter
Track
Manager
• Assigns detections to tracks
• Creates new tracks
• Updates existing tracks
• Removes old tracks
• Predicts and updates state of track
• Supports linear, extended, and
unscented Kalman filtersIntroduction to Automated
Driving System Toolbox
19:27
![Page 20: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/20.jpg)
35
Radar
Camera
IMU
CA
NCAN Tx CAN FD
EthernetTCP/IP
How can I test my sensor fusion algorithm with live data?
![Page 21: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/21.jpg)
36
Radar
Camera
IMU
CA
NCAN Tx CAN FD
EthernetTCP/IP
Test forward collision warning algorithm with live data from
vehicle
FCW application
FCW algorithm
Visualization
Read
sensor data stream
and video stream
CAN
Rx
TCP/IP
Vision Object
Radar Object
Lane
Vehicle Speed
Yaw Rate
Video frame
![Page 22: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/22.jpg)
37
Test forward collision warning algorithm with live data from
“surrogate” vehicle
Transmitter
Vision Object
Radar Object
Lane
Vehicle Speed
Yaw Rate
Video frame
Recorded
messages
Recorded video
FCW application
FCW algorithm
Visualization
Read
sensor data stream
and video stream
Vision Object
Radar Object
Lane
Vehicle Speed
Yaw Rate
Video frame
TCP/IP
CAN
Rx
CAN FD
CAN Tx
EthernetTCP/IP
![Page 23: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/23.jpg)
38
Send live CAN FD and TCP/IP data
![Page 24: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/24.jpg)
39
Receive live CAN FD and TCP/IP data
![Page 25: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/25.jpg)
40
Generate C/C++ code for algorithm
![Page 26: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/26.jpg)
41
Stream live CAN FD and TCP/IP data into compiled algorithm code
![Page 27: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/27.jpg)
44
Learn more about developing sensor fusion algorithms
▪ Design algorithm with
multi-object tracker and
recorded vehicle data
Automated Driving
System ToolboxTM
▪ Generate C/C++
code from algorithm
which includes a
multi-object tracker
MATLAB CoderTM
▪ Stream CAN FD data to
prototype algorithm on
your laptop
Vehicle Network ToolboxTM
![Page 28: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/28.jpg)
45
How can you use MATLAB and Simulink to develop
control algorithms?
Planning
Path planning
Perception
Deep learning
Sensor fusion
with live data
Control
Sensor models &
model predictive control
![Page 29: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/29.jpg)
46
Automated Driving System Toolbox introduced:
Synthesizing scenarios to test sensor fusion algorithms
Introduction to Automated
Driving System Toolbox
26:10
![Page 30: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/30.jpg)
48
Adaptive Cruise Control with Sensor Fusion
Simulate closed loop system with radar/vision detections,
sensor fusion, and model-predictive control
![Page 31: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/31.jpg)
49
Synthesize detections to test sensor fusion and
model-predictive controller
![Page 32: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/32.jpg)
52
Synthesize lane detection with Vision Detection Generator
![Page 33: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/33.jpg)
53
Create highway double curve with drivingScenario
▪ Driver waypoints simulate distraction at curvature changes
![Page 34: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/34.jpg)
54
Simulate distracted driver
![Page 35: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/35.jpg)
55
Simulate lane keep assist at distraction events
![Page 36: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/36.jpg)
56
Compare distracted and assisted results
▪ Detect lane departure
and maintain lane
during distraction
![Page 37: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/37.jpg)
60
Simulate lane following by increasing minimum safe distance
![Page 38: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/38.jpg)
62
Graphically edit scenarios with Driving Scenario Designer
![Page 39: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/39.jpg)
63
Export MATLAB code to generate scenarios
![Page 40: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/40.jpg)
64
Explore what is required to follow high curvature paths
![Page 41: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/41.jpg)
66
Learn about synthesizing sensor detections to develop
control algorithms with these examples
▪ Edit roads, cuboid actors,
and sensors with
Driving Scenario Designer App drivingScenarioDesigner
▪ Simulate and
generate C++ for
model-predictive control
with lane detections
▪ Simulate and
generate C++ for
model-predictive control and
sensor fusion algorithms
![Page 42: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/42.jpg)
67
Learn about modeling vehicle dynamics to develop control
algorithms with these examples
▪ Simulate vehicle
dynamics for closed
loop design
Vehicle Dynamics BlocksetTM
▪ Co-simulate with Unreal
Engine and to set actor
positions get camera image
Vehicle Dynamics BlocksetTM
![Page 43: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/43.jpg)
68
How can you use MATLAB and Simulink to develop
planning algorithms?
Control
Sensor models &
model predictive control
Perception
Deep learning
Sensor fusion
with live data
Planning
Path planning
![Page 44: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/44.jpg)
69
Robotics System Toolbox introduced:
Connectivity with the ROS ecosystem
▪ Communicate via ROS
to integrate with
externally authored ROS
components
▪ Communication with
Gazebo to visualize and
simulated system
▪ Follow path for differential
drive robot with ROS
based simulator
![Page 45: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/45.jpg)
70
We are investing in design and simulation of path planning for
automobiles
Rapidly-exploring Random Tree (RRT*)
![Page 46: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/46.jpg)
71
Learn about developing path planning algorithms
with these examples
▪ Plot map tiles using
World Street Map (Esri)
Automated Driving
System ToolboxTM
▪ Plan path for automobile
given pre-defined map
Automated Driving
System ToolboxTM
▪ Simulate V2X
communication to assess
channel throughput
LTE System ToolboxTM
![Page 47: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/47.jpg)
72
Perception Control
Planning
Examples of how you can use MATLAB and Simulink to develop
automated driving algorithms
Control
Planning
Perception
Deep learning
Path planning
Sensor models &
model predictive control
Sensor fusion
with live data
![Page 48: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/48.jpg)
73
MathWorks can help you customize MATLAB and Simulink for
your automated driving application
▪ Web based ground
truth labeling▪ Consulting project with Caterpillar
▪ 2017 MathWorks Automotive
Conference
▪ Lidar ground truth
labeling▪ Joint presentation with Autoliv
▪ SAE Paper 2018-01-0043
▪ 2018 MathWorks Automotive
Conference
▪ Lidar sensor model for
Unreal Engine▪ Joint paper with Ford
▪ SAE Paper 2017-01-0107
![Page 49: Automated Driving Development with MATLAB and Simulink · Learn about synthesizing sensor detections to develop control algorithms with these examples Edit roads, cuboid actors, and](https://reader034.vdocuments.site/reader034/viewer/2022042314/5f022bb07e708231d402ec12/html5/thumbnails/49.jpg)
74
How can we help you can use MATLAB and Simulink to develop
automated driving algorithms?
Perception Control
Planning
Control
Planning
Perception