building adas system from scratch
TRANSCRIPT
Building ADAS SYSTEM from scratchAlex Myakov, Chief CV Advocate
Yury Gorbachev, CV Integration & Optimization leadSeptember, 2016
Who we are ?Itseez was acquired by IoTG/Intel in July, 2016Itseez was founded in 2005:
• 3 ex- Intel co-Founders + 1 Principal Engineer• OpenCV development and support (2005-present)• OpenVX initiative leaders: v1.0 and v1.1 were
published in October, 2014 and in May, 2016SW Products: ADAS, Facense, AcceleratedCV (ACV)Skills: CV algorithms, HW specific optimization, 3+ years of deep learning (DL)Industries: automotive, security, robotics, wearables,etc
Building ADAS System from Scratch
The Dream and Ambition: • Create state-of-the-art software based front
camera ADAS algos• License such algos to Tier-1s and OEMs
Starting point (late 2013):• Strong knowledge of CV• Strong knowledge of
embedded/optimization• Good knowledge of cameras/sensors/optics• No ADAS/automotive specific knowledge
Building ADAS solution from scratch
Strategy:• Highly portable CV algos (pure ARM optimized
code)• Open SW platform (Android, Linux)• Simplest system possible• Cheapest COTS components
• Camera (optics, sensor, packaging)• HW platform
• Easy/fast installation in any car with no dependence on car parameters
• Automatic or simple calibration
ADAS Project Timeline
2013• TSR• Demo
platform: Nexus 4
2014• +LDW• +FCW• Demo
Platform: Asus Transformer Tablet + Android OS + Standalone Camera
2015• +PD• Demo
Platform: TK1+ Linux OS + Standalone Camera
• Demos with QNX:• CES 2015• TU Update
2015
2016• PoCs:• +Semantic Road
Segmentation• +Obstacle
Detection• +Driver
Monitoring• Demos with
QNX:• CES 2016
Demo Setup
Camera:• 1M• 1280H x 800V• HDR/WDR• 30 fps
Embedded Platform
USB 3.0 USB 2.0
CAN
Snap-and-go concept:• Simple and fast installation in any
car• No dependence on car parameters• Automatic calibration or simple
calibration
Lessons Learned
We expected ADAS to be just another CV application !• we ended up running into and solving lots of
issues
• SW development/testing paradigm• HW issues• Datasets
ADAS Solution SW ArchitectureTSR LDW FCW PD
Common Image Processing Pipeline + AutocalibrationOpenCV
IPP AcceleratedCV (ACV)x86 ARM
CV algo prototyping on desktopLab testing/CI on server
Execution on targetLive test/benchmarking
SW design approachADAS algorithms are purely software based:• Possible to design and test on desktops• Purely based on OpenCV• No special software skills are required (GPU,
DSP, etc)• Flexible and upgradable
Solved platform compatibility issues• No vendor provides cross-platform CV
framework• OpenCV is limited in supporting this
Created AcceleratedCV (ACV) library to address platform compatibility issue
Continuous integration
Any change in ADAS algorithms or processing pipeline requires complete re-evaluation• Detection/processing quality on the entire
dataset• Performance figures for each ADAS algorithm
Benefits from pure SW based approach• Quality evaluation on servers/cloud for entire
dataset• Performance benchmarking on multiple HW
targets• Reduces test time from days to hours!
DatasetsCV algorithms require datasets for design and testing:• No available commercial datasets• Research datasets cannot be used for products
Our own datasets for each ADAS algo were created: • Different conditions (rain, snow, sun)• Geographical locations• Dataset annotation and management tools • Many days of driving + many months of
annotation
The market offering of quality annotated datasets is still very limited !
Datasets stats
TSR: 2.5K good unique signs PD: 83K+ pedestrian bounding boxes FCW: 5K+ different cars and ca 1K trucksLDW: 0.5M+ boundaries
HW IssuesHW issues are caused by consumer “gradeness” of components Temperature issues:• Camera overheating -> skipped or corrupted
frames• HW platform overheating -> throttling
System issues:• throttling and unpredicted system behavior
under heavy processing loads
Mechanical issues:• USB 3.0 cable connectors get loose and break
Where is Deep Learning in our algos?Original ADAS algos were based on classical CV• Embedded platforms were too weak and not
able to provide required performance Gflops• Datasets were too small to yield quality DL
results• DL technology was not fully there
Conventional CV + small DL networks:• PD: validation of PD results based on classical
CV – increase DR and reduce FA rate• Driver monitoring: conventional face detector
+ DL based headpose estimation
Semantic Road Segmentation - PoC
Obstacle Detection using SfM - PoC
•We estimate 3D coordinates using points tracking and vehicle speed.
•Obstacles are calculated as clusters of points above the road plane.
Semantic segmentation + Obstacle Detection
Deep Learning - Silver Bullet ?
• Great new CV tool• Large datasets• A few orders of magnitude more
compute than classic CV• FuSa implications
