3D Object Recognition Using Computer Vision

VanGogh Imaging, Inc.

Kenneth LeeCEO/Founder

klee@vangoghimaging.com

Corporate Overview

Founded in 2007, located in McLean VA

Mission: “Provide easy to use, real-time 3D computer vision (CV) technology for embedded and mobile applications” – 2D to 3D for better visualization, higher reliability, and accuracy– Solve problems that require spatial measurements (e.g. parts inspection)

Target customer: Application and System Developers– Enhance existing product or develop new products

Product: ‘Starry Night’ 3D-CV Middleware (Unity Plugin)– Operating Systems: Android and Linux– 3D Sensor: Occipital Structure and Intel RealSense– Processors: ARM and Xilinx Zynq

Our focus– Object recognition– Feature detection– Analysis (e.g., measurements)

Potential Applications

3D Printing Parts Inspection Robotics

Entertainment

Automotive Safety

Security

Medical Imaging

Challenges for ImplementingReal-Time 3D Computer Vision

– Busy uncontrolled real-world environment– Limited processing power and memory– Noisy and uncalibrated low-cost scanners– Difficult to use libraries– Hard to find proficient computer vision engineers– Lack of standards– Large development investment

Starry Night Unity Plugin(patent pending)

Starry Night Video:https://www.youtube.com/watch?v=IZX-9PH7Erw&feature=youtu.be

The ‘Starry Night’ Template-Based3D Model Reconstruction

Reliable - The output is always a fully-formed 3D model with known

feature points despite noisy or partial scans

Easy to use – Fully automated process

Powerful – Known data structure for easy analysis and measurement

Fast – Real-time modelingInput Scan (Partial) + Reference Model = Full 3D Model

3D Object Recognition Algorithm

for mobile and embedded Devices

Challenges - Scene

Busy scene, object orientation, and occlusion

Challenges - Platform

Mobile and Embedded Devices– ARM – A9 or A15, <2G RAM – Existing libraries were built for laptop/desktop platform– GPU processing is not always available

Previous Approaches

(2D) Texture-Based Methods– Color-based → depends heavily on lighting or color of the object– Machine learning → robust, but requires training for each object– Neither method provides transform (i.e., orientation)

(3D) Methods– Hough transform and geometric hashing → slow– Geometric hashing → even slower– Tensor matching → not good for noisy and sparse scene– Correspondence-based methods using rigid geometric descriptors

– The models must have distinctive feature points which is not true for most models (i.e., cylinder)

Tried

General Concept for CV-BasedObject Recognition

Reference Object

Descriptor

Scene

Compare

Distance & Normal

Distance & Normal ofRandom Sample Points

Match CriteriaFine-Tune Orientation

LocationTranspose

Block Diagram

Model Descriptor (Pre-Processed)

Sample all point pairs in the model that are

separated by the same distance D

Use the surface normal of the pair to group them into the

hash tablet

key

(α1,β1,Ω1) P1, P2 P3, P4

(α2,β2,Ω2) P5, P6 P7, P8 P9, P10 P11, P12

(α3,β3,Ω3) P13, P14

Note: In the bear example, D = 5 cm which resulted in 1000 pairs

Note: The keys are angles derived from the normal of the points.alpha(α) = first normal to second pointbeta(β) = second normal to first pointomega(Ω) = angle of the plane between two points

Object Recognition Workflow

Grab Scene

Sample point pair w/ distance D using

RANSAC

Generate key using same hash function

Use key to retrieve similarly oriented

points in the model & rough transform

Match criteria to find the best match

Use ICP to refine transform

Note: The example scene has around 16K points

Note: We iterated this sampling process 100 times

Note: Entire process can be easily parallelized

Very Important: Multiple models can be found using a single hash table, for example, sampled point pair in the scene

Implementation

Result

Object Recognition Video:https://www.youtube.com/watch?v=h7whfei0fTw&feature=youtu.be

Object Recognition Examples

* CONFIDENTIAL * 18

Adaptive 3D Object Recognition Algorithm

Resize and Reshape

Object Recognitionfor Different Sizes & Shape

Objects in the real world are not always identical

Similarity Factor, S%, can be used to denote % of shape difference– This allows recognition of object that’s similar but does not have the

exact shape as the reference model

Size Factor, Z%, can be used to note the % size the object can recognize– This allows recognition of object that’s of different sizes from the

reference model

General Approach

Dynamically resizes the reference model

Dynamically reshapes the reference model– Uses our ‘Shape-based Registration’ technique

Hence, the reference model is ‘deformed’ to match the object in the scene

Results in very robust object recognition

The end reference model best represents the object in the scene both in size and shape

Block Diagram – Adaptive Object Recognition with feedback

Reference model is iteratively modified with every new frame until it converges into the same object in the scene

Note: Currently in the process of being implementedand will be available in Version 1.2 later this year

Object RecognitionPerformance Numbers

Reliability (w/ bear model)

Reliability– % false positives – depends on the scene

– Clean scene: <1%– Noisy scene: 5% (1 out of 20 frames)

– % negative results (cannot find the object)– Clean scene: <1%– Noisy scene: 10% (also takes longer)

Effect of orientation on success ratio– Model facing front: >99%– Model facing backwards: >99%– Model facing sideways (narrower): 85%

Performance - Mobile

Performance on Cortex A-15 2GHz ARM (on Android mobile)– Amount of time it takes to find one object

– Single thread: 2 seconds– Multi-thread & NEON: 0.3 second

– Amount of time it takes to find two objects– Single thread: 2.5 seconds– Multi-thread & NEON: 0.5 second

Note: Effective use of NEON led to significant performance gains of X2.5 for certain functions

Hardware Acceleration Using FPGA

• Xilinx Zynq SoC provides 20 to 1,000 parallel voxel processors depending on the size of the FPGA

Zynq

ARM

FPGA

Processor 1

Processor 1

Processor 1

Processor 1

Processor 20+

voxel

voxel

voxel

voxel

voxel

scan

Hardware Acceleration:FPGA (Xilinx Zynq)

Select Functions to Be Implemented in Zynq– FPGA: Matrix operations– Dual-core ARM: Data management + Floating point– Entire implementation done in C++ (Xilinx Vivado-HLS)

Performance:Embedded Using FPGA

Note: Currently, only 30% of the computationally intensive functions are implemented on the FPGA with the rest still running on ARM A9. Speed will be much improved once the remaining high-intensity functions are transferred to the FPGA.

Performance on Xilinx Zynq (Cortex A-9 800 MHZ + FPGA)– Amount of time it takes to find one object

– Zynq 7020: 0.7 second– Zynq 7045 (est.): 0.1 second

– No test results for two objects, but should scale the same way as for the ARM

Future

The chosen algorithm works well in most real-world conditions

The chosen algorithm is tolerant to size and shape differences respect to the reference model

The chosen algorithm can find multiple objects at the same time with minimal additional processing power

Additional improvements in performance are needed– Algorithm– Application-specific parameters (e.g., size of the model descriptor)– ARM - NEON– Optimize the use of FPGA core

Summary

Key implementation issues– Model descriptor– Data structure– Sampling technique– Platform

IMPORTANT– Both ARM & FPGA provide the scalability

Therefore– Real-time 3D object recognition was very difficult but

successfully implemented on both mobile and embedded platforms!

LIVE DEMO AT THE Xilinx BOOTH!

Resources

www.vangoghimaging.com

Android 3D printing: http://www.youtube.com/watch?v=7yCAVCGvvso

“Challenges and Techniques in Using CPUs and GPUs for Embedded Vision” by Ken Lee, VanGogh Imaging—http://www.embedded-vision.com/platinum-members/vangogh-imaging/embedded-vision-training/videos/pages/september-2012-embedded-vision-summit

“Using FPGAs to Accelerate Embedded Vision Applications”, Kamalina Srikant, National Instruments— http://www.embedded-vision.com/platinum-members/national-instruments/embedded-vision-training/videos/pages/september-2012-embedded-vision-summit

“Demonstration of Optical Flow algorithm on an FPGA”—http://www.embedded-vision.com/platinum-members/bdti/embedded-vision-training/videos/pages/demonstration-optical-flow-algorithm-fpg

* Reference: “An Efficient RANSAC for 3D Object Recognition in Noisy and Occluded Scenes” by Chavdar Papazov and Darius Burschka. Technische Universitat Munchen (TUM), Germany.