towards embedded computer vision邁向嵌入式電腦視覺

Wang, Yuan-Kai(王元凱)Electronic Engineering Department, Fu Jen Univ. (輔仁大學電機工程系)

Email: [email protected]

URL: http://www.ykwang.tw

2010/05/14

Towards Embedded

Computer Vision

本著作採用創用CC 「姓名標示」授權條款台灣3.0版

mailto:[email protected]�

http://www.ykwang.tw/�

http://creativecommons.org/licenses/by/3.0/tw/�

王元凱 Towards Embedded Computer Vision p.

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

2

Contents1. Embedded Systems2. Embedded Computer Vision3. Entertainment Robot (CPU)4. Embedded Vision Sensor (CPU)5. Portable Vision Device (DSP)6. Smart Video Surveillance (FPGA)7. Summary & Outlook



3

1. Embedded Systems



4

Evolution of ComputerPast Now Future

• Embedded System is a computer that is• Special-purpose• Light, Thin, Short, Small⇒ Limited resources



55

Embedded Systems "Without" Sensors

資料來源：資策會MIC ITIS計畫整理



66

Embedded Systems "With" Sensors

GPSNavigation

Wii Roomba

Exoskeleton



77

Embedded Systems "With" Image Sensors

IntelligentRobot

AugmentedReality

DARPAGrand

Challenge

SurfaceComputing

http://images.google.com.tw/imgres?imgurl=http://www.avland.co.uk/sony/ers7/ers7lrg.jpg&imgrefurl=http://www.avland.co.uk/sony/ers7/ers-7.htm&h=987&w=906&sz=73&hl=zh-TW&start=3&sig2=DGvNILdLM9J2NvxSgzOptg&tbnid=55XewzOMLKglcM:&tbnh=149&tbnw=137&ei=3iX1RvG3DaX4gQPaqdnYDA&prev=/images%3Fq%3Dsony%2Baibo%2Bers7%26gbv%3D2%26svnum%3D10%26hl%3Dzh-TW�



8

2. Embedded Computer Vision (ECV)



9

What Is ECV Embedded compute vision Implements computer vision algorithms

on low-cost, low-power, constrained hardware

Constrained hardware Low-speed CPU Low capacity memory No floating-point (FPU) Low-resolution image sensor



10

Embedded Computer Vision Embedded System + Camera

+ Computer Vision Algorithm

ImageProcessing

ImageRecognition

ImageCapturing



11

Why Smart Camera (1/2) Front-end processing An example for video surveillance

Classical stationary camera

Smart camera

IOImage Inc.



12

Why Smart Camera (2/2) In-node processing:

Vision sensor network Distributed vision system Camera networks Use multiple cameras

to analyze the scene Benefit Less problems with

occlusion Challenge Distributed processing

and reasoning



13

International Activities (1/2) Special conferences IEEE Int. Conf. Distributed Smart Cameras

Special journal issues IEEE Journal of Selected Topics in Signal

Processing, vol. 2, no. 4, Aug. 2008 EURASIP Journal on Embedded Systems,

Short courses in important CV conferences CVPR07&08: Distributed vision processing in

smart camera networks ESC07: Embedded CV and smart cameras ICASSP09: Distributed processing in smart

cameras



14

International Activities (2/2) Research projects and Lab. Princeton Univ./Georgia Tech.:

Embedded Systems Lab., Wayne Wolf Stanford Univ.

Wireless sensor networks Lab. UCLA, CMU, MIT Delft Univ. of Technology

SmartCam Project Graz Univ. of Technology



15

Three Ways for ECV CPU (Central Processing Unit) ARM, PowerPC

DSP (Digital Signal Processor) TI, ADI, NXP

FPGA (Field Programmable Gate Array) Altera, Xilinx

http://images.google.com/imgres?imgurl=http://dehalastech.com/images/bigolxilinx.jpg&imgrefurl=http://dehalastech.com/consulting.html&h=1728&w=2592&sz=567&hl=en&start=31&sig2=KXuegL1CvTndSqfzE8Mp0g&um=1&tbnid=z5r68j88_4v03M:&tbnh=100&tbnw=150&ei=UW4dSKqxE6WiswKx8MHcCw&prev=/images%3Fq%3Dxilinx%2Bfpga%26start%3D20%26ndsp%3D20%26um%3D1%26hl%3Den%26sa%3DN�



16

Embedded CPU Embedded CPU = Low-power CPU ARM

Drawbacks of embedded CPU for computer vision No FPU, usually fixed-point Speed: 60MHz ~ 600MHz

Therefore it is usually developed for (video) sensor networks



17

Embedded CPU: MeshEye Stanford MeshEye ARM 7 (55MHz), ZigBee node 3 image sensors 30x30 grayscale x 2 640x480 color x 1

Object detection

(http://wsnl.stanford.edu/smartcam.html)

"MeshEye:a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance", IPSN-SPOTS, 2007

http://wsnl.stanford.edu/smartcam.html�



18

Embedded CPU: CMUCam CMU CMUcam3 ARM7

ECV applications Robotic vision, color tracking,

histogram processing, face detection

(http://www.cmucam.org)

http://www.cmucam.org/�



19

DSP DSP is good for signal processing SIMD structure for filtering processing

However, Computer Vision needs extreme DSP + video port Media processors: powerful DSP Parallelism: VLIW Faster memory, DMA Wide data bus

ECV applications Face detection, face recognition, license

plate recognition, vehicle tacking,



20

DSP : TRICam "Visual surveillance on DSP-based

embedded platform," Graz Univ. of Technology, 2008(Phd. dissertation) TI C6414 (600MHz) Applications: Adaboost face detection, vehicle

detection, license plate detection



21

FPGA (1/3) Customizable hardware for parallelism Reconfigurable computing Flexible hardware design by HDL codes on

FPGA.



22

FPGA (2/3) "Hardware, Design and Implementation

Issues on a FPGA-Based Smart Camera," ICDSC, 2007



23

FPGA (3/3) ECV application:

object tracking Template matching

by MAE(Maximum Absolute Error)



24

Hybrid: MCU+CPLD UCLA Cyclops MCU: Atmega128 CPLD: image processing ECV app.: Hand gesture recognition

http://research.cens.ucla.edu/projects/2007/Multiscaled_Actuated_Sensing/Cyclops/

http://research.cens.ucla.edu/projects/2007/Multiscaled_Actuated_Sensing/Cyclops/�





25

Hybrid: CPU+DSP (1/2) "Distributed Embedded Smart Cameras

for Surveillance Applications," IEEE Computer, 2006.

Developed for traffic surveillance



26

Hybrid: CPU+DSP (2/2)



27

SOC - Xetal Philips: Xetal SIMD processor InCa WiCa Xetal3 + 8051 Stereo sensors (640x480) 50 GOPS @ 600mWatt ZigBee node C++ programming

Used by Stanford Univ.,Delft University of Technology

ECV Applications: edge detection, face detection, hough transform, gesture recognition, depth estimation, ...



28

Challenges for ECV Algorithm refinement Parallel computation Function partition, Multi-threading

Stream processing Memory hierarchy optimization

Hardware design Pipeline, SIMD, board design

Optimized programming skills Fixed-point arithmetic Memory management Intrinsic commands



29

Our Current Results Entertainment robot (CPU) Sony AIBO robot with 64-bit RISC CPU Application: Game playing,

Face detection/recognition, Facial expression recognition

Smart vision sensor: (CPU) Self-made sensor with ARM7 Application : Face detection, Robot

Portable vision device (DSP) Self-made device with Dualcore DSP Application: gesture recognition

Smart video surveillance (FPGA) Background subtraction with FPGA Application: video surveillance



30

3. Entertainment Robot

CPU Solution



31

Two AIBO Models Hardware CPU: 64-bit RISC

576 MHz/192Mhz RAM: 64MB/32MB Flash: 32MB 20 motors

Camera: CMOS sensor Resolution:

280 × 160 10 fps

ERS-7

ERS220A



32

Sensors



33

Software Operating System Aperios (Embedded Linux)

Development tools C++ GCC 3.4.4 on Linux Libraries: OpenR, Tekkotsu



34

Our AIBO Pet Rolling Dice Face Detection Face Recognition Facial Expression Recognition



35

Rolling Dice (1/3)



36

Rolling Dice (2/3) Detect the dice by color detection Using Gaussian mixture model (GMM)

and EM algorithm to model colors

Originalimage

Detectionresult

GMM Morphology+CCL

11( ( ) ( ))2

1/21

( | )

12

Ti i i

N x m x m

ii i

p x

eωπ

−− − ∑ −

=

=

∑∑

c



37

Rolling Dice (3/3)



38

Face Detection (1/2)



39

Face Detection (2/2) Algorithm:

Adaboost face detection Proposed by Viola and Jones in 2001 Cascaded weak classifiers



40

Face Recognition (1/3)



41

Face Recognition (2/3) Eigenface approach (PCA)



42

Face Recognition (3/3)



43

Facial Expression Recognition 1/3



44




45


3 expressions Happy, Surprise, Angry

Video-based method Feature: optical flow,

common vector flow Classifier: Hidden Markov model

Well done in small-resolution images 140 * 120 (~ QCIF, 176 * 144)



46

4. Embedded Vision Sensor

CPU Solution



47

FJUCam It is a self-made camera Camera module + embedded system 3S: Small, Smart, Sensing

Components of the FJUCam ARM7 TDMI 32-bit, 60MHz, 64KB RAM, 128KB

ROM CMOS sensor: OV6620 (CIF 50fps) CIF(352x288), QCIF, 8-bit RGB



48

What Is FJUCam (1/3)

• Power consumption: 1W

• Power sources:• 5V DC current • 5V Cell Battery

• Weight: 35gm•Size:6 x 4.5 x 5 (cm)(W x H x D)

Three Modules1. Main board, 2. Lens module

3. Storage module



49


49

Switch

PowerButton/ISP

Serial port

ARM7Microcontroller

Frame buffer/FIFO

Back Side

Front Side

Main Board

Other interfaces: RS232x2, SPI, I2C, GPIO



50


Lens Module Storage Module



51

Software Development environment C Language PC Windows + Cygwin + GCC cc3 library (open source developed by

CMU)



52

Face Detection The Adaboost algorithm for face detection Proposed by Viola and Jones in 2001 Cascaded weak classifiers(21 cascades)

Algorithm refinement Reduced to 5 cascades Fixed-point arithmetic Stream processing for only 64KB memory

utilization

FJUCam Face Detection

Image Display



53

Cyclops Robot Color tracking



54

5. Portable Vision Device:X-EYE

DSP Solution



55

Goal of X-Eye

55



56

Components

滑鼠

鍵盤

SD卡 USB-WIFI卡

讀卡機USB-RS232控制線

微投影機

自製外殼

USB連接線

顯示器

觸控面板

筆電

移動電源

BeagleBoard

CameraUSB

Hub



57

1st Generation Prototype



58

Photo Mode

Switch Command: Mode switch

Capture Command: capture images



59

Manage Mode

59

Original Photos

Next Command

Previous Command

Switch Command 2(to photo)



60

Hardware (1/2)

6060

Peripheral I/O•USB Host

•JTAG

•DVI-D video out

•S-Video out

•SD/MMC+

•Stereo in/out

•RS-232 serial1

•Alternate power

•USB 2.0 HS OTG

USER

OMAP3530 Processor•600MHz Cortex-A8

•NEON+VFPv3•16KB/16KB L1•256KB L2•430MHz C64x+ DSP•32K/32K L1•48K L1D•32K L2

•Power VR SGX GPU•64K on-chip RAMPOP Memory•256MB LPDDR RAM•256MB NAND flash

RESET

7.6 cm



61

Hardware (2/2)

2010.04.2561



62

Software (1/2)



63

Software (2/2)

2010.04.25 X-Eye

軟體名稱版本功用簡述

Gesture Command Module 1.0 手勢辨識

OpenCV 1.0 影像處理

FFMpeg 0.5.1 視(音)訊邊解碼

QT 4.6.2 視窗介面

VMWare 6.5.3 WinXP下安裝Linux工具

Ubuntu (host) 9.04 安裝交叉編譯環境

Ubuntu (client) 9.10 BB上的filesystem

Kernel (client) 2.6.29 BB上的kernel63



64

Algorithm (1/2) Gaussian Mixture Model (GMM) Model four colors

Expectation Maximization (EM) Parameter estimation of GMM

64

11( ( ) ( ))2

1/21

1( | )2 || ||

Ti i i

N x m Cov x m

ii i

p x c eCov

ωπ

−− − −

=

=∑

1

( | , )( )

( | , )

t t ti j i i

ij it t tp j p p

p

p x m CE z

p x m C

ω

ω=

=

∑

1

1

1 ( )N

ti ij

jE z

Nω +

=

= ∑ 11

1

1 ( )N

ti ij jt

ji

m E z xNω

++

=

= ∑1 1 1

11

1 ( )[( )( ) ]N

t t t Ti ij j i j it

ji

C E z x m x mNω

+ + ++

=

= − −∑

E Step M Step



65

Algorithm (2/2) Color Identification

Performance optimization by Look Up Table (LUT) for real-time

Gesture Recognition Four gestures: capture, switch, next,

previous,

65

ˆ arg max ( | ), 1 ~j

jcc P x c j k= =



66

6. Smart Video Surveillance

FPGA Solution



67

Our Experience

FPGA

Background subtraction, ...• 2.8 GHz Intel CPU• Software: C/C++• Frame rate: 10 fps



68

Background SubtractionCurrent Frame

-

BackgroundImage Update

Post Processing

Motion Object Image

Background Image

1+kB

kB

1+kP

1+kM),(),(

),(

1

1

yxByxPyxM

kk

k

−= +

+

),()1(),(),(

1

1

yxPyxByxB

kk

k

+

+

−+= αα

Speed up by (1) Circuit design, (2) Paralization



69

Background Subtractionby FPGA (1/3)

Parallelism: 7-level pipeline SIMD with stream processing



70


Hardware: Altera Cyclone II 2C35 Design: Verilog HDL with Quartus II Tools

Background New Frame ResultFrame rate

• Background module : 368 fps• Whole system : 51 fps



71


Comparisons PC: 2.8GHz CPU, C implementation FPGA can speed up 500 times

Clock(Hz) FPS

CPU

FPGA

2.8G

25M

51

10



72

7. Summary and Future Research Directions



73

Summary (1/2) Embedded CPU is not appropriate for

computer vision Although CPU has great flexibility and

programming environment Its architecture is interrupt-based Designed for I/O-process usage Not for data-intensive computing, such as

DSP and image/video processing



74

Summary (2/2) High-performance processor is

necessary for computer vision Clock rate is not the crucial point But SIMD and algorithm parallelism

do



75

Questions Embedded compute vision Low-cost, low-power,

minimal hardware High-Performance

computer vision Fast speed without cost,

power, and hardware constraints

From contradiction to convergence ?!

ConstrainedResource

AbundantResource

Yes by multicore

Contradiction



76

Challenges (1/2) Algorithm decomposition Function decomposition Partition serial part and parallel part Data flow analysis and

data dependency analysis Parallelism Loop unrolling Multithreading Pipeline



77

Challenges (2/2) Performance analysis method

For efficiency improvement Implementation efforts Choose a good embedded platform for

computer vision Software issues Hardware issues

Programming skills Multi-threading



78

Future Research Directions Multicore framework DSP + CPU : FJUCam2 Advantage: Using C language Challenge: algorithm parallelism

FPGA + CPU: Advantage

Reconfigurable multicore Less Verilog, more C

Challenge: hardware/software co-design GPGPU Advantage:

240~512 cores Using C language

Challenge: algorithm parallelism



79

FJUCam2 (1/2) Next generation FJUCam (MPSoC) Adopt multicore technology DSP(Media) + CPU

Processor

RAMImageResolution

FJUCam1

FJUCam2

64KB

128MB

CIF

VGA

60MHz 600MHz

98/2

99/6



80

FJUCam2 (2/2) Algorithms going to be developed

for FJUCam2 Color tracking Gesture recognition Face tracking and recognition Event detection Video summarization Sleep monitoring Distributed vision processing



81

Reconfigurable Multi-Core FPGA + CPU + Linux

CMOS CCD Capture

RAW to

RGB

RGB to YBackground Subtraction

Morphology

SDRAM0

VGA

Nios II Processer

Avalon Bus

CMOS Controller

SDRM1 Controller

SSRAM Controller

DM9000A Controller

Flash Controller

SDRAM1Flash SSRAM DM9000A

PC

Internet

DE2_70

FPGA Chip

Hardware Software



82

The End

Free for Any Questions



83

Our Development Boards

Celoxica RC10+DKAltera DE2-70

TI DSP DaVinci 6446Beagle boardXscale270



84

Robotic Vision

http://www.amazon.com/gp/product/images/B0007SXAI4/ref=dp_image_0/104-3882231-3347955?ie=UTF8&n=165793011&s=toys-and-games�



本簡報授權聲明此簡報內容採用 Creative Commons 「姓名標示 - 非商業性台灣 3.0 版」授權條款

歡迎非商業目的的重製、散布或修改本簡報的內容，但請標明： (1)原作者姓名：王元凱； (2)圖標示：

簡報中所取用的部份圖形創作乃截取自網際網路，僅供演講者於自由軟體推廣演講時主張合理使用，請讀者不得對其再行取用，除非您本身自忖亦符合主張合理使用之情狀，且自負相關法律責任。

http://creativecommons.org/licenses/by-nc-nd/3.0/deed.zh_TW�







http://www.islab.tw/�

http://creativecommons.org/licenses/by/3.0/tw/�

towards embedded computer vision邁向嵌入式電腦視覺

Education

embedded computer visionp

embedded computer vision

embedded cpu embedded

embedded computer vision3

embedded cv

embedded systems2

embedded vision sensor

ecv embedded compute