SMART SURVEILLANCE USING PDA

BASEM MUSTAFA ABDURAHAMAN MUFTAH

A dissertation submitted in partial fulfillment of the

requirements for the award of the degree of

Master of Engineering

(Electrical-Electronics & Telecommunication)

Faculty of Electrical Engineering

Universiti Teknologi Malaysia

November, 2006

iii

“To my beloved Father, Mother, Brothers and Sisters”

iv

ACKNOWLEDGEMENTS

Praise to Allah, the Most Gracious and Most Merciful, Who has created the

mankind with knowledge, wisdom and power.

Special thanks to go to PM.DR.Syed Abd Rahman AL-Attas for giving this

opportunity to work under his supervision and for sharing his great knowledge and

experience with me.

I would like to convey my deepest gratitude to Mr. Usman Ullah Sheikh for

his guidance to complete this research. Appreciation is also extend to all people who

gave the author heartfelt corporation and shared their knowledge and for giving some

their valuable time.

Finally, my biggest gratitude is to my family, for their endless love, emotional

support and belief in me .Without them I would never come up to this stage.

v

ABSTRACT

The aim of this research is to develop a fast and reliable surveillance system

based on a personal digital assistant (PDA) device. This is to extend the capability of

the device to detect moving objects which is already available in personal computers.

Secondly, to compare the performance between Background subtraction (BS) and

Temporal Frame Differencing (TFD) techniques for PDA platform as to which is

more suitable. This project is designed for Pocket PC devices (windows Mobile 2003

SE or windows CE 4.2) meant for indoor use only. The system is capable to operate at

5 frames per second. The entire surveillance program has been developed and

programmed using Embedded Visual C++ taking the advantage of the built-in camera.

In order to reduce noise and to prepare frames for the moving object detection part,

each frame is first converted to a gray-scale representation and then smoothed using a

Gaussian low pass filter. Two moving object detection schemes i.e., BS and TFD

have been analyzed. TFD is done by taking the difference between the current frame

and the previous frame. On the other hand BS is accomplished by taking the

difference between subsequence frame and the background frame. The background

frame is updated by using Infinite Impulse Response (IIR) filter so that the

background frame is adapted to the varying illuminate conditions and geometry

settings. In order to reduce the effect of noise pixels resulting from frame differencing

morphological filters erosion and dilation are applied. In this work, it has been found

that TFD technique is more suitable for motion detection purpose than the BS in term

of speed. On average TFD is approximately 170 ms faster than the BS technique.

vi

ABSTRAK

Tujuan utama penyelidikan ini adalah untuk membangunkan satu sistem

pengawasan yang cepat dan boleh diharap serta beroperasi dengan penolong peribadi

digital (PDA). Ini adalah untuk meningkatkan kebolehan peranti ini untuk mengesan

objek bergerak, iaitu satu kebolehan yang sudah sedia ada pada komputer peribadi.

Objektif kedua adalah untuk membandingkan pencapaian teknik penolakan latar

belakang (BS) berbanding dengan teknik penolakan kerangka semasa (TFD) yang

dilaksanakan di atas platform PDA dan menentukan teknik yang mana lebih sesuai.

Projek ini direka untuk peranti Pocket PC (Windows Mobile 2003 SE atau Windows

CE 4.2) dan hanya untuk kegunaan di dalam bangunan sahaja. Sistem in berkebolehan

beroperasi pada kadar 5 kerangka per saat. Keseluruhan aturcara ini dibangunkan

menggunakan Embedded Visual C++ dan memanfaatkan kamera digital dalaman.

Bagi mengurangkan hingar dan menyediakan kerangka untuk pengesanan objek

bergerak, setiap kerangka ditukarkan kepada skala kelabu dan ditapis dengan penapis

lulus rendah Gaussian. Teknik TFD merupakan operasi penolakan antara kerangka

semasa dengan kerangka terdahulu, manakala teknik BS pula mengambil hasil

penolakan antara kerangka semasa dan kerangka latar belakang. Latar belakang

dikemas-kini menggunakan penapis impuls terbatas (IIR) agar latar belakang boleh

menyesuai terhadap perubahan pencahayaan dan keadaan geometri. Operasi

morfologi penghakisan dan pengembangan digunakan untuk mengurangkan hingar

hasil operasi penolakan kerangka. Di dalam penyelidikan ini, didapati apabila faktor

masa diambil kira, teknik TFD adalah lebih berkesan untuk pengesanan objek

bergerak berbanding teknik BS. Secara puratanya, TFD adalah 170ms lebih pantas

berbanding dengan teknik BS.

vii

TABEL OF CONTENTS

CHAPTER TITLE PAGE

TITLE i

DECLARATION ii

DEDICATION iii

ACKNOWLEDGMENT iv

ABSTRACT v

ABSTRAKT vi

TABLE OF CONTENTS vii

GLOSSARY OF TABLES x

GLOSSARY OF FIGURES xi

LIST OF NOTATIONS xiv

LIST OF ABBREVIATIONS xv

LIST OF APPENDICES xvi

1 INTRODUCTION

1.1 Introduction 1

1.2 PDA 3

1.2.1 Palm PDAs 4

1.2.2 Pocket PC 4

1.2.3 Smartphones 4

viii

1.3

The comparison between Pocket PC and Palm

PDA 5

1.4 Windows Mobile for Pocket PC 6

1.5 Surveillance using PDA 7

1.6 Objective 7

1.7 Scope of the work 8

1.8 Project outline 8

2 LITERATURE REVIEW

2.1 Overview 10

2.2 Surveillance based on PDA 10

2.2.1 System Demonstration 13

2.2.2 Conclusion 13

3 METHODOLOGY

3.1 Introduction 14

3.2 System Overview 14

3.3 Why Pocket PC 15

3.4 Image Capturing 16

3.5 Preprocessing 17

3.1 Introduction 14

3.5.1 Grayscale 17

3.5.2 Smooth (Gaussian) filter 17

3.6 Moving Object Detection 20

3.6.1 Overview 20

3.6.2 Optical flow 20

3.6.3 Temporal Frame Differencing 22

3.6.4 Background Subtraction 23

3.6.5 Adaptive Background 23

3.7 Threshold 24

3.8 Filtering Noise 25

3.8.1 Erosion 25

3.8.2 Dilation 26

ix

3.9 Tracking Object 27

3.9.1 Projection Histogram Technique 28

3.10 Program Architecture 29

3.10.1

Overview of Programming

Languages Used For PDA

Platform

29

3.10.2 Program Flow Chart 31

3.10.3

Graphical User Interface of The

Program 33

3.10.4 Multi-Threading 34

4 RESULTS

4.1 Introduction 36

4.2 Equipment used for project Development 36

4.3 Preprocessing Result 37

4.3.1 Grayscale Results 37

4.3.2 Gaussian filter Results 38

4.4 Moving Object Detection Results 38

4.4.1 FD algorithms Results 39

4.4.2 Background Subtraction Results 40

5 CONCLUSION & SUMMARY

5.1 Summary 49

5.2 Conclusion 50

5.3 Suggestions for Future Work 50

References 50

Appendix A 52

Appendix B 59

Appendix C 66

Appendix D 69

Appendix E 70

x

GLOSSARY OF TABLES

TABLE NO. TITLE PAGE

1.1 Comparison between PocketPC and Palm 5

xi

GLOSSARY OF FIGURES

FIGURE NO. TITLE PAG

E

1.1 Types of PDA 3

1.2 Pocket PC platform 6

1.3 Evolution of Mobility & connectivity 7

2.1 The system architecture of Surveillance Server 11

2.2 The layered structure of the PDA Watcher 12

2.3 The snapshots of PDA watch in Palm Emulator 13

3.1 Overall system architecture block diagram 15

3.2 HP iPAQ hx4700 PocketPC 16

3.3 Digital camera card (HP Photosmart mobile camera) 16

3.4 2-D Gaussian distribution with mean (0, 0) and σ =1. 18

3.5 Gaussian filters coefficients 19

3.7 The mask of Gaussian filter 19

3.7 An image with its optical flow vectors 22

3.8 Erosion operation 26

3.9 Dilation operation 27

3.10 Summations of rows and columns to obtain the vertical and

horizontal projection 29

3.11 Program flow chart 32

3.12 Program screenshot 35

4.1 Grayscale image 37

4.2 An image filtered by Gaussian filter 38

4.3 Result of TFD 39

xii

4.4 Result of BS 40

4.5 (a) The background image

(b) The adapted background whenα =0.5 (after 2 seconds)

(c) The final adaptation of background (after 5 seconds)

41

4.6 (a) The background image

(b) The adapted background whenα =0.1 (after 5 seconds)

(c) The final adaptation of background (after 10 seconds)

42

4.7 (a) The result of threshed (T=32).

(b) T=60

(c) T=100

43

4.8 The result of Morphological operators (Erosion and

followed by Dilation) 44

4.9 Computation time foe the processes of the system 45

4.10 The performance comparison between BS and TFD 46

xiii

GLOSSARY OF NOTATIONS

R Value of Red color.

G Value of Green color.

B Value of Blue color.

T Threshold value (32)

tI The current frame.

1−tI The previous frame.

2σ The variance of Gaussian distribution

),( yxf Input image

),( yxg Output image (binary image).

)(yPv Projection of the vertical axis

)(xPh Projection of the horizontal axis

xiv

LIST OF ABBREVIATIONS

1GSS First Generation Surveillance Systems

2GSS Second Generation Surveillance Systems

3GSS Third generation surveillance systems

BS Background subtraction

FPS Frames Per Second

GUI Graphical user interface

HTTP Hypertext Transfer Protocol

IIR Infinite Impulse Response

J2ME Java2 Micro Edition

JMF Java Media Framework

MO Moving Object

PDA Personal Digital Assistant

RMI Remote Method Invocation

TFD Temporal Frame Differencing

WTP Watching Transmission Protocol

xv

LIST OF APPENDICES

APPENDIX TITLE PAGE

A C++ DECLARATIONS 52

B CAMERA SDIO SDK SPECIFICATION 59

C SYSTEM SPECIFICATION 66

D PROGRAM SCREENSHOTS 69

E OTHER COMMERCIAL SOFTWARE 70

1

CHAPTER 1

INTRODUCTION

1.1 INRODUCTION

Video surveillance systems have long been in use to monitor security

sensitive areas. The history of video surveillance consists of three generations of

systems which are called 1GSS, 2GSS and 3GSS.

The first generation surveillance systems (1GSS, 1960-1980) were based on

analog sub systems for image acquisition, transmission and processing. They

extended human eye in spatial sense by transmitting the outputs of several cameras

monitoring a set of sites to the displays in a central control room. They had the major

drawbacks like requiring high bandwidth, difficult archiving and retrieval of events

due to large number of video tape requirements and difficult online event detection

which only depended on human operators with limited attention span.

The next generation surveillance systems (2GSS, 1980-2000) were hybrids in

the sense that they used both analog and digital sub systems to resolve some

drawbacks of its predecessors. They made use of the early advances in digital video

2

processing methods that provide assistance to the human operators by filtering out

spurious events. Most of the work during 2GSS is focused on real-time event

detection.

Third generation surveillance systems (3GSS, 2000- ) provide end-to-end

digital systems. Image acquisition and processing at the sensor level, communication

through mobile and fixed heterogeneous broadband networks and image storage at

the central servers benefit from low cost digital infrastructure.

Unlike previous generations, in 3GSS some part of the image processing is

distributed towards the sensor level by the use of intelligent cameras that are able to

digitize and compress acquired analog image signals and perform image analysis

algorithms like motion and face detection with the help of their attached digital

computing components.

The ultimate goal of 3GSS is to allow video data to be used for online alarm

generation to assist human operators and for offline inspection effectively. In order to

achieve this goal, 3GSS will provide smart systems that are able to generate real-time

alarms defined on complex events and handle distributed storage and content-based

retrieval of video data.

The making of video surveillance systems “smart” requires fast, reliable and

robust algorithms for moving object detection, classification, tracking and activity

analysis. Starting from the 2GSS, a considerable amount of research has been

devoted for the development of these intelligent algorithms.

3

1.2 PDA

PDA is an abbreviation for Personal Digital Assistant, a handheld device that

combines computing, telephone/fax, Internet and networking features. A typical

PDA can function as a cellular phone, fax sender, Web browser and personal

organizer. Unlike portable computers, most PDAs began as pen-based, using a stylus

rather than a keyboard for input. This means that they also integrated handwriting

recognition features. Some PDAs can also react to voice input by using voice

recognition technologies. PDAs of today are available in either a stylus or keyboard

version. The most of PDAs can be categorized as shown in Figure 1.1 in three types:

Figure 1.1: Types of PDA

4

1.2.1 Palm PDAs:

Palm PDAs originally developed by the Palm Company but now available

from other companies such as Acer, Sony and Handspring.

1.2.2 Pocket PC

These PDAs run a version of Windows specifically designed for handhelds.

As such, it is initially more familiar to a PC user. HP, Fujitsu-Siemens, DELL are

popular manufacturers.

1.2.3 Smartphones

Mobile phones use Windows Mobile or Symbian as operating system

developed by the Palm Company but now available from other companies such as

Acer, Sony and Handspring.

1.3 The comparison between Pocket PC and Palm PDA

Differences between Pocket PC (Windows CE or Windows Mobile OS) and Palm

PDA (Palm OS) are shown in Table 1.1:

5

Table 1.1: Comparsion between PocketPC and Palm

Feature PocketPC

(Windows CE)

Palm PDA

(Palm OS)

User Interface Action

Clicks (clicks required

to perform a function)

1 1

User Interface

Application Launching

Menu, submenu is icons

after drilling down icons only

User Interface

Application Menus Always visible

Requires user to click on

menu button

Fonts Regular and Bold 3 different font sizes per

application

Handwriting

Recognition Shortcuts No Yes

E-Mail - Attachments Yes, and HTML No

Pocket apps (Word and

Excel) w/desktop file

format support

Yes No

Sound levels variable 3

Sound level options System System, Alarm, Game

IrDA Auto Receive no yes

Sync log on device no yes

Security password password, private items,

forgotten password

Find by column, word, or

application

General - searches all apps

and data

Storage Model Directory, flattened My

Documents invisible to user

Multitasking Yes No Multitasking

6

1.4 Windows Mobile for Pocket PC

Microsoft recently announced Windows Mobile, a new global brand for

Microsoft software for mobile devices such as Pocket PCs and Smartphones. The

launch of Windows Mobile software extends the Windows brand to the Pocket PC

and Smartphone mobile device categories. The new Windows Mobile brand also

helps customers more readily understand the consistent user experience they can

expect from the software inside Pocket PCs and Smartphones. The new branding also

reflects Microsoft's commitment to the mobile space in bringing its mobile device

software into the Windows brand family.

Figure 1.2: Pocket PC platform.

7

1.5 Surveillance using PDA

PDAs have become devices many of us use on a daily basis. They are

relatively inexpensive; offer a many of applications while also being small, portable,

and easy to use. Many standard PDA interfaces have been developed for a wide

range of applications. In addition to standard applications such as word processing,

calendar management, and calculators, some more specialized systems have been

developed. For example, character recognition [1, 2, 3], educational tools [4, 5], and

face recognition [6].

Therefore, it seems relatively obvious that PDAs may represent a good base

for supervision. Their small size, processing capability, and portability make them

suitable platform for surveillance.

Figure 1.3: Evolution of Mobility & connectivity

8

1.6 Objectives

The main objective of the project to design and develop a fast and reliable

surveillance system based on PDA. This is to extend the capability of the device to

detect the moving of objects which is already available in personal computers. The

second objective is to compare the performance between BS and TFD techniques for

PDA platform

1.7 Scope of the work

In developing a system for smart surveillance for PDA platform, the scope of the

current research has been defined as follows:

• The system is designed for Pocket PC only

(Windows Mobile 2003 SE, Windows CE 4.2 or higher)

• Indoor use only with stationary PDA.

• The system operates at 5 FPS.

• The entire surveillance program has been developed and programmed using

Embedded Visual C++.

1.8 Project Outline

The project is organized into chapters. The outline is as following:

9

Chapter 1-Introduction

The first chapter provides readers a first glimpse at the basic aspects of the

research undertaken such as objectives, copes, problem statement.

Chapter 2-Literature Review

The second chapter gives an insight to the previous work regarding to the

surveillance based on PDA platform.

Chapter 3- Methodology

This chapter presents an overall system methodology and steps must be taken

in consideration for moving object detection.

Chapter 4-Results

This chapter shows the final results for Moving Object detection based on

PDA platform.

Chapter 5- Conclusion

This chapter deals with summary and conclusion of the research. Last of all

some realistic extensions as well as possible enhancement for the research are

provided.

50

REFERENCES

[1] Z. Luo and C.H. Wu, “A Chinese Character Recognition Interface for

Mobile Communication Devices Using Fuzzy Logic and Unit Extraction”,

Proceedings of the IEEE IECON 22nd International Conference on

Industrial Electronics, Control, and Instrumentation, August 1996. 572-577

[2] H. Kang and H.J. Kim, “Design of an Interface on PDA for Korean”, IEEE

Transactions on Consumer Electronics, Vol.46, Issue: 3, pp.834-838,

August 2000.

[3] Yu-Fei Ma, Hong-Jiang Zhang. Detecting Motion Object By Spatio-

Temporal Entropy Microsoft Research, China 5F, Sigma Center, 49 Zhi

Chun Road Beijing, China.

[4] J. Breitbart, D. Balakrishnan, and A. Ganz, “Pocket-IMPACT Software for

Delivering Online Courseware on a PDA: Challenges, Design Guidelines

and Implementation”, Proceedings of IEEE Frontiers in Education

Conference, Vol.1, pp. T3F-5, November 2002.

[5] X. Vila, A. Riera, E. Sanchez, M. Lama, D. L. Mureno, “A PDA-based

Interface for a Computer Supported Educational System”, Proceedings of

the 3rd IEEE International Conference on Advanced Learning

Technologies, pp.12-16, July 2003.

[6] J. Yang, X. Chen, and W. Kunz, “A PDA-based Face Recognition System”,

Proceedings of the Sixth IEEE Workshop on Applications of Computer

Vision, pp. 19-23, December 2002.

[7] Sheng-Tun Li, Huang-Chih Hsieh, Ly-Yen Shue, Wen-Shen Chen “PDA

Watch for Mobile Surveillance Services". Department of Information

Management National Kaohsiung First University of Science Technology

2 Juoyue Rd., Nantz District, Kaohsiung 811, Taiwan.

51

[8] Gregory A. Baxes. 1996. |Digital Image Processing: Principles and

Applications. New York: John willey & Sons.

[9] A. J. Lipton, H. Fujiyoshi, and R.S. Patil. Moving target classification and

tracking from real-time video. In Proc. of Workshop Applications of

Computer Vision, pages 129–136, 1998.

[10] Jeff Prosise. Programming Windows with MFC. Redmond, Washington:

Microsoft Press. 1999.

[11] Richardh.Carverkuo Chungtai. Modern Multithreading. Hoboken, New

Jersey .John Wiley & Sons. 2006.