USING THE S60 MOBILE PHONE USING THE S60 MOBILE PHONE PLATFORM FOR REMOTE PLATFORM FOR REMOTE

MONITORING WITH IMAGE MONITORING WITH IMAGE PROCESSING AND CAMERA PROCESSING AND CAMERA

CONTROLCONTROL

By

Kittinut Khumkiatkul

THESIS COMPETITION

ContentContent

1. Introduction

2. Local server module

3. Mobile Client module

4. Experiment and results

5. Conclusion

1.1 Background1.1 Background

1. Ability to automatically classify the object in the frame : In practical use, manual classification is susceptible for the human error. The image processing is needed to help the human for classification task

2. Accessibility of the system beyond the internet : Frequently, the users stays where the internet is unavailable, so they can not access the monitoring system. The mobile device is needed to strengthen system accessibility

The surveillance system is to transmit the event (signal) from the local to remote location for the monitoring purpose. To effectively monitor the event, there are two importance points to be concerned:

1.1 Background (Cont.)1.1 Background (Cont.)

This research is proposed to serve these requirement.• To do classification work, the local server is installed in the local site for image processing tasks.• The accessibility of the system can be increased by using Symbian mobile client with the API

Symbian mobile client

Local server

Network camera

Internet

LAN

Web browser

Manual monitoring

1.2 Concept of study1.2 Concept of study

Main component

1. Network camera

2. Local server

3. Web server

4. Web browser

5. Symbian Mobile client

1.3 Objective1.3 Objective

• Construct the local server for image processing project in MATLAB

• Construct the web server for data transferring between local components and mobile module

• Build an API for S60 mobile client to communicate with the local component via EDGE

To enable mobile client communicate with local server and network camera and do the classification work, three items have to be done

2. Local Server module2. Local Server module

2.1 General concept

2.2 Network Camera System

2.3 Human Detection Project

2.4 Adaptive background to environment

2. General concept: Target2. General concept: Target

• Implement network camera system, comprising of image flow, image database, web browser and stream

• Develop the program to detect the object in the image and classify whether it is the human or not

• Develop the program to generate alarm when the object is classified to human

There are three major parts in the system

• Network camera

• Web server

• Local server

2.1 General concept: Architecture2.1 General concept: Architecture

Web Server

Send the image to FTP1

Get the image from FTP2

3

Store image to web server and its storage location

4

Local server

Network camera

TCP/IP

TCP/IP

Process image

2.2 Network camera system2.2 Network camera system

The storage location:ftp://stweb.ait.ac.th/www

Frequency of taking photo:Every 2 seconds

Image size:160 x 120 pixels

Setting items

Web browser

2.3 Human 2.3 Human Detection ProjectDetection Project Start

1) Acquire Image from web server

2) Pre-process image

End

3) Feature Extraction

5) Store image on database and FTP

4) Analyze the feature and Classify

Finish loop ?

Yes

No

Is there any change?

Yes

No

1) Image acquisition1) Image acquisition

• The Human Detection program on local server acquires image from web server (via its ftp)

• Since the image name is time-related name (ISE-YYYYMMDD-HHMMSS.jpg), the Existence test method is used to guess it to acquire image- The current time: 20080514-103019- Test to load image name by using “ISE-20080514-103019.jpg”- Increase 1 sec. if that image does not exist.

• By using the web server as the image database, the system is free from setup when the IP of local server changes

2) Image pre-processing2) Image pre-processing

This process is to adjust the image to the suitable format. There are two interesting steps

1. Color-integrated segmentation: It is to segment the object from to background.

2. Morphological operation: It is to improve the quality of the object shape, compactness and filtering the small particle objects.

2.1) Color-integrated Segmentation 2.1) Color-integrated Segmentation techniquetechnique

This is the upgrade version of Background subtraction technique (prior method) by means of YIQ color model to increase the robustness for lighting and complicated background

Background Current image

Background subtraction technique

Color-integrated segmentation technique

Changes in luminance and the reflection generate many disturbances to segmentation. They cause background more

complex since they are the part of value in R, G and B component. The luminance should be separated from color.

Algorithm comparison: Prior TechniqueAlgorithm comparison: Prior Technique

Background – The current image

Convert to binary image

Start

End Background subtraction technique

Lighting effect on that component

R1,G1,B1

R1 = R1actual + LR1G1 = G1actual + LG1B1 = B1actual + LB1

RGB model

Y1 = Y1actual + LY1I1 = I1actualQ1 = Q1actual

YIQ model

Y=luminanceI, Q = mixture of color

Convert to gray scale image

Compare background to the current image

pixel

BG is Significantly higher than CI

Same color (same I and Q: YIQ)

That pixel is background: Set it to

zero

CI is Significantly higher than BG

The current image - Background

Background – The current image

Convert to binary image

End

Start

The Backgroun

d subtraction technique

yes

no

yes

no

yes

no

Algorithm comparison: Upgraded Algorithm comparison: Upgraded TechniqueTechnique

Color-integrated segmentation technique

The object in segmented image is full with the damages, holes and irrelevant objects. Therefore, this process is used to improve the object quality for the further process

2.2) Morphological operation2.2) Morphological operation

Segmented image Pre-processed image

• Dilate to repair the damage parts• Filter the irrelevant objects• Fill the holes

Morphological operation(Dilation and Erosion)

The unique differences between human and non-human, called Feature, are used to classify the object. Three features that have the consistency for all condition (gender, race, etc.) are selected

3) Feature extraction3) Feature extraction

• Geometrical feature

• Object Skeleton

• Skin pixel extraction

3.1) Geometrical feature3.1) Geometrical feature

Shape and size of the object are extracted and used as the one of criteria of classification. The geometric grids are applied to count the pixel, passed by line to

define the width and height of the object

Segmented image with geometric grid

Size: Amount of the white pixels in the image

The horizontal grid is used to indicate the width in a pixel

The vertical grid is used to indicate the height in a pixel

Width: Amount of pixels, passed by horizontal grid

Height: Amount of pixels, passed by Vertical grid

Shape: ratio between the longest line and shortest line in both ver. And hor.

• Human size is always larger than 12%: The experiment show that the minimum percentage when sample human

are in the frame at 5 meter distance from the camera.• Human shape is always triangle or vertical rectangular

shape.

Sample pattern Vertical rectangular shapeTriangle shape

Regarding to the research assumption that the position of camera is in range 0.5 - 2 meters, there

are 2 unique characteristics of human as the following:

This feature shows the angle of object axis that human always has the vertical skeleton due to the direct backbone, but those of non-human object does not

3.2) 4-line Skeleton Recognition method3.2) 4-line Skeleton Recognition method

The extracted line of skeleton

The longest line

The 2nd longest

line

Preprocessed image The skeleton of human

• The object is applied Zhang-Suen skeletonization method to extract the skeleton line• Four longest lines are extracted from the skeleton using Hough transform and create table of the case of human and non-human

This skin color is the major feature since the skin color properties is rarely shown in non-human object. To extract the skin, it is to classify pixel using these process:

• Create color database (r, g, b) that is collected from the sample image has the 5 separated group (2 for skin and 3 for background)

• Each pixel is checked with mean of each group for the distance of color component

3.3) Skin pixel extraction3.3) Skin pixel extraction

The distance equation:

BGR

Bb

BGR

Gg

BGR

Rr

bbggrrD spspsp

,,

)()()( 222

• Pixel is the member of the group that has the smallest distance between that pixel to its mean value

1.01.0

1.01.0

1.01.0

rr

Skin2Skin2

bbbb

Skin1Skin1

BG1BG1

BG2BG2

BG3BG3

This pixel is the member

of Skin1 group

There are 3 major reasons to change background

• There is rotating command from mobile phone

• Time reaches 30 minutes

• Luminance changes

2.4 Adaptive background to 2.4 Adaptive background to environmentenvironment

2.4 Adaptive Background: Lighting 2.4 Adaptive Background: Lighting changeschanges

The Current Background

There is a change signal

The Current Background

3. Mobile Client Module3. Mobile Client Module

3.1 General concept

3.2 Web Server

3.3 S60 Mobile Application

3.1 General concept: Architecture 3.1 General concept: Architecture

Local Server Module

Local server

Network camera

S60 Client

Client browser

View the processed image

Playback stream

Web Server ftp://stweb.ait.ac.th/

WWW

http://stweb.ait.ac.th/

~st105000

Mobile client Module

Control camera

HTTPS

HTTPS

TCP/IP

TCP/IP

There are three major parts in the system

1. Is addressed at http://stweb.ait.ac.th/~105000

3.2 Web server3.2 Web server

3. Send SMS automatically to user’s mobile client

2. Permits mobile phone to communicate with local site

• Connect via HTTP and GPRS• Get the image from its database• Issue the camera control command

The PHP script, server-side script, works on the web server when it is request from client to do some action such as redirecting to url, posting message, and others There are 3 main PHP scripts.

1. Rotating command: Control the direction of the camera movement by redirecting to the url of camera controlling botton and save the log to web server

up.php, down.php, right.php, left.php are the code that used for each direction

2. Search image: This is used to search the image from its name, ISE-YYYYMMDD-HHMMSS to allow mobile phone getting the most recent image without local server

3. SendSMS: Send the automatically to user’s mobile client

3.2 Web server3.2 Web server Programs: Main PHP Programs: Main PHP scriptscript

3.2 Web server: SMS Configuration3.2 Web server: SMS Configuration

GSMPROVIDERSMS SMS GATEWAY

WEB SERVER

LOCAL SERVERSendsms.php

seachimage.phpor

direction.php

MOBILE CLIENT

3.3 S60 mobile Application3.3 S60 mobile Application

HTTPClientExample is developed on S60 3rd edition Maintenance Release platform

The Carbide.C++ v1.2 is used as IDE

Load the recent image

Show the recent image

Playback the stream

Control the camera

Mobile client user

Function of API: Load the recent image Show the recent image Playback the stream Control the camera Respond to SMS

3.3 S60 Mobile Application: Mobile 3.3 S60 Mobile Application: Mobile InteractionInteraction

1. Select action

2. Do action

3.

4. HTTP Get

UI Engine HTTP API Web Server PHP

a. Request PHP

c. result

d. result

5. HTTP response

6. Receive data

7. Result

8. Show result *

UserNetwrok camera

* In case of camera controlling function, go back to step 4th to get the recent image.

3.3 S60 Mobile Application: Class 3.3 S60 Mobile Application: Class InteractionInteraction

ClientAppClientApp Document

ClientAppUi

ClientEngine

SMSReceiver

ClientImage Engine

Browser.exe

Client Container

3.3 S60 Mobile Application: API function3.3 S60 Mobile Application: API function

• Show the recent image

Request searchimage.php

Receive image data

Acquire the most recent image

Save as recent image

Store the image

Web server(http://stweb.ait.ac.th/

~st105000/recentim.jpg)Mobile client

Local server

Network camera

Request RTSP

Receive the RTSP stream

rtsp://ipaddress:port/live.sdp

• Playback the stream• Control the camera

Request direction.php url

Receive the result log

Get the command log

Change background

Redirect to direction url Send

rotating log1

2

3

4

a

b

1

a

2

b

c

4.1 Image processing Results4.1 Image processing Results

Background Current image Pre-processed image

Geometric feature Skin feature Skeleton feature

4.1 Image processing: Human 4.1 Image processing: Human classification (1)classification (1)

START

H1-H5 NH1-NH8

> 10 %

Skeleton Case

No match

Non-Human

Non-Human

> 12% < 12%

< 10%Skin pixel

percentage

Object Size percentage

Next

20%

H1

17%

Man

Triangular or Vertical

rectangular shape

Other shapes

Human

Width > 2 and Height < ½ of

image

Others

Object Shape

1

Object Shape

2

Non-Human

Next

Man

4.1 Image processing: Human 4.1 Image processing: Human classification (2)classification (2)

After detecting the human, local server sends the SMS to user’s mobile client

4.2 S60 Mobile Application: Response to 4.2 S60 Mobile Application: Response to SMSSMS

Standby in background

Receive new message

The incoming SMS match to

“Detect Human !”

Show image if SMS match

Run application

5. Research Summary: Completed task5. Research Summary: Completed task

Local Server moduleLocal Server moduleMobile Client moduleMobile Client module

Create API for mobile client Display the recent image Playback the stream Control the camera Respond to the emergency SMS

Setup web server and write script

Be the image database for mobile client

Redirect control command to network camera

Send emergency SMS to client

Implement network camera sys

Send the image to FTP periodically

Be accessed by web browser Be controlled from mobile clientProgram Human detection

project Design image flow chart and

acquisition system Design segmentation technique Create and collect color

database Select the consistent feature

and design algorithm to extract them

Design the decision tree for classification unknown object

Setup video database sys

5. Research summary: Challenge5. Research summary: Challenge

1. Low communication bandwidth: Owing to the common frame between stream and image, the smallest size is chosen as the restricted bandwidth. Moreover, it delays the program execution in some stage

Image size – The resolution is reduced, so some image detail is missing and it makes analysis technique more complicated

5. Research Summary: Challenge (Cont.)5. Research Summary: Challenge (Cont.)

2. The communication between camera and web server is not reliable. Sometimes, the image can not be stored into ftp; moreover, it takes a several seconds for storing during low connection. The effect was shown acquisition process

Image Acquisition – The image acquisition function, trial & error technique, is needed to wait for the image. It should be optimized. If the algorithm is too fast search it has never found the image. If it is to slow, it will waste the execution time

5. Research Summary: Challenge (Cont.)5. Research Summary: Challenge (Cont.)

3. Variety of the skin and cloth of human and background: These affect to the segmentation and feature extraction as the follow detail

Segmentation – Some cloth that is nearly same as background becomes the part of background and the object is able to camouflage to the complex background. It is essential to use the integrating of object properties to solve it

Skin pixel extraction – The cloth that is similar to skin color can be extracted as the skin and some skin color can not be extracted as the skin. The color database is needed to be carefully collected

Thank you