5. video object tracking and processing

1

5. Video Object Tracking and Processing

To achieve augmented reality, computer generated graphics should be shown together with the live video

In addition, the generated graphics should be changed according to the live video contents

Hence we need to be able to understand the contents of the live video

References:1. Gonzalez, Rafael C., Digital image processing, Prentice-Hall, 2002. 2. The Mandelbrot DAZIBAO, “Lesson 5 – The HSV Colorspace”,

http://www.mandelbrot-dazibao.com/HSV/HSV.htm.3. Ogre Tutorials, http://www.ogre3d.org/wiki/index.php/Ogre_Tutorials

2

Project Requirements

In this project, a few color balls are randomly placed on the racing track

You are required to develop computer programs that can detect the color balls in the live video obtained from the camera installed on the robot car

Computer graphics should be generated to cover the areas in the video where the balls are found

The size and position of the graphics should be adjusted according to the distance and relative direction between the balls and the robot car (i.e. the camera)

Department of ELECTRONIC AND INFORMATION ENGINEERING

5. Video Object Tracking and Processing by Dr Daniel Lun

3

Procedure for Detecting the Color Balls Grab a frame of the live video Convert the video data from RGB color space to HSV

color space HSV color space is more close to human description to colors

Select all pixels in the image that matches with the color of the ball Assume there are no other objects in the video that have the

same color and similar shape Remove the isolated pixels (optional)

Assume noise in the images may have similar color as the ball but exists in the form of isolated pixels

Locate the center of the remaining pixels Assume the remaining pixels all belong to the ball of interested



4

RGB Color Space For the current digital displays, each pixel is made of a Red, a

Green and a Blue sub-pixel Each sub-pixel is activated by a byte of data, i.e. a number

between 0 and 255 With these three bytes, the computer can generate 256x256x256

= 16,777,216 combinations If R = x, G = y and B = z, we can plot an RGB cube as follows:

While we have a huge amount of possibilities, it is also tricky to define a precise color by adjusting the three components: If R=G=B=0, we have black If R=G=B=255, we have white If R=G=255 and B=0, we have yellow But how about the others???

5

HSV Color Space HSV codification provides an

intuitive method for color selection. Closer to human perception: The blend of the three

components is defined by a single parameter called "Hue"

The "Saturation" parameters selects how grey or pure the color will be

The "Value" parameter defines the brightness of the color

The HSV Color Space can be visualized as a cone



6

RGB to HSV Conversion

r – value of Redg – value of Greenb – value of Bluemax – the greatest of (r,

g, b)min – the smallest of (r,

g, b)h [0, 360) is the hue ∈

angle in degrees, ands, v [0, 100] are the ∈

saturation and value

bifgr

gifrb

bgandrifbg

bgandrifbg

if

h

max,240minmax

60

max,120minmax

60

max,360minmax

60

max,0minmax

60

minmax0

otherwise

ifs

,100max

minmax

0max,0

100255

maxv



7

Why HSV? The color of an object as shown in the video can differ quite

significantly from what is seen using naked eyes Many environmental factors can affect the color of an object in

the video Exposure of the camera White balance setting Ambient lighting Reflection of the spot lights around

Theoretically, an object of red color should have the RGB value of (255,0,0)

In practice, one can never find such value although the object is really red in color when seeing it with naked eyes

Some tolerance should be allowed if we want to detect an object of red color in the video. But the problem is how to set this tolerance



8

Why HSV (cont)?

We should consider an object in the video as red in color if it is “rather red” to “very red”

Besides, a red object should be considered as red no matter it is pictured in a dark environment or a bright environment

How to set such a tolerance? By using the HSV codification

Hue (H) – Define a set of Red color Saturation (S) – Define how Red it is Value (V) – Define how bright it is



9

Why HSV (cont)?

Result of using the Paint.NET

• In RGB color space, for every R value, there can be a set of G and B values that will give you a red color

• Difficult to adjust if a set of Red colors are to be defined

• In HSV color space, H defines a set of Red colors

• S defines the saturation• V defines the brightness • A possible set of values for

Red can be• H < 25 & H > 335• S > 45• V > 60

• In HSV color space, H defines a set of Red colors

• S defines the saturation• V defines the brightness • A possible set of values for

Red can be• H < 25 & H > 335• S > 45• V > 60



10

Original Video Frame

A blue ball and a red ball are put in front of the camera



11

Select the color We set a tolerance as

follows:?? < H < ??; S > ??; V > ??

We compare the HSV value of every pixel of the video frame to see if it falls within the tolerance Yes – mark it as green No – no action

No matter how good is your estimation, there must be some pixels selected not belong to the object of interest They are considered as noise

Noise



12

Denoising

Noise on the image is often considered as isolated dots or islands

It can be reduced by using a simple denoising algorithm as follows

• For every 5x5 block, if most pixels have color falls within the tolerance, consider them as a part of the object

• For this block, only a few pixels are green. Hence they are noise

• For this 5x5 block, all of them have been marked as green. They are a part of the object

• Should carry out the above test for the whole image

• For this 5x5 block, all of them have been marked as green. They are a part of the object

• Should carry out the above test for the whole image

13

Denoising Result The above algorithm

should be carried out pixel-by-pixel for the whole image

As a result, most of the noise are removed

Depends on the actual environment, the denoising performance may not be as good as that

Need long time fine-tuning

Noise are removed



14

Locate the object

The location of the ball can be easily estimated by summing up all marked pixels row-by-row and column-by-column and look for their peaks The peak

For each column, if a marked pixel is found, add 1

For each row, if a marked pixel is found, add 1

…

…

15

Restriction of the algorithm

Certainly, the algorithm can only detect one blue ball If there are multiple blue balls, multiple peaks will

result Become very difficult to determine which peak

corresponds to which blue ball The difficulty also applies to the situation that there

are other large objects that are also blue in color Multiple balls with different color is not a problem

One can detect balls one color at a time



16

Replace the detected object with 3-D graphics

To achieve augmented reality, we need to combine graphics with live video

If the position of the object is detected, we can replace it with 3-D graphics, such as fire generated by Ogre particle system



17

Create a fire in Ogre// The particle "Examples/Smoke" is in the file // Example.particle// Assume mParticleSys and mParticleNode are two private // variables of EIE330projectAppmParticleSys = mSceneMgr->createParticleSystem("Smoke",

"Examples/Smoke");mParticleNode = mSceneMgr->getRootSceneNode() ->createChildSceneNode();mParticleNode->setVisible(false);

// Default - invisiblemParticleNode->setPosition(0, 0, -140);

// Set the position of the firemParticleNode->attachObject(mParticleSys);

:

mParticleSys->setDefaultDimensions(220, 220);// Set the size of the fire

18

Resize the fire

Big fire!



19

Changing the size and orientation of graphics For a real object on the

screen, its size and orientation will change with the change of the camera position

If a 3-D graphic is used to replace a real object on the screen, its size and orientation should also change with the change of the camera position

Camera is closer to the objectCamera is closer to the object



20

Changing the size and orientation of graphics

Camera is moved to the right hand side of the screen

Camera is moved to the right hand side of the screen

Camera is moved to the left hand side of the screen

Camera is moved to the left hand side of the screen



21

How do we know the relative direction and distance of camera?

By using the markers Each ball and the robot car will have a marker

associated with it In Lab 6, you are required to use the markers’ data

to estimate the relative distance and direction between the blue ball and the camera

Need to replace the blue ball with a 3-D object that will change in size and orientation according to the position of the camera

marker

marker



PC

Camera

22

Task for Lab 5

Modify the function TextureSystem::UpdateTexture()

to implement the following:

Change the color space Select the pixels with red and blue

color

Denoise the selected pixels Detect the position of the red ball Create a fire at the position of the red ball on the screen

23

Task for Lab 6 (1st part) Change the size and

orientation of an Ogre head with respect to the relative position of the camera, which is given in the first part of Lab 6

Procedure Create a Ogre head in EIE330ProjectApp::createScene() Implement the function

EIE330ProjectApp::placingObject(float distance, float direction);// change the size and orientation of the Ogre head// based on the input parameters



24

Task for Lab 6 (2nd part) The blue ball will be replaced by a 3-D object of your

choice on the screen By making use of the markers’ data, estimate the

relative distance and angle between the blue ball and the camera

Modify the function placingObject() such that it will change the size and orientation of the 3-D object based on the estimated relative distance and angle

marker

marker

PC

Camera

25

More instructions

For Lab 5 and Lab 6, each team submits one lab report and does one demo for each lab

No more lecture for Lab 6 Once finishing Lab 5, students can start Lab 6

immediately. The lab sheets for Lab 5 and Lab 6 are available in WebCT

Both Lab 5 and Lab 6 do not need robot car and wireless camera

Use that time to fine-tune your robot car. Don’t wait until the last week!



5. video object tracking and processing

Documents