video object tracking with classification and recognition of objects

Video Object Tracking with Classification and Recognition of

Objects

ByManish Khare

Under the Supervision ofDr. Rajneesh Kumar Srivastava

Department of Electronics and Communication, University of Allahabad

First progress Presentation

Introduction about Object Tracking

Object tracking is an important task within the field of computer vision[1].

There are three key steps in video analysis: Detection of moving objects of interest Tracking of such objects from frame to frame Analysis of object tracked to recognize their behavior

The use of object tracking is pertinent in the tasks of: motion-based recognition, automated surveillance, video indexing, human-computer interaction etc.

First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

2

Introduction about Object Tracking (contd..)

Usual input for a motion analysis system is a temporal image sequence with a corresponding amount of processed data.

Motion analysis requires comprehensive information about moving object(s) in video sequences, which include segmentation and tracking of individual object from occluded scene of video.

Video Object Tracking combines two phases of processing: Recognition and Classification of moving objects. Tracking of moving objects.

During the phase of recognition and classification of moving objects, we classify the type of object. (For e.g. object is Car, Human Body, some machine, etc…).

3


Application of Tracking System

The areas where this recognition and tracking system surveillance can be used are: monitoring of people in crowded area such as Shopping

Mall, Temples or several commercial buildings. monitoring of the people to ensure that they are within the

norms such as in secured banking. military and police. educational and manufacturing industries.

4


Shortcoming in Existing Tracking Systems

The existing methods often require special markers attached with the objects, which prevent the different applications, other problems with the existing tracking systems are-

They depend on data from limited field of view. i.e. fixed camera with limited view.

Human operators are required to monitor activities. Require a lot of human intervention to track the object

the same object in case the multiple cameras are used.

5


Modules in My Work

The goal of the my research is to track multiple objects from video sequence, The main parts of the proposed research are as follows-

Automatic Segmentation in video sequence. Recognition of objects. Classification of objects. Detection and Removal of Shadows of objects in video. Tracking of objects in video.

6


Work Going on this time for my Work

Presently I am working on following two modules of my proposed research work:

Automatic Segmentation in video sequence. Detection and Removal of Shadows of objects in

video.

7


Automatic Segmentation

The goal of the first step of the proposed work is to segment image sequence automatically into regions that are meaningful with respect to our application.

Presently I am trying to segment medical images (specially Medical images with some noise and blur), after this I will segment real images and then proceed to image sequences.

8


Image Segmentation

Segmentation subdivides an image into its constituent regions or objects [2].

The level to which the subdivision is carried depends on the problem being solved.

Image segmentation algorithms generally based on one of two basic properties of intensity values:

Discontinuity Similarity

9


Image Segmentation (Contd..)

Our hypothesis is to use multi-resolution automatic segmentation in combination with conventional edges and region based segmentations as well as Partial Differential Equation with level set methods.

Level set methods have their own importance in segmentation due to its accuracy and fast speed [3].

I am using level set methods for my proposed work.

10


Level set method

Introduced by Osher- Sethian in year 1988.Level set method is a way to denote active contours.Basically it is used in fluid mechanics, but it can also

be used in imaging sciences.Vision and image segmentation

Malladi-Sethian-Vermuri (1994) Chan and Vese (1999)

Very few work have been done in image and video processing by using this technique so this is very challenging area.

This depend on position, time, the geometry of interface, and the some energy function[4].

11


Level set method (contd..)12


Illustration of level set method and the contour change

[Chieh-Ling Huang(2009)]


Level set method (contd..)

Advantages: It can easily represent complicated contour changes,

for ex. When the contour splits into two or develops holes inside.

Easily know whether a point is inside or outside the contour by checking Ф value.

How to evaluate the contour value:

[Chieh-Ling Huang(2009]

14


Difference of Level set method from Fast Marching Technique

Fast Marching technique are designed for problems in which energy function never changes sign during processing, so that the contour is always moving forward or backward[5].

Level set Methods are designed for problems in which energy function can be positive in some place and negative in others, so that the front can move forwards in some places in others[5].

Level set significantly slower than fast marching method.

15


Level set method based image segmentation (Literature Survey)

Very few work have been done for image segmentation using level set method up to this time.

Work done for image segmentation by using level set method. Active contour without edges [T.F.Chan et al.(2001)][8] Level set evolution without re-initialization [Chunming Li et al.(2005)][10] Moment based Level set method for image segmentation [Juan Jhou et al.

(2006)][14] Level set method for image segmentation based on Bayesian Analysis [Xu Jing et al. (2008)][11] Edge extraction with level set method for image segmentation [Zhang Junru et al.(2008)][13] Shape based Level set method for image segmentation [Chieh Ling Huang

(2009)][15] Level set method based on overall Information of image [Li Min et al.(2010)]

[12] Distance Regularized Level set Evolution [Chunming Li et al.(2011)][9]

16


Active Contour Without Edges

This paper propose a model for active contour to detect objects in a given image based on technique of contour evolution.

This model can detect objects whose boundaries are not necessarily defined by gradient

Here, the initial curve can be anywhere in the images, the interior contour automatically detected.

This is oldest method for images segmentation using level set method.

17


Shape based Level set method

level set method proposed by chan and vese can not work well on some specific shape.

So, this paper add the shape knowledge into segmentation method.

This algorithm can work with medical images, temperature images etc.

Algorithm for this method is: Establish the initial shape model. Initialize the level set method. Adjust the shape model by current level set function. Determine the contour of shape. Compute the distance of the contour of shape by dilation operation. Update the level set function. Repeat the step 3-6 until the object is not segmented.

18


Edge extraction with level set method

This paper deals with remote sensing images using level set methods.

There are several advantages and disadvantage of extract edges using this method- This method is very flexible, we can just extract the edge

that we want and we can have some control, when the curves moves.

When we use edge detectors, sometimes we will meet a lot of properties of how to track the edges, while using level set method, the curve is continuous which make it easy to be tracking.

One disadvantage – because the algorithm is much more complicated, it take very long time for processing.

19


Level set evolution without Re-initialization

This paper proposed a new Variational formulation for geometric active contours.

This method can easily implemented by using simple finite difference scheme.

Here larger time step can be used to speed up the curve evolution.

Variational formulation consists of an internal energy term that penalizes the deviation of the level set function a signed distance function and a external energy term that derives the motion of the zero level set toward the desired image features, such as object boundaries.

20


Level set method based on Bayesian Classifier

this paper deals with color images, in which level set method applied.

Traditional level set method used gray level images. Traditional method only set gray-level gradient information as the stopping power of contour lines to defined velocity function, which will distort when applied into color images.

This paper re-designed a speed function based on color gradient function to color image to replace the traditional method of gray scale gradient by combining regional color characteristics.

By using Bayesian classification, we can improve the speed function as integrate the regional color characteristics make use of processing image, regional color information to access & determine the contour lines for reach at goal.

21


22

Results of some Existing Algorithms (Using Level Set Method)

Initial Contour 200 Iteration 450 Iteration 650 Iteration Result of [Chunming et al. (2005)]

Initial 100 400 600 Contour Iteration Iteration Iteration

Results of [Li jun Zhang et al. (2006)]


23

Results of some Existing Algorithms (Using Level Set Method)

Initial Contour 100 Iteration 200 Iteration 300 Iteration Result of [Chunming et al. (2009)]

Results of [Xu Jing et al. (2008)]


Results of some other Existing Algorithms

24

Bragman Algorithm for Segmentation of image


Results of some other Existing Algorithms (Contd.)

25

Fast Global Minimization of the Active Contour model base Image Segmentation



26

Image segmentation by boundary extraction method



27

Global Region-Based Image Segmentation Method


My Proposed Approach for Image Segmentation

Take RGB/GRAY Image as input image. Convert it into GRAY Image. Apply Gaussian Convolution function for smoothness. Apply Dirac function into smooth Image. Initialize Level set energy function in image region R. [Negative contour value inside the region, Positive contour value outside the region and Zero contour value is at the region.] For iteration 1 to user define

Call level set function with updated value (achieved by previous iteration). Update contour with updated value.

Stop for loop either when iteration completed or when all contour value reach at zero value(neither at positive contour value nor at negative contour value).

Stop Process.

28


Achieved Result by Proposed Approach

29

Original Image Initial Contour 15000 Iteration 22000 Iteration

Original Image Initial Contour 14000 Iteration 20000 Iteration First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Achieved Result by Proposed Approach (contd..)

30




Shadow Detection and Removal

Shadow detection is the next important component of the proposed work. By detecting the shadow, we can reduce the possibility of partial occlusion problem easily. Shadow detection is an important step in video surveillance and monitoring system, shadow point are often classified as object points causing errors in segmentation and tracking of moving objects.

For this approach, firstly I am trying to detect and remove shadow from still image, then I will proceed to images sequences (video).

31


Shadow Detection

What is Shadow? A shadow is an area where direct light from a light source

cannot reach due to obstruction by an object. A shadow occurs when an object partially or totally

occludes direct light from a source of illumination.

Types of Shadow. Self Shadow:

This shadow occurs in the portion of a object which is not illuminated by direct light.

Cast Shadow:This shadow is the area projected by the object in the direction of direct light.

32


A B C[Figure From Li Xu et. al. (2006)]

A - shows scene image with both cast and self shadows;

B - gives an example of cast shadow of two photographers on a grass field;

C - shows an example of self shadow


Shadow Detection (Contd..)

Self Shadow is again divided into two parts: Shading. Interreflection.

Cast Shadow is again divided into two parts: Umbra. Penumbra.

34

• Umbra, is the darkest part of the shadow. In umbra, the light source is completely occluded.

• Penumbra, is the region in which only a portion of the light source is obscured by the occluding body. [Figure From Sanjeev Kumar et. al. (2010)]


Why the need for shadow removal?

Shadows cause tracking, segmentation or recognition algorithms to fail.

Shadows have proven to be a large source of error in the detection and classification of vehicles.

Real images with shadows can‘t be used for image synthesis.

35


Literature Survey about Shadow Detection and Removal

Very few work have done for shadow detection and removal in case of images up to this time.

There are five approaches based on shadow properties to detect and removal of shadow:

Model Based Technique. Image Based Technique. Colour/Spectrum Based Technique. Texture Based Technique. Geometry Based Technique.

36


Model Based Technique

In Model based technique, the geometry and illumination of the scene are assumed to be known.

This includes the camera localization, the light source direction, and the geometry of observed objects, from which a priori knowledge of shadow areas is derived.

In most applications the geometry of scene and/or light sources are unknown [23].

This technique is oldest in all techniques.

37


Image Based Technique

Image based technique makes use of certain image shadow properties such as colour (or intensity), shadow structure (umbra and penumbra), boundaries, etc., without any assumption about the scene structure.

If any of that information is available, it can be used to improve the detection process performance.

According to Salvador et. al. [24] shadow do not change the surface texture, surface marking tend to continue across a shadow boundary under general viewing conditions.

According to C. Jiang et. al. [25], in some colour components (or combination of them) no change is observed whether the region is shadowed or not, this is invariant to shadows

38


Colour/Spectrum Based Technique

The Colour based technique attempts to describe the colour change of shaded pixel and find the colour feature that is illumination invariant.

K. Siala [26] consider the pixel’s intensity change equally in RGB colour components and a diagonal model proposed to describe the colour distortion of shadow in RGB space.

Cucchiara [27] investigated the Hue-Saturation-Value (HSV) colour property of cast shadows, and it is found that shadows change the hue component slightly and decrease the saturation component significantly. The shadow pixels cluster in a small region that has distinct distribution compared with foreground pixels. The shadows are then discriminated from foreground objects by using empirical thresholds on HSV color space.

39


Texture Based Technique

The principle behind the textural model is that the texture of foreground objects is different from that of the background, while the texture of the shaded area remains the same as that of the background.

The several techniques have been developed by using this technique.

The technique of Leone et. al. [17,18] is a good approach and it is based on the observation that shadow regions present same textural characteristics in each frame of the gray-level video sequence and in the corresponding adaptive background model.

The technique proposed D. Xu. [28] include the generation of initial change detection masks and canny edge maps.

40


Geometry Based Technique

Geometric Model based technique makes use of the camera location, the ground surface, and the object geometry, etc., to detect the moving cast shadow,

The Hsieh [29], Gaussian shadow model was proposed to detect the shadow of pedestrian. The model is parameterized with several features including orientation, mean intensity, and center position of a shadow region with the orientation and centroid position being estimated from the properties of object moments.

This technique some has similarity with the model based technique.

41


Results of Some Existing Algorithms

Results of [Sanjeev Kumar et. Al. (2010)]This approach is based on Colour Based Technique

42


Results of Some Existing Algorithms (Contd..)

A - Original images with shadows.

B – The reconstructed shadow images based on our method.

C - The recovered shadow free images.

These are Results of

[Li Xu et. al. (2006)]

This approach is based on Image Based Technique.

43

A B C


Results of Some Existing Algorithms (Contd..)

Result of [Li Xu et. al. (2008)]This approach is based on Model Based Technique

44

Result of [Clement Fredembach et. al.

(2006)]

This approach is based on Texture Based Technique.


My Proposed Approach for Detecting and Removing Shadow

Take RGB/GRAY image with shadow.Remove noise by using any non-linear filter.Calculate average Colour of image for determine

effect of shadow in each of the dimension of Colour.[colours in shadow regions have larger value than the average, while

colours in non-shadow regions have smaller value than the average values.]

Construct some threshold to extract shadow regions.

Use level set method for evaluate energy function for shadow are removal.

Finally we got shadow free image.

45


Result of this proposed approach and other modules/methods, which I will do up to next progress presentation will be show in next presentation.

46


References

1. Alper Yilmaz, Omar Javed, Mubarak Shah: “Object tracking: A survey” ACM Computing Surveys (CSUR) Volume 38 Issue 4, 2006.

2. Rafael Gonzalez and R. E. Woods: “Digital Image Processing” Pearson Education, 2nd edition, 2002.

3. Stanley Osher,Nikos Paragios: “Geometric Level Set Methods in Imaging, Vision, and Graphics” Springer publication, 1st edition, 2003.

4. Amar Mitiche, Ismail Ben Ayed: “Variational and Level Set Methods in Image Segmentation” Springer publication, 1st edition, 2011.

5. J.A. Sethian: “Level Set Methods and Fast Marching Methods - Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science” , Cambridge University Press, 1999.

6. M. Kass, A. Witkin, and D. Terzopoulos, “Snakes - Active Contour Models'' International Journal of Computer Vision, 1(4): 321-331, 1987.

7. T. F. Chan & L. A. Vese: “A Multiphase level set framework for image segmentation using the Mumford and Shah model” International Journal of Computer Vision 50(3), 271–293, 2002.

8. T. F. Chan & L. A. Vese: “Active contours without edges” IEEE Transactions on Image Processing, 10(2), 266-277, 2001.

9. C. Li, C. Xu, C. Gui, and M. D. Fox: "Distance Regularized Level Set Evolution and its Application to Image Segmentation", IEEE Trans. Image Processing, vol. 19 (12), 2010.

47


References (Contd..)

10. C. Li, C. Xu, C. Gui, and M. D. Fox: "Level Set Evolution Without Re-initialization: A New Variational Formulation", CVPR 2005.

11. Xu Jing, Wu Jian, Ye Feng, Cui Zhi-ming: “A Level Set Method for Color Image Segmentation Based on Bayesian Classifier” Proceeding of International Conference on Computer Science and Software Engineering, 886 - 890, 2008.

12. Li Min, Xu Xiangmin, Qian Min, Wang Zhuocai: “A Fast Level Set Segmentation Method Based on the Overall Information of Image” Proceeding of 2nd International Symposium on Information Engineering and Electronic Commerce (IEEC), 1-4, 2010.

13. Zhang Junru, Jiang Xuezhonga: “Research on Edge Extraction With Level Set Method” International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B4. Beijing 2008.

14. Juan Zhou, Lixiu Yao, Jie Yang, Chunming Li: “Moment based level set method for image segmentation” Proceeding of 15th IEEE International Conference on Image Processing (ICIP), 1069 - 1072, 2008.

15. Chieh-Ling Huang: “Shape-Based Level Set Method for Image Segmentation” Proceeding of Ninth International Conference on Hybrid Intelligent Systems, HIS '09. 243-246, 2009.

16. Li-jun Zhang, Xiao-juan Wu, Zan Sheng: “A Fast Image Segmentation Approach based on Level Set Method”8th International Conference on Signal Processing, 2006.

17. A. Leone, C. Distante, F. Buccolieri: “A texture-based approach for shadow detection” Proceeding of IEEE Conference on Advanced Video and Signal Based Surveillance, 371 - 376, 2005.

48



18. A. Leone, C. Distante, N. Ancona, E. Stella, P. Siciliano: “Texture analysis for shadow removing in video-surveillance systems” Proceeding of IEEE International Conference on Systems, Man and Cybernetics, 6325 - 6330 vol.7, 2004.

19. Li Xu, Feihu Qi, Renjie Jiang, Yunfeng Hao, Guorong Wu, Li Xu, Feihu Qi, Renjie Jiang, Yunfeng Hao, and Guorong Wu.: “Shadow detection and removal in real images: A survey” Technical report, Shanghai Jiao Tong University, 2006.

20. Li Xu, Feihu Qi, Renjie Jiang: "Shadow Removal from a Single Image," Proceeding of Sixth International Conference on Intelligent Systems Design and Applications (ISDA'06) Volume 2, 1049-1054, 2006.

21. Sanjeev Kumar and Anureet Kaur: “Shadow Detection And Removal in Colour Images Using Matlab” International Journal of Engineering Science and Technology. Vol. 2(9), 4482-4486, 2010.

22. C. Fredembach, G. Finlayson: “Simple Shadow Removal” 18th International Conference on Pattern Recognition(ICPR), Hong Kong, 832 - 835, 2006.

23. D. Roller, K. Daniilidis and H. H. Nagel: “Model-based object tracking in monocular image sequences of road traffic scenes” International Journal of Computer Vision, Volume 10, Number 3, 257-281, 1993.

24. E. Salvador, A. Cavallaro, T. Ebrahimi: “Shadow identification and classification using invariant color models” Proceeding of IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP '01), 1545 - 1548 vol.3, 2001.

25. C. X. Jiang, M. O. Ward: “Shadow Segmentation and Classification in a Constrained Environment” CVGIP: Image Understanding Volume 59, Issue 2, 213-225, 1994.

49



26. K. Siala, M. Chakchouk, F. Chaieb, O. Besbes: “Moving shadow detection with support vector domain description in the color ratios space” Proceedings of the 17th International Conference on Pattern Recognition (ICPR 2004), 384 - 387 Vol.4, 2004.

27. R. Cucchiara, C. Grana, M. Piccardi, A. Prati: “Detecting moving objects, ghosts, and shadows in video streams” IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI), Vol. 25(10), 1337 - 1342, 2003.

28. Dong Xu, Xuelong Li, Zhengkai Liu and Yuan Yuan: “Cast shadow detection in video segmentation” Pattern Recognition Letters Vol. 26(1), 91-99, 2005.

29. Jun-Wei Hsieh, Wen-Fong Hu,Chia-Jung Chang and Yung-Sheng Chen: “Shadow elimination for effective moving object detection by Gaussian shadow modeling” Image and Vision Computing, Vol. 21(6), 505-516, 2003.

50


Thanks

video object tracking with classification and recognition of objects

Education

tracking of objects

recognition of objects

classification of objects

tracking of moving objects

use of object tracking

tracking of individual

video analysis

multiple objects