Continuously Adaptive Mean SHIFTBy ERIC | Published: SEPTEMBER 18, 2013

In the Motion Capture course lecture this week, teachers talked about camShiftalgorithm, I heard about this algorithm since about three or four years before,but I never tried it, and during that class I thought I must do it this time.

CamShift is a tracking algorithm, which is based on MeanShift algorithm, whatcamShift do is nothing but do meanShift in every single frame of a video, andrecord the results we got by meanShift.

CamShift algorithm includes these three parts:

1. Back Projection2. MeanShift3. Track

and I will simply explain each of these steps in this blog.

1. Back Projection.Back projection is a method which using the histogram of an image to show upthe probabilities of colors may appear in each pixel. Let’s see how to get the backprojection of an image using OpenCV.

First we transform the picture space to HSV space (or any space which include aH channel that represent the hue of each pixel, of course, value of hue is between0 to 180, you can see more info in wiki.) Secondly, we split the H channel out, asa single grayscale image, and get its histogram, and normalize it. Thirdly, use“calcBackProject()” function to calculate the back projection of the image.

Let me use an example to explain how we get the back projection.

1 cvtColor(image, hsv, CV_BGR2HSV);2 int ch[]=0,0;3 hue.create(hsv.size(), hsv.depth());4 mixChannels(&hsv, 1, &hue, 1, ch, 1);5 calcHist(&hue, 1, 0, Mat(), hist, 1, &hsize, &phranges);6 normalize(hist, hist, 0,255, CV_MINMAX);7 calcBackProject( &hue, 1, 0, hist, backproj, &phranges, 1,

true );

SearchTo search, type and hit enter

Tag Cloudactive contour Algorithm alpha beta

pruning Armadillo Bilateral Filter BlurC++ camShift Cluster CNN ComputerVision Convolutional Neural Networks

Deep Learning deep networkDynamic Programming GradientDescent image blending Image Processingimage pyramid LeetCodeMachine LearningMatlab meanShift MNIST neuralnetworks NLP Octave OpenCVPanorama PCA Poisson blending Poolingpyramid blending RANSAC Recurrent NeuralNetworks RNNs seam carving SIFT snakeSoftmax Sparse Autoencodertrack UFLDL Unsupervised LearningZCA

PagesAbout Me

CategoriesAlgorithmDynamic ProgrammingGraphicsLeetCodeMachine LearningMathsNLPOpenCVSomething elseTwaddleUncategorizedVision

ArchivesJune 2015April 2015March 2015October 2014September 2014July 2014

Eric Yuan's BlogPerstando et Praestando

Home About Me

If this is our input image, we can see it is a colorful mosaic picture.As we talked above, transform the picture into HSV space and here is the huechannel.

This is its histogram looks like.

“calcBackProject()” function actually calculate the weight of each color in thewhole picture using histogram, and change the value of each pixel to the weightof its color in whole picture. For instance, if one pixel’s color is, say yellow, andthe color yellow’s weight in this picture is 20%, that is, there are 20% of pixels’color in the whole picture is this kind of yellow, we change this pixel’s value fromyellow to 0.2 (or 0.2*255 if using integer), by doing this method to all pixels, weget the back projection picture.

June 2014May 2014April 2014March 2014February 2014January 2014November 2013October 2013September 2013

FriendsJeky Cui's blogQi Cai's Blog

2. MeanShiftWhat is meanShift? MeanShift is nothing but an algorithm which finding modesin a set of data samples representing an underlying probability density function(PDF) in R^N. It is a nonparametric clustering technique which does not requireprior knowledge of the number of clusters, and does not constrain the shape ofthe clusters. I will not show you those annoying formulae, I’ll show you how itworks by graph and words.

Imagine we are in a space, and the dimension of this space is d, (of course, dmaybe bigger than 2) and there are a lot of points in this space, what we aregoing to do is clustering these points. We now make a sphere which center is anyof the points, and radius is, say, h. Because we are in a high­dimensional space,so what we just made is a high­dimensional sphere. By now, every single pointinside this space can be seen as a vector (directed line), and the sum(normalized) of these vectors is what we called mean shift. By this mean shiftvector, we can get the current mass center.

In c++, the current mass center can be calculated like this:

Above is the core of meanShift algorithm, so the whole algorithm is:a. Initialize the sphere, including the center and radius.

01 M00 = 0.0;02 M10 = 0.0;03 M01 = 0.0;04 for(int i = 0; i < probmap.height; i++)05 for(int j = 0; j < probmap.width; j++)06 M00 += probmap.at(i, j);07 M10 += i * probmap.at(i, j);08 M01 += j * probmap.at(i, j);09 10 11 Center_x = M01 / M00;12 Center_y = M10 / M00;

b. Calculate the current mass center.c. Move the sphere’s center to mass center.

d. repeat step b and c, until converge, that is, current mass center after calculate,is the same point with center of sphere.

In OpenCV, meanShift function is something like this:

In which, imgProb is 2D object probability distribution, which is the result of backprojection above; windowIn is CvRect of window initial size; numIters meansthat if the algorithm iterates this many times, stop, that prevents in some casesthe function just repeats and repeats; windowOut is the location, height andwidth of converged window.

3. TrackThe last step is tracking, if we have a video, or frames captured by our webcamera, what we need to do is just use meanShift algorithm to every single frame,and the initial window of each frame is just the output window of the priorframe.By using OpenCV camshift() function, we can get a RotatedRect, which isdefined in OpenCV like:

1 CV_IMPL int2 cvMeanShift( const void* imgProb, CvRect windowIn,3 CvTermCriteria criteria, CvConnectedComp*

comp );

01 class RotatedRect02 03 public:04 // constructors05 RotatedRect();06 RotatedRect(const Point2f& _center, const Size2f& _siz

e, float _angle);07 RotatedRect(const CvBox2D& box);08 // returns minimal up‐right rectangle that contains th

e rotated rectangle

Interesting, except center, size, we can also get an angle of the rectangle, whichmeans we can track the orientation of our target, this is a very useful feature.

TestingFirst, I tried to track my Kendama balls, it works well.

Moreover, see, I’m drawing with my face!

09 Rect boundingRect() const;10 // backward conversion to CvBox2D11 operator CvBox2D() const;12 // mass center of the rectangle13 Point2f center;14 // size15 Size2f size;16 // rotation angle in degrees17 float angle;18 ;

« Can we make a Transparent screen? Seam Carving »

This entry was posted in Algorithm, OpenCV and tagged camShift, meanShift, OpenCV.Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

16 Comments

saraPosted April 20, 2014 at 11:10 am | Permalink

hi i want to ask a question can i use this to find goalpost in soccer video ???

Reply

EricPosted April 20, 2014 at 1:46 pm | Permalink

Hey Sara,

I don’t think it’s a good way to find goalpost using mean shift. It’s nice fortracking object which you already recognized, but can hardly findsomething new. For your situation, maybe you should extract features ofgoalpost, as soon as you find it, use mean shift for tracking it.

Reply

Lakshay Garg

Posted November 5, 2014 at 5:06 pm | Permalink

Try reading about the Hough Line transform

Reply

hassanPosted December 18, 2014 at 6:40 am | Permalink

you must search of morphological operations to find the skeleton(basicshape) of the object. In your case it is a goal post so you shall have askeleton of that and perform morphological operations for that.You haveto work a little.Regards

Reply

FarazPosted May 11, 2014 at 1:12 pm | Permalink

I am using opencv code for mean shift calculation but if the object is not in the screenthe program crashes .how to solve this?

Reply

GeorgePosted August 26, 2014 at 3:59 pm | Permalink

In your code, line 6 includes I(i, j) which is never defined. Bradski calls it the “pixel(probability) value.” How do you calculate it?Thank you.

Reply

EricPosted August 27, 2014 at 11:27 pm | Permalink

I edited that code for more clear, yes it is actually a probability map, wecan get it by doing Back Projection. CheckOpenCV/samples/cpp/camshiftdemo.cpp

Reply

mohammadPosted December 15, 2014 at 2:02 pm | Permalink

do have any idea about speed tracking of object in every frame.thank you .

Reply

EricPosted December 15, 2014 at 4:48 pm | Permalink

Hi, for speed calculating, we should know the world coordination, meansthe camera should be calibrated, then we can figure out the speed ofobjects by project the distance in image into real world coordination (we

should also know the time between the two images have been taken, say,40ms). Maybe LK method can do the distance calculation part well.

Reply

ShilohPosted January 18, 2015 at 9:09 am | Permalink

Hi, is it possible to use cam shift for multiple objects? Can you explain a little bit whatshould I do to achieve that?

Reply

PeerapongPosted February 10, 2015 at 10:16 pm | Permalink

I want to ask you some question. In case of we have only gray scale image, Thisalgorithm can be processed or can’t? thank you

Reply

EricPosted March 10, 2015 at 2:37 pm | Permalink

Hi Peerapong,

I think so, because for 3­channel images, we just use the H channel (hue),because for RGB representation, the correlation among the three channelsis high; however, by using gray scale image, the correlation thing is notnecessarily to be concerned anymore, and we can directly get thehistogram of the image, and do back projection.

Reply

SirajPosted February 16, 2015 at 12:59 am | Permalink

in case of cameshif algorithm, you have colculated only the back projection… but howto use the CamShift algorithm….i have capture vedio from my came and want to trac theface in vedio….how i can do this..???Thanks

Reply

EricPosted March 10, 2015 at 2:40 pm | Permalink

Hey Siraj,

If you are using OpenCV, you may want to checkopencv/samples/cpp/camshiftdemo.cppAlso, there is python version of the code inside python folders

Reply

Powered by WordPress. Built on the Thematic Theme Framework.

BrucePosted March 18, 2015 at 5:00 am | Permalink

Hi Eric,

Have you considered the performance difference between the MeanShift and theCamshift?From my observation, both of their efficiency is not acceptable. I tried them on my i7,8G RAM notebook, that’s very slow for tracking. Anyway, I am considering toimplement it on GPU. Thanks~

Reply

erenPosted May 11, 2015 at 3:56 am | Permalink

Hi Eric,

Do you have the implementation of the tutorial code shared? I would like to try it on myown, if it works or not.

Thanks.

Reply

Post a Comment

Your email is never published nor shared. Required fields are marked *

Name *

Email *

Website

Comment

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronymtitle=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""><s> <strike> <strong>

Post Comment