Intel Open Source Computer Vision Library Version 4.0-Beta Installation and Getting Started Guide for Windows

Troy L. McDaniel

http://www.public.asu.edu/~tmcdani

Center for Cognitive Ubiquitous Computing

Arizona State University

1

Table of Contents

1. Introduction 2

2. Installation 2

2.1. Installing DirectX and DirectX Extras 2

2.2. Installing OpenCV v4.0-Beta 3

3. Setting Up Visual C++ 6.0 For OpenCV 4

3.1. Setting Up the System PATH Environmental Variable 4

3.2. Specifying Directories in Visual C++ 6.0 6

3.3. Creating a Visual C++ Project That Uses OpenCV 9

4. Getting Started With OpenCV 18

5. Troubleshooting 19

5.1. Error: cannot open include file <filename.h> 19

5.2. Error: unresolved external symbol <symbol> 19

5.3. Error: bad number of channels/unsupported format 20

5.4. Missing Files 20

5.5. Other Hints 20

6. Example Programs 20

6.1. Displaying Video From a Web Cam (Web Cam Required) 20

6.2. Edge Detection (Web Cam Required) 21

6.3. Motion Detection (Web Cam Required) 22

6.4. Pyramidal Lucas-Kanade Motion Tracker with Optical Flow Field Visualization

(Web Cam Required) 24

7. Future Work 29

8. Final Words 29

2

1. Introduction

The Intel Open Source Computer Vision Library, or just OpenCV, is a library of image

processing and computer vision algorithms. It’s open source so anybody can contribute to it.

OpenCV utilizes Intel’s Image Processing Library (IPL), which is a set of low-level image

processing functions. OpenCV uses DirectX, which is a set of APIs developed by Microsoft for

creating multimedia applications and games.

OpenCV is quickly gaining popularity for developing real-time computer vision

applications. Some examples of applications include face recognizers, object recognizers, and

motion trackers, just to name a few. The library has especially gained popularity in the computer

vision research community. It allows researchers to get demos or research projects up and

running quickly, and take advantage of the large collection of algorithms that are already

available.

This guide will take you step-by-step through the installation of OpenCV and the setup of

Visual C++ 6.0. This guide covers installation on Windows XP; however, OpenCV may be

installed on other operating systems including Windows 95/98/2000/NT and Linux. Furthermore,

we use here Visual C++ 6.0, but .NET is possible as well. After installation and setup, some

advice on getting started with learning OpenCV is given. Also, this guide offers a

troubleshooting section where some common problems are solved, as well as a section listing

some example programs to help better familiarize you with OpenCV.

2. Installation

This section covers installing DirectX 9.0 SDK Update (December 2004), DirectX 9.0

SDK Update (December 2004) Extras, and OpenCV v4.0-Beta.

2.1. Installing DirectX and DirectX Extras

Required to run OpenCV is Microsoft DirectX 9.0 SDK Update (December 2004) and

Microsoft DirectX 9.0 SDK Update (December 2004) Extras. (Later versions of the update

and extras will work as well.)

Go to http://www.microsoft.com/downloads/details.aspx?FamilyID=4e825a37-0c94-

4421-9ec8-156e52525d11&displaylang=en. The top of the page should read “DirectX 9.0 SDK

Update – (December 2004)”. Click the download button on the right hand side. Click save and

save the file to the desktop, or some temporary folder if desired. A self-extracting zip file called

“dxsdk_dec2004.exe” of size 222 MB will be downloaded. Once this file has successfully

downloaded, double click it and follow the instructions to install it. Next, we’ll install the extras.

Go to http://www.microsoft.com/downloads/details.aspx?FamilyID=a8667cc6-22ca-

4195-a4fa-71dec1f454f9&displaylang=en. The top of the page should read “DirectX 9.0 SDK

Update (December 2004) Extras”. Follow the same instructions as before to download the file. A

self-extracting zip file called “dxsdk_dec2004_extras.exe” of size 35.1 MB will be downloaded.

Once this file has successfully downloaded, double click it and extract the files to a temporary

folder. Created at this location will be a folder called “Extras”. Take this folder and put it in the

3

DirectX 9.0 SDK folder, which by default should have the location C:\Program Files\Microsoft

DirectX 9.0 SDK (December 2004).

This completes the installation of DirectX. Where DirectX is stored is very important

since later we’ll be specifying paths so Visual C++ 6.0 can find DirectX. Next, we began the

installation of Open CV v4.0-Beta.

2.2. Installing OpenCV v4.0-Beta

In this section, we’ll install OpenCV v4.0-Beta. Even though the library will be installed

at the end of this section, we still won’t be able to use it. Using the library requires setting up

Visual Studio to work with it, which takes some work. The main OpenCV site is

http://www.intel.com/research/mrl/research/opencv/. You should familiarize yourself with this

site. Furthermore, joining Yahoo! OpenCV group is a very smart move; it is a major OpenCV

resource.

From the main OpenCV site, click on “Download” under “Featured Content” on the right

hand side. Then click on the download link, i.e., http://sourceforge.net/projects/opencvlibrary/. We’re

now at the OpenCV project page at SourceForge. The top of the page should read “Project: Open

Computer Vision Library: Summary”. Scroll down to “Latest File Releases”. Find the package

called “opencv-win”, and click the download button. Fig. 2-1 shows which file to download.

Figure 2-1: Download List.

On the next page, click “OpenCV_b4a.exe” under “beta4”. Now, choose a close mirror

site to download the file from. With respective to your selected mirror site, click on the picture of

a binary file under the column labeled “Download”. Click save and save the file to a temporary

folder. A self-extracting zip file of size 12.4 MB called “OpenCV_b4a.exe” will be downloaded.

Double click the file and follow the instructions to install OpenCV v4.0-Beta, which

requires at least 36.2 MB. To make life easier, install OpenCV in the default folder, i.e.,

“C:\Program Files\OpenCV”. Leave the check box under additional tasks checked so the system

will add the specified path to the system PATH variable. (Actually, this feature doesn’t seem to

work in the current release of OpenCV, so we’ll go in and add in manually later.) That’s it! In

the next section, we’ll set up Visual C++ 6.0 to work with OpenCV.

4

3. Setting Up Visual C++ 6.0 For OpenCV

We now begin the process of setting up Visual C++ 6.0 to work with OpenCV. The

explanation found here is meant for users with minimal experience in Visual Studio, and thus,

details will be carefully explained. We begin with setting up the system PATH variable.

3.1. Setting Up the System PATH Environmental Variable

Although we selected the option to take care of what we’re about to do during

installation, this feature seems to not be working correctly in the current release of OpenCV. But

that’s not a problem; we’ll just do it manually. From the taskbar, click ”start” and then click

“Control Panel”. Switch to “Classic View” (upper left corner) if the current view is “Category

View”. Once in “Classic View”, double click on the “System” icon to bring up the “System

Properties” dialog box. Click the “Advance” tab. And then click the “Environmental Variables”

button towards the bottom of the dialog (Fig. 3-1).

Figure 3-1: “System Properties” Dialog Box.

The “Environmental Variables” dialog box should now be open. Under the second list,

i.e., “System Variables”, find the variable called “Path”. Select it, and then click “Edit” (Fig. 3-

2).

5

Fig. 3-2: “Environmental Variables” Dialog Box.

Add the path of the OpenCV “bin” folder in the “Variable value” text box. To do this, we

may append the path to the end of the list, making sure that semicolons separate the entries (Fig.

3-3). If the default destination was chosen during installation, your path should be “C:\Program

Files\OpenCV\bin”. If not, find the path to OpenCV’s “bin” folder, and add it in the “Variable

value” text box. Click “OK” on all open dialogs, and close the Control Panel. You must now

restart your computer for the PATH variable to be updated. Now, OpenCV will be able to

find the required DLLs it needs. If you need to run an application on another computer that

doesn’t have OpenCV set up, you’ll have to bring not only the application but the DLLs as

well. This completes setting up the PATH variable. Next, we’ll set up our directory listings.

Figure 3-3: ‘Edit System Variable” Dialog Box.

6

3.2. Specifying Directories in Visual C++ 6.0

Visual C++ 6.0 must be able to find files such as include, source, executable, and library

files. We need to go in and manually tell the IDE where to find the ones it needs.

Start Microsoft Visual C++ 6.0. The programming environment without a loaded

project/file is depicted in Fig. 3-4.

Figure 3-4: Microsoft Visual C++ 6.0 IDE

7

From the menu, click “Tools”, and then “Options…”, (Fig. 3-5). The “Options” dialog

box should now be shown.

Figure 3-5: Go To “Tools” Then “Options…”.

In the “Options” dialog box, find the “Directories” tab and click it. The “Options” dialog

box with the “Directories” tab selected is shown in Fig. 3-6. Currently, we’re looking at the

directories specified for the include files. You can switch to other file types such as library

files by clicking the arrow under “Show directories for”. Right now, let’s leave it on “Include

files”.

Visual C++ should have it’s include files already set up. If there is no folder for the

include files of DirectX 9.0 SDK (December 2004) listed, then specify it. If you chose the

default folder, your path will be the same as shown in Fig. 3-6. To add another entry, click where

the blue highlight is in Fig. 3-6. Don’t type the entry directly; instead, browse for DirectX’s

“include” folder. And finally, move this entry to the top of the list by clicking, holding, and

dragging it. The entry for the DirectX “include” folder must be at the top of the “Include

files” directory list.

8

Figure 3-6: “Options” Dialog Box With the “Directories” Tab Selected

We have more directories to specify. Assuming the default destination folder during

installation, add the following directories. (Your directories may be different if you chose to

install OpenCV in a different location. If so, change the paths accordingly.)

• C:\Program Files\OpenCV\cv\include

• C:\Program Files\OpenCV\cxcore\include

• C:\Program Files\OpenCV\otherlibs\highgui

9

The directories for the needed include files are finished. Next, switch to “Library files”.

The listing for the library files is shown in Fig. 3-7.

Figure 3-7: Library Files Listing.

Visual C++ should have it’s library files already set up. If there is no directory for the

library files of DirectX 9.0 SDK (December 2004) listed, then specify it. If you chose the default

directory, your path will be the same as shown in Fig. 3-7. Go ahead and add it as we did before

if it’s missing. (Don’t forget to move it to the top as well.) Finally, add the directory “C:\Program

Files\OpenCV\lib” to the list of directories.

Switch to “Executable files”. Same as before, check for a DirectX listing. If it is missing,

add “C:\Program Files\Microsoft DirectX 9.0 SDK (December 2004)\Utilities\Bin\x86”, and

move it to the top. Lastly, add “C:\PROGRAM FILES\OPENCV\BIN”.

Specification of the directories is finished. Be sure to click “OK” on the “Options”

dialog box, or you’ll be doing all of this over again. In the next section, we’ll get a project up

and running.

3.3. Creating a Visual C++ Project that Uses OpenCV

Let’s go ahead and get a project up and running. The settings we specify in this section

are specific to a project. If we create another project, these settings will have to be made to that

one as well. However, the settings we made in the previous section, i.e., adding the new

directories, are inherited by all Visual C++ projects.

10

Start Microsoft Visual C++ 6.0. From the menu, click “File” and then click “New…”

(Fig. 3-8).

Figure 3-8: Go to “File” Then “New…”.

11

A dialog box for creating a new file/project/etc. will appear. We’re interested in creating

a new project, so click on the “Projects” tab if it’s not already selected. Several fields are on the

right hand side. The following is all depicted in Fig. 3-9. For “Project name”, enter

“imageViewer”. You may choose any location you wish. We recommend a folder on your

desktop called “imageViewer” to make things simple. Check “Create new workspace” if it’s not

checked. Check “Win32” as the platform if it’s not checked. Now, select “Win32 Console

Application” from the project list on the left, as shown in Fig. 3-9. Finally, click “OK”.

Figure 3-9: Dialog Box For Creating New Files/Project/Etc.

12

A second dialog box will appear asking about what type of console application we wish

to create. Leave “An empty project” selected and click “Finish” (Fig. 3-10). Once “Finish” is

clicked, a skeleton project will be created with no files. Another window will appear informing

you about the successful creation of a skeleton console application project. Click “OK”.

Figure 3-10: “Console Application Type” Dialog Box

13

Before creating the actual application, we need to modify the project settings to our

liking. From the menu, click “Projects” and then click “Settings…” (Fig. 3-11).

Figure 3-11: Go To “Projects” Then “Settings…”.

14

Once “Settings…” is clicked, the “Project Settings” dialog box will appear. Under

“Settings For” at the top left, switch to “All Configurations” (Fig. 3-12).

Figure 3-12: All Configurations.

Click the “Link” tab, and then select “General” for the “Category” (Fig. 3-13).

Figure 3-13: Go To “Link” Then Select “General”.

15

Under “Object/library Modules” (Fig. 3-14), add the following libraries.

• cv.lib

• cxcore.lib

• highgui.lib

• strmbase.lib

Figure 3-14: Object/library modules.

16

Once the libraries have been entered, click “OK”. Now, we’ll add a file to our project.

From the menu, click “File” and then click “New…”, as before. Instead of the “Projects” tab, we

need the “Files” tab, so click it (Fig. 3-15). From the choices, choose “C++ Source File”. On the

right hand side, enter a file name. (We use “imageViewer” as a file name). The location should

already be specified. “Add to project” should already be checked, and set to “imageViewer”. If

not, change it. Once done, click “OK”.

Figure 3-15: Adding a File.

After clicking “OK”, the file should be open and ready for coding. Type the following

code into the source file.

#include <cv.h> /* required to use OpenCV */ #include <highgui.h> /* required to use OpenCV's highgui */ void main(int argc, char *argv[]) { IplImage* image=0; /* pointer to an image */

if(argv[1] != 0) image = cvLoadImage(argv[1], 1); // 1 for color if(image != 0) { cvNamedWindow("Display", CV_WINDOW_AUTOSIZE); // create a window cvShowImage("Display", image); // show image in window cvWaitKey(0); // wait until user hits a key cvDestroyWindow("Display"); } }

17

After entering the code, compile the program and then build the executable, either a

Debug or Release version. To compile, from the menu click “Build” and then click “Compile

<filename.cpp>” (Fig. 3-16.). Once compilation is complete, from the menu click “Build” and

then click “Build <filename.exe>” (Fig.3-16). If you receive errors, either 1) review the program

to make sure it was entered correctly; 2) skim through the guide to see if you missed any steps or

did any steps incorrectly; or 3) visit section 5 for an outline of popular issues.

Figure 3-16: Compiling and Building

To run this program, the command line must be used. From the taskbar, click “start” and

then click “Run…”. Type “cmd”. Using the “change directory” command (“cd”), change your

current directory to the directory holding your executable. Once you’ve reached the correct

folder, type “imageViewer <filename>”, where filename is the file name of the image you wish

to display. Of course you’ll need an image before you can display one. Probably the easiest way

to get an image if you don’t have one is by simply finding one on the Web (Google’s image

search works fine).

Fig. 3-17 depicts the series of commands we issued to display an image called “lena.jpg”.

Remember that you’ll have to set up your project like we did in this section each time you create

a new one (minus the code of course). In the next section, you’ll “learn how to learn” OpenCV.

18

Figure 3-17: Viewing Lena, An Image Made Popular By Image Processing Researchers.

4. Getting Started With OpenCV

By now, we’ve installed and set up OpenCV. We’ve even created a simple project. Now,

it’s time to “learn how to learn” OpenCV. A question every single newcomer asks is “Where do I

begin?”. OpenCV is a very large library, and therefore can quickly become overwhelming. This

section attempts help you get started with OpenCV.

Open the “OpenCV” folder. Next, open the “docs” folder. In this folder are a number of

documents. Begin with “faq.htm”. This document answers a number of popular questions. Next,

visit “index.htm”. This is a very important resource that you’ll be visiting time and time again. In

fact, you should make a shortcut to it on your desktop. Carefully read through “index.htm”. At

the end of “index.htm” are four reference manuals, i.e., “CXCORE”, “CV”, “HighGUI”, and

“CVCAM” reference manuals, and an extensive, however outdated, reference manual for the

entire library. Use the first four reference manuals as your APIs when developing applications.

The outdated manual is good for understanding the algorithms used in OpenCV, but not for use

as an API.

What we recommend is to first read through the “CXCORE” reference manual. This

manual is an excellent starting place. Here, you’ll learn about the basic data structures used by

OpenCV; the different operations on Arrays such as accessing elements, copying Arrays, etc.;

dynamic data structures such as sequences and trees; how to draw shapes or display text; and

other topics as well.

19

The next file to visit is the “HighGUI” reference manual. This API is very useful since it

allows you to use DirectShow without any knowledge of the DirectShow API. Finally, you

should visit the “CV” reference manual. Here you’ll find many of OpenCV’s image processing

and computer vision algorithms, but not all as we’ve found out. If you can’t find an

implementation of an algorithm you’d like to use, check the Yahoo! OpenCV forum, discussed

next, before reinventing the wheel.

The Yahoo! OpenCV forum a truly invaluable resource. It’s very useful for getting help

with OpenCV, or communicating with other computer vision researchers that are using OpenCV.

The last step in our Getting Started section is to visit the “samples” folder, i.e.,

“C:\Program Files\OpenCV\samples\c”. Here, you’ll find a number of useful and interesting

sample programs that you can learn from. The next section gives some advice on

troubleshooting.

5. Troubleshooting

Many things can go wrong whilst setting up Visual C++ 6.0 for OpenCV. Furthermore,

for newcomers it is quite challenging writing the first, somewhat complex, program in OpenCV.

This section attempts to outline many of the problems we experienced during our own process of

learning OpenCV.

5.1. Error: cannot open include file <filename.h>

Obviously, Visual C++ 6.0 cannot find an “include” file. See section 3.2, and make sure

you’ve included those paths mentioned. It’s possible to use other projects besides just these, for

example, “cvcam” or “cvaux”. If you forgot to include the path to a particular project’s “include”

files, you’ll have to add it like we did before. If you’re not sure where an “include” file is, you

may do a search for it (ctrl+f).

5.2. Error: unresolved external symbol <symbol>

Here, there’s a good chance you forgot to include a “library” file in the “Object/library

module” field of the “Project Settings” Dialog Box. See section 3.3, and make sure you’ve used

“strmbase.lib”, “cxcore.lib”, “cv.lib”, and “highgui.lib”, at the minimum. If you want to use

other projects such as “cvcam” or “cvaux”, you’ll of course have to add additional libraries like

we did in section 3.3.

20

5.3. Error: bad number of channels/unsupported format

Popular error many newcomers receive. Basically, the number of channels corresponds to

color channels. For example, 1 is grayscale, 3 is BGR. Certain functions accept only images of a

certain number of channels. Furthermore, functions may accept images of only a particular bit

depth. As an example, Sobel will accept only grayscale images with a bit depth of 16 bits.

5.4. Missing Files

You may be missing a DLL or a library file of a particular project, for example, “cvcam”.

You’ll have to manually go in and compile/build the respective project. These files can be

accessed in the “_make” folder within the “OpenCV” folder. Open up the appropriate workspace

and find the project in question. Make sure you set the project as the active project as well as

setting the configuration of the project to your liking. Now, you’re ready to compile and build.

5.5. Other Hints

Read and make sure you understand the data structures you’re using perfectly.

Furthermore, you should understand precisely what a function accepts, does, and its end result.

Due to seemingly infinite number of problems you might encounter during your use of OpenCV,

we cannot enumerate every situation here. If you cannot find help here, please see the very

helpful “faq.htm”, or the life-saving Yahoo! OpenCV forum.

6. Example Programs

In this section, we’ll cover some example programs to further your knowledge of

OpenCV. Each program requires a web cam or some other video recording device whose video

data can be downloaded in real-time to a computer via a USB cable. The first program is a

simple program to feed in video from a web cam and display it. The next program performs

some spatial processing on the fed in video. The third program performs some temporal

processing on the fed in video. And the fourth program is a tracking application with

visualization of the optical flow field. You should advance in order since the programs are in

order of difficulty. Enjoy!

6.1. Displaying Video From a Web Cam (Web Cam Required)

This simple program will allow video fed in from a web cam to be display on the

monitor. See section 3.3 to learn how to create a project and set it up properly. (Note: This file

need not be run from the command line as we did before.)

21

#include <cv.h> #include <highgui.h> void main(int argc,char *argv[]) { int c; IplImage* color_img; CvCapture* cv_cap = cvCaptureFromCAM(-1); // -1 = only one cam or doesn't matter cvNamedWindow("Video",1); // create window for(;;) { color_img = cvQueryFrame(cv_cap); // get frame if(color_img != 0) cvShowImage("Video", color_img); // show frame c = cvWaitKey(10); // wait 10 ms or for key stroke if(c == 27) break; // if ESC, break and quit } /* clean up */ cvReleaseCapture( &cv_cap ); cvDestroyWindow("Video"); }

6.2. Edge Detection (Web Cam Required)

This program applies a spatial filter, namely the Sobel operator, to detect vertical edges.

As an aside, you may modify the filter if you wish to detect horizontal edges. Furthermore, it’s

possible to use Sobel to detect both types of edges. See section 3.3 to learn how to create a

project and set it up properly. (Note: This file need not be run from the command line as we did

before.)

#include <cv.h> #include <highgui.h> void main(int argc,char *argv[]) { int c; IplImage* color_img=0; IplImage *gray_img=0; IplImage *sobel_img16=0; IplImage *sobel_img8=0; CvCapture* cv_cap = cvCaptureFromCAM(-1); /* create windows */ cvNamedWindow("Video", CV_WINDOW_AUTOSIZE); cvNamedWindow("Edges", CV_WINDOW_AUTOSIZE); /* get a frame and use it to allocate space and define specs for other images */ color_img = cvQueryFrame(cv_cap);

22

if(!color_img) return; gray_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); sobel_img16 = cvCreateImage( cvSize(color_img->width,color_img->height),

IPL_DEPTH_16S, 1); sobel_img8 = cvCreateImage( cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); /* origin (start) of image must be the same */ gray_img->origin = color_img->origin; sobel_img16->origin = color_img->origin;

sobel_img8->origin = color_img->origin; for(;;) {

color_img = cvQueryFrame(cv_cap); // get a frame if(!color_img) break; /* convert to grayscale */ cvCvtColor(color_img, gray_img, CV_BGR2GRAY); /* apply Sobel operator; requires as 16 bit destination */ cvSobel(gray_img, sobel_img16, 1, 0, 3); /* display requires an 8 bit image, so we must convert it */ cvConvertScale(sobel_img16, sobel_img8, 1, 0);

/* display images */ cvShowImage("Edges",sobel_img8); cvShowImage("Video", color_img); c = cvWaitKey(10); // wait for 10 ms or key stroke if(c == 27)

break; // if ESC, break and quit } /* clean up */ cvReleaseCapture(&cv_cap); cvDestroyAllWindows(); }

6.3. Motion Detection (Web Cam required)

This next program performs some temporal processing on video data to detect motion.

This program basically computes the difference between two frames. See section 3.3 to learn

how to create a project and set it up properly. (Note: This file need not be run from the command

line as we did before.)

23

#include <cv.h> #include <highgui.h> #include <math.h> void main(int argc,char *argv[]) { int c, i, j; IplImage* color_img=0; IplImage* gray_img=0; IplImage* prev_img=0; IplImage* curr_img=0; IplImage* difference=0; CvCapture* cv_cap = cvCaptureFromCAM(-1); /* Create windows */ cvNamedWindow("Video", CV_WINDOW_AUTOSIZE); cvNamedWindow("Output", CV_WINDOW_AUTOSIZE); /* Allocate space for other images */ color_img = cvQueryFrame(cv_cap); if(!color_img) return; gray_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); prev_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); curr_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); difference = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); gray_img->origin = color_img->origin; prev_img->origin = color_img->origin; curr_img->origin = color_img->origin; difference->origin = color_img->origin; /* convert to grayscale */ cvCvtColor( color_img, gray_img, CV_BGR2GRAY ); /* get a current image now so we'll have a previous image when we begin */ cvCopy(gray_img, curr_img, 0); for(;;) { color_img = cvQueryFrame(cv_cap); // get a frame if(!color_img) break;

/* convert to grayscale */ cvCvtColor( color_img, gray_img, CV_BGR2GRAY ); /* current becomes previous and new frame because current */ cvCopy(curr_img, prev_img, 0); cvCopy(gray_img, curr_img, 0);

24

/* compute the difference */ for(i = 0; i < color_img->height; i++) for(j = 0; j < color_img->width; j++) (difference->imageData + i*color_img->width)[j] = abs((prev_img->imageData + i*color_img->width)[j] - (curr_img->imageData + i*color_img->width)[j]); /* show images */ cvShowImage("Video", color_img); cvShowImage("Output", difference); c = cvWaitKey(10); // wait for 10 ms or keystroke if(c == 27) // ESC break; } /* clean up */ cvReleaseCapture(&cv_cap); cvDestroyWindow("Video"); cvDestroyWindow("Output"); }

6.4. Pyramidal Lucas-Kanade Motion Tracker with Optical Flow Field Visualization (Web cam Required)

Tracker based on motion using a famous algorithm. Click on the video to create “track

balls”. The display with the arrows is the optical flow field. Bigger arrows indicate larger

displacements/velocities. See section 3.3 to learn how to create a project and set it up properly.

(Note: This file need not be run from the command line as we did before.)

#include <cv.h> #include <highgui.h> #include <math.h> #include <stdio.h> const int MAX_COUNT = 2000; int count = 0, startOfTrackPoints = 0; CvPoint2D32f* points[2] = {0,0}; IplImage *color_img=0; /* Listener for mouse clicks */ void OnMouseClick(int event, int x, int y, int flags) {

if(color_img->origin) y = color_img->height - y; if(event == CV_EVENT_LBUTTONDOWN && count < MAX_COUNT) { points[1][count] = points[0][count] = cvPoint2D32f(x, y); // Add point count++; } }

25

void main(int argc,char *argv[]) { int c, i, j, step, win_size, flags, color; char* status = 0; CvCapture* cv_cap = cvCaptureFromCAM(-1); IplImage *gray_img, *prev_img, *curr_img; IplImage *pyramid, *prev_pyramid, *optical_flow_field; CvPoint p1, p2, p3, p4, p5; printf("Welcome to Troy's LK demo.\n"); printf("Click on the 'Video' window to add trackballs.\n"); printf("The 'Field' window shows the optical flow with larger \n"); printf("arrows corresponding to larger displacements.\n\n"); printf("Keyboard commands:\n"); printf(" ESC to quit\n"); printf(" c to clear all the trackballs\n"); printf(" n to cycle through the different trackball colors\n"); /* Initialization and space allocation */ color_img = cvQueryFrame(cv_cap); if(!color_img) return; gray_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); optical_flow_field = cvCreateImage( cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); prev_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); curr_img = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); pyramid = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); prev_pyramid = cvCreateImage(cvSize(color_img->width,color_img->height),

IPL_DEPTH_8U, 1); gray_img->origin = color_img->origin; optical_flow_field->origin = color_img->origin; prev_img->origin = color_img->origin; curr_img->origin = color_img->origin; pyramid->origin = color_img->origin; prev_pyramid->origin = color_img->origin; points[0] = (CvPoint2D32f*)cvAlloc(MAX_COUNT*sizeof(points[0][0])); points[1] = (CvPoint2D32f*)cvAlloc(MAX_COUNT*sizeof(points[0][0])); status = (char*)cvAlloc(MAX_COUNT); flags = 0; step = 10; win_size = 10; color = 0;

26

/* Create windows */ cvNamedWindow("Video", CV_WINDOW_AUTOSIZE); cvNamedWindow("Field", CV_WINDOW_AUTOSIZE); /* Set up mouse event listener */ cvSetMouseCallback("Video", OnMouseClick); /* Initialize the points to the points on the image separated by distances of 10 pixels */ for(i = 0; i < color_img->height; i+=step) { for(j = 0; j < color_img->width; j+=step) { points[0][count] = cvPoint2D32f(j, i); points[1][count] = cvPoint2D32f(j, i); count++; // count the number of points } } /* Where track points begin */ startOfTrackPoints = count; /* Use the "first" frame */ color_img = cvQueryFrame(cv_cap); if(!color_img) return; cvCvtColor( color_img, gray_img, CV_BGR2GRAY ); /* Smooth to improve accuracy */

cvSmooth(gray_img, gray_img, CV_GAUSSIAN, 3, 0, 0); cvSmooth(gray_img, gray_img, CV_GAUSSIAN, 3, 0, 0); cvSmooth(gray_img, gray_img, CV_GAUSSIAN, 3, 0, 0); cvCopy(curr_img, prev_img, 0); cvCopy(gray_img, curr_img, 0); /* Begin the main infinite loop until break */ for(;;) { color_img = cvQueryFrame(cv_cap); if(!color_img) break; cvCvtColor( color_img, gray_img, CV_BGR2GRAY );

/* Smooth to improve accuracy */ cvSmooth(gray_img, gray_img, CV_GAUSSIAN, 3, 0, 0);

cvSmooth(gray_img, gray_img, CV_GAUSSIAN, 3, 0, 0); cvSmooth(gray_img, gray_img, CV_GAUSSIAN, 3, 0, 0); cvCopy(curr_img, prev_img, 0); cvCopy(gray_img, curr_img, 0); cvCopy(pyramid, prev_pyramid, 0); for(i = startOfTrackPoints; i < count; i++) points[0][i] = points[1][i];

27

/* Calculate the optical flow using pyramidal iterative LK */ cvCalcOpticalFlowPyrLK(prev_img, curr_img, prev_pyramid, pyramid, points[0], points[1], count, cvSize(win_size, win_size), 3, status, 0, cvTermCriteria(CV_TERMCRIT_ITER|CV_TERMCRIT_EPS,20,0.03), flags);

/* Threshold to ignore noise */ for(i = startOfTrackPoints; i < count; i++) if(abs((int)(points[0][i].x - points[1][i].x)) > 50 ||

abs((int)(points[0][i].y - points[1][i].y)) > 50) points[1][i] = points[0][i]; /* Set flags */ flags |= CV_LKFLOW_PYR_A_READY; /* Refresh the optical flow field */ for (i = 0; i < optical_flow_field->height; i++) for (j = 0; j < optical_flow_field->width; j++) (optical_flow_field->imageData+

optical_flow_field->widthStep*i)[j] = (char)255; /* Draw the optical flow field; the greater the velocity, the bigger the respective arrow and arrowhead will be for better visualization */ for (i = 0; i < startOfTrackPoints; i++) { /* Threshold velocities: not drawing very small or very large velocities */

if((abs((int)(points[0][i].x - points[1][i].x)) >= 1 || abs((int)(points[0][i].y - points[1][i].y)) >= 1) &&

(abs((int)(points[0][i].x - points[1][i].x)) <= 20 && abs((int)(points[0][i].y - points[1][i].y)) <= 20)) {

/* Arrow */ cvLine(optical_flow_field, cvPointFrom32f(points[0][i]), cvPointFrom32f(points[1][i]), cvScalar(0, 0, 0,0), 1, 8, 0); /* Arrowhead */ p1 = cvPoint((int)(cvPointFrom32f(points[0][i]).x +

0.7*(cvPointFrom32f(points[1][i]).x - cvPointFrom32f(points[0][i]).x)), (int)(cvPointFrom32f(points[0][i]).y + 0.7*(cvPointFrom32f(points[1][i]).y - cvPointFrom32f(points[0][i]).y)));

p2 = cvPoint((int)(cvPointFrom32f(points[0][i]).x + 0.7*(cvPointFrom32f(points[1][i]).x -

cvPointFrom32f(points[0][i]).x)), (int)cvPointFrom32f(points[1][i]).y);

p3 = cvPoint((int)cvPointFrom32f(points[1][i]).x, (int)(cvPointFrom32f(points[0][i]).y +

0.7*(cvPointFrom32f(points[1][i]).y - cvPointFrom32f(points[0][i]).y)));

p4 = cvPoint((int)(p1.x + 0.5*(p2.x-p1.x)), (int)(p1.y + 0.5*(p2.y-p1.y)));

p5 = cvPoint((int)(p1.x + 0.5*(p3.x-p1.x)), (int)(p1.y + 0.5*(p3.y-p1.y)));

cvLine(optical_flow_field, cvPointFrom32f(points[1][i]),

28

p4, cvScalar(0, 0, 0,0), 1, 8, 0); cvLine(optical_flow_field, cvPointFrom32f(points[1][i]),

p5, cvScalar(0, 0, 0,0), 1, 8, 0); } else { /* Draws a dot */ cvLine(optical_flow_field, cvPointFrom32f(points[0][i]), cvPointFrom32f(points[0][i]), cvScalar(0, 0, 0,0), 1, 8, 0); } } for(i = startOfTrackPoints; i < count; i++) { /* Draw trackballs */ if(color == 0) cvCircle(color_img, cvPoint((int)points[1][i].x, (int)points[1][i].y),

5, cvScalar(0,255,0,0), 3, 8, 0); else if(color == 1) cvCircle(color_img, cvPoint((int)points[1][i].x, (int)points[1][i].y),

5, cvScalar(255,0,0,0), 3, 8, 0); else cvCircle(color_img, cvPoint((int)points[1][i].x, (int)points[1][i].y),

5, cvScalar(0,255,255,0), 3, 8, 0); } /* Show images */ cvShowImage("Video", color_img); cvShowImage("Field", optical_flow_field); /* Wait */ c = cvWaitKey(10); /* Check for ESC */ if(c == 27) break; else if(c == 'c') startOfTrackPoints = count; else if(c == 'n') { color++; color = color % 3; }

} /* Release windows */ cvReleaseCapture(&cv_cap); cvDestroyWindow("Video"); cvDestroyWindow("Field"); }

29

7. Future Work

As newer versions of OpenCV appear, we’ll try to keep this guide updated as much as

possible. Also, we’ll continue expanding the troubleshooting section as we run into other

problems many others are experience. Problems will include not only installation and setup

problems, but problems dealing with computer vision algorithms in the OpenCV library.

8. Final Words

We hope this guide has helped you. If we help at least one person, then it was worth

writing. Thanks and happy OpenCV-ing!