wacv2012: tutorial: introduction to pyvision for computer vision applications
TRANSCRIPT
![Page 1: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/1.jpg)
Introduction to PyVision for Computer Vision Applications
David Bolme and Stephen O’Hara
Thursday, January 12, 12
![Page 2: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/2.jpg)
System Architecture
PyVision
Applications:Face Recognition
OpenCVScipyPIL
Python
Sessions 3 & 4
Session 2
Session 1
2
Applications:Video Analytics
Thursday, January 12, 12
![Page 3: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/3.jpg)
System Architecture
PyVision
Applications:Face Recognition
OpenCVScipyPIL
Python
Sessions 3 & 4
Session 2
Session 1
3
Applications:Video Analytics
Thursday, January 12, 12
![Page 4: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/4.jpg)
Session 1 Goals
• Install Virtual Box and Appliance.
• Hands on help.
• Introduction to Python for computer vision.
• PIL, NumPy, SciPy, OpenCV
4
Thursday, January 12, 12
![Page 5: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/5.jpg)
Installation Flash Drive
• Virtual Box for Windows, Mac OS, and Linux
• PyVision Virtual Appliance
• Tutorial Slides
5
Thursday, January 12, 12
![Page 6: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/6.jpg)
Virtual Box Appliance
6
https://www.virtualbox.org/wiki/Downloads
• Ubuntu Linux
• python, scipy, numpy, pil,
• Eclipse with pydev&subclipse
• FireFox
• PyVision
• CSU Face Baseline
• iPython + Html Notebook
• R (Statistics) Thursday, January 12, 12
![Page 7: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/7.jpg)
Things to know...
7
• Username: pyvision
• Password: pyvision• Ubuntu 11.10
• 32bit Single Processor
• 1 GB Ram
• 16 GB Hard Drive
Thursday, January 12, 12
![Page 8: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/8.jpg)
Installation Requirements
• Python (2.7 recommended)
• Python Imaging Library (PIL)
• NumPy and SciPy
• OpenCV (ver 2.2 or 2.3)
• PyVision
• Optional:
• IPython
• Matplotlib
PyVision
8
Thursday, January 12, 12
![Page 9: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/9.jpg)
Hands On Installation
Thursday, January 12, 12
![Page 10: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/10.jpg)
Quick Introduction Python
Thursday, January 12, 12
![Page 11: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/11.jpg)
Benefits of Python• Similar syntax/functionality to MATLAB through Scipy
• Supports modern programming language constructs
• Interfaces to OpenCV, LibSVM, and many other open source libraries
• Quick and easy prototyping
• Free
11
Thursday, January 12, 12
![Page 12: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/12.jpg)
Popularity• Python is one of the most
popular and fastest-growing dynamic languages
• Graphic is from:http://blog.revolutionanalytics.com/2010/12/programming-languages-ranked-by-popularity.html
12
Thursday, January 12, 12
![Page 13: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/13.jpg)
File: TutorialControl.py
def foo(a,b): ''' A function that adds two numbers ''' return a + b
# Count from 0 to 9i = 0while i < 10:
print "Number:",i,
if i % 2 == 0: print "even" else: print "odd"
i = foo(i, 1)
Indentation / Control
• Indentation determines block structure.
• Use colon “:” instead of braces
• Set your text editor to use spaces instead of tabs.
• Reference Counting / GC
13
Thursday, January 12, 12
![Page 14: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/14.jpg)
Results
Number: 0 evenNumber: 1 oddNumber: 2 evenNumber: 3 oddNumber: 4 evenNumber: 5 oddNumber: 6 evenNumber: 7 oddNumber: 8 evenNumber: 9 odd
14
Thursday, January 12, 12
![Page 15: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/15.jpg)
File: TutorialMain.py
def add(a,b): return a + b
class Add: def __init__(self,a,b): self.val = a+b
def getValue(self): return self.val
if __name__ == '__main__': # execute this code if this # is the main script print "Hello World!!!" print "2 + 3 =",add(2,3) my_obj = Add(2,3) print "Add(2,3)=", my_obj.getValue()
The “main” script
• Basic script structure.
• Executes from top to bottom.
• “__main__” if statement
• Arguments: sys.argv
• Functions “def”
• Classes “class”
• “self” parameter
15
Thursday, January 12, 12
![Page 16: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/16.jpg)
Results
Hello World!!!2 + 3 = 5Add(2,3)= 5
16
Thursday, January 12, 12
![Page 17: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/17.jpg)
File: TutorialTypes.py
if __name__ == '__main__': print "2 + 3 =", 2 + 3 print "2. + 3. =", 2. + 3. print "'2' + '3' =", '2' + '3' print "(2,) + (3,) =", (2,) + (3,) print "[2] + [3] =", [2] + [3] print "dictionary:", {'two':2,3:'three',(2,3):5} print "int('2') + 3 =", int('2') + 3 print "'2' + 3 =", '2' + 3
Data and Types
• object - myobj = MyClass()
• int - 1
• float - 1.0
• str / buffer - “Hello World”
• list - [1,2.0,”three”,myobj]
• dict - {“key”:”val”, 203:myobj}
17
Thursday, January 12, 12
![Page 18: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/18.jpg)
Results2 + 3 = 52. + 3. = 5.0'2' + '3' = 23(2,) + (3,) = (2, 3)[2] + [3] = [2, 3]dictionary: {3: 'three', (2, 3): 5, 'two': 2}int('2') + 3 = 5'2' + 3 =Traceback (most recent call last): File "/Users/bolme/Documents/workspace/FaceRec/src/experiments/tutorials/TutorialTypes.py", line 9, in <module> print "'2' + 3 =", '2' + 3TypeError: cannot concatenate 'str' and 'int' objects
18
Thursday, January 12, 12
![Page 19: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/19.jpg)
File: TutorialHelp.py
import numpy as np
a = np.array([1.,2.,3.,4.])
print a
print type(a)
print dir()
print dir(a)
help(a)
Introspection and Help
• print - print object info
• type(object) - get the object type.
• dir() - list variables, functions, etc in current scope.
• dir(object) - list members/methods of object.
• help(object/module) - get help on an object, function, or module.
19
Thursday, January 12, 12
![Page 20: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/20.jpg)
Results[ 1. 2. 3. 4.]<type 'numpy.ndarray'>['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'a', 'np']['T', '__abs__', '__add__', '__and__', ..., 'all', 'any', 'argmax', 'argmin',...]Help on ndarray object:
class ndarray(__builtin__.object) | ndarray(shape, dtype=float, buffer=None, offset=0, | strides=None, order=None) | | An array object represents a multidimensional, homogeneous array | of fixed-size items. An associated data-type object describes the | format of each element in the array (its byte-order, how many bytes it | occupies in memory, whether it is an integer, a floating point number, | or something else, etc.)20
Thursday, January 12, 12
![Page 21: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/21.jpg)
Matrix Manipulation• Numpy is the numeric python library
• Scipy has additional scientific programming packages, is superset of Numpy
• “ndarray” type, optional “matrix” type
• Scipy linalg package
• http://www.scipy.org/NumPy_for_Matlab_Users
21
Thursday, January 12, 12
![Page 22: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/22.jpg)
iPython and PyLab
• iPython is an enhanced interactive python interpreter
• iPython Notebook
• PyLab is built on iPython and aims to be an interactive workspace for scientific programming
• Matplotlib is a MATLAB-syntax plotting facility for python
22
Thursday, January 12, 12
![Page 23: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/23.jpg)
Interactive Demonstration
23
Thursday, January 12, 12
![Page 24: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/24.jpg)
Standard Library Highlights• Operating System - os, os.path
• Shell - shutil
• Binary Data - struct
• Math - math, cmath, random
• Object Serialization - pickle
• XML - ElementTree,dom,sax
• DB/Tables - csv, bsddb,sqlite3
• Compression: zlib,bz2,zipfile,tarfile
• Security: md5,sha,ssl
• Time: time, calendar, ...
• Multiple CPU: multiprocessing
• Networking: socket
• Web: urllib, email, htmllib, ftplib
• Other: unittest, string, copy, sys
24
Thursday, January 12, 12
![Page 25: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/25.jpg)
Third Party Libraries
• Interfaces to C, Java, Matlab, R ...
• Web services, Databases, Networking, XML ...
• Scientific Computing, Machine Learning, cuda ...
• GUI: wxPython, Qt, Gnome, Cocoa, Windows...
• Bindings to most popular open source resources.
25
Thursday, January 12, 12
![Page 26: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/26.jpg)
System Architecture
26
PyVision
Applications:Face RecognitionSessions 3 & 4
Session 2
Applications:Video Analytics
OpenCVScipyPIL
PythonSession 1
Thursday, January 12, 12
![Page 27: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/27.jpg)
Session 2 Goals
• Introduction to PyVision
• Basic datatypes
• Tools for understanding algorithms
• Using NumPy / SciPy
• Using OpenCV
27
Thursday, January 12, 12
![Page 28: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/28.jpg)
Face Detection in OpenCV#!/usr/bin/python"""This program is demonstration for face and object detection using haar-like features.The program finds faces in a camera image or video stream and displays a red box around them.
Original C implementation by: ?Python implementation by: Roman Stanchak, James Bowman"""import sysimport cvfrom optparse import OptionParser
# Parameters for haar detection# From the API:# The default parameters (scale_factor=2, min_neighbors=3, flags=0) are tuned # for accurate yet slow object detection. For a faster operation on real video # images the settings are: # scale_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING, # min_size=<minimum possible face size
min_size = (20, 20)
image_scale = 2haar_scale = 1.2min_neighbors = 2haar_flags = 0
def detect_and_draw(img, cascade): # allocate temporary images gray = cv.CreateImage((img.width,img.height), 8, 1) small_img = cv.CreateImage((cv.Round(img.width / image_scale),! ! ! cv.Round (img.height / image_scale)), 8, 1)
# convert color input image to grayscale cv.CvtColor(img, gray, cv.CV_BGR2GRAY)
# scale input image for faster processing cv.Resize(gray, small_img, cv.CV_INTER_LINEAR)
cv.EqualizeHist(small_img, small_img)
if(cascade): t = cv.GetTickCount() faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0), haar_scale,
28
Thursday, January 12, 12
![Page 29: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/29.jpg)
Face Detection in OpenCV cascade, cv.CreateMemStorage(0), haar_scale, min_neighbors, haar_flags, min_size) t = cv.GetTickCount() - t print "detection time = %gms" % (t/(cv.GetTickFrequency()*1000.)) if faces: for ((x, y, w, h), n) in faces: # the input to cv.HaarDetectObjects was resized, so scale the # bounding box of each face and convert it to two CvPoints pt1 = (int(x * image_scale), int(y * image_scale)) pt2 = (int((x + w) * image_scale), int((y + h) * image_scale)) cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)
cv.ShowImage("result", img)
if __name__ == '__main__': print "hello world"
parser = OptionParser(usage = "usage: %prog [options] [filename|camera_index]")
parser.add_option("-c", "--cascade", action="store", dest="cascade", type="str", help="Haar cascade file, default %default", default = "../data/haarcascades/haarcascade_frontalface_alt.xml") (options, args) = parser.parse_args()
print "load cascade" cascade = cv.Load(options.cascade) print "Print help" if len(args) != 1: parser.print_help() sys.exit(1)
input_name = args[0] print input_name if input_name.isdigit(): capture = cv.CreateCameraCapture(int(input_name)) else: capture = None
cv.NamedWindow("result", 1)
if capture: frame_copy = None
29
Thursday, January 12, 12
![Page 30: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/30.jpg)
Face Detection in OpenCV while True: frame = cv.QueryFrame(capture) if not frame: cv.WaitKey(0) break if not frame_copy: frame_copy = cv.CreateImage((frame.width,frame.height), cv.IPL_DEPTH_8U, frame.nChannels) if frame.origin == cv.IPL_ORIGIN_TL: cv.Copy(frame, frame_copy) else: cv.Flip(frame, frame_copy, 0) detect_and_draw(frame_copy, cascade)
if cv.WaitKey(10) >= 0: break else: image = cv.LoadImage(input_name, 1) detect_and_draw(image, cascade) cv.WaitKey(0)
cv.DestroyWindow("result")
30
Thursday, January 12, 12
![Page 31: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/31.jpg)
File: TutorialFaceDetect.py
import pyvision as pvimport pyvision.face.CascadeDetector as cd
if __name__ == '__main__': detector = cd.CascadeDetector() cam = pv.Webcam() while True: frame = cam.query() rects = detector(frame) for rect in rects: frame.annotateRect(rect) frame.show()
Face Detection Demo• PyVision Philosophy:
• PyVision is designed for researchers.
• Algorithms should have simple interfaces and intelligent defaults.
• Support standard datatypes.
• Using OpenCV, SciPy, and PIL together should be easy.
• Results should be easy to understand and debug.
31
Thursday, January 12, 12
![Page 32: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/32.jpg)
File: TutorialEyeDetect.py
import pyvision as pvimport pyvision.face.CascadeDetector as cdimport pyvision.face.FilterEyeLocator as ed
face_detect = cd.CascadeDetector()eye_detect = ed.FilterEyeLocator()
im = pv.Image("face.png",bw_annotate=True)
faces = face_detect(im)eyes = eye_detect(im,faces)
for face,eye1,eye2 in eyes: im.annotatePolygon(face.asPolygon(), width=4) im.annotatePoints([eye1,eye2]) im.show(delay=0)
Eye Detection
• Read image from disk: pv.Image().
• bw image to make annotations stand out.
• Thicker detection rectangle using Polygon and width=4.
• Also detect eyes.
32
Thursday, January 12, 12
![Page 33: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/33.jpg)
Result
33
Thursday, January 12, 12
![Page 34: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/34.jpg)
What PyVision Provides• Read and convert common data types: video, image,
matrix, rects, points, ...
• Common computer vision functions: preprocessing, transforms, detectors, interest points, motion detection, surf, lda, pca, svm, ...
• Analysis and Visualization: Annotation, Plots, Logs, Montage...
• Integration with OpenCV34
Thursday, January 12, 12
![Page 35: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/35.jpg)
PyVision Directory Structure
Thursday, January 12, 12
![Page 36: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/36.jpg)
Points, Rects, Images, Videos, ...
36
• pv.Point - A point (x,y,[z,[w]])
• pv.Rect - A rect (x,y,w,h)
• pv.Image - JPG, PNG, TIF, ...
• pv.ImageBuffer - Set of images
• pv.Video - AVI, MOV, M4V, IP network cameras
• pv.Webcam - USB Webcams
• and other classes that implement a video interface...
Thursday, January 12, 12
![Page 37: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/37.jpg)
PyVision Image Class
37
• Easily convert to common formats
• Maintain annotations separate from source image
• Convenience methods for loading, saving, displaying, resizing, cropping, and other common operations
Thursday, January 12, 12
![Page 38: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/38.jpg)
import pyvision as pvimport PIL, cvilog = pv.ImageLog()
im = pv.Image("baboon.jpg")
pil = im.asPIL()gray = pil.convert('L')thresh = PIL.Image.eval(gray, lambda x: 255*(x>127.5) ) ilog(pv.Image(thresh),"PILThresh")
mat = im.asMatrix2D() thresh = mat > 127.5ilog(pv.Image(1.0*thresh),"ScipyThresh")
cvim = im.asOpenCVBW()dest=cv.CreateImage(im.size,cv.IPL_DEPTH_8U,1)cv.CmpS(cvim,127.5,dest,cv.CV_CMP_GT)ilog(pv.Image(dest),"OpenCVThresh")
ilog.show()File: TutorialThresh.py
PIL, SciPy, and OpenCV
• Use im.as<Format> to get PIL, Scipy, and OpenCV images.
• Perform operations using preferred library.
• Convert back using pv.Image()
• Note: Scipy format matrices are transposed so that mat[x,y] correspond to the x and y image axis.
38
Thursday, January 12, 12
![Page 39: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/39.jpg)
Results
39
PIL SciPy OpenCV
Thursday, January 12, 12
![Page 40: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/40.jpg)
File: TutorialAffine.py
import pyvision as pv
if __name__ == '__main__': im = pv.Image("face.png") eye1,eye2 = pv.Point(140,165),... out1,out2 = pv.Point(64,128),... im.annotatePoints([eye1,eye2]) im.show(delay=0) affine = pv.AffineFromPoints(eye1,eye2, out1,out2,(256,320)) tile = affine(im) tile.show(delay=0) affine = pv.AffineRotate(3.1415,(256,320), center=pv.Point(128,160))*affine; tile = affine(im) tile.show(delay=0)
Affine Transformations
• affine = pv.AffineTransform(matrix,size)
• new_im = affine(old_im)
• new_pts = affine(old_pts)
• both = affine1*affine2
• affine.invert(pts)
• Helper Functions: pv.AffineFromPoints, pv.AffineFromRect, pv.Affine[Scale,Rotate,Trans...]
• pv.PerspectiveTransform40
Thursday, January 12, 12
![Page 41: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/41.jpg)
Results
41
Thursday, January 12, 12
![Page 42: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/42.jpg)
Annotation and Logging Results
Thursday, January 12, 12
![Page 43: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/43.jpg)
File: --Example.py--
import pyvision as pvimport scipy as spif __name__ == '__main__': im = pv.Image(sp.zeros((128,128)))
pts = [pv.Point(48,55),pv.Point(80,55)] im.annotatePoints(pts) elipse = pv.CenteredRect(64,64,96,96) im.annotateEllipse(elipse) im.annotatePolygon([pv.Point(48,90), pv.Point(80,90),pv.Point(64,100)]) im.annotateLabel(pv.Point(40,36),"MMM") im.annotateLabel(pv.Point(72,36),"MMM") im.annotateLabel(pv.Point(58,64),"db") im.show(delay=0)
Image Annotation
• Implemented in PIL.
• Annotate images with points, rects, circles, ellipses, polygons, lines, and labels.
• A separate copy of the image is created within the object just for annotations.
• Supports colors and other drawing options: color = “red” or “#FF0000”
43
Thursday, January 12, 12
![Page 44: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/44.jpg)
Result
44
Thursday, January 12, 12
![Page 45: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/45.jpg)
File: TutorialLogsPlots.py
import pyvision as pv
ilog = pv.ImageLog()im = pv.Image("baboon.jpg")ilog(im,"Baboon")
table = pv.Table()table[1,"image"] = im.filenametable[1,"width"] = im.size[0]table[1,"height"] = im.size[1]ilog(table,"ImageData")print table
plot = pv.Plot(title="Some Dots and Lines");plot.points([[3.5,7.1],[1,1],[5.5,2]],shape=2)plot.lines([[5.5,7.5],[2,3],[3.3,7]])ilog(plot,"MyPlot")
ilog.show()
Logs, Tables, Timers
• pv.ImageLog - A collection of images, tables, plots, etc that is saved to disk for later analysis.
• pv.Table - Tabular data that support pretty printing and csv.
• pv.Timer - Time functions and processes.
• pv.Plot - line and scatter plots.
45
Thursday, January 12, 12
![Page 46: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/46.jpg)
Results
46
Thursday, January 12, 12
![Page 47: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/47.jpg)
PyVision Video Interface Classes
47
Thursday, January 12, 12
![Page 48: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/48.jpg)
System Architecture
PyVision
Applications:Face Recognition
OpenCVScipyPIL
Python
Sessions 3 & 4
Session 2
Session 1
48
Applications:Video Analytics
Thursday, January 12, 12
![Page 49: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/49.jpg)
Video Processing
49
• Background Subtraction
• Motion Detection
• Optical Flow
• People Detection
• VideoStreamProcessor abstract class
Thursday, January 12, 12
![Page 50: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/50.jpg)
PyVision Video Stream Processing
50
Thursday, January 12, 12
![Page 51: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/51.jpg)
Application to Action Recognition
51
• Action Recognition using Manifold Distances
• Y.M. Lui et al., CVPR 2010 and FG 2011,S. O’Hara et al., FG2011 and AVSS 2011
• Current Work: Subspace ForestRecognition via Approximate Nearest Neighbor in a Grassmann manifold space [details and source code will be available pending publication, under review]
Thursday, January 12, 12
![Page 52: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/52.jpg)
Demonstration Overview• Data Sets
• KTH Actions
• Cambridge Gestures
• UCF Sports
• Scripts for using data set and applying evaluation protocol
52
Thursday, January 12, 12
![Page 53: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/53.jpg)
Demonstration
• Placeholder for demonstration of running the action recognition protocol on the Gestures data set.
53
Thursday, January 12, 12
![Page 54: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/54.jpg)
Comparative Performance
54
KTH Actions AccuracySubspace ForestProduct Manifold
97.996
CMOF (Guo 2010)MSTF (Gilbert 2009)
97.494.5
UCF Sports AccuracySubspace ForestTangent Bundle
91.388
AFMKL (Wu 2011)DTM (Bregonzio 2010)
91.386.9
Cambridge Gestures AccuracySubspace ForestTangent Bundle
Product Manifold
89.89188
TCCA (Kim 2007) 82
CSU Action Recognition:• Product Manifold - Lui et al. CVPR 2010• Tangent Bundle - Lui et al. FG 2011• Subspace Forest - O’Hara et al. (in review)Note: Comparisons between methods on a given dataset use a common evaluation protocol.UCF: leave-one-out, KTH: Schuldt’s partitioning,Gestures: Train on Set 5; Test on 1-4.
Thursday, January 12, 12
![Page 55: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/55.jpg)
System Architecture
55
PyVision
Applications:Face Recognition
OpenCVScipyPIL
Python
Sessions 3 & 4
Session 2
Session 1
3
Applications:Video Analytics
Thursday, January 12, 12
![Page 56: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/56.jpg)
A Good Challenge Problem
56
Today
Time / Effort / $$$
Per
form
ance
Challenge Problem
Thursday, January 12, 12
![Page 57: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/57.jpg)
The Dataset: GBU Challenge ProblemThe Good - 98%Easiest Matches
The Bad - 75%Average Matches
The Ugly - 15%Hardest Matches
• Challenge problem created to encourage improvement on difficult to match cases (Bad & Ugly)
• All three partitions contain the same number of images from 437 people.
• Most images are “high quality frontal images” collected at University of Notre Dame
• No image pairs are collected on the same day.
57
Thursday, January 12, 12
![Page 58: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/58.jpg)
Are these images really difficult?
580%
4%
8%
12%
16%
AlgorithmFusion 2006 LDA-IR LRPCA KDA KFA V1-Like A EBGM LBP PCA 1992
3%2%2%
6%
2%
6%7%
11%
15%
Verif
icat
ion
Rat
e@
0.0
01 F
AR
Thursday, January 12, 12
![Page 59: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/59.jpg)
The Idea:Update and Tune Well-Known
Algorithms
Thursday, January 12, 12
![Page 60: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/60.jpg)
LRPCA Workflow
60
Align Image
Extract Region 0 SQI/Normalize PCA/Whitening
Concatenate Vectors
Select/Weight
Create Face Record
Extract Region 1 SQI/Normalize PCA/Whitening
Extract Region 2 SQI/Normalize PCA/Whitening
Extract Region 13 SQI/Normalize PCA/Whitening
… … ...
Phillips P.J., Beveridge J.R., Draper B.A., Givens G.H., O’Toole A.J., Bolme D.S., Dunlop J., Lui Y.M., Sahibzada H., and Weimer S. An Introduction to the Good, the Bad, and the Ugly Face Recognition
Challenge Problem. Automatic Face and Gesture Recognition. 2011
Thursday, January 12, 12
![Page 61: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/61.jpg)
The System: Local Region PCA
• Modernized version of the PCA algorithm.
• Establishes open-source baseline performance for GBU.
• Includes advances in preprocessing: Self Quotient Image
• Analysis of multiple face regions.
61
Thursday, January 12, 12
![Page 62: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/62.jpg)
LRPCA Eigenbasis
...
...
...
...
...
62
Thursday, January 12, 12
![Page 63: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/63.jpg)
LDA-IR Flow
63
Align Image
Extract Red Normalize LDACohort Norm
Create Face Record
Extract I Normalize LDA
Yui Man Lui, David S. Bolme, P. Jonathon Phillips, Bruce A. Draper, and J. Ross Beveridge. Pushing LDA Performance on the Good, the Bad, and the Ugly Face. Journal of Research of the National Institute of Standards and Technology, (Under Review)
Thursday, January 12, 12
![Page 64: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/64.jpg)
LDA-IRRed Channel
I Channel
64
Thursday, January 12, 12
![Page 65: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/65.jpg)
File: --Example.py--
class FaceRecognitionAlgorithm: def addTraining(self,label,im,rect,l,r): raise NotImplementedError() def addCohort(self,im,rect,leye,reye): pass # NO OP def train(self,ilog=None): raise NotImplementedError() def getFaceRecord(self,im,rect,leye,reye): raise NotImplementedError() def similarity(self,face_rec1,face_rec2): raise NotImplementedError() def similarityMatrix(self,probes,targets): raise NotImplementedError()
class FaceRecord: def __init__(self, rect, leye, reye): ...
Algorithm Interface
65
• Currently ignores face detection rect.
• getFaceRecord, similarity, and similarity matrix are required for testing.
• addTraining, addCohort, train are only needed if you are using the CSU training framework.
• __init__: setup/initialization
• preprocess: alignment, norm
Thursday, January 12, 12
![Page 66: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/66.jpg)
Source Code:Interface and Algorithms
Thursday, January 12, 12
![Page 67: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/67.jpg)
Running the Algorithms
• Both algorithm include a trained algorithm stored in a python pickle (.pkl) file.
• Three testing scripts for each algorithm in the “bin” directory. They conform to the BEST interface.
• Setup Image Directories and Eye Coordinates.
• “./scripts/gbu_test_reduced.sh” or “./scripts/gbu_test_manual.sh” runs it all including R plots.
• Test Script: “./scripts/lfw_tests.sh”
67
Thursday, January 12, 12
![Page 68: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/68.jpg)
Source Code:Run lfw_test.sh
Thursday, January 12, 12
![Page 69: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/69.jpg)
Training the Algorithms
• Training is optional.
• LRPCA: “scripts/train_lrpca.sh”
• LDA-IR: “src/facerec2010/train_lda.py”
69
Thursday, January 12, 12
![Page 70: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/70.jpg)
CSU Baseline Performance
0
25
50
75
100
Good Bad Ugly
7.4
24.0
64.6
11.4
48.3
84.1
15.1
79.2
98.4
Fusion *LDA-IRLRPCA (FG 2011)
* Jan 2010 Version70
Thursday, January 12, 12
![Page 71: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/71.jpg)
CSU Baseline ROCs
710.
00.
20.
40.
60.
81.
0False accept rate
Verif
icat
ion
rate
0.001 0.01 0.1 1.0
GoodBadUgly
● 0.11
● 0.49
● 0.84
0.0
0.2
0.4
0.6
0.8
1.0
False accept rate
Verif
icat
ion
rate
0.001 0.01 0.1 1.0
GoodBadUgly
● 0.07
● 0.24
● 0.63
LRPCA LDA-IR
Thursday, January 12, 12
![Page 72: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/72.jpg)
Automatic Face Detection
• Face Detection - OpenCV Viola and Jones
• Eye Detection - Average of Synthetic Exact Filters (ASEF)
72**Bolme D.S., Draper B.A., and Beveridge J.R. ``Average of Synthetic
Exact Filters'', Computer Vision and Pattern Recognition, 2009
* P. Viola and M.J.Jones. Robust real-time face detection. Int. J. Computer Vision, 57(2):137–154, 2004
Thursday, January 12, 12
![Page 73: WACV2012: Tutorial: Introduction to PyVision for Computer Vision Applications](https://reader033.vdocuments.site/reader033/viewer/2022050904/544e5f27b1af9f23638b4d96/html5/thumbnails/73.jpg)
Question / Answer
Thursday, January 12, 12