Evaluation of the stability of SIFT keypoint correspondence across cameras

or .. “can we put a ‘C’ in SIFT?”

max van kleek6.869: learning and interfacesthursday may 11, 2005

ubiquitous computing: computers (and cameras) are everywhere!

little sister: follow-me-around user modeling

object correspondence acrossvarying cameras/lighting/scenes is an important subgoal

applications: buliding a personal life log for yourself, interest profiling, social network mining, health care

identifying objects with local features: the SIFT transform

Orientation histogram = SIFT feature vector

recognizing objects

identified and orientedkeypoints used to votefor pose orientationsin a hough transform

pose-and-scale space

keypoint correspondence

1. % keypoints detected2. stability of orientation histogram

object orientation (away from frontal parallel)

object deformation

lighting direction, intensity, shading

lens distortion, sharpness, ccd “quality”, noise, capture artifacts

Mikolajczyk, K., C. Schmidt “A Performance Evaluation of Local Descriptors”, CVPR ‘03

Lowe, D.G. “Distinctive Image Features fromScale-Invariant Keypoints”, ICJV 2004.

Me! well, sort of..

???

cameras vary widely in sizes, configurations, capabilities, and prices

the experiment

Logitech QC Express

640x48016-bit colorYUV4:2:2AGC, Auto Exposuremanual focusUSB iface$15

Logitech QC Pro 3000

CCD by Phillips640x48016-bit colorYUV4:2:2, RGBAGC, Auto ExposureAuto WBmanual focusUSB iface$50

Sony EVI-D30Steerable NTSCcamera,DV capture card

720x480 luminance,less for colorRaw DV AGC, Auto ExposureAuto focus

~$300 + $200

Nikon Coolpix 990Digital still camera,

2048x1536RGBAuto Gain, Auto WB, Auto ExposureAuto focus

$1000 -> $500

experiment setup:5 incandenscent lights12 ft between camera and subject

acquiring image sets

background10 images stationary

2 front 2 face right 2 face left

for each camera:

= 16 images/cam * 4 cameras = 48

320x240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100;default camera settings except disabled AGC, disabled AE (locked to optimal settings)

algorithm for keypoint correspondence

source image contrast stretch over whole set

contrast-stretchedbackground images

background model(mean)

-

find(p(img) < epsilon)

compute_sift_points

dilate fg mask with a disc strel

intersectionfilter out bgkey points

keeping onlyrelevant keypointsby intersecting keypoints with foreground pts

image A w/ sift keypoints

orientation histograms for each keypoint in A

orientation histograms for each keypoint in B

image B w/ sift keypoints

orientation histograms for each keypoint in A

orientation histograms for each keypoint in B

match keypoints using nearest-neighbor in SIFT space

orientation histograms for each keypoint in A

orientation histograms for each keypoint in B

match keypoints using nearest-neighbor in SIFT space

1.1 -> 1.2 1.2 -> 1.1

sanity check: same (dv) camera, slightly different pose 1.1: 15 keypoints detected1.2: 11 keypoints detected

6 properly assigned8 in common

5 properly assigned8 in common

4.1 -> 1.1 1.1 -> 4.1

nikon coolpix versus sony steerable 4.1: 18 keypoints detected1.1: 15 keypoints detected

5 properly assigned10 in common

5 properly assigned10 in common

4.1 -> 3.1 3.1 -> 4.1

nikon coolpix versus qc pro 4.1: 18 keypoints detected3.1: 17 keypoints detected

2 properly assigned10 in common

5 properly assigned10 in common

4.1 -> 2.1 4.1 -> 2.1

nikon coolpix versus qc express 4.1: 18 keypoints detected2.1: 22 keypoints detected

2 properly assigned10 in common

0 properly assigned10 in common

other results:

qc pro vs qc exp (3.1 -> 2.1) 3.1->2.1 : 17 / 22 4 correct out of 6 in common2.1->3.1: 22 / 17 0 correct out of 6 in common

poor reproducibility with qcs?

qc exp test (is the qc exp just too noisy?)2.1->2.2 : 22/20 1 correct out of 8 in common2.2->2.1 : 20/22 2 correct out of 8 in common

yes.

qc pro reproducibility3.1->3.2: 17/24 6 correct out of 9 in common3.2->3.1 : 24/17 7 correct out of 9 in common

angle test using qc pro3.1->3.4 : 0 correct out of 0 in common

sensitive to out-of-plane rotation

experiment setup:5 incandenscent lights3 ft between camera and robot

320x240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100;default camera settings except disabled AGC, (locked to optimal settings)

• parameters: – bins / histogram

– pixels / quadrant

– quadrants / keypoint

– gaussian dropoff covariance

keypoint splitting / multiple primary gradient directions

source keypoint merging

histogram blurring