11

Anne JorstadAnne Jorstad

AMSC 664AMSC 664University of MarylandUniversity of Maryland

Spring 2008Spring 2008Final ReportFinal Report

Leaf Classification from Leaf Classification from Local Boundary AnalysisLocal Boundary Analysis

Advisor: Dr. David Jacobs, Computer ScienceAdvisor: Dr. David Jacobs, Computer Science

22

BackgroundBackground

• Electronic Field Guide for PlantsElectronic Field Guide for Plants

33

BackgroundBackground

• Current System: Current System: – Inner-Distance Shape Context (IDSC)Inner-Distance Shape Context (IDSC)

• Measures the shortest distance between two points on a Measures the shortest distance between two points on a path contained entirely within a figurepath contained entirely within a figure

• Good for detecting similarities between deformable Good for detecting similarities between deformable structuresstructures

44

BackgroundBackground

• Current System: Current System: – All shape information is compared at a global level, All shape information is compared at a global level,

no specific consideration of edge typesno specific consideration of edge types

Cephalanthus occidentalis Cephalanthus occidentalis (smooth boundary)(smooth boundary)

Carpinus carolinianaCarpinus caroliniana(serrated boundary)(serrated boundary)

55

Problem StatementProblem Statement

Use local boundary information to make Use local boundary information to make classification decisions that complement the classification decisions that complement the existing system.existing system.

66

The AlgorithmThe Algorithm

• Input:Input:

• Capture Capture

boundary curve:boundary curve:

77

The Algorithm: WaveletsThe Algorithm: Wavelets

• Discrete wavelet transformDiscrete wavelet transform– In: vector of pointsIn: vector of points– Out: two vectors, each half original lengthOut: two vectors, each half original length

• Approximation coefficients: Approximation coefficients: – general spatial informationgeneral spatial information

• Detail coefficients: Detail coefficients: – local detail informationlocal detail information

– Repeat for multiple scalesRepeat for multiple scales

88

The Algorithm: WaveletsThe Algorithm: Wavelets• Model leaf by its detail coefficients Model leaf by its detail coefficients

over several scalesover several scales

Approximations, Approximations, continually subtracting continually subtracting out detail informationout detail information

InputInput

99

The Algorithm: DataThe Algorithm: Data

• Forget leaves:Forget leaves:– Each boundary point: Each boundary point:

• Lose one degree of freedom in preserving rotation invarianceLose one degree of freedom in preserving rotation invariance

– For 3 wavelet scales, leaf is ~2000 5-D pointsFor 3 wavelet scales, leaf is ~2000 5-D points

• Combine data for all leaves:Combine data for all leaves:– #leaves x ~2000 5-D points#leaves x ~2000 5-D points

• Group all points into meaningful clustersGroup all points into meaningful clusters

1010

The Algorithm: ClusteringThe Algorithm: Clustering

• Goal: Sort points into “buckets” to get a Goal: Sort points into “buckets” to get a unique distribution for each leaf speciesunique distribution for each leaf species

• K-Means Clustering: K-Means Clustering:

group all points into 36group all points into 36

representative clustersrepresentative clusters

1111

The Algorithm: Distribution The Algorithm: Distribution ComparisonComparison

• Distribution of individual leaf’s 2000 Distribution of individual leaf’s 2000 points over the 36 clusters represents leafpoints over the 36 clusters represents leaf

Leaf image and corresponding histogram for (a) Corylus americana, (b) Corylus americana, different example, (c) Asimina triloba

(a) (b) (c)

1212

• Compare distributions between leaves using the Compare distributions between leaves using the chi-squared distance:chi-squared distance:

wherewhere

• Smallest distance defines best matchSmallest distance defines best match– New leaf is assigned the species of the closest matchNew leaf is assigned the species of the closest match

The Algorithm: Distribution The Algorithm: Distribution ComparisonComparison

1313

ValidationValidation• Training data: 20 species, 10 examples of each Training data: 20 species, 10 examples of each

→→ 200 leaves200 leaves

10 serrated species10 serrated species 10 smooth species10 smooth species

1414

ValidationValidation

• Test data: same 20 species, 5 new examples of eachTest data: same 20 species, 5 new examples of each

• Nearest-Neighbor ClassificationNearest-Neighbor Classification

• Species classification: 46% correctSpecies classification: 46% correct

• Serration classification: 100% correctSerration classification: 100% correct– closest match was to species with appropriate serrationclosest match was to species with appropriate serration

1515

ValidationValidation

• Test data: same 20 species, 5 new examples of eachTest data: same 20 species, 5 new examples of each

• Nearest-Neighbor ClassificationNearest-Neighbor Classification

• Species classification: 46% correctSpecies classification: 46% correct

• Serration classification: 100% correctSerration classification: 100% correct– closest match was to species with appropriate serrationclosest match was to species with appropriate serration

Local serration Local serration information IS information IS being captured!being captured!

1616

Combining ResultsCombining Results

• Original IDSC results on same data set:Original IDSC results on same data set:– Species correct: 62%Species correct: 62%– Serration correct if species wrong: 53%Serration correct if species wrong: 53%

• No better than chanceNo better than chance

• How to combine wavelet distances with IDSC How to combine wavelet distances with IDSC distances?distances?

1717

Combining ResultsCombining Results

• GivenGiven

andand

• Want to find:Want to find:

1818

Naïve Bayes ClassificationNaïve Bayes Classification

• From Bayes’ Rule:From Bayes’ Rule:

• Can now calculate all relevant probabilities Can now calculate all relevant probabilities from training datafrom training data

1919

Naïve Bayes ClassificationNaïve Bayes Classification

• Wavelet distances Wavelet distances → binary serration value→ binary serration value– Add small linear smoothing termAdd small linear smoothing term

• IDSC distances IDSC distances → species ranked in order → species ranked in order from nearest to farthestfrom nearest to farthest– Add Gaussian smoothing termAdd Gaussian smoothing term

2020

Validation ResultsValidation Results

• Test on same 20 species, 5 examples of eachTest on same 20 species, 5 examples of each

• Adding serration information has improved Adding serration information has improved overall classification results!overall classification results!

2121

Full Data SetFull Data Set

• 245 species, 7481 leaves245 species, 7481 leaves

• Binary serration assignment no longer makes Binary serration assignment no longer makes sense:sense:

2222

Linear OptimizationLinear Optimization

• Find best linear weighting of distances:Find best linear weighting of distances:

• Train over previousTrain over previous

training settraining set

alpha

% c

orr

ect

2323

Full Data SetFull Data Set

• Nearest-Neighbor Classification over allNearest-Neighbor Classification over all

7481 leaves7481 leaves

– Wavelet alone: 20% correctWavelet alone: 20% correct– IDSC alone: 54% correctIDSC alone: 54% correct– Combined: 64% correctCombined: 64% correct

2424

In PracticeIn Practice

• Electronic field guide displays top 5, 10 or 20 Electronic field guide displays top 5, 10 or 20 matchesmatches

• Calculate correct % in top Calculate correct % in top nn matches, matches,

for for n = 1, …, 20n = 1, …, 20

2525

In PracticeIn Practice

# matches considered

% c

orre

ct

2626

In PracticeIn Practice

• Need results in near real-timeNeed results in near real-time– Otherwise no benefit over paper field guidesOtherwise no benefit over paper field guides

• Running timeRunning time– Preprocessing: (several hours)Preprocessing: (several hours)

• Determine cluster centersDetermine cluster centers• Determine distributions for each leafDetermine distributions for each leaf

– On the spot: (0.92 seconds)On the spot: (0.92 seconds)• Calculate single distributionCalculate single distribution• Compare to all distributions in systemCompare to all distributions in system

2727

ConclusionsConclusions

• Wavelets do capture local serration Wavelets do capture local serration informationinformation

• Wavelet + IDSC classification does a better Wavelet + IDSC classification does a better overall job than the original IDSC aloneoverall job than the original IDSC alone

• Calculations can be done in real time to make Calculations can be done in real time to make the system realistic to usethe system realistic to use

2828

ReferencesReferences

• Gaurav Agarwal, Haibin Ling, David Jacobs, Sameer Shirdhonkar, W. John Kress, Rusty Russell, Peter Gaurav Agarwal, Haibin Ling, David Jacobs, Sameer Shirdhonkar, W. John Kress, Rusty Russell, Peter Belhumeur, Nandan Dixit, Steve Feiner, Dhruv Mahajan, Kalyan Sunkavalli, Ravi Ramamoorthi, Sean Belhumeur, Nandan Dixit, Steve Feiner, Dhruv Mahajan, Kalyan Sunkavalli, Ravi Ramamoorthi, Sean White. “First Steps Toward an Electronic Field Guide for Plants”. Taxon, vol. 55, no. 3, Aug. 2006. White. “First Steps Toward an Electronic Field Guide for Plants”. Taxon, vol. 55, no. 3, Aug. 2006.

• Cene C.-H. Chuang, C.-C. Jay Kuo. “Wavelet Descriptor of Planar Curves: Theory and Applications”. Cene C.-H. Chuang, C.-C. Jay Kuo. “Wavelet Descriptor of Planar Curves: Theory and Applications”. IEEE Transactions of Image Processing, Vol. 5, No. 1, January 1996.IEEE Transactions of Image Processing, Vol. 5, No. 1, January 1996.

• Pedro F. Felzenszwalb, Jushua D. Schwartz. “Hierarchical Matching of Deformable Shapes”. IEEE Pedro F. Felzenszwalb, Jushua D. Schwartz. “Hierarchical Matching of Deformable Shapes”. IEEE Conference on Computer Vision and Pattern Recognition, 2007.Conference on Computer Vision and Pattern Recognition, 2007.

• Haibin Ling, David Jacobs. “Using the Inner-Distance for Classification of Articulated Shapes.” CVPR, Haibin Ling, David Jacobs. “Using the Inner-Distance for Classification of Articulated Shapes.” CVPR, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, 2005.Recognition, Vol. 2, 2005.

• Jitendra Malik, Serge Belongie, Thomas Leung, Jainbo Shi. “Contour and Texture Analysis for Image Jitendra Malik, Serge Belongie, Thomas Leung, Jainbo Shi. “Contour and Texture Analysis for Image Segmentation”. International Journal of Computer Vision, vol. 34, no. 1, July 2001.Segmentation”. International Journal of Computer Vision, vol. 34, no. 1, July 2001.

• Stephane Mallat. “A Wavelet Tour of Signal Processing”. Academic Press, Chestnut Hill, Massachusetts, Stephane Mallat. “A Wavelet Tour of Signal Processing”. Academic Press, Chestnut Hill, Massachusetts, 1999.1999.