Efficient Part-Based Recognition of Multiple Object Classes

Object Class Recognition Systems

• Model Representation

2

Object Class Recognition Systems

• Model Representation

• Learning Algorithm

3

Object Class Recognition Systems

• Model Representation

• Learning Algorithm

• Recognition Algorithm

4

Object Class Recognition Systems

• Model Representation– Part-based model

• Learning Algorithm

• Recognition Algorithm

5

Object Class Recognition Systems

• Model Representation– Part-based model

• Part appearance

• Learning Algorithm

• Recognition Algorithm

6

Object Class Recognition Systems

• Model Representation– Part-based model

• Part appearance• Part position

• Learning Algorithm

• Recognition Algorithm

7

Object Class Recognition Systems

• Model Representation– Part-based model

• Part appearance• Part position

• Learning Algorithm

• Recognition Algorithm

8

History of Part-BasedObject Recognition

TemplateMatching

Part-Based

ConstellationModel

k-FansProposedAlgorithm

Fergus et al. 2003, 2005

Crandall et al. 2005, 2006

Burl et al. 1998

Bag of Parts

Hoffman 1999

9

Template Matching

10

Templ Part

Const

k-Fan this

BoP

Template Matching

11

Templ Part

Const

k-Fan this

BoP

Template Matching

12

Templ Part

Const

k-Fan this

BoP

Template Matching

13

Templ Part

Const

k-Fan this

BoP

Template Matching

14

Templ Part

Const

k-Fan this

BoP

Template Matching

15

Templ Part

Const

k-Fan this

BoP

Template Matching

16

Templ Part

Const

k-Fan this

BoP

Template Matching

17

Templ Part

Const

k-Fan this

BoP

Template Matching

18

Templ Part

Const

k-Fan this

BoP

Template Matching

19

Templ Part

Const

k-Fan this

BoP

Template Matching

20

Templ Part

Const

k-Fan this

BoP

Template Matching

21

Templ Part

Const

k-Fan this

BoP

Template Matching

22

Templ Part

Const

k-Fan this

BoP

Template Matching

23

Templ Part

Const

k-Fan this

BoP

Template Matching

24

Templ Part

Const

k-Fan this

BoP

Template Matching

25

max

Templ Part

Const

k-Fan this

BoP

Template Matching

26

Templ Part

Const

k-Fan this

BoP

Template Matching

27

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels)

28

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

29

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust?

30

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust?

31

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust?

32

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust?

33

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust?

34

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust?

35

Templ Part

Const

k-Fan this

BoP

Template Matching

• Efficient? (N = # pixels) – Yes, O(N)

• Robust? – No, inflexible

36

Templ Part

Const

k-Fan this

BoP

Template vs. Part-Based

Trade-off

RobustEfficient

37

Templ Part

Const

k-Fan this

BoP

Part-Based

38

Templ Part

Const

k-Fan this

BoP

Part-Based

39

Templ Part

Const

k-Fan this

BoP

Part-Based

40

Templ Part

Const

k-Fan this

BoP

Part-Based

41

Templ Part

Const

k-Fan this

BoP

Part-Based

42

Templ Part

Const

k-Fan this

BoP

Part-Based

• Part appearance (template matching)

43

Templ Part

Const

k-Fan this

BoP

Part-Based

• Part appearance (template matching)

• Part location (fully connected)

44

Templ Part

Const

k-Fan this

BoP

Part-Based

45

Templ Part

Const

k-Fan this

BoP

Part-Based

46

Templ Part

Const

k-Fan this

BoP

Part-Based

47

Templ Part

Const

k-Fan this

BoP

Part-Based

48

Templ Part

Const

k-Fan this

BoP

Part-Based

49

Templ Part

Const

k-Fan this

BoP

Part-Based

50

Templ Part

Const

k-Fan this

BoP

Part-Based

51

Templ Part

Const

k-Fan this

BoP

Part-Based

52

Templ Part

Const

k-Fan this

BoP

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels)

53

Templ Part

Const

k-Fan this

BoP

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels)

54

Templ Part

Const

k-Fan this

BoP

P

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels)

55

Templ Part

Const

k-Fan this

BoP

P

N

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels)

56

Templ Part

Const

k-Fan this

BoP

P

N

…

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels)

57

Templ Part

Const

k-Fan this

BoP

P

N

N…

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels) – No, O(PN^P)

58

Templ Part

Const

k-Fan this

BoP

P

N

N…

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels) – No, O(PN^P)• Robust?

59

Templ Part

Const

k-Fan this

BoP

P

N

N…

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels) – No, O(PN^P)• Robust? Yes!

60

Templ Part

Const

k-Fan this

BoP

P

N

N…

Part-Based

• Part appearance (template matching)

• Part location (fully connected)• Efficient? (P = #parts, N = #pixels) – No, O(PN^P)• Robust? Yes!• All following algorithms try to approach the accuracy of

this method, while gaining efficiency

61

Templ Part

Const

k-Fan this

BoP

P

N

N…

Constellation Model

Trade-off

Dense Image (n=N)

RobustEfficient

62

Templ Part

Const

k-Fan this

BoP

Constellation Model

Trade-off

Sparse Interest Points (n<<N) Dense Image (n=N)

RobustEfficient

63

Templ Part

Const

k-Fan this

BoP

Constellation Model

• Efficient O(Pn^P) (n = # interest points, P = # parts)

64

Templ Part

Const

k-Fan this

BoP

Constellation Model

• Efficient O(Pn^P) (n = # interest points, P = # parts)

• Approximation:– Sparse image: only consider interest points

(n<<N)

65

Templ Part

Const

k-Fan this

BoP

Constellation Model

• Efficient O(Pn^P) (n = # interest points, P = # parts)

• Approximation:– Sparse image: only consider interest points

(n<<N)

• Interest point detector too general– regions of the image discarded without

considering particular parts that may be there

66

Templ Part

Const

k-Fan this

BoP

Bag of Parts

Trade-off

RobustEfficient

67

Templ Part

Const

k-Fan this

BoP

Bag of Parts

Trade-off

RobustEfficient

68

Templ Part

Const

k-Fan this

BoP

Bag of Parts

69

Templ Part

Const

k-Fan this

BoP

Bag of Parts

• Part appearance (template matching)

70

Templ Part

Const

k-Fan this

BoP

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

71

Templ Part

Const

k-Fan this

BoP

Bag of Parts

72

Templ Part

Const

k-Fan this

BoP

Bag of Parts

73

Templ Part

Const

k-Fan this

BoP

Bag of Parts

74

Templ Part

Const

k-Fan this

BoP

Bag of Parts

75

Templ Part

Const

k-Fan this

BoP

Bag of Parts

76

Templ Part

Const

k-Fan this

BoP

Bag of Parts

77

Templ Part

Const

k-Fan this

BoP

Bag of Parts

• Part appearance (template matching)

• Part location (ignored, disconnected)

max

max

Templ Part

Const

k-Fan this

BoP

78

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels)

79

Templ Part

Const

k-Fan this

BoP

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels)

80

Templ Part

Const

k-Fan this

BoPN

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels)

81

Templ Part

Const

k-Fan this

BoPN

P

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

82

Templ Part

Const

k-Fan this

BoPN

P

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

83

Templ Part

Const

k-Fan this

BoP

• Robust?

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

84

Templ Part

Const

k-Fan this

BoP

• Robust?

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

85

Templ Part

Const

k-Fan this

BoP

• Robust?– No localization

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

86

Templ Part

Const

k-Fan this

BoP

• Robust?– No localization

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

87

Templ Part

Const

k-Fan this

BoP

• Robust?– No localization

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

88

Templ Part

Const

k-Fan this

BoP

• Robust?– No localization

– More likely to find false detections

Bag of Parts

• Part appearance (template matching)• Part location (ignored, disconnected)

• Efficient? (P = #parts, N = #pixels) – Yes, O(NP)

89

Templ Part

Const

k-Fan this

BoP

• Robust?– No localization

– More likely to find false detections

– …but still in common use

k-Fans

RobustEfficient

90

Templ Part

Const

k-Fan this

BoP

k-Fans

RobustEfficient

91

Templ Part

Const

k-Fan this

BoP

k-Fans

RobustEfficient

92

Templ Part

Const

k-Fan this

BoP

k-Fans

…

RobustEfficient

93

Templ Part

Const

k-Fan this

BoP

k-Fans

…

Trade-off

RobustEfficient

94

Templ Part

Const

k-Fan this

BoP

1-Fans

95

Templ Part

Const

k-Fan this

BoP

1-Fans

96

Templ Part

Const

k-Fan this

BoP

1-Fans

97

Templ Part

Const

k-Fan this

BoP

1-Fans

98

Templ Part

Const

k-Fan this

BoP

1-Fans

99

Templ Part

Const

k-Fan this

BoP

1-Fans

100

Templ Part

Const

k-Fan this

BoP

1-Fans

101

Templ Part

Const

k-Fan this

BoP

• Look at the probability model for locations

1-Fans

102

Templ Part

Const

k-Fan this

BoP

• Look at the probability model for locations

1-Fans

103

Templ Part

Const

k-Fan this

BoP

• Look at the probability model for locations

104

Templ Part

Const

k-Fan this

BoP

1-Fans

105

Templ Part

Const

k-Fan this

BoP

1-Fans

...

106

Templ Part

Const

k-Fan this

BoP

1-Fans

...

107

Templ Part

Const

k-Fan this

BoP

1-Fans

...

108

Templ Part

Const

k-Fan this

BoP

1-Fans

...

109

Templ Part

Const

k-Fan this

BoP

1-Fans

...

110

Templ Part

Const

k-Fan this

BoP

1-Fans

...

111

Templ Part

Const

k-Fan this

BoP

1-Fans

...

112

Templ Part

Const

k-Fan this

BoP

1-Fans

...

113

Templ Part

Const

k-Fan this

BoP

1-Fans

...

114

Templ Part

Const

k-Fan this

BoP

1-Fans

...

115

Templ Part

Const

k-Fan this

BoP

1-Fans

...

116

Templ Part

Const

k-Fan this

BoP

1-Fans

...

117

Templ Part

Const

k-Fan this

BoP

1-Fans

...

118

Templ Part

Const

k-Fan this

BoP

1-Fans

...

119

Templ Part

Const

k-Fan this

BoP

1-Fans

...

120

Templ Part

Const

k-Fan this

BoP

1-Fans

...

121

Templ Part

Const

k-Fan this

BoP

1-Fans

...

122

Templ Part

Const

k-Fan this

BoP

1-Fans

...

123

Templ Part

Const

k-Fan this

BoP

1-Fans

...

124

Templ Part

Const

k-Fan this

BoP

1-Fans

...

125

Templ Part

Const

k-Fan this

BoP

1-Fans

...

126

Templ Part

Const

k-Fan this

BoP

1-Fans

...

1-Fans

Templ Part

Const

k-Fan this

BoP

1-Fan

+

… …

128

Templ Part

Const

k-Fan this

BoP

• Efficient? (N = # pixels, P = # parts)

1-Fans

129

Templ Part

Const

k-Fan this

BoP

• Efficient? (N = # pixels, P = # parts)

1-Fans

130

Templ Part

Const

k-Fan this

BoP

N

• Efficient? (N = # pixels, P = # parts)

1-Fans

131

Templ Part

Const

k-Fan this

BoP

P

N

• Efficient? (N = # pixels, P = # parts)

1-Fans

132

Templ Part

Const

k-Fan this

BoP

P

N

N

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2)

1-Fans

133

Templ Part

Const

k-Fan this

BoP

N

P

N

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2)– Use Dynamic Programming:

1-Fans

134

Templ Part

Const

k-Fan this

BoP

N

P

N

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2)– Use Dynamic Programming:

• Precompute DTj image in O(N)

1-Fans

135

Templ Part

Const

k-Fan this

BoP

N

P

N

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2)– Use Dynamic Programming:

• Precompute DTj image in O(N)

• Then add the A1 and DTj images

1-Fans

136

Templ Part

Const

k-Fan this

BoP

N

P

N

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2) O(NP) – Same as bag of parts!– Use Dynamic Programming:

• Precompute DTj image in O(N)

• Then add the A1 and DTj images

1-Fans

137

Templ Part

Const

k-Fan this

BoP

N

P

N

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2) O(NP) – Same as bag of parts!

• Robust?

1-Fans

138

Templ Part

Const

k-Fan this

BoP

• Efficient? (N = # pixels, P = # parts) Yes, O(PN^2) O(NP) – Same as bag of parts!

• Robust? – Yes, better than bag of parts.

1-Fans

139

Templ Part

Const

k-Fan this

BoP

Aren’t We Done?

• Object Recognition is efficient and robust. Can’t we stop here?

Aren’t We Done?

• Object Recognition is efficient and robust. Can’t we stop here?

• What about…

Aren’t We Done?

• Object Recognition is efficient and robust. Can’t we stop here?

• What about…– Detecting multiple object classes? (M = #

objects… think 30,000) O(MNP)

Recap: History of Part-BasedObject Recognition

TemplateMatching

Part-Based

ConstellationModel

k-FansProposedAlgorithm

Fergus et al. 2003, 2005

Crandall et al. 2005, 2006

Burl et al. 1998

Bag of Parts

Hoffman 1999

143

AccurateEfficient

Proposed Algorithm

1-fan

Sparse Appearance Image(thresholded)

Does not rely on ageneralinterest point detector

144

Templ Part

Const

k-Fan this

BoP

145

Proposed Algorithm

Templ Part

Const

k-Fan this

BoP

146

Proposed Algorithm

Templ Part

Const

k-Fan this

BoP

147

Proposed Algorithm

Templ Part

Const

k-Fan this

BoPThis is sufficientto solve 0-fans:

148

Proposed Algorithm

Templ Part

Const

k-Fan this

BoPThis is sufficientto solve 0-fans:

max

max

Proposed Algorithm

• How do we threshold appearances in sublinear time (i.e. < O(MNP)? M = # objects, N = # pixels, P = # parts/object)

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

• How do we threshold appearances in sublinear time (i.e. < O(MNP)? M = # objects, N = # pixels, P = # parts/object)

• View thresholded appearance detection as an R-nearest neighbor problem

Templ Part

Const

k-Fan this

BoP

R-Nearest Neighbors

• What is the set of points in a database that are within a radius R from the query point q?

Templ Part

Const

k-Fan this

BoP

q

R

R-Nearest Neighbors

• What is the set of points in a database that are within a radius R from the query point q?

Templ Part

Const

k-Fan this

BoP

q

R-Nearest Neighbors

• What is the set of points in a database that are within a radius R from the query point q?

Templ Part

Const

k-Fan this

BoP

q

• The space part appearances (high dimensions)

R-Nearest Neighbors

q

• The space part appearances (high dimensions)

• Assume part appearances are identically distributed spherical Gaussians (i Σi =cI)

R-Nearest Neighbors

q

• The space part appearances (high dimensions)

• Assume part appearances are identically distributed spherical Gaussians (i Σi =cI)

• Database points are the means (μi)

R-Nearest Neighbors

q

• The space part appearances (high dimensions)

• Assume part appearances are identically distributed spherical Gaussians (i Σi =cI)

• Database points are the means (μi)

• 1-Fan part appearances can be expressed this way

R-Nearest Neighbors

q

• Okay, how fast can it be solved?

R-Nearest Neighbors

q

• Okay, how fast can it be solved?– In low dimensions (<8), kD trees solve it in sublinear time

R-Nearest Neighbors

q

• Okay, how fast can it be solved?– In low dimensions (<8), kD trees solve it in sublinear time

– But the conjectured “curse of dimensionality” prevents it from being solved efficiently for high dimensions

R-Nearest Neighbors

q

• Okay, how fast can it be solved?– In low dimensions (<8), kD trees solve it in sublinear time

– But the conjectured “curse of dimensionality” prevents it from being solved efficiently for high dimensions

• Locality Sensitive Hashing solves the problem approximately,– misses some points with probability 1-δ

– Solves it in O(nd1/c+o(1))

– Trades off probability of false negative for efficiency

R-Nearest Neighbors

...

162

Proposed Algorithm

Templ Part

Const

k-Fan this

BoP

163

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

164

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

165

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

166

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

167

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

168

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

169

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

170

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

Now we can do this efficiently

171

...

Templ Part

Const

k-Fan this

BoP

Proposed Algorithm

How do we do these efficiently?

Proposed Algorithm

• Lazy distance transforms, and HA*LD• Coming soon