probabilistic context free grammars for representing action song mao november 14, 2000

46
Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Upload: eleanor-mcgee

Post on 30-Dec-2015

220 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Probabilistic Context Free Grammars for Representing Action

Song Mao

November 14, 2000

Page 2: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Action Understanding in Computer Vision

Interpretation of basic movements Sitting, walking , running …

Description of motion of several objects Recognition of gestures High-level event

Page 3: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

High-level Events

Consists of primitives For car drop-off event: car-enter, car-stop,

person-enter, person-exit, etc.

Spatio-temporal structure & constraint Semantically defined activities Span extended periods of time Multi-object interactions

Page 4: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Approaches

Statistical techniques Syntactic techniques Methods that combine the two techniques

Page 5: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Statistical Techniques

Classifying pattern by assuming an statistical model Tennis stroke recognition Gesture recognition Visual language recognition

Advantages Real world data are noisy in nature (signal noise) Uncertainty in observation (sensor noise)

Page 6: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Statistical Techniques (cont.)

Disadvantages Insufficient data Semantic ambiguity Temporal ambiguity Known structure

Page 7: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Syntactic Techniques

Describe pattern structure Formal grammar Context free grammar (CFG) Stochastic context free grammar (SCFGs) Parsing

Page 8: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Combine the Two Techniques

Independent primitives detection using statistical techniques

Actions (structured primitives) recognition by syntactic techniques Parsing primitives by SCFGs Removing ambiguity by parsing SCFGs Correcting errors (substitution, insertion, deletion) by

adding SKIP rules and penalty function to SCFGs

Page 9: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Decoupling Primitive Detection and Primitive Structuring

Page 10: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Parsing

What is Parsing?

The process of taking an input and producing some sort of structure for it.(Jurafsky & Martin)

Structure assigned by Context Free Grammar (CFG) / Stochastic Context Free Grammar (SCFG)

Page 11: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Parsing Approaches

Top-down approach Bottom-up approach Dynamic programming approach

Cocke-Younger-Kasami parser (CYK) Graham-Harrison-Ruzzo parser (GHR) Earley parser

Page 12: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Context Free Grammars (CFG)

A set of non-terminal symbols A set of terminal symbols A set of productions P of form

Start symbol Directly derivation:

if ,

A

N

)(, NNA S

A ,)(, N A

Page 13: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Context Free Grammar (Cont.)

Derivation: . . .,

A language generated by a grammar

,, 3221 ,1 mm

m

1

},|{ wSwwLG

G

Page 14: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Stochastic Context Free Grammar (SCFG)

Modify production as: Where is the rule

probability of the production from a Context-Free Grammars (CFGs)

Rules are conditionally independent

pA

)|( AAPp A

)|()|()|(

),,|,,(

CCPBBPAAP

CBACBAP

Page 15: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Earley Parsing Algorithm

A set of states for each position in the input Dot denotes the current input position A state with the dot at the right most position is a

complete state A state produced by prediction is a predicted state A state produced by completion is a completed state

Page 16: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Earley Parsing Algorithm (cont.)

A State:

Prediction:

:

,

, :

ik Yi

Y

YXi

YXi k :

Page 17: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Earley Parser (cont.)

Scanning: Completion:

:1 : aXiaXi kk

:

, :

, :

YXiYi

YXjk

j

k

Page 18: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

An Example

VP VerbVP VerbVP

ProperNounNP Noun Det NP

flightNoun VPS

thatDet VP NPAux S

bookVerb VP NPS

state set (0) (1) Book (2) that (3) flight

NP Verb VP:0

Verb VP:0

VP S:0

VP NPAux S:0

ProperNoun NP:0

NounDet NP:0

VP NP S:0

predicted

S 0 :0

0

0

0

0

0

0

0

0

ProperNoun NP:1

NounDet NP:1

predicted

NP VerbVP:1

VPS:1

VerbVP:1

completed

bookVerb:1

scanned

1

1

0

0

0

0

Noun Det NP:2

completed

thatDet:2

scanned

1

1

VPS:3

NP VerbVP:3

NounDet NP:3

completed

flightNoun:3

scanned

0

0

1

2

Page 19: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Earley-Stolcke Parser (1)

A state Forward probability

Inner probability

Earley path: a sequence of states needed to reach the current state

Length of path: number of scanning states

],[ : YXi k : state in start that length of path ( Y X k k i Pk

path

path

k YXiP ) : statein ends that ilength ofpath (

) : statein end and YXi k

Page 20: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Earley-Stolcke Parser (2)

Prediction

where

],[ : ,

],[ :

ik Yi

Y

ZXi

)(

)(),() :(,

YP

YPYZRZXi Lk

Page 21: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Compute

Left-recursion in grammar Possibly infinite prediction loop that accumulate probability

computation Example: Left Corner relation:

aAAaA ,YXYX L iff ,

...)()(),(10

YZPYZPYZR LLL

LR

)...()( 121 YYYYZPYZP kL

k

L

Page 22: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Compute(cont.)

LR

Matrix form

Computed once for the grammar, and used at each iteration of the prediction step

1

0

10 )( ...

k

LLLL PIPPR

Page 23: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Earley-Stolcke Parser (3)

Scanning

Completion

where

],[ :1 ],[ : aXiaXi kk

],[ : , ],[ :

, ],[ :

ZXi

Yi

ZXjk

j

k

):(),() :(

):(),() :(

jUk

jUk

YiYZRZXj

YiYZRZXj

Page 24: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Compute

Unit production: Infinite completion by unit production

e. g.

Unit production relation matrix Similarly as computing in prediction step

UR

YXYX U iff ,

ABaABA , ,

UP

1

0

10 )( ...

k

UUUU PIPPR

LR

Page 25: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Uncertainty in the Input

Source of the input symbols is probabilistic Modify scanning of the Earley-Stolcke parser

Address the substitution error

0)( t.s. ,

],[ :1 ],[ :

aPa

aXiaXi kk

)() :(

)() :(

aPXi

aPXi

k

k

Page 26: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

An Example for a Grammar acbabcA |

Page 27: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Insertion and Deletion

Use a robust form grammar of

Includes all repetitions of all terminals Set small Penalize derivation consuming less terminals

... | | | ... | | SKIPcSKIPbcbSKIP

G G

) ( ruleSKIPP

CBA bCA

G :G

SKIP b | b SKIP | bB

Page 28: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Enforcing Consistency (1)

Types of consistency Temporal consistency Spatial consistency Object identity consistency

Add 2 vector valued state variables low mark high mark

Containing the data for computing distance penalty between two joining states

:l:h

Page 29: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Enforcing Consistency (2)

Prediction

Scanning

],[ : ,

],[ :tti

k SSYiY

hlZXi

else ],[ :1

if ],[ :1

],[ :

ak

aak

k

hlaXi

hlaXi

hlaXi

Page 30: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Enforcing Consistency (3)

Completion

: distance penalty function Computed based on high mark of completed

state and low mark of completing state

):(),() :()(

):(),() :()(

jUk

jUk

YiYZRZXjdf

YiYZRZXjdf

)(df

Page 31: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Choice of

Sever penalty: step function e. g.

Softer penalty: exponential function e. g.

)(df

else ,1

0 ,0 )(

ddf

2

)(d

Cedf

Page 32: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Application:Vedio Surveillance of Parking Lot

Outdoor environment – occlusions and lighting change

Static cameras Real-time performance Labeling activities and person-vehicle

interactions in a parking lot Handling simultaneous events

Page 33: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Known Structure, Uncertain Elements

Activities as sequences of primitives represented by SCFG Car drop-off, car pick-up Dancing

Input primitives are uncertain Uncertain observation of primitive Noisy symbols

Page 34: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Approach

First detect primitives using statistical method Tracker Event generator

Then Recognize activity by parsing input stream of uncertain primitives (partial tracks) by an SCFG parser

Page 35: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

System Overview

Page 36: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

System Overview (cont.)

Tracker Assign identity to the moving objects Collects the trajectory data into partial tracks

Event generator Maps partial tracks onto predetermined set of events

Parser Labels sequences of events by parsing using a SCFG Enforce consistency constraint

Page 37: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Tracker

Object found Assign a unique ID Track changes in objects’ appearance, position,

velocity Based on the data, assign each object a class label

(e.g. a car or a person) Object lost Object exit

Page 38: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Event Generator

Based on data from tracker Object-enter Object-found Object-exit Object-lost Object-stopped

Initially, tracker can not figure out class label, When object exit, tracker has enough information to

assign a class label to the object

Page 39: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

An Example of Generating Events

Page 40: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Parsing Events

Page 41: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Sample Stochastic Context-Free Grammar

Page 42: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Tracker and Event Generator Data for Parser States

Tracker event generator provides data for “low mark” and “high mark” of parser states

f: frame number t: timing stamp (x,y): location (dx,dy): velocity

h

h

h

h

h

h

l

l

l

l

l

l

dy

dx

y

x

t

f

h

dy

dx

y

x

t

f

l

Page 43: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Distance Penalty Function

: high mark data of state being completed :low mark of the completing state

Where is the predicted position of the object at time

)(

else ,)()(

exp

0)( if ,0

),,,(

1211

22

12

221

ttdrrr

rrrr

tt

rrttf

p

pT

pp

2t

111 ,, drrt22 , rt

Page 44: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

An Example

Page 45: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Events Data for Drive-In and Drop-Off Activities

Page 46: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000

Vedio Frame Illustration

Person passed through Person drove in Person drop off Car passed through