Applications to Science

Pietro PeronaCalifornia Institute of Technology

NSF Workshop - Frontiers in VisionCambridge, 23 Aug 2011

Friday, August 26, 2011

Goals

• A few examples

• Implications for machine vision

• Lessons learned

• NSF’s role

Friday, August 26, 2011

Plan

• Intro (5’)

• Sketch of a few success stories (50’)

• Discussion (10’)

Friday, August 26, 2011

‘Lunging’ (view from top)

Friday, August 26, 2011

Why measure behavior

Friday, August 26, 2011

Why measure behavior

• Genes <<>> Brains <<>> Behavior

Friday, August 26, 2011

Why measure behavior

• Genes <<>> Brains <<>> Behavior

• Ethology

Friday, August 26, 2011

Why measure behavior

• Genes <<>> Brains <<>> Behavior

• Ethology

• What is behavior?

Friday, August 26, 2011

Adapted from:Kravitz et al.PNAS April 16, 2002 vol. 99 no. 85664–5668

Fly behavior(as we understand it today)

Friday, August 26, 2011

Friday, August 26, 2011

Friday, August 26, 2011

Detection performance

[Dankert et al., Nature Methods, April 2009]

Friday, August 26, 2011

Phenotyping

[Dankert et al., Nature Methods, 2009]Friday, August 26, 2011

Ethograms

[Dankert et al., Nature Methods, April 2009]Friday, August 26, 2011

images, trajectories

pose, movemes, actions, activities, objects, scenes

plans, goals, behavior, relationships ...

SENSO

RYPSY

CH

OLO

GY

Perception

World

interaction, cooperation, competition

Friday, August 26, 2011

group-level goals and plans

individual goals and plans

motor programs

sensor-based control

IMAGING,TRACKING

RECOGNITION

THEORY OF PSYCHOLOGY

SOCIAL NETWORK

PREFRONTAL CORTEX

MOTOR CORTEX

SPINAL CORD

MO

TOR

PLA

NN

ING

Action

INDIVIDUAL

THEORY OF SOCIOLOGY

images, trajectories

pose, movemes, actions, activities, objects, scenes

plans, goals, behavior, relationships ...

SENSO

RYPSY

CH

OLO

GY

Perception

World

interaction, cooperation, competition

Friday, August 26, 2011

Lessons learned

• Image deluge in science

• Doing better than the scientists

• Payoffs in science, not in MV (short term)

‣ Must work as scientist‣ Students must be interested in science too‣ Publish in unfamiliar venues‣ CV publications are suspicious

• Benefit to MV: new challenges and datasets

• Benefit to PI: fun, learning

Friday, August 26, 2011

Basic research needed

• Tracking, detection and identification

• Parts and pose

• Hierarchical models (for time series)

• Unsupervised discovery of categories

• Weakly supervised learning

Friday, August 26, 2011