wbc master thesisdefense

Protect us against infection

Lookout for signs of disease

Microscopic scale: 7,000 to 25,000 in a drop of blood

2

“The little warriors of your body”

1. Flowing

2. Rolling

3. Adhering

3Microscopy Video of white blood cells

Biologists learn cell behaviors by tracking a few cells

4

Tedious

Expensive

Subjective

Little Effort

Inexpensive

Objective

5

Ray et al. [2002]

Active Contour

Cui et al. [2005]

Monte Carlo

Mukherjee et al. [2004]

Level Set Analysis

6

Eden et al. [2005]SmoothnessConstraints

Li et al. [2005]Lineage

Construction

Smith et al. [2008]Probabilistic

Formalization

7

8

Ray Cui Mukherjee Eden Smith Li Ours

Tracker Type S S M M M M M

Presentation C C L P P P P

Entries/ Exits

Occlusion

Various Motions

Collision

Optimal Solution

S : single cell M : multiple cells C: contour L: level set P: point

As many cells move at a wide range of speed

9

In a collision, cell motion and appearance

1. could be different

2. changeabruptly

10

Eden et al. [2005]

Our Method11

broken tracks

robust tracks

having separate collision statesto describe cells inside and outside of collisions

testing multiple hypothesesof cell motion and appearance as transitions between abrupt motion patterns.

To improve tracking accuracy of colliding cells by:

12

Classify each pixel in the image as a Cell or Background13

14

Variation in cell appearances within an image

Varied appearance of a cell over time

time

15

Idea: combine many “rules of thumb” to a highly accurate prediction rule.

Input: visual features from training samples.

Schema: maintain a strategy to determine “rules of thumb” using weight distribution.

Output: a single strong classifier which is a linear combination of the set of weak classifiers.

1. Mean Intensity2. Standard Dev. of Intensity3. Radial Mean

Decision Rules on feature scores16

1. Scan each pixel p in the image

2. Compute image feature vector V from a window centered around p

3. Classify p as a Cell pixel if the feature score in Vsatisfies the learned decision rule; otherwise classify p as a Background pixel.

4. Cluster groups of Cell pixels into cell observation.

17

1. Predictusing multiple hypotheses

2. Correspondpredictions and measurements

3. Updatebased on the current state

18

19

1. Extensively used for tracking.

2. Estimator of the state that is optimal.

3. Consist of two steps:

• Predict: projects next time step state.

• Correct: incorporates a new measurement into a priori estimate to obtain an improved posteriori estimate.

Kalman ?

smooth abrupt

H00: No collision – No collisionH11: Collision – Collision

H01: No collision – CollisionH10: Collision – No collision

Collision States 20

21

State Vector*:

Observation Vector:

Collision States:

Hypotheses:

State Transition:

Motion and Appearance Models:

state transitionmatrix

control input matrix

process noise vector

~N(0,Qs)

(for )

control input vector

to predict the state in the next frame

Multiple Hypotheses

No Collision-No Collision

H00

No Collision

– Collision

H01

Collision –Collision

H11

Collision –No Collision

H10

Motion

Position KalmanCollision location

From collision

speed

From restoredvelocity

Velocity KalmanApproaches

zeroApproaches

zeroRestore to

pre-collision

Appearance Area ConstantSum of

colliding cells

Sum of colliding

cells

Restore to pre-collision

22

23

0. Initial state

24

1. Predict motion

25

2. Predict collision

26

3. Get measurements

27

4. Get errors in position & area

28

5. Correspond predictions to measurements

no measurement

no predictioncorrespond to H00

correspond to H01

29

6. Update

lost cell

new cell

non-colliding cell

colliding cells

30

7. Repeat for the next time step

stay colliding

split away

keep moving

start moving

keep moving

31

32

33

34

1. Smoothness Constraints (SC)

2. Single Hypothesis (SH)

3. Multiple Hypotheses (MH)

Same detection results

Same blood flow directions

Same tracking parameters (SH & MH)

35

Recall : TP / (TP + FN)

Precision: TP / (TP + FP)

36

Dataset SC SH MH

Colliding positions 7.11 7.08 6.91

Non-colliding positions 5.63 5.99 5.78

All positions 5.82 6.15 5.95

RMSE : Root mean squared errors of position (pixel)

1. SH introduces additional error in positions.

2. MH does not introduce any additional error.

3. Estimating colliding cells’ positions is more difficult.

-0.03 -0.17

+0.36 -0.21

+0.33 -0.20

37

Dataset SC SH MH

Colliding positions 36 63 72

Non-colliding positions 53 76 79

All positions 49 73 77

PTP: Percentage of Tracked Positions (%)

1. Large improvement in colliding positions.

2. Improvement overall.

3. Tracking colliding cells is more difficult.

+27 +9

+23 +3

+24 +4

+28

MH SC SH

38

39

The effect of collision duration on tracking

1. Exclude SC from being considered for collision.

2. Classify colliding positions into bins based on the number of frames of the collision.

40

The effect of collision duration on RMSE

41

The effect of collision duration on PTP

42

The impact of detection on tracking

Data with good detection results before and after collision (+/- 2 frames)

43

The impact of detection on RMSE

Dataset SH MH

Colliding positions 7.08 6.91

Colliding positions with good detection

6.95 5.86

-0.13 -1.05

-0.17

-1.09

1. Different improvement between dataset.

2. Different improvement between methods.

44

The impact of detection on PTP

Dataset SH MH

Colliding positions 63 72

Colliding positions with good detection

70 88

+7 +16

+9

+18

1. Large improvement between dataset.

2. Large different between methods.

3. MH achieves high performance in tracking.

Our Method

45

1. Add more features to improve detection.

46

2. Incorporate a probabilistic approach to transition between collision states.

47

3. Expand to track cells with more complex motions and behaviors.

48

49

1. A method for tracking colliding cells.

2. Incorporate Kalman filter and multiple hypotheses for each collision state.

3. Improve 28% in tracked position coverage

compared to a previous work .

4. Achieve 88% in tracked position coverage in

tracking colliding cells.

50

S. J. Schmugge, S. Keller, N. Nguyen, R. Souvenir, T. H. Huynh, M. Clemens, M. C. Shin. "Segmentation of Vessels Cluttered with Cells using a Physics based Model". 11th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), New York, September 6-10 2008.

N. Nguyen, S. Keller, T. Huynh, M. Shin. “Tracking Colliding Cells”. IEEE Workshop on Applications of Computer Vision (WACV), Snowbird, UT December 07-09 2009.

to be submitted to IEEE Transactions on Medical Imaging

N. Nguyen, S. Keller, Eric Norris, T. Huynh, M. Shin. “Tracking Colliding Cells in Intravital Microscopy Images”.

51

52

Eden et al. [2005]

Our Method

53

1. Predict motion

2. Predict collision

3. Get measurements

4. Get errors in position & area

5. Match with minimal error

State Vector*:

Observation Vector:

Collision States:

Hypotheses:

Motion and Appearance Models:

state transitionmatrix

control input matrix

process noise vector

~N(0,Qs)

(for )

control input vector

to predict the state in the next frame

Measurement Model:

measurement transition matrix

measurement noise vector

~N(0,R)54

Predicted State Vector:

Predicted Covariance:

H00 H01 H11 H10

A

B

*

State Vector of cell i :

Zero Matrix Zero Matrix

Zero Vector Zero Vector

55

X1

X2

X3

Xk

Cell Detector

z1

z3

zn

z2

x’1a

x’1b

x'2a

x’2b

x’3a

x’3b

x’ka

x’kb

HФ

HФ

HФ

HФ

HФ

HФ

HФ

HФ

z1x’1a

x’2b z3

znx’kb

x’3a

z2

Cell 3 is missing

New cell has entered

X1

X2

X3

Ck

X4

Calculate error between all possible pairs

Sm & On

Criteria: Position & Area.

Assign pairs via Greedy Search

2. Correspondence

56

Predicted State Vector:

Hypothesized Measurement Vector:

measurement transition matrix

Error of hypothesis :

observation from the detector

weight vector

Rule 1:

Rule 2:

error threshold of Unlikely (i, k) pair

Stop corresponding condition:

57

Remaining Observation : new cell

leukocyte typicaldiameter

area

Remaining Cell :

Not corresponded for 3 frames:

Updated State Vector :

Kalman gain

Updated Covariance:

depends on the cell current state s Eliminate the affect cause by abrupt change in collision

58

59

Robert Schapire algorithm (1996).

Idea: Combine many “rules of thumb” a highly accurate

prediction rule.

Maintain a strategy to determine “rules of thumb” weight.

Terms: Learner = Hypothesis = Classifier.

Weak Learner: <50% error rate.

Strong Learner: linear combination of weak learner.

60

Plastic AppleReal Apple

61

Weight

Color

62

Weak Learner

Optimal Guessing line

Wrong guess

Wrong

guess

63

Right guess

Gets smaller

Get smaller

Wrong guess

Gets larger

64

More weighted ones

get classify correctly

Wrong guess

Wrong guess

65

Wrong guess

Gets larger

Right guess

Gets smaller

smaller

smaller

66

67

68

1st

2nd

3rd

4th

69

Decision

Boundary

Real Apple

Region

Plastic Apple

Region

70

4. Get errors in position & area

71

7. Repeat for the next time step

NoCollision-

CollisionDuring

CollisionPost

Collision

Position KalmanCollision location

From collision

speed

From restoredvelocity

Velocity KalmanApproaches

zeroApproaches

zeroRestore to

pre-collision

Area ConstantSum of

colliding cells

Sum of colliding

cells

Restore to pre-collision

72

73

74

Dataset SC SH MH

All cells .49 .73 .77

Non-colliding cells .53 .76 .79

Colliding cells .40 .63 .72

Treated colliding cells .51 .70 .88

+.24

+.23

+.23

+.21

+.04

+.03

+.09

+.18

75

The impact of detection on PTP

Dataset SC SH MH

Colliding positions 40 63 72

Colliding positions with good detection

51 70 88

+7 +16

1. Large improvement between dataset.

2. Large different between methods.

3. MH achieves high performance in tracking.

+11

1. Mean Intensity2. Standard Dev. of Intensity3. Radial Mean

Decision Rules on feature scores76

77

Ray[2004]

Cui[2006]

Eden[2005]

Smith[2008]

Li[2008]

Dzyubachyk

[2010]

Ours[2010]

Tracker Type Single Single Multiple Multiple Multiple Multiple Multiple

Imaging Modality in-vivo in-vivo in-vivo ex-vivo in-vitro in-vitro in-vivo

Entries/ Exits

Occlusion

Various Motions

Collision