advanced artificial intelligence object trackingmi.cau.ac.kr/teaching/lecture_aai/ot.pdf ·...

46
Advanced Artificial Intelligence Creative Design | Chung-Ang University | Narration: Prof. Jaesung Lee Object Tracking Hello I’m sungmin and master student of the cvml. I will address on the object tracking task. Memo Here Sungmin cho

Upload: others

Post on 17-Jul-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Advanced Artificial IntelligenceC

reativ

e Desig

n | C

hu

ng

-An

g U

niv

ersity | N

arra

tion

: Pro

f. Ja

esun

g L

ee

Object Tracking

Hello I’m sungmin and master student of the cvml.

I will address on the object tracking task.

Mem

o H

ere

Sungmin cho

Page 2: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

This is my contents.

Tracking, Tracking’s dataset and How to evaluate tracking

And 4, 5, 6 chapters are related to case studies.

and let you know the application of tracking

Mem

o H

ere

1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Applications

Contents

Page 3: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

Do you know the difference between with detection and tracking?

Detection is finding all object that we want to find at all frames.

Tracking is keeping track of the object that we decide the target at first frame.

Mem

o H

ere

Object detection vs object tracking

• Detection

• Tracking

Detection Tracking

• object detection

vs object

tracking

• data association

• Single object

traking vs Multi

object tracking

• Tracking

Now

I am

talk

ing a

bou

t this co

nten

t.

Tracking 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

Page 4: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

Tracking is such a data association problem.

Data association is to discover interesting relations between variables in large databases.

If we get appropriate bounding boxes, tracking could be an data association between interframe objects

Mem

o H

ere

Data association

• Tracking is such a data association problem.• object detection

vs object

tracking

• data association

• Single object

traking vs Multi

object tracking

• Tracking

Now

I am

talk

ing a

bou

t this co

nten

t.

Tracking

Page 5: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee5

Video is composed of a series of video frames, and there are temporal and spatial similarities

(similar in location, size, and shape of objects in an image) between adjacent video frames.

Tracking uses these history information to track specific targets

Mem

o H

ere

Data association

• video is composed of a series of video frames

• there are temporal and spatial similarities between adjacent frames.

• Tracking uses these history information.

• object detection

vs object

tracking

• data association

• Single object

traking vs Multi

object tracking

• Tracking

Now

I am

talk

ing a

bou

t this co

nten

t.

Tracking

Page 6: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee6

In Tracking, There are single object tracking and multi-object tracking.

It depends on the number of tracked objects.

Mem

o H

ere

Single object traking vs Multi object tracking

Single Object Tracking Multi Object Tracking

• object detection

vs object

tracking

• data association

• Single object

traking vs

Multi object

tracking

• Tracking

Now

I am

talk

ing a

bou

t this co

nten

t.

Tracking

Page 7: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee7

In conclusion Tracking can be viewed as a data association,

and is to keep track of an object using first frame bounding box.

And we use the information that there are temporal and spatial between adjacent video frames.

Mem

o H

ere

Tracking

• Keep track of an object using first frame bounding box

time

• object detection

vs object

tracking

• data association

• Single object

traking vs Multi

object tracking

• Tracking

Now

I am

talk

ing a

bou

t this co

nten

t.

Tracking

Page 8: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee8

There are lots of tracking datasets.

Unlike detection's data set, class is not important.

If there are both sequences and their bounding box, it can be tracking dataset.

In recent years, data in a more diverse and wilder environment has emerged, especially in autonomous driving.

Mem

o H

ere

Tracking dataset

• Tracking dataset

• OTB dataset

• Mot dataset

Now

I am

talk

ing a

bou

t this co

nten

t.

Data 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

Page 9: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee9

Especially OTB dataset is the object tracking benchmark.

This classifies each sequence by assigning attributes that will be challenging characteristic.

For example OCC is the occlusion and OV is the out of view and so on..

Mem

o H

ere

OTB dataset

• Tracking dataset

• OTB dataset

• Mot dataset

Now

I am

talk

ing a

bou

t this co

nten

t.

Data

Page 10: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

0

MOTChanllenge dataset is for the multiple object tracking.

Unlike the usual single tracking dataset,

The MOTChanllenge dataset provides the object detection results

The ID of the object of the previous frame is traced so that it can have the id of the next frame.

Single trackers also predict bbox, whereas multi-object target tracking using this dataset, finds just id in given state.

Mem

o H

ere

Mot dataset

• Tracking dataset

• OTB dataset

• Mot dataset

Now

I am

talk

ing a

bou

t this co

nten

t.

Data

https://motchallenge.net/

Object bounding box

Page 11: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee11

There are two main ways we evaluate the tracker.

The first is to examine the overlap and the second is to compare the distances between the centers.

Mem

o H

ere

Good Tracker

• How do we know good Tracker?

• Overlap

Intersection ground truth bbox with

predicted bbox

• Location Error

The distance between ground truth bbox’s center point and

predicted bbox’s center point.

• Good Tracker

• Overlap

• Location Error

Now

I am

talk

ing a

bou

t this co

nten

t.

Evaluation 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

Page 12: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

2

Ope means one pass evaluation.

Changing the iou threshold, measure the success rate at that time.

the success rate is the percentage of results that exceed the threshold.

The value in the lower left box is the success rate when the threshold is 0.5.

Mem

o H

ere

Overlap

• OPE

one pass evaluation

• Success plots of OPE

Change the iou's threshold and measure the success rate.

If iou cross threshold, it will be successful.

• Good Tracker

• Overlap

• Location Error

Now

I am

talk

ing a

bou

t this co

nten

t.

Evaluation

Page 13: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

3

This time, changing the threshold of the center distance, mesuere center distance less than that distance,

it is judged to be precise.

The value in the lower right box is the precision when the threshold is 20 pixels.

In both two measurement, the higher the value, the better.

Mem

o H

ere

Location Error

• OPE

one pass evaluation

• Precision plot of OPE

Change the location error threshold and measure

the success rate. If location error (center dis)

is lower threshold, it will be precise.

• Good Tracker

• Overlap

• Location Error

Now

I am

talk

ing a

bou

t this co

nten

t.

Evaluation

Page 14: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

4

This is the trend of object tracking

Right side section is trackers using correlation filter.

And Left side section is tracker that use deep learning.

We will see the boxed trackes.

Mem

o H

ere

How many trackers?

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

Page 15: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

5

What is the correlation? In Wikipedia, Cross correlation is defined as above.

The meaning of this expression is the sum of the products of each signal within the range that the signal can have.

We can see that it has a peak value in similar parts.

And if you expand it to 2 dimensions, an image can have a value with a peak in a similar place.

We can use this charictericstic of correlation for tracking.

Mem

o H

ere

Correlation

What is the correlation?• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 16: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

6

Let’s see the filter frame work.

There are two step approach.

First is the prediction that use filter and find the location of the object.

Second is update step that use Bayes theorem to modify prediction pdf based on current measurement.

Mem

o H

ere

Filtering

Filtering Framework

• Two step approach

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Prediction

(using filter)

update

(filter)

Page 17: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

7

We can use FFT to get the correlation.

With fast fourier transform, you can easily obtain correlation by simply multiplying.

The inverse Fourier calculation can be obtained by multiplying the Fourier operation of f by complex conjucation of the

Fourier operation of h.

Mem

o H

ere

FFT

We can use FFT to get the correlation

In fourrier domain,

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 18: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

8

Mosse means minimum output sum of squared error.

Mosse algorithm’s goal is to find H* that statisfy these equation.

gi is 2d gaussian that has sigma 2 of f.

Fi is the FFT of input image f, and Gi is FFT of gi

So, The value H* is obtained so that the correlation value with Fi is similar to Gi.

Mem

o H

ere

MOSSE

We ca use FFT to get the correlation

In fourrier domain,

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 19: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee1

9

I'll explain it step by step.

First filter update step.

Input image fi can create the gi that be used gaussian 2d.

And fi, gi tranforms Fi, Gi by FFT, and we can find H* that minize that equation.

So we can get H.

Mem

o H

ere

MOSSE

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 20: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

0

And this is Prediction step.

Using The inverse Fast Fourier Transforms of the element-wise multiplication of Fi and H.

We can get the correlation of Input image fi using H.

We can predict the next frame’s location of object.

Mem

o H

ere

MOSSE

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 21: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

1

This is the filter frame work again.

Prediction and update are executed alternately through the sequence over and over again.

We can get the location of objcets. Using this correlation filter methods.

Mem

o H

ere

MOSSE

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 22: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

2

Mosse is an algorithm released in 2010, so performance is not so good.

However, there are still a lot of correlation trackers with good performance.

Mosse’s Precision has a value of 0.399 and Success rate has a value of 0.291 on the OTB 100.

Mem

o H

ere

Quantitative result

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 23: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

3

This is a demo video of actual code.

Take a look.

Mem

o H

ere

Qualitative reseult

• Correlation

• Filtering

• FFT

• MOSSE

• Quantitative

result

• Qualitative

reseult

Now

I am

talk

ing a

bou

t this co

nten

t.

CF

Page 24: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

4

The second method is MD net using deep learning techniques. Mem

o H

ere

MD Net 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Page 25: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

5

Mdnet means Multi-domain Convolutional Neural Networks for Visual Tracking.

Tracking objects is various. For example there are cup, juice, face, car, and athlete and so on..

It is important to get a general representation of tracked object regardless of the kinds of objects.

So MD net trains using K sequences to get the general representations.

Mem

o H

ere

MD Net

• Mdnet means Multi-domain Convolutional

Neural Networks for Visual Tracking

• Tracking address various objects.

• General representataion is important.

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Page 26: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

6

Hard negative mining is a technique used when training, which makes it robust by using hard problems.

If we want to fine rabbit positive images is rabbit images and negative image is dog images but hard negative image is

Confusing image of rabbits.

And an image that is a muffin but difficult to distinguish from a puppy.

We use these hard negative data, we can get the robust networks.

Mem

o H

ere

Hard negative mining

positive negative hard negative• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Page 27: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

7

At training phase, this network try to get the general representation features using various sequences.

There are two types of input.

Sampling positive and negative images and it can be the inputs.

At domain spefic layers train the weight of characteristic of specific object.

And Shared Layers train the weight of general representations.

Mem

o H

ere

Multi-domain learning (offline-train)

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Negative

Positive

Page 28: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

8

After passing through the network for each sampled positive and negative image,

we use Binary cross entropy Loss to learn to distinguish between positive and negative well.

And Use the k Sequences, at each sequences, domain specific layers are swaped.

But shared layers are shared.

Mem

o H

ere

Multi-domain learning (offline-train)

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Negative

Positive

Binary cross entropy

Page 29: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee2

9

The At test phase, input, also puts a positive sample (the bounding box of the first frame) and a negative sampling that

has high iou (hard negative).

And it removed the domain specific layer and integrated it into one andd trained.

We then use the linear regression model to predict the bounding box with features that passed through convolution 3.

Mem

o H

ere

Test online

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Hard Negative

Positive

Bbox regression

Classification

Page 30: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

0

Md net showed excellent performance.

In the OTB100 data set, both the Precision and the Success rate were the first among the latest trackers of 2015.

Precison is 0.909 and Success rate is 0.678.

Mem

o H

ere

Qualitative Result

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Page 31: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

1

I also tried to run MD net code. Take a look. Mem

o H

ere

Quantitative Result

• MDnet?

• Hard negative

mining

• Multi-domain

learning

(train-offline)

• Test online

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

MDNET

Page 32: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

2

Let's look at siamese, a branch of tracking. Mem

o H

ere

Siamese

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

Page 33: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

3

Siamese means being a Siamese twin ( very similar )

network for first frame, and network for detection frames appearance’s are similar

It is important to combine features for first frame and fetures for sequences frames.

Using two features and do correlation operations and then find the location of objects.

Mem

o H

ere

Why Siamese?

Siamese means being a Siamese twin ( very similar )

network for first frame, and network for detection frames appearance’s are similar

siamese FC siamese RPN

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 34: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

4

Tracking does not only fetch the bounding box but also its surroundings.

So there is an methd to fetch an appropriate size fetch for each algorithm.

In Siamese RPN, template frame is created by

And Detection frame is created by dt

Mem

o H

ere

RPN Data

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 35: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

5

Siamese rpn trained with a huge number of datasets.

The dataset is a YouTube bounding boxes dataset with 240,000 videos and a capacity of over 2TB.

Mem

o H

ere

RPN Data

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 36: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

6

At Training phase, we know first frame bounding box and sequence’s bounding box and we get the template frame and

detection frame using data acquisition method.

So using alex net we can get the each feature map, the network is expanded in the same way as the figure, and

classification and bounding box regression prediction are performed.

Mem

o H

ere

Train

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 37: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

7

Classificaion loss is binary cross entropy and bounding box regression loss use faster rcnn bounding loss.

Cross entropy loss is classification between background and foregrouds.

Bounding box regression loss is to predict bounding box. (width, height, center x, center y)

An then backpropacate and update we can train the weight for this tracking.

Mem

o H

ere

Train

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 38: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

8

This is Qualitative Result.

siamese rpn was also among the best trackers of the time.

The success rate was 0.637 and the precision was 0.851 in the OTB 100

Mem

o H

ere

Qualitative Result

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 39: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee3

9

This is the Quantitative Result for Siamese RPN.

Originally this algorithms uses lots of datasets but I can’t get this dataset so

This result are little unstable.

Mem

o H

ere

Quantitative Result

• Why Siamese?

• RPN Data

• Train

• Qualitative

Result

• Quantitative

Result

Now

I am

talk

ing a

bou

t this co

nten

t.

Siamese

Page 40: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

0

Let talk about the applications of traking. Mem

o H

ere

Applications of tracking. 1

2

3

4

5

Tracking

6

7

Data

Evaluation

Correlation Filter

MD Net

Siamese

Application

• autonomous

vehicle

• surveillance

camera

• Sport analisis

• Naver AutoCam

Now

I am

talk

ing a

bou

t this co

nten

t.

Application

Page 41: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

1

Autonomous vehicles must have tracking.

Because every moment we have to track down to know the distance to the adjacent car.

In addition, it is necessary to detect the surrounding situation and to recognize the situation

in the risk of small object tracking.

Mem

o H

ere

Autonomous vehicle

• autonomous

vehicle

• surveillance

camera

• Sport analisis

• Naver AutoCam

Now

I am

talk

ing a

bou

t this co

nten

t.

Application

Page 42: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

2

Surveillance cameras need technology to track people.

We can track criminals or track people who are in danger by tracking them.

In particular, surveillance cameras require low resolution and robust tracking technology

because of their low resolution.

Mem

o H

ere

Surveillance camera

• autonomous

vehicle

• surveillance

camera

• Sport analisis

• Naver AutoCam

Now

I am

talk

ing a

bou

t this co

nten

t.

Application

Page 43: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

3

Tracking technology is also required in Sport analysis.

because each player can be measured the distance.

And we can know the individual skill and condition of each player.

It is also a field that has been studied extensively recently for the sport pattern and analysis.

Mem

o H

ere

Sport analysis

• autonomous

vehicle

• surveillance

camera tracking

• Sport analysis

• Naver AutoCam

Now

I am

talk

ing a

bou

t this co

nten

t.

Application

Page 44: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

4

Naver has developed an application that automatically creates a “Jiccam” for each idle group named AutoCam.

These applications use tracking technology to keep track of each one.

Mem

o H

ere

Naver AutoCam

• autonomous

vehicle

• surveillance

camera tracking

• Sport analysis

• NaverAutoCam

Now

I am

talk

ing a

bou

t this co

nten

t.

Application

Page 45: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

5

Q & A

Page 46: Advanced Artificial Intelligence Object Trackingmi.cau.ac.kr/teaching/lecture_aai/OT.pdf · 2019-06-14 · Advanced Artificial Intelligence g-e Object Tracking ... Tracking • Keep

Crea

tive D

esign

| Ch

un

g-A

ng

Un

iversity

| Narra

tion

: Pro

f. Ja

esun

gL

ee4

6

Thank you for listening