1. introductiondmqm.korea.ac.kr/uploads/seminar/open set recognition... · 2020-01-03 · - 20 / 72...

발표자 : 백승호

- 2 / 72 -

1. Introduction

- 4 / 72 -

- 5 / 72 -

- 6 / 72 -

- 7 / 72 -

- 8 / 72 -

English Math

ScienceArt

- 9 / 72 -

English Math

ScienceArt

- 10 / 72 -

Training

- 11 / 72 -

Test

- 12 / 72 -

Test

- 13 / 72 -

Test

- 14 / 72 -

Test

- 15 / 72 -

2. Recent Advances in Open Set Recognition

- 17 / 72 -

Knowns Unknowns

Kn

ow

ns

Un

kn

ow

ns

Knowns Knowns

(KKC)

Knowns Unknowns

(KUC)

Unknowns Knowns

(UKC)

Unknowns Unknowns

(UUC)

Knowledge of State(Data available)

Kn

ow

led

ge o

f D

om

ain

(Un

der

stan

din

g)

High Low

Low

Hig

h

Data 有

Label 無

Data 有

Label 有

Data 無

Label 有

Data 無

Label 有

- 18 / 72 -

Training

- 19 / 72 -

Data 無

Label 有

Knowns Unknowns

Kn

ow

ns

Un

kn

ow

ns

Knowns Knowns

(KKC)

Knowns Unknowns

(KUC)

Unknowns Knowns

(UKC)

Unknowns Unknowns

(UUC)

Knowledge of Domain(Understanding)

Kn

ow

led

ge o

f St

ate

(Dat

a av

aila

ble

)

High Low

Low

Hig

h

Data 有

Label 無

Data 有

Label 有

Data 無

Label 有

- 20 / 72 -

Data 無

Label 有

Knowns Unknowns

Kn

ow

ns

Un

kn

ow

ns

Knowns Knowns

(KKC)

Knowns Unknowns

(KUC)

Unknowns Knowns

(UKC)

Unknowns Unknowns

(UUC)


Kn

ow

led

ge o

f St

ate

(Dat

a av

aila

ble

)

High Low

Low

Hig

h

Data 有

Label 無

Data 有

Label 有

Data 無

Label 有

- 21 / 72 -

Data 無

Label 有

Knowns Unknowns

Kn

ow

ns

Un

kn

ow

ns

Knowns Knowns

(KKC)

Knowns Unknowns

(KUC)

Unknowns Knowns

(UKC)

Unknowns Unknowns

(UUC)


Kn

ow

led

ge o

f St

ate

(Dat

a av

aila

ble

)

High Low

Low

Hig

h

Data 有

Label 無

Data 有

Label 有

Data 無

Label 有

- 22 / 72 -

Data 無

Label 有

Knowns Unknowns

Kn

ow

ns

Un

kn

ow

ns

Knowns Knowns

(KKC)

Knowns Unknowns

(KUC)

Unknowns Knowns

(UKC)

Unknowns Unknowns

(UUC)


Kn

ow

led

ge o

f St

ate

(Dat

a av

aila

ble

)

High Low

Low

Hig

h

Data 有

Label 無

Data 有

Label 有

Data 無

Label 有

심해물고기 ??

- 23 / 72 -

Data 無

Label 有

Knowns Unknowns

Kn

ow

ns

Un

kn

ow

ns

Knowns Knowns

(KKC)

Knowns Unknowns

(KUC)

Unknowns Knowns

(UKC)

Unknowns Unknowns

(UUC)


Kn

ow

led

ge o

f St

ate

(Dat

a av

aila

ble

)

High Low

Low

Hig

h

Data 有

Label 無

Data 有

Label 有

Data 無

Label 有

심해물고기 ??아몰랑~

- 24 / 72 -

Training Testing Goal

Traditional classification KKC KKC Classifying KKC

Classification

with Reject OptionKKC KKC

Classifying KKC &

rejecting samples of

low confidence

One-class Classification

(Anomaly Detection)

KKC & few or none outliers

from KUCs

KKC &

few or none outliersDetecting outliers

One-shot Learning

(Few-shot Learning)

KKC & a limited number of

UKCs’ samplesUKC Identifying UKC

Zero-shot LearningKKC &

semantic-informationKKC & UKC Identifying KKC & UKC

Open Set Recognition KKC KKC & UKCIdentifying KKC &

rejecting UKC

SettingTask

- 25 / 72 -

Train Test [ Goal ]

Dog

Cat

Traditional classification

Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning

Open Set Recognition

SettingTask

- 26 / 72 -

Train Test [ Goal ]

Dog

Cat


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

SampleProbability

ClassDog Cat

1 0.97 0.03 Dog

2 0.01 0.99 Cat

3 0.23 0.77 Cat

4 0.37 0.63 Cat

𝑝𝑖 =𝑒𝑍𝑖

σ𝑗=1𝑘 𝑒𝑍𝑗

ሿL𝑎𝑏𝑒𝑙 𝑖𝑛𝑑𝑒𝑥 = 𝑎𝑟𝑔𝑚𝑎𝑥[𝑝1, 𝑝2, … , 𝑝𝑘

- 27 / 72 -

Train Test [ Goal ]

Dog

Cat

Unknown


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 28 / 72 -

Train Test [ Goal ]

Dog

Cat

Unknown

SampleProbability

ClassDog Cat

1 0.97 0.03 Dog

2 0.01 0.99 Cat

3 0.23 0.77 Cat

4 0.37 0.63 Unknown

Class = { reject, if Softmax(xi) < 𝜃

argmaxli , if Softmax(xi) ≥ 𝜃

𝜃 = ThresholdTraditional classification

Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 29 / 72 -

Train Test [ Goal ]

Dog

Unknown

Outlier


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 30 / 72 -

Train Test [ Goal ]

Dog

Unknown

Outlier


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 31 / 72 -

Train Test [ Goal ]

Dog

Unknown

Outlier


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 32 / 72 -

Train Test [ Goal ]

Dog

Unknown

Outlier


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 33 / 72 -

Train Test [ Goal ]

Dog

Unknown

Outlier


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 34 / 72 -

Train Test [ Goal ]

Dog

Unknown

Outlier


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

- 35 / 72 -

Train Test [ Goal ]

Knife


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

Ship

Air craft

- 36 / 72 -

Train Test [ Goal ]

Knife


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

Ship

Air craft

- 37 / 72 -


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

[ Why Zero Shot ? ]

1) AI 전성시대 & Data 홍수의시대(너무많은 Class가 필요)

2) 일부 Class에 대한 Label은 전문가만이가능

3) 결국기존 Classification의 성능강화

송이

영지

표고

흰주름

- 38 / 72 -


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

[ Why Zero Shot ? ]




송이

영지

표고

흰주름

- 39 / 72 -


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

[ Why Zero Shot ? ]




송이

영지

표고

흰주름

- 40 / 72 -

Train Test


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

Husky

Bull

Goal

Husky

Bull

Unknown

- 41 / 72 -

Train Test


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

Husky

Bull

Goal

Husky

Bull

Unknown

- 42 / 72 -

Train Test


Classification

with Reject Option


(Anomaly Detection)

One-shot Learning

(Few-shot Learning)

Zero-shot Learning


SettingTask

Husky

Bull

Goal

Husky

Bull

Unknown

- 43 / 72 -

Testing

- 44 / 72 -

Training

3. What I do in DMQA ?

- 46 / 72 -

[ Structure ] [ How to find 𝜃 ]

- 47 / 72 -

Convolution + ReLU Max pooling 2x2

Input WBM

… …

40x40x1

20x20x64

10x10x128

5x5x256

Normal

Center

Edge

Full fail

Probability> Threshold

Probability > Threshold Probability < Threshold

Fully Connected

Scratch

Ring

225 0 9 0 0 0

5 78 5 1 0 0

16 4 272 4 0 0

1 1 3 20 1 0

0 3 2 1 5 0

0 2 0 15 68 0

Tru

e la

bel

Rin

g(N

ew)

Full

fail

Edge

Cen

ter

No

rmal

Scra

tch

Predict label

Original Classification Proposed Classification (by max) Proposed Classification (by stdev)

Number of samples found a new pattern The number of samples that require reclassification

Normal Center Edge Full fail Ring(New)

Scratch

199 0 4 0 0 31

3 69 1 0 0 16

4 2 256 1 0 33

0 1 1 16 0 8

0 1 0 1 3 6

0 0 0 2 36 47

Tru

e la

bel

Rin

g(N

ew)

Full

fail

Edge

Cen

ter

No

rmal

Scra

tch

Predict label


Scratch

205 0 7 0 0 22

3 73 2 0 0 11

5 2 260 1 0 28

0 1 1 17 0 7

0 2 0 1 4 4

0 0 0 3 43 39

Tru

e la

bel

Rin

g(N

ew)

Full

fail

Edge

Cen

ter

No

rmal

Scra

tch

Predict label


Scratch

- 48 / 72 -

성과 한계점

1) 신규 불량 패턴 등장 인식 가능

2) 잘못된 분류의 감소

(후속 처리 품질 위험 감소)

1) 지정된 범주의 이중 불량 발생

탐지 불가

2) 신규 불량 패턴의 종류를 고려하지

않고 모두 신규 패턴(1 Class)으로 분류

(2개 이상의 신규 패턴 등장 ??)

4. DOC : Deep Open Classificationof Text Documents

- 50 / 72 -

- 51 / 72 -

…

Normal

Donut

Edge

Location

Center

각 Class 별로 Sigmoid를 최종활성화함수로갖는다.

(고전적인 Multi-task 분류기는 Softmax 사용)

각범주별확률을따로계산

- 52 / 72 -

…

Normal

Donut

Edge

Location

Center




- 53 / 72 -

…

Normal

Donut

Edge

Location

Center




- 54 / 72 -

…

Normal

Donut

Edge

Location

Center

Y

0

0

1

0

0

Loss 1

Loss 3

Loss 5

Loss 2

Loss 4

Total loss = σ𝑖=15 𝐿𝑜𝑠𝑠 𝑖

- 55 / 72 -

…

Normal

Donut

Edge

Location

Center

Y

0

0

1

0

0

Loss 1

Loss 3

Loss 5

Loss 2

Loss 4


- 56 / 72 -

…

Normal

Donut

Edge

Location

Center

Y

0

0

1

0

0

Loss 1

Loss 3

Loss 5

Loss 2

Loss 4


𝜃𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙

𝜽DOC

- 57 / 72 -

…

Normal

Donut

Edge

Location

Center

Y

0

0

1

0

0

Loss 1

Loss 3

Loss 5

Loss 2

Loss 4


𝜃𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙

- 58 / 72 -

- 59 / 72 -

1) Class i의예측확률 𝒑(𝒚 = 𝒍𝒊|𝒙𝒋, 𝒚𝒋=𝒍𝒊)이

가우시안분포의절반을따른다고가정

1

- 60 / 72 -



1

2) 기존의점 𝒑(𝒚 = 𝒍𝒊|𝒙𝒋, 𝒚𝒋=𝒍𝒊)에대해

평균 “1”에미러링된 점을만든다

𝒎𝒊𝒓𝒓𝒐𝒓𝒆𝒅 𝒑𝒐𝒊𝒏𝒕 = 𝟏 + (𝟏 − 𝒑(𝒚 = 𝒍𝒊|𝒙𝒋, 𝒚𝒋=𝒍𝒊))

ex) 𝒑(𝒚 = 𝒍𝒊|𝒙𝒋, 𝒚𝒋=𝒍𝒊)→ 𝒎𝒊𝒓𝒓𝒐𝒓𝒆𝒅 𝒑𝒐𝒊𝒏𝒕 = 𝟏. 𝟐

- 61 / 72 -



1





- 62 / 72 -



1





- 63 / 72 -



1





- 64 / 72 -



1





- 65 / 72 -



1





3) 기존점과생성된점을사용하여표준편차 (𝝈)를추정

𝝈

- 66 / 72 -



1





3) 기존점과생성된점을사용하여표준편차 (𝝈)를추정

𝝈

4) 𝜽𝑫𝑶𝑪 𝒕𝒊 = 𝒎𝒂𝒙 𝟎. 𝟓 , 𝟏 − 𝜶 ∙ 𝝈𝒊

𝜶는일반적으로 3을사용

𝜽𝑫𝑶𝑪(𝒕𝒊) = 𝒎𝒂𝒙(𝟎. 𝟓 , 𝟏 − 𝜶 ∙ 𝝈𝒊)

- 67 / 72 -

일반적인 Sigmoid func.의임계값은 0.5이다

Train에 참가하지않은 Unseen 표본에대한확률을 고려必

DOC는잠재적인 Unseen 표본에대한확률을고려 하여임계값을재설정

각임계값은 Class 별로차등 적용

- 68 / 72 -

…

Normal

Donut

Edge

Location

Center

- 69 / 72 -

Data-set 1 Data-set 2

DOCU. News Review

Class 20 50

# of Samples(in 1 class)

1000 1000

Train60%

Test30%

Valid10%

- 70 / 72 -

Unknown을많이가지고있더라도성능저하가적다

DOC가 DOC(𝒕 = 𝟎. 𝟓)보다성능이높다

5. Conclusion

- 72 / 72 -

감사합니다.