Deep Anomaly Detection

Kang, Min-GukMingukkang1994@gmail.com

Jan. 16, 2019

1/47

Contents

1. Introduction

2. Conventional Anomaly Detection

3. Deep Anomaly Detection

4. What is important?

2/47

3/47

Introduction of Anomaly Detection(Vision data only)

1. Introduction

Anomaly Detection is the process of identifying the new or unexplained set of data to determine

if they are within the norm or outside of it.

4/47

1. Introduction

OODD: Out Of Distribution Detection!

5/47

Patient’s Heart rate data

1. Introduction(Time Series Data)

QTUM Coin

6/47

Assembly Inspection

1. Introduction(Vision Data)

Welding Defect

7/47

1. Introduction(Vision Data)

8/47

Welding Defect

1. Introduction

9/47

Wasserstein GAN

VAE

Conventional Anomaly Detection(Vision data only)

10/47

Before dealing with Anomaly Detection, It is essential to identify the definition of the problem.

→ Domain Knowledge

2. Conventional Anomaly Detection

11/47

PCA

2. Conventional Anomaly Detection

https://www.tableau.com/resource/top-10-big-data-trends-2017

outlier

12/47

KDE

2. Conventional Anomaly Detection

https://www.tableau.com/resource/top-10-big-data-trends-2017

Normal

outlier

𝒇𝒉 𝒙 =𝟏

𝒏

𝒊=𝟏

𝒏

𝑲𝒉(𝑿 − 𝑿𝒊) =𝟏

𝒏𝒉

𝒊=𝟏

𝒏

𝑲(𝑿 − 𝑿𝒊

𝒉)

13/47

2. Conventional Anomaly Detection

Support Vector Data Description(SVDD)

https://link.springer.com/article/10.1023/B:MACH.0000008084.60811.49

Isolation forests(IF)

https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf

14/47

https://en.wikipedia.org/wiki/Anomaly_detection#cite_note-12

15/47

Deep Anomaly Detection(Vision data only)

NeuralNetworks

𝑍1

𝑍2

Scatter

3. Deep Anomaly Detection(Representation Learning)

16/47

NeuralNetworks

𝑍1

𝑍2

Scatter

17/47

3. Deep Anomaly Detection(Representation Learning)

NeuralNetworks

𝑍1

𝑍2

NeuralNetworks

𝑍1

𝑍2

18/47

3. Deep Anomaly Detection(Representation Learning)

normal

abnormal

① Non linear Classifier

Deep One-Class Classification

http://proceedings.mlr.press/v80/ruff18a.html

DCAE(Deep Convolutional Autoencoder)

Deep Anomaly Detection Using Geometric Transformation

https://arxiv.org/pdf/1805.10917.pdf

3. Deep Anomaly Detection(Representation Learning)

19/47

NeuralNetworks

𝑍1

𝑍2

NeuralNetworks

L1, L2 distance

3. Deep Anomaly Detection(Reconstruction Error)

20/47

Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery(IPML, 2017)

Adversarially Learned One-Class Classifier forNovelty Detection(CVPR, 2018)

NeuralNetworks

𝑍1

𝑍2

NeuralNetworks

High L1, L2 distance

3. Deep Anomaly Detection(Reconstruction Error)

21/47

Generator Discriminator 0(fake) or 1(real)

𝑃𝑑𝑎𝑡𝑎(𝑥)

𝐺(𝑧)

3. Deep Anomaly Detection(GANs)

22/47

3. Deep Anomaly Detection(GANs)

23/47

Generator

Generator𝜃

Discriminator∅ 1

𝐺(𝑧)

𝐺𝑙𝑜𝑠𝑠 = 𝑎𝑟𝑔𝑚𝑖𝑛𝜃𝔼𝑧 ~ 𝑝(𝑧)[log(1 − 𝐷(𝐺(𝑧)))]

3. Deep Anomaly Detection(GANs)

24/47

Generator𝜃

Discriminator∅ 1

𝐺(𝑧)

𝐺𝑙𝑜𝑠𝑠 = 𝑎𝑟𝑔𝑚𝑖𝑛𝜃𝔼𝑧 ~ 𝑝(𝑧)[log(1 − 𝐷(𝐺(𝑧)))]

3. Deep Anomaly Detection(GANs)

25/47

3. Deep Anomaly Detection(GANs)

26/47

Discriminator

D(G(z)) = 0

𝑃𝑑𝑎𝑡𝑎(𝑥)

𝐺(𝑧)

𝐷𝑙𝑜𝑠𝑠 = 𝑎𝑟𝑔𝑚𝑎𝑥∅𝔼𝑥 ~ 𝑝𝑑𝑎𝑡𝑎 𝑙𝑜𝑔𝐷 𝑥 + 𝔼𝑧 ~ 𝑝(𝑧)[log(1 − 𝐷(𝐺(𝑥)))]

D(X) = 1

Generator𝜃

Discriminator∅

3. Deep Anomaly Detection(GANs)

27/47

3. Deep Anomaly Detection(GANs)

28/47

Repeat Over and Over

D(G(z)) = 0.5

𝑃𝑑𝑎𝑡𝑎(𝑥)

𝐺(𝑧)

D(X) = 0.5

Generator𝜃

Discriminator∅

3. Deep Anomaly Detection(GANs)

29/47

DCGenerator

DCDiscriminator 0(fake) or 1(real)

𝑃𝑑𝑎𝑡𝑎(𝑥)

𝐺(𝑧)

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

30/47

Sampler(Normal instances)

3. Deep Anomaly Detection(AnoGAN)

Generator

Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

31/47

Generator

𝑮 𝒁𝟏

Sampler(Normal instances)

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

32/47

normal Instance

𝑮 𝒁𝟏

𝑿𝒕𝒂𝒓𝒈𝒆𝒕

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

Generator

33/47

𝑳𝒂𝒏𝒐_𝟏 = | 𝑿𝒕𝒂𝒓𝒈𝒆𝒕 − 𝑮 𝒁𝟏 |

normal Instance

𝑮 𝒁𝟏

𝑿𝒕𝒂𝒓𝒈𝒆𝒕

Generator

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

34/47

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

Discriminator 𝑳𝒂𝒏𝒐_𝟐 = |𝑫( 𝑿𝒕𝒂𝒓𝒈𝒆𝒕) − 𝑫(𝑮 𝒁𝟏 )|

𝐿𝑎𝑛𝑜𝑚𝑎𝑙𝑦 = (1 − 𝜆) × 𝐿𝑎𝑛𝑜_1 + 𝜆 × 𝐿𝑎𝑛𝑜_2

𝑳𝒂𝒏𝒐_𝟏 = | 𝑿𝒕𝒂𝒓𝒈𝒆𝒕 − 𝑮 𝒁𝟏 |

normal Instance

𝑮 𝒁𝟏

𝑿𝒕𝒂𝒓𝒈𝒆𝒕

Generator

35/47

Fix

Fix

Update latent vector to reduce 𝑳𝒂𝒏𝒐𝒎𝒂𝒍𝒚!

Discriminator 𝑳𝒂𝒏𝒐_𝟐 = |𝑫( 𝑿𝒕𝒂𝒓𝒈𝒆𝒕) − 𝑫(𝑮 𝒁𝟏 )|

𝐿𝑎𝑛𝑜𝑚𝑎𝑙𝑦 = (1 − 𝜆) × 𝐿𝑎𝑛𝑜_1 + 𝜆 × 𝐿𝑎𝑛𝑜_2

𝑳𝒂𝒏𝒐_𝟏 = | 𝑿𝒕𝒂𝒓𝒈𝒆𝒕 − 𝑮 𝒁𝟏 |

normal Instance

𝑮 𝒁𝟏

𝑿𝒕𝒂𝒓𝒈𝒆𝒕

Generator

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

36/47

Discriminator 𝑳𝒂𝒏𝒐_𝟐 = |𝑫( 𝑿𝒕𝒂𝒓𝒈𝒆𝒕) − 𝑫(𝑮 𝒁𝟏 )|

𝐿𝑎𝑛𝑜𝑚𝑎𝑙𝑦 = (1 − 𝜆) × 𝐿𝑎𝑛𝑜_1 + 𝜆 × 𝐿𝑎𝑛𝑜_2 ↓↓

𝑳𝒂𝒏𝒐_𝟏 = | 𝑿𝒕𝒂𝒓𝒈𝒆𝒕 − 𝑮 𝒁𝟏 |

normal Instance

𝑮 𝒁𝟏

𝑿𝒕𝒂𝒓𝒈𝒆𝒕

Generator

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

37/47

Discriminator 𝑳𝒂𝒏𝒐_𝟐 = |𝑫( 𝑿𝒕𝒂𝒓𝒈𝒆𝒕) − 𝑫(𝑮 𝒁𝟏 )|

𝐿𝑎𝑛𝑜𝑚𝑎𝑙𝑦 = (1 − 𝜆) × 𝐿𝑎𝑛𝑜_1 + 𝜆 × 𝐿𝑎𝑛𝑜_2 ↑↑

𝑳𝒂𝒏𝒐_𝟏 = | 𝑿𝒕𝒂𝒓𝒈𝒆𝒕 − 𝑮 𝒁𝟏 |

Abnormal Instance

𝑮 𝒁

𝑿𝒕𝒂𝒓𝒈𝒆𝒕

Generator

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

38/47

Generated Images are similar

to the normal images.

3. Deep Anomaly Detection(AnoGAN)Unsupervsed Anomaly Detection with Generative Adversarial Networks to Guide marker discovery (IPML, 2017)

39/47

Anomaly Detection Feature Extraction How do we extract representation of data well?

3. Deep Anomaly Detection(DSVDD)Deep One-Class Classification(ICML, 2018)

40/47

Anomaly Detection Feature Extraction How do we extract representation of data well?

3. Deep Anomaly Detection(DSVDD)Deep One-Class Classification(ICML, 2018)

𝑆𝑜𝑓𝑡 𝑆𝑉𝐷𝐷 = 𝑚𝑖𝑛𝑅,𝑊 𝑅2 +1

𝑣𝑛σ𝑖=1𝑛 max{0, | 𝜙 𝑥𝑖;𝑊 − 𝑐||2 − 𝑅2 +

𝜆

2σ𝑙=1𝐿 ||𝑊𝑙||𝐹

2

𝑆𝑉𝐷𝐷 = 𝑚𝑖𝑛𝑊

𝑖=1

𝑛

| 𝜙 𝑥𝑖;𝑊 − 𝑐2 | +𝜆

2

𝑙=1

𝐿

||𝑊𝑙||𝐹2

41/47

Anomaly Detection Feature Extraction How do we extract representation of data well?

3. Deep Anomaly Detection(DSVDD)Deep One-Class Classification(ICML, 2018)

𝑆𝑜𝑓𝑡 𝑆𝑉𝐷𝐷 = 𝑚𝑖𝑛𝑅,𝑊 𝑅2 +1

𝑣𝑛σ𝑖=1𝑛 max{0, | 𝜙 𝑥𝑖;𝑊 − 𝑐||2 − 𝑅2 +

𝜆

2σ𝑙=1𝐿 ||𝑊𝑙||𝐹

2

𝑆𝑉𝐷𝐷 = 𝑚𝑖𝑛𝑊

𝑖=1

𝑛

| 𝜙 𝑥𝑖;𝑊 − 𝑐2 | +𝜆

2

𝑙=1

𝐿

||𝑊𝑙||𝐹2

42/47

3. Deep Anomaly Detection(DSVDD)Deep One-Class Classification(ICML, 2018)

43/47

What is important?

44/47

4. What is important?

1. Domain Knowledge

2. GAN을 이용한 Anomaly Detection은 아직은 별루다.

But Mode Problem을 잘 이용하면 재미있는 것을 할 수 있을지도…

3. Anomaly Detection은 Representation을 잘 추출하는게 관건이다.

(Reconstruction, Classification)

4. 추출한 잠재변수를 잘 사용해서 비정상 점수를 잘 뽑아내야한다.

45/47

We are Hiring

• Selected as Deep Learning Best Practice at NVIDIA AI Conference 2018 Keynote.

• Use Cases for “Accelerate AI with Synthetic data using generative adversarial networks” at Strata Data Conference 2018 NY.

• ICPR 2018 Contests on Object Detection in Aerial Images 2위, 국방부 주관 제 20차 M＆S 발전 세미나 우수 논문상 수상

• CVPR 2017 NTIRE Challenge Task 2,3,4,5 위 수상, Nips 2017 DeeoArt.io Poster Contest 1등상 수상

• Kaggle “DSTL Satellite Imagery Feature Detection” Silver medal 수상

• 이외에도 SIA의 연구 내용은 CVPR, ICPR, ACML, ICML, ACM SIGSPATIAL 등 학회에서 발표되었습니다.

46/47

https://github.com/MINGUKKANG/SIA_

Thank you!

47/47