neural doodle

Neural  Doodle    By  Mark  Chang  

This  Work  

•  Title:    – Seman;c  Style  Transfer  and  Turning  Two-­‐Bit  Doodles  into  Fine  Artwork    

•  Author:  – Alex  J.  Champandard    

•  Ins;tu;on:  – nucl.ai  Research  Laboratory  

•  URL:  – hKp://arxiv.org/abs/1603.01768    

Previous  Works  

•  A  Neural  Algorithm  of  Ar;s;c  Style  – Leon  A.  Gatys,  Alexander  S.  Ecker  and  MaKhias  Bethge    – hKp://arxiv.org/abs/1508.06576  

•  Combining  Markov  Random  Fields  and  Convolu;onal  Neural  Networks  for  Image  Synthesis.  – Chuan  Li  and  Michael  Wand    – hKp://arxiv.org/abs/1601.04589  

A  Neural  Algorithm  of  Ar;s;c  Style  

content  

style  

artwork  

hKp://www.slideshare.net/ckmarkohchang/neural-­‐art-­‐english-­‐version    

Combining  Markov  Random  Fields  and  Convolu;onal  Neural  Networks  …  content   style   artwork  

Patch  Based  v.s  Gram-­‐based  

gram-­‐based   patch-­‐based  content  

style  

Patch-­‐Based  Style  Transfer  

Width*Height  

Depth  style  

VGG19  canvas  

Width*Height  Depth  

Filter    Responses  VGG19  

Most  Similar  Patch  

Minimize  the  Difference  

Patch-­‐Based  Style  Transfer  

Style:  Width*Height  

Depth  �(x)

x

xs �(xs)

i(�(xs))

Filter  Responses:  

Patch  of    Filter  Response:  

Canvas:  Width*Height  

Depth   i(�(x))

Filter  Responses:  

Patch  of    Filter  Response:  

Patch-­‐Based  Style  Transfer  

Es(�(x),�(xs)) =mX

i=1

k i(�(x))� NN(i)(�(xs))k2

NN(i) = argmini

i(�(x)) · j(�(xs))

| i(�(x))| · | j(�(xs))|

Width*Height  

Width*Height  

Depth  

Most  Similar  Patch:  

Style  Loss  Func;on:  

Patch-­‐Based  Style  Transfer  

Content  Loss  Func;on:  

Regularizer:  

Ec(�(x),�(xc)) = k�(x)� �(xc)k2

Total  Loss:  

R(x) =X

i,j

(xi,j+1 � xi,j)2 + (xi+1,j � xi,j)

2

E = ↵Es(�(x),�(xs)) + �Ec(�(x),�(xc)) + �R(x)

Content:  xc

Seman;c  Style  Transfer    content  

style  

seman;c  maps  

artwork  

Patch-­‐Based  With  Seman;c  Maps  

content  

style  

patch-­‐based  only  

patch-­‐based  with  seman;c  

maps  

seman;c    maps  

Seman;c  Style  Transfer    

content  

style  

seman;c    maps  

Pixel  Labeling  

Seman;c  Style  Transfer    

Width*Height  

Depth  

Filter    Responses:  VGG19  

style  

seman;c    map  of  style  

Average  Pooling  

Width*Height  

Depth  

Pooling  Result:  

�(xs)

ms

concatenate  

Width*Height  

Depth  

Ss = �(xs)kkms

weight  of  seman;cs  

Seman;c  Style  Transfer    

style  

seman;c    map  of  style  

canvas  

seman;c    map  of  content  

Most  Similar  Patch:  

Style  Loss  Func;on:  

SSs

NN(i) = argmini

i(Ss) · j(S)

| i(Ss)| · | j(S)|

Es(S,Ss) =mX

i=1

k i(S)� NN(i)(Ss)k2

Seman;c  Style  Transfer    

Total  Loss:  

style  

seman;c    map  of  style  

canvas  

seman;c    map  of  content  

SSs

content  

E = ↵Ec(�(x),�(xc)) + �Es(S,Ss) + �R(x)

xc

x

Image  Analogy  

Image  Analogy  

Total  Loss:  

Ss S

style  seman;c    

map  of  style   canvas  seman;c    

map  of  content  

No  Content  Loss  

E = ↵Ec(�(x),�(xc)) + �Es(S,Ss) + �R(x)

Image  URL  

hKp://arxiv.org/abs/1601.04589  

hKp://arxiv.org/abs/1603.01768  

Source  Code  

•  Neural  Doodle  – hKps://github.com/alexjc/neural-­‐doodle  

•  Patch-­‐Based  Style  Transfer  – hKps://github.com/chuanli11/CNNMRF  

About  the  Speaker  

•  Email:  ckmarkoh  at  gmail  dot  com  •  Blog:  hKp://cpmarkchang.logdown.com  •  Github:  hKps://github.com/ckmarkoh  

Mark  Chang  

•  Facebook:  hKps://www.facebook.com/ckmarkoh.chang  •  Slideshare:  hKp://www.slideshare.net/ckmarkohchang  •  Linkedin:  hKps://www.linkedin.com/pub/mark-­‐chang/85/25b/847