image inpainting via generative multi-column convolutional neural networks · 2019-11-29 · image...

1

Image Inpainting via Generative Multi-column Convolutional Neural Networks Yi Wang 1 Xin Tao 1,2 Xiaojuan Qi 1 Xiaoyong Shen 2 Jiaya Jia 1,2 1 The Chinese University of Hong Kong 2 YouTu Lab, Tencent Introduction Our Model Experiments Target • Estimating suitable pixel information to fill holes in images. Motivations A typical inpainting method focuses on • Features for generation: to consider features as a group of different components and combine both global semantics and local textures. • Reliable similar patches: using neighboring patches to regularize the generated ones. • Spatial-variant constraints: pixels close to area boundary are with few choices, while the central part can be less constrained. ID - MRF regularization To take MRF-like regularization only in the training phase. For neural patches and extracted from and respectively, their relative similarity is: RS , = exp( (,) ℎ×( max ∈ , +) ), and it is further normalized as: RS , = RS , σ ∈ RS , . The final ID-MRF loss: ℒ =ℒ 4_2 + σ =3 4 _2, where ℒ = −log( 1 σ ∈ max ∈ RS(, )). Spatial variant reconstruction loss To propagate the confidence of known pixels to unknown ones, = ∗ ഥ ⊙ , where ഥ =−+ −1 and 0 =. Our confidence-driven loss is: ℒ = ||( − ( , ; )) ⊙ || 1 Adversarial loss ℒ = − ~ℙ ; + ~ℙ [(|| △ ( ) ⊙ || 2 − 1) 2 ] User studies Quantitative Evaluation Our Model Overall framework GMCNN: generative multi-column convolutional neural network; CE: context encoder; MSNPS: multi-scale neural patch synthesis; CA: contextual attention. Input CE MSNPS CA Ours Input CA Ours Input CA Ours Input CA Ours

Upload: others

Post on 29-May-2020

9 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Image Inpainting via Generative Multi-column Convolutional Neural Networks · 2019-11-29 · Image Inpainting via Generative Multi-column Convolutional Neural Networks Yi Wang1 Xin

Image Inpainting via Generative Multi-column Convolutional Neural Networks

Yi Wang1 Xin Tao1,2 Xiaojuan Qi1 Xiaoyong Shen2 Jiaya Jia1,2

1The Chinese University of Hong Kong 2YouTu Lab, Tencent

Introduction Our Model Experiments

Target• Estimating suitable pixel information to fill

holes in images.

MotivationsA typical inpainting method focuses on

• Features for generation: to consider features as a

group of different components and combine both

global semantics and local textures.

• Reliable similar patches: using neighboring

patches to regularize the generated ones.

• Spatial-variant constraints: pixels close to area

boundary are with few choices, while the central

part can be less constrained.

ID-MRF regularization

To take MRF-like regularization only in the

training phase. For neural patches 𝐯 and 𝐬 extracted

from 𝐘𝑔𝐿 and 𝐘𝐿 respectively, their relative

similarity is: RS 𝐯, 𝐬 = exp(𝜇(𝐯,𝐬)

ℎ×( max𝐫∈𝜌𝐯 𝐘𝐿

𝜇 𝐯,𝐫 +𝜖)), and

it is further normalized as:

RS 𝐯, 𝐬 =RS 𝐯, 𝐬

σ𝐫∈𝜌𝐯 𝐘𝐿 RS 𝐯, 𝐫

.

The final ID-MRF loss:

ℒ𝑚𝑟𝑓 = ℒ𝑀 𝑐𝑜𝑛𝑣4_2 + σ𝐭=34 𝑐𝑜𝑛𝑣𝐭_2,

where ℒ𝑀 𝐿 = −log(1

𝑍σ𝐬∈𝐘𝐿max

𝐯∈𝐘𝑔𝐿RS(𝐯, 𝐬)).

Spatial variant reconstruction lossTo propagate the confidence of known pixels to

unknown ones,

𝐌𝑤𝑖 = 𝑔 ∗ ഥ𝐌𝑖 ⊙𝐌,

where ഥ𝐌𝑖 = 𝟏 −𝐌+𝐌𝑤𝑖−1 and 𝐌𝑤

0 = 𝟎.

Our confidence-driven loss is:

ℒ𝑐 = ||(𝐘 − 𝐺( 𝐗,𝐌 ; 𝜃)) ⊙𝐌𝑤||1

Adversarial lossℒ𝑎𝑑𝑣 = −𝐸𝐗~ℙ𝐗 𝐷 𝐺 𝐗; 𝜃

+𝜆𝑔𝑝𝐸𝐗~ℙ𝐗[(|| △𝐗 𝐷(𝐗)⊙𝐌𝑤||2 − 1)2]

User studies

Quantitative Evaluation

Our Model

Overall framework

GMCNN: generative multi-column convolutional neural network; CE: context

encoder; MSNPS: multi-scale neural patch synthesis; CA: contextual attention.

Input CE MSNPS CA Ours

Input CA Ours Input CA Ours

Input CA Ours

U-Finger: Multi-Scale Dilated Convolutional Network for ...€¦ · U-Finger: Multi-Scale Dilated Convolutional Network for Fingerprint Image Denoising and Inpainting Ramakrishna

Supplementary Materials: Generative Image Inpainting with ... · our proposed contextual attention to borrow information at distant spatial locations. D. Network Architectures In

Generative adversarial networks and convolutional …...Generative adversarial networks and convolutional neural networks based weather classification model for day ahead short-term

Image Inpainting via Generative Multi-column Convolutional

Joint Inpainting of RGB and Depth Images by Generative ...hvrl.ics.keio.ac.jp/hachiuma/poster_fujii_final.pdf · •Inpainting using pixels in the image [3] ØNo need of other camera

Deep Generative The Success of Models - GitHub …Generative modeling: Auto-regressive models General idea is to factorise the joint distribution: and use neural networks (e.g., convolutional

A Survey of Image Synthesis and Editing with Generative ......LAPGAN. Laplacian generative adversarial networks (LAPGAN)[30] are composed of a cascade of convolutional GANs with the

A Simple Convolutional Generative Network for Next Item … · 2018-11-30 · real-world scenarios produce a large number of “images" of differ-ent sizes, where traditional convolutional

Fully Convolutional Networks and ... - web.inf.ufpr.brweb.inf.ufpr.br/vri/wp-content/uploads/sites/7/2019/08/lucio2018fully… · Fully Convolutional Networks and Generative Adversarial

Capsule Architecture as a Discriminator in Generative Adversarial …€¦ · Variational Autoencoder VAE Deep Learning DL Convolutional Neural Network CCN Generative Adversarial

Deep Generative Models: GANs and VAE · Deep convolutional generative adversarial networks Auxiliary classifier GANs Odena, A., Olah, C., & Shlens, J. (2016). Conditional image synthesis

Perceptual Generative Adversarial Networks for Small Object ......for style transfer and inpainting, respectively. The idea of using GANs for unsupervised representation learning was

Generative Image Inpainting with Submanifold Alignment Inpainting GAN - IJCAI2019.pdf · Generative Image Inpainting with Submanifold Alignment Ang Li 1, Jianzhong Qi , Rui Zhang1,

Image Inpainting via Generative Multi-column with the aid

U-Finger: Multi-Scale Dilated Convolutional Network for ...faculty.cse.tamu.edu/ajiang/Publications/2018/ECCV_Chalearn.pdf · natural image denoising/inpainting/super resolution [6,10,11,17,18],

Shape Inpainting using 3D Generative Adversarial …graphics.usc.edu/cgit/publications/papers/weiyuewa_iccv17.pdfShape Inpainting using 3D Generative Adversarial Network and Recurrent

Generative Image Inpainting With Contextual Attentionopenaccess.thecvf.com/content_cvpr_2018/papers/Yu_Generative_Image_In... · Generative Image Inpainting with Contextual Attention

FPD-M-net: Fingerprint Imagechalearnlap.cvc.uab.es/.../eccv18_presentation_sukesh.pdfIIIT Hyderaba d FPD-M-net: Fingerprint Image Denoising and Inpainting Using M-Net Based Convolutional

Invertibility of Convolutional Generative Networks …...Invertibility of Convolutional Generative Networks from Partial Measurements Fangchang Ma* MIT [email protected] Ulas Ayaz MIT [email protected]

1 Region-wise Generative Adversarial Image Inpainting for ...Fig. 1: Image inpainting results for large missing areas (discontiguous at the top row, and contiguous at the bottom row),

A Simple Convolutional Generative Network for Next Item ... · using residual network to wrap convolutional layer(s) by residual block.To the best of our knowledge, this isalsothe

A Simple Convolutional Generative Network for Next Item

Image Inpainting via Generative Multi-column Convolutional …papers.nips.cc/paper/7316-image-inpainting-via... · 2019-02-19 · Image Inpainting via Generative Multi-column Convolutional

Generative Adversarial Networks (GANs) - Ian Goodfellotions. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to gen-erate highly compelling images of

Fingerprint Inpainting with Generative ModelsFingerprint Inpainting with Generative Models s1739461 s1313429 s1771851 (Group 107) Abstract We propose and test two generative models

Generative Image Inpainting with Submanifold Alignment · alignment for improving structural consistency, and 2) a patch-level alignment for improving textural details. Experimental

Deep Quantization: Encoding Convolutional … Quantization: Encoding Convolutional Activations with Deep Generative Model Zhaofan Qiu, Ting Yao, and Tao Mei University of Science and

A Deep Convolutional Generative Adversarial Networks … · In this paper, we present a semi-supervised learning method that is based on the standard deep convolutional generative

Conditional generative adversarial nets for convolutional ... · Conditional generative adversarial nets for convolutional face generation Jon Gauthier Symbolic Systems Program, Natural

Hao-Wen Dong (Herman) - Convolutional Generative ......stand B-7 Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation Hao-Wen Dong and

Combining Generative and Discriminative Representation ... · PDF file1 Combining Generative and Discriminative Representation Learning for Lung CT Analysis with Convolutional Restricted

Introduction to Generative Adversarial Networks · Traditional MLE Approach ... convolutional generative adversarial networks. 2015. (3) Mirza, Mehdi and Osindero, Simon. Conditional

MIDINET: A CONVOLUTIONAL GENERATIVE ADVERSARIAL … · MidiNet: A Convolutional Generative Adversarial Network for ... genre specicity Bach chorale ... tionally chord progressions

Generative adversarial networks and convolutional neural ...webx.ubi.pt/~catalao/ECM-D-18-04970R2-Revised.pdf1 Generative adversarial networks and convolutional 2 neural networks based

Hypothetical Exploration on Generative Adversarial Networks · Convolutional Neural Networks, Generative Adversarial Net- works, Laplacian Pyramid of Adversarial Networks. I. INTRODUCTION