deconvolution networks - control
TRANSCRIPT
![Page 1: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/1.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Deconvolution Networks
Johan Brynolfsson
Mathematical StatisticsCentre for Mathematical Sciences
Lund University
December 6th 2016
1 / 27
![Page 2: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/2.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Deconvolution Neural Networks
2 / 27
![Page 3: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/3.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Image Deconvolution
I ”True Deconvolution”I Using neural network to deconvolve blurring in an image.I For the unblurring to be effective, large convolutional kernels must be
used. This however is both hard to optimise and expensivecomputationally.
I L. Xu et al presents a method in A Deep Convolutional NeuralNetwork for Image Deconvolution.
3 / 27
![Page 4: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/4.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Image Deconvolution
I A simple model is used to model the blurring
y = x ∗ k (1)
I The image can be reconstructed using convolution
x = F−1 (F (y) /F (k)) = F−1 (1/F (k)) ∗ y (2)
I A Wiener deconvolution is proposed
x = F−1(
1F (k)
{|F(k)|2
|F(k)|2 + 1SNR
})∗ y = k† ∗ y (3)
4 / 27
![Page 5: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/5.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
x = k† ∗ y (4)
I For the unblurring to be effective, large convolutional kernels must beused. This however is both hard to optimise and expensivecomputationally, even with the model.
I By introducing the noise term k† becomes compact with finitesupport.
I The kernel separability is achieved using singular value decomposition,making the problem 1D
x = k† ∗ y =∑
jsj · uj ∗
(vT
j ∗ y)
(5)
5 / 27
![Page 6: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/6.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Image Deconvolution
x = k† ∗ y =∑
jsj · uj ∗
(vT
j ∗ y)
(6)
6 / 27
![Page 7: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/7.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Image Deconvolution
7 / 27
![Page 8: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/8.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Image Deconvolution
8 / 27
![Page 9: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/9.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Deconvolution Networks for Semantic Segmentation
9 / 27
![Page 10: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/10.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I Instead of only wanting a class prediction we also want a localisationof where in the image the object(s) is(are).
I One way to solve this is to produce 5 values - image class + twobounding box coordinates.
10 / 27
![Page 11: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/11.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Semantic Segmentation
11 / 27
![Page 12: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/12.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I Very naive approach: Pixel by pixel classification by sending separatepatches into neural network.
I Non-robust and noisy
12 / 27
![Page 13: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/13.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I Less naive approach proposed by Sermanet et al in IntegratedRecognition, Localisation and Detection using ConvolutionalNetworks
I Pixel by pixel classification by sending separate patches into neuralnetwork.
I Repeat using multiple different size of boxes
13 / 27
![Page 14: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/14.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I Farabet et al proposes a multi-scale setup in Learning HierarchicalFeatures for Scene Labeling, together with an image sectionsegmentation
14 / 27
![Page 15: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/15.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Deconvolution
I Disliked name, but seems to have have stuckI Also called transpose convolution, fractional strided convolution, or
backwards convolution.
15 / 27
![Page 16: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/16.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I Long et al proposes deconvolution in Fully Convolutional Networksfor Semantic Segmentation
16 / 27
![Page 17: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/17.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Unpooling
I Used to sample up the image between deconvolutionI First proposed to be used naively, by up-sampling each pixel to e.g.
2x2 pixels.
17 / 27
![Page 18: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/18.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Unpooling
I Noh et al proposes in an improved unpooling in LearningDeconvolution Network for Semantic Segmentation
I In each max-pooling layer, the coordinates of max-value is stored.I In a corresponding unpooling layer, values from a previous layer are
entered into stored coordinated, setting the rest of the pixels to zero.
I By doing this, localisation information is included in the unpooling.
18 / 27
![Page 19: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/19.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Deconvolution Networks
I Noh et al proposes a VGG 16-layer net, with classification layerremoved
I The structure is reversed, with deconvolution and unpooling, toproduce a semantic segmentation
19 / 27
![Page 20: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/20.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I An example of deconvolution/unpooling
20 / 27
![Page 21: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/21.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Neural network deconvolution for auto-encoding
I The deconvolution network may also be used for auto encoding,where the image itself is given as ground-truth
https://github.com/mikesj-public/convolutional_autoencoder/tree/master
21 / 27
![Page 22: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/22.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
Dose planning system
22 / 27
![Page 23: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/23.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
MRI
23 / 27
![Page 24: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/24.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
MRI
24 / 27
![Page 25: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/25.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I Physicians spend ∼ 1h / scan segmenting!I Huge amount of data ( 500x500x250 = 62’500’000 voxels )I Noisy dataI Poor resolutionI Arbitrary intensitiesI Tumors, hernias, prosthetics, tatoos etc.
25 / 27
![Page 26: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/26.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
I A successful solution is to extract data around each voxel
I Use multiple networksI Cure segmentation using shape awareness
26 / 27
![Page 27: Deconvolution Networks - Control](https://reader030.vdocuments.site/reader030/viewer/2022013012/61cefa64de21f80a02423a70/html5/thumbnails/27.jpg)
Image deconvolution using Neural NetworksDeconvolution Neural Network for Semantic Segmentation
HOMEWORK(One of the following is sufficient)
I Install and test the deconvolution network for semantic segmentation( caffe )https://github.com/HyeonwooNoh/DeconvNet
I Install and test deconvolution for auto-encoding and compare withfully connected auto-encoding for the mnist dataset ( theano )https://github.com/mikesj-public/convolutional_autoencoder/tree/master
27 / 27