nips2015読み会: ladder networks
Post on 16-Apr-2017
10.951 Views
Preview:
TRANSCRIPT
Ladder Network
Semi-supervised learningwith Ladder NetworksNIPS, 2016/1/20Preferred Networks, @mattya1089
2
(@mattya1089)Preferred NetworksChainer-goghChainer-DCGAN
Deep Generative ModelsSemi-supervised learning with deep generative models (Kingma et al., 2014) Improving Semi-Supervised Learning with Auxiliary Deep Generative Models(Maaloe et al., 2015) Virtual Adversarial TrainingDistributional smoothing with virtual adversarial training (Miyato et al., 2015)Ladder Networks ()Semi-supervised learning with Ladder network (Rasmus et al., 2015) Deconstructing the ladder network architecture (Mohammad et al., 2016)
3
(semi-supervised learning)4
(semi-supervised learning)5
()
(semi-supervised learning)6
(semi-supervised learning)7
50%()
(semi-supervised learning)8
50%()
(semi-supervised learning)9
()
x: y: 16.4%60000 : 0.82 -> 0.82%x33
Ladder NetworkLateral connection
xxx~ytReconstruction lossSupervised lossOKxdenoisingCombination
denoising
Ladder Network
xxx~ytReconstruction lossSupervised lossReconstructionSemi-supervisedh1h2h1~h2~100 label: 16.4 -> 1.86%60000 : 0.82 -> 0.73%y~
Ladder Networkxxx~ytReconstruction lossSupervised lossVATh1h2h1~h2~100 label: 1.86 -> 1.69%60000 : 0.73 -> 0.61%NoiseNoiseNoiseNoisey~
Ladder NetworkReconstruction lossLadder Networkxxx~ytReconstruction lossSupervised lossh1h2h1~h2~100 label: 1.69 -> 1.09%60000 : 0.61 -> 0.61%NoiseNoiseNoiseNoisexxyh1h2Encodery~DecoderPriorLossweightFully-supervised
Ladder Network
Feed-forward NNDecoderReconstructionLateral connectionReconstruction loss100 labels25.8% 23.0%16.4%1.86%1.69%1.09%60000 labels1.18% 0.82%0.82%0.73%0.61%0.61%NN
Generative models, VAT, Ladderdeep unsupervised32LadderLateral connectionReconstructionLateral connectionGenerative modelsMNIST The proposed model is simple and easy to implement with many existing feedforward architectures(`)
Semi-supervised learning with Ladder network (Rasmus et al., 2015) Deconstructing the ladder network architecture (Mohammad et al., 2016) Semi-supervised learning with deep generative models (Kingma et al., 2014) Improving Semi-Supervised Learning with Auxiliary Deep Generative Models(Maaloe et al., 2015) Distributional smoothing with virtual adversarial training (Miyato et al., 2015)
https://github.com/mattya/chainer-semi-supervisedVATLadder Network (1.4%1.1%)ChainerLadder34
Ladder NetworkEncoderBatch Normalization
Ladder NetworkLateral connectionDecoderEncoderDecoderDecoderBNscale, shift
Ladder NetworkEncoderz~DecoderuCNNDenoisingz~iui
Ladder NetworkReconstruction LosscleanzreconstructzLateralzBatch normalizationminibatchEncoderCleanNoisyreconstructionzcleanNormalizationcleanz
Ladder NetworkAdamLearning rateepoch1000.0021500[0,1]DecodertopsoftmaxLinearBiasClean encoderBackpropWeight decay
top related