Upload File

a machine learning perspective on neural networks and learning tools tom schaul

  • Home
  • Documents
  • A machine learning perspective on neural networks and learning tools Tom Schaul
21
A machine learning perspective on neural networks and learning tools Tom Schaul

Upload: reyna-neale

Post on 31-Mar-2015

219 views

Category:

Documents


2 download

Report

Tags:

TRANSCRIPT

Page 1: A machine learning perspective on neural networks and learning tools Tom Schaul

A machine learning perspective on neural networks and learning tools

Tom Schaul

Page 2: A machine learning perspective on neural networks and learning tools Tom Schaul

2

Overview

PyBrain: training artificial neural networks for classification, (sequence) prediction and control

1. Neural networks– Modular structure– Available architectures

2. Training– Supervised learning– Optimization– Reinforcement learning (RL)

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 3: A machine learning perspective on neural networks and learning tools Tom Schaul

3

Disclaimer

• Only version 0.3, you may encounter– inconsistencies– bugs– undocumented “features”

• But growing– 10+ contributors– 100+ followers (github, mailing list)– 1000+ downloads

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 4: A machine learning perspective on neural networks and learning tools Tom Schaul

4

(Our) Neural Networks

• No spikes• Continuous activations• Discrete time steps

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 5: A machine learning perspective on neural networks and learning tools Tom Schaul

5

Network Structure: Modules

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Module

input

output

Parametersparameters

input error

output error

Derivativesderivatives

Page 6: A machine learning perspective on neural networks and learning tools Tom Schaul

6

Network Structure: Connections

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Module

input

output

input error

output error

Module

input

output

input error

output error

Module

input

output

input error

output error

FullConnection

Page 7: A machine learning perspective on neural networks and learning tools Tom Schaul

7

Network Structure:Graphs, Recurrency, Nesting

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Module

Module Module

Module

Module Module

Page 8: A machine learning perspective on neural networks and learning tools Tom Schaul

8

Network Components: Modules

• Module types– layers of neurons• additive or multiplicative• sigmoidal squashing functions• stochastic outputs

– gate units– memory cells (e.g. LSTM cells)– …

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 9: A machine learning perspective on neural networks and learning tools Tom Schaul

9

Network Components: Connections

• Connection – Fully connected or sparse– Time-recurrent– Weight-sharing– may contain parameters– …

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 10: A machine learning perspective on neural networks and learning tools Tom Schaul

10

Network Architectures

• Feed-forward networks, including– Deep Belief Nets– Restricted Boltzmann Machines (RBM)

• Recurrent networks, including– Reservoirs (Echo State networks)– Bidirectional networks– Long Short-Term Memory (LSTM) architectures– Multi-Dimensional Recurrent Networks (MDRNN)

• Custom-designed topologies4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 11: A machine learning perspective on neural networks and learning tools Tom Schaul

11

Overview

1. Neural networks– Modular structure– Available architectures

2. Training– Supervised learning– Optimization– Reinforcement learning (RL)

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 12: A machine learning perspective on neural networks and learning tools Tom Schaul

12

Training: Supervised Learning

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Module

input

output

Parametersparameters

input error

output error

Derivativesderivatives

compare to target

gradientupdate on

parameters

Backpropagation

Page 13: A machine learning perspective on neural networks and learning tools Tom Schaul

13

Training: Black-box Optimization

• fitness function based on e.g. MSE, accuracy, rewards

• multiple fitness values: multi-objective optimization

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Black box

update parameters

fitness

Parametersparameters

BlackBoxOptimizer

Page 14: A machine learning perspective on neural networks and learning tools Tom Schaul

14

Optimization Algorithms

• (Stochastic) Hill-climbing• Particle Swarm Optimization (PSO)• (Natural) Evolution Strategies (ES)• Covariance Matrix Adaptation (CMA)• Genetic Algorithms (GA)• Co-evolution• Multi-Objective Optimization (NSGA-II)• …4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 15: A machine learning perspective on neural networks and learning tools Tom Schaul

15

Training: Reinforcement Learning

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Agent

action observationreward

Environment

stateaction

EnvironmentTask

Expe

rimen

t

Page 16: A machine learning perspective on neural networks and learning tools Tom Schaul

16

RL: Agents, Learners, Exploration

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

action observationreward

LearningAgent

Module

Learner

DataSetExplorer

Page 17: A machine learning perspective on neural networks and learning tools Tom Schaul

17

RL: Learning Algorithms and Exploration

• Value-based RL– Q-Learning, SARSA– Fitted-Q Iteration

• Policy Gradient RL– REINFORCE– Natural Actor-Critic

• Exploration methods– Epsilon-Greedy– Boltzmann– State-Dependent Exploration

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 18: A machine learning perspective on neural networks and learning tools Tom Schaul

18

RL: Environments and Tasks

• 2D Mazes (MDP / POMDP)• Pole balancing

• 3D environments (ODE, FlexCube)

• Board games (e.g. Atari-Go, Pente)

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 19: A machine learning perspective on neural networks and learning tools Tom Schaul

19

Also in PyBrain

• Unsupervised learning and preprocessing• Support Vector Machines (through LIBSVM)• Tools– Plotting / Visualization– netCDF support– XML read/write support

• arac: fast C version

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

Page 20: A machine learning perspective on neural networks and learning tools Tom Schaul

20

References

• Source download, documentationwww.pybrain.org

• Mailing list (200+ members)groups.google.com/group/pybrain

• Feature requestsgithub.com/pybrain/pybrain/issues

• CitationT. Schaul, J. Bayer, D. Wierstra, Y. Sun, M. Felder, F. Sehnke, T. Rückstieß and J. Schmidhuber. PyBrain. Journal of Machine Learning Research, 2010.

4th FACETS CodeJam Workshop - Tom Schaul - PyBrain

http://groups.google.com/group/pybrain
http://groups.google.com/group/pybrain
http://groups.google.com/group/pybrain
http://groups.google.com/group/pybrain
Page 21: A machine learning perspective on neural networks and learning tools Tom Schaul

21

Acknowledgements

Justin BayerMartin FelderThomas RückstiessFrank SehnkeDaan Wierstra

and many more who contributed code, suggestions, bug fixes …

… and you for your attention!4th FACETS CodeJam Workshop - Tom Schaul - PyBrain


Neural Optimizer Search with Reinforcement Learning - … · Neural Optimizer Search with Reinforcement Learning ... Neural Optimizer Search with Reinforcement Learning ... Our approach

Self-Organizing Neural Systems Based on Predictive Learning Neural Systems... · Self-organizing neural systems based on predictive learning ... We review the basic TD-learning model

7 Neural Networks Learning

Neural network retuning and neural predictors of learning ...Neural network retuning and neural predictors of learning success associated with cello training Indiana Wollmana,b,c,1,

Learning Algorithm and Neural Networks

Neural Information Flow: Learning neural information

Machine Learning Michael M. Richter Neural Networks Email ...pages.cpsc.ucalgary.ca/~mrichter/ML/ML 2010/Neural nets/neural-nets 2(2010).pdfMachine Learning Michael M. Richter Neural

Keywords: Machine learning, reinforcement learning, artificial neural … · Keywords: Machine learning, reinforcement learning, artificial neural network, mobile robot, control,

Neural Network Learning Without Backpropagation

Explanation-Based Neural Network Learning for Robot …papers.nips.cc/paper/614-explanation-based-neural-network-learnin… · explanation-based neural network learning (EBNN),

Tom Schaul, Dan Horgan, Karol Gregor, Dave Silver...Universal Value Function Approximators. Tom. Schaul, Dan Horgan, Karol Gregor, Dave Silver

CSCE 636 Neural Networks (Deep Learning)Deep Learning and Neural Network Input Neural Network Output What neural network is doing: computing (often transformation of features/representations,

Neural Network Learning Rules

Neural network for machine learning

Supervised Learning in Neural Networks

Unsupervised Learning With Neural Nets Deep Learning and Neural Nets Spring 2015

Neural Networks and Deep Learning

Deep Learning in Neural Networks: An Overviewpeople.idsia.ch/~juergen/DeepLearning2July2014.pdf · Deep Learning in Neural Networks: An Overview ... numerous neural network experts

Stochastic Neural Networks Deep Learning and Neural Nets Spring 2015

Interference and Generalization in Temporal Difference ... · learning (Lopez-Paz & Ranzato,2017;Schaul et al.,2019), it forms the Neural Tangent Kernel (Jacot et al.,2018), it is

Machine Learning Deep Learning Neural nets

Neural Networks for Machine Learning Lecture 3a Learning

neural network learning

IV. Neural Network Learning

Deep Learning in Neural Networks: An Overviejuergen/DeepLearning28May2014.pdf · Deep Learning in Neural Networks: An Overview ... 1 Introduction to Deep Learning ... Neural Fitted

Artificial Neural Networks Lect3: Neural Network Learning rules

Unsupervised Neural Networks Learning Rules, BCM Learning Rule

Artificial Neural Networks Connectionist Learning Machinesci.louisville.edu/kerem/evraklar/sunum/CECS694_01_seminar.pdfArtificial Neural Networks Connectionist Learning Machines

Perceptrons and Learning Learning in Neural Networks

Convolutional neural networks for image classificationceur-ws.org/Vol-2546/paper06.pdf · Keywords: machine learning, deep learning, neural network, recognition, convolutional neural

RobustFill: Neural Program Learning under Noisy I/O · RobustFill: Neural Program Learning under Noisy I/O ... est problems in machine learning and artificial intelligence ... Neural

Artificial Neural Network Supervised Learning

Deep Learning: Feedforward Neural Nets and Convolutional ... · PDF fileDeep Learning: Feedforward Neural Nets and Convolutional Neural Nets Piyush Rai Machine Learning (CS771A) Nov

Reinforcement Learning Architectures for Deep · Dueling Network Architectures for Deep Reinforcement Learning Paper by: Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc

A Learning Analog Neural Network Chip with …papers.nips.cc/paper/778-a-learning-analog-neural...A Learning Analog Neural Network Chip with Continuous-Time Recurrent Dynamics 861