![Page 1: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/1.jpg)
An introduction to artificial neural networks and how to use them
Christopher AlbertMax-Planck-Institut für Plasmaphysik, 85748 Garching
NMPP Seminar, 25.09.2018
25.09.18 e-mail: [email protected] 1
![Page 2: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/2.jpg)
Main sources
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 2
[1] François Chollet. Deep learning with Python (Pearson, 2017)(developer of Keras)
•Machine learning for artistshttps://ml4a.github.io/
• Keras bloghttps://blog.keras.io/
• Kaggle (datasets, algorithms, challenges)https://www.kaggle.com/
![Page 3: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/3.jpg)
What is machine learning?
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 3
•Machine learning algorithms find rules relating input and output data
Relation of Machine learning to programming, AI and artificial neural networks (after [1])
Classical programming
Machine learning
Rules
Rules
Input data
Input data
Output data
Output data Artificial intelligence
Machine learning
Artificial neural
netwoks
![Page 4: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/4.jpg)
Types of machine learning
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 4
• Supervised learning• Training with correct input/output pairs• Classification• Regression / interpolation
• Unsupervised learning: machine is “on its own“• Correct output and rules unknown • Clustering
•Mixed forms existsource: https://xkcd.com/1838/, license: https://xkcd.com/license.html
![Page 5: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/5.jpg)
Some machine learning techniques
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 5
• Classical interpolation and regression• Linear regression, splines
• Stochastic methods including uncertainty• Polynomial chaos expansion, Gaussian processes
• Decision trees• Random forest, gradient boosting machine
• Clustering methods• k-Means, EM algorithm
• Artificial neural networks• Convolutional networks (recognition), LTSM (time series prediction),
autoencoder (dimensionality reduction), and many more
Rule of thumb: the moreflexible a method, themore difficult it is tointerpret its parameters.
![Page 6: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/6.jpg)
Classification and regression
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 6
• Take map !of input " ∈ $ to output % ∈ &
!:$ → &" ↦ % = !(")
• Examples:• Simulation with input parameters " ↦ result %• Pixels " ∈ 1…16 0⋅2 of an image ↦ digits % ∈ {0…9}• Microphone signal "(7) ↦ words % ∈ dictionary
•Machine learning: find an approximate map 8! ≈ !
![Page 7: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/7.jpg)
Classification and regression
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 7
•Machine learning: find an approximate map !"($;&) ≈ "($)• Depends on parameters & (e.g. spline coefficients)• Should be fast to evaluate• Should provide insight into features
• Training (parameter estimation / fit)• Based on given training data { $*, "($*) }• Minimize ‖.( !" $*, & − " $* ,&)‖ w.r.t. & for some norm ‖ ⋅ ‖• Loss function . may include weight-based regularisation to avoid overfitting
• Validation• Check correctness on test data $1 ≠ $*• Measures of goodness: ‖.‖ or classification accuracy in %
![Page 8: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/8.jpg)
Artificial neural network
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 8
• Neural network = graph• Neurons = nodes• Dendrites = edges towards nodes• Activation function !• Weights "
!#
"$#
"%#
"
"
![Page 9: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/9.jpg)
Artificial neural network
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 9
• Neural network = directed acyclic graph with summation rules• Neurons = nodes that sum input• Dendrites = edges carrying weights for node input summation• Activation function !" = transformation to add nonlinearity (prescribed)• Weights #"$ and bias %$ = (statistical) model parameters
Summation at each node, e.g. &' = !'(%' + &+#+' + &,#,')
!+
!,
!'
#+'
#,'&,
&+
!"
#+"
#,"
%'%+
%+
&'
![Page 10: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/10.jpg)
Layers and overall map
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 10
•Map generated by the network, including bias ̅"# $ = "#('# + $):*+ = ,-+ .;0 = ̅"+ ∘ 0+2 ̅"2 ∘ ⋯ ∘ 045 ̅"5 ∘ 054.4
. *Layers:
input hidden output hidden
depth
![Page 11: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/11.jpg)
Example
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 11
• Let’s have a look on https://ml4a.github.io/ml4a/neural_networks/#regression
![Page 12: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/12.jpg)
Different activation functions
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 12
Sour
ce: W
ikip
edia
, Lau
ghsin
thes
tock
s[CC
BY-
SA 4
.0]
![Page 13: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/13.jpg)
Are artificial neural networks intelligent?
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 13
• Artificial neural networks are not intelligent per se.
• Shared features with the brain:• Neurons and dendrites• Emergence: complex behavior based on simple constituents• Convolutional networks model the way we think the brain filters information
• How the brain is different:• Bigger. Human brain: 10s of billions of neurons with 1000s of inputs for each.• More complex structure. Brain has feedback loops, ANNs usually feed-forward.
![Page 14: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/14.jpg)
Where to use artificial neural networks?
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 14
•Why Google, Facebook, Apple (Big Data) love neural networks• Have huge amounts of unstructured data for free• Need fast evaluation (mobile devices, real-time search, etc.) • Can do training in big datacenters
• Experiment and modelling at IPP• Data often structured and expensive to produce• Fast evaluation e.g. for real-time control, optimisation, parameter studies • Can do training in big datacenters
• It depends on the specific problem to solve which tools to use
![Page 15: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/15.jpg)
What is TensorFlow?
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 15
• Dataflow programming framework developed by Google (C++, Python)• High level frontend: Keras (now included)• Runs fast on parallel GPU/TPU architectures
• Included analysis tool: TensorBoard
![Page 16: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/16.jpg)
How to build networks yourself
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 16
• Install Python environment• NumPy/SciPy/Matplotlib for Matlab-like functionality• TensorFlow (Alternative: Theano) with Keras frontend
for high performance artificial neural networks• scikit-learn for different machine-learning methods and tools
• Anaconda Python distribution already includes most of it,use conda install -c conda-forge tensorflow• Spyder is a good editor similar for Matlab• Jupyter Notebook for Mathematica-like notebooks• Look at examples, e.g. from https://blog.keras.io/
![Page 17: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/17.jpg)
Example: Autoencoder
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 17
•We would like to build a tool for dimensionality reduction•Map from !-D to !-D data, but only " < ! dimensions relevant• Introduce „bottleneck“ layer of order " in neural network
• Let‘s start our own environment on http://localhost:8888/tree/Dropbox/ipp/neuralnet/jupyter• Diagnostics (TensorBoard) on http://localhost:6006/
![Page 18: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/18.jpg)
Examples for machine learning and neural networks at IPP
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 18
• Projects at TOK• Daniel Schäfer: MSc thesis (Zohm) with TU Eindhoven,
Fast neural network surrogate model for QuaLiKiz (turbulence)
• [AI-at-IPP], meetings organised by Lennart Bock
• Expertise at TUM• Tobias Neckel: Uncertainty quantification
• Nils Thuerey: Neural networks, computer graphics
• Projects at NMPP• Helmholtz project: reduced complexity models (Albert, Tyranowski, v.Toussaint)
• TODO: Coster, Hoenen, Luk, Preuss, v.Toussaint
• Dirk Nille: PhD thesis, high-res/smooth divertor IR thermography via Bayes
• Tell me if you know more!
![Page 19: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/19.jpg)
Initial ideas on symplectic neural networks
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 19
• A linear transformation stays linear with the right network…• Criterion: linearity of layers via linear activation functions
• … (non-linear) symplectic transform stays symplectic if done correctly!• Criterion: symplecticity of (combination of) layers• Interpolate mechanical system. Input: initial conditions, Output: final conditions
• Idea based on symplectic Euler integrator for canonical ! = ($, &)
!(()*) = !(() + ℎ -.∇0($( , & ()* ) (Variant 1)
.∇0($ ()* , & ( ) (Variant 2)
• Must be explicit (separable Hamiltonian 0) for forward network
see Deco&Brauer, Neural Networks 8, 525 (1995) with focus on entropy
![Page 20: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/20.jpg)
Initial ideas on symplectic neural networks
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 20
• Approximate real Hamiltonian map by many simple ones
• Activation function due to Hamiltonian !" in each node,# = −&!"&' , ) = −&!"&* , (Euler: 2⋅4= 1,2⋅6 = ℎ)
• First guess: harmonic oscillator, but yields linear map• Second guess: pendulum, “half-harmonic” oscillator, or “quendulum”
!" =*92 − cos ' , !" =
*92 + '
9
2 Θ('), !" = − cos * − cos '
![Page 21: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/21.jpg)
Example: image recognition with convolutional net
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 21
https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html(”very little data” means 1000s ofimages artificially blown up to10s of 1000s of training pairs)
Keras offers convolutional layersto detect features on pictures in a translationally invariant way
Also videos: Methods existto recognize if someone is packing or un-packing the trunk of a car
![Page 22: An introduction to artificial neural networks and how to ...ert/assets/neuralnet/Albert... · •Spyderis a good editor similar for Matlab ... Examples for machine learning and neural](https://reader034.vdocuments.site/reader034/viewer/2022042118/5e97b47405ca205a031ac981/html5/thumbnails/22.jpg)
Conclusion
25.09.18 Introduction to artificial neural networks, Christopher Albert <[email protected]>, NMPP, IPP Garching 22
• Choice of machine-learning method is highly problem-specific• Current boom has made a lot of user-friendly tools available• Extending methods still requires low-level work
• Artificial neural networks for classification and regression• Working on unstructured data• Involve hard optimisation problem while training• Once trained, extremely fast to evaluate (in parallel)
• The machine is only clever if you are
Thank you for your attention!
Talk available on https://itp.tugraz.at/~ert/teaching