intro to deep learning w/ clojure

Intro to Deep Learning w/ ClojureIt’s Difficult to Make Predictions.Especially About the Future.

@JulioBarrosConsultantE-String.com

@JulioBarros http://E-String.com 1

Hypothesis

Given that Python dominates ML/DL;Is there a valid use case for Clojure?TL;DR -- It depends


I Hope So

... and I'm not the only one.


Hype or Reality?


"It is a renaissance, it is a golden age,"

"Machine learning and AI is a horizontal enabling layer. It will empower and improve every business, every government organization, every philanthropy — basically there's no institution in the world that cannot be improved with machine learning."— Bezos


Every industry can expect to be transformed by Artificial Intelligence


Image Classification

Justin Johnson, Andrej Karpathy, Li Fei-Fei - Stanford@JulioBarros http://E-String.com 7

My dog Rex


Object Detection


Image Captioning


Dense Captioning


Healthcare

Near or better than human level performance.


Business

— Law & Finance— Text, audio, image, video understanding— Churn prediction, customer segmentation— Product recommendations— Manufacturing, maintenance and control— Many many more


Artificial Intelligence

Artificial Intelligence (AI) - the study of "intelligent agents". Reasoning, knowledge representation, planning, robotics, etc.

— Artificial Narrow Intelligence (ANI)— Artificial General Intelligence (AGI)— Artificial Superintelligence (ASI)


Machine Learning

Machine Learning (ML) - Programs that learn from the data and make predictions.


Types of prediction

Regression - continuous values

Classification - discreet values/labels


Types of ML Algorithms

Supervised - trained on labeled data (regression or classification).

Unsupervised - trained on unlabeled data (clustering, segmentation).

Reinforcement - learns based on outcomes/results of actions.


Deep Learning

Deep Learning (DL) - ML/AI using artificial neural networks (ANNs)

You might be thinking ...

What? Isn't that just a neural net? Didn't we show those don't work. Twice.


Third time is a charm

Capabilities of neural nets have changed dramatically due to advancements in:

— Data

— Algorithms

— Hardware


Neurons: Biologically inspired

1942 McCulloch and Pitts1957 Rosenblatt

(A (+ b (apply + (map * x w)))


Activation Function

Introduces non linearity

— Historicaly: Identity, Step, Tanh, Sigmoid

— Currently: Rectified linear Unit (relu), Softmax


Universal Approximation Theorem (1989)

... a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions ...


Deep Neural Nets

A net with more than one hidden layer.


VGG16


GoogleLeNet


Training

1) Initialize weights randomly

2) Make prediction

3) Measure error (loss)

4) Adjust weights in the right direction

5) GOTO 2

Repeat over and over and over again with your training data


Backpropagation

Finding the right direction.


Backpropagation

To know the right direction calculate the gradient of the loss function with respect to each weight. Multiply by the error and a learning rate to get the delta.

Don't worry. The libraries do it for you.


Millions of Knobs Parameters


A Simple Neural Net in Cortex

(def network-architecture [(layers/input 11 1 1 :id :x)

(layers/linear 64) (layers/relu)

(layers/linear 1 :id :y)])


Wine Quality Data

11 features, 1 target column


UCI Machine Learning Repository

https://archive.ics.uci.edu/ml/datasets/Wine+Quality

Source: Paulo Cortez, University of Minho, Guimarães, Portugal, http://www3.dsi.uminho.pt/pcortez A. Cerdeira, F. Almeida, T. Matos and J. Reis, Viticulture Commission of the Vinho Verde Region(CVRVV), Porto, Portugal @2009


Normalized / Scaled data (Standardized)


Demo

https://github.com/thinktopic/cortexhttps://github.com/JulioBarros/clj-dl-demo


How Do We Work With Images

Well, images are just numbers/data.

Though numbers close to each other are more related.@JulioBarros http://E-String.com 35

Convolutional Layers

Similar to correlations from signal processing or filters from photoshop.

A small NxN filter is slid over and convolved/correlated with the image. Learns to find features.

Then lower level features are combined into higher level features.


Common Types of Layers

— input / output— fully connected (dense)— activation - relu, softmax— convolutional— maxpool— flatten— drop out


Types of ANN

1. Dense Neural Net (DNN)2. Convolutional Neural Net (CNN)3. Recurrent Neural Net (RNN)4. Everything else


Programming Abstractions

Level Python CLJ/JVM

DSL/API Keras, Lasagne, TF-Slim Cortex, ???

Computation Graph, Backprop, Autograd

Tensorflow, Theano, Torch, Pytorch, Caffe, MXNet, CNTK

Cortex, dl4j, Java bindings

Matrix Math CUDA (cuDNN), Eigen3 Neandertal, core.matrix, vectorz-clj


Production Considerations

— Target hardware environment— GPU(s)— Powerful multicore server— Mobile device— Embedded

— Desired latency / scalability— Data pipeline— Software engineering practices@JulioBarros http://E-String.com 40

Don't Underestimate the Last Two

We need to consider running and maintaining the models in production.


Approaches

— Python based environment— Clojure based environment— Generate C++ binaries— API calls to third party— API calls to microservices


Challenges with DL

— Needs lots of data. Labeled data is expensive.— Lacks explain-ability— Performance requirements - training and inference— Max performance unclear— Best architecture unclear


Benefits of DL

— Handles much of the feature engineering

— Handles complex (non linear) problems

— Advancements coming quickly


Recommendations

Do not be intimidated by ANNs or the math.

Start with Keras (and Tensorflow or Theano) tutorials (or maybe Pytorch).

Later choose language/framework as needs dictate.


Resources

Andrew Ng's Coursera Course and Fast.AI Mooc

Deep Learning Book - Goodfellow, Bengio and Courville

Meetups- Portland-Data-Science-Group- Portland-Machine-Learning-Meetup- Portland-Deep-Learning


https://www.coursera.org/learn/machine-learning

http://course.fast.ai

http://www.deeplearningbook.org

https://www.meetup.com/Portland-Data-Science-Group/

https://www.meetup.com/Portland-Machine-Learning-Meetup/

https://www.meetup.com/Portland-Deep-Learning/

Thank you! Questions?

[email protected] @JulioBarros@JulioBarros http://E-String.com 47

intro to deep learning w/ clojure

Technology