azure 機器學習 - 使用python, r, spark, cntk 深度學習

Azure Machine Learning

– 其他篇

台灣微軟

技術傳教士

吳宏彬

8/25/2016

什麼是R語言

Open Source “lingua franca”

Analytics, Computing, Modeling

Global Community

Millions of users 7000+ Algorithms, Test Data & Evaluations

Can be Scaled to Big Data,

Big Analytics

Ecosystem

Scalability

Polls of data miners and analytics professionals on their software

choices since 2007Source: http://blog.revolutionanalytics.com/2013/10/r-usage-skyrocketing-rexer-poll.html

R is developed and contributed by open source community

CRAN – the Comprehensive R Archive Network Package repository of R

7500+ packages, covering all aspects of statistical analysis, machine learning, natural language processing …

Still exponentially growth

Free!

Source: http://r4stats.com/2014/04/07/r-continues-its-rapid-growth/

1.Seasonal ARIMA

2.Non Seasonal

ARIMA

3.Seasonal ETS

4.Non -Seasonal ETS

5.Average of Seasonal

ETS and Seasonal

ARIMA

Mean Error (ME) - Average forecasting error (an error is the difference between the

predicted value and the actual value) on the test dataset

Root Mean Squared Error (RMSE) - The square root of the average of squared errors of

predictions made on the test dataset.

Mean Absolute Error (MAE) - The average of absolute errors

Mean Percentage Error (MPE) - The average of percentage errors

Mean Absolute Percentage Error (MAPE) - The average of absolute percentage errors

Mean Absolute Scaled Error (MASE)

Symmetric Mean Absolute Percentage Error (sMAPE)

DatasizeIn-memory

In-memory In-Memory or Disk Based

Speed of

AnalysisSingle threaded Multi-threaded

Multi-threaded, parallel

processing 1:N servers

SupportCommunity Community Community + Commercial

Analytic

Breadth &

Depth

7500+ innovative analytic

packages7500+ innovative analytic

packages

7500+ innovative packages

+ commercial parallel high-

speed functions

LicenseOpen Source

Open Source

Commercial license.

Supported release with

indemnity

Microsoft

R Open

Microsoft

R Server

Support standard Python library types such as

Pandas data frames and NumPy arrays.

Execute the Python code is based on Anaconda

2.1, It comes with close to 200 of the most

common Python packages (as NumPy, SciPy and

Scikits-Learn )

Output generate images from MatplotLib

KNN

21

What is Spark?

Data is growing faster than processing

speeds

Only solution is to parallelize data

processing on large clusters

Example: HDInsight

Fast, expressive cluster computing system compatible with Apache

Hadoop

• Works with any Hadoop-supported storage system (HDFS, S3, Avro, …)

Improves efficiency through:

• In-memory computing primitives

• General computation graphs

Improves usability through:

• Rich APIs in Java, Scala, Python

• Interactive shell

Spark was initially started by Matei Zaharia at UC Berkeley AMPLab

in 2009, was open sourced in 2010 and donated to Apache in 2013

Up to 100× faster

Often 2-10× less code

What is Spark?

Spark for Azure HDInsight

Spark Node

Spark Node

Spark Node

Spark Node

Spark Node

Storage Layer

Decision Maker

Decision Maker

Decision

Maker

Spark Cluster

clients

Spark Notebooks

Using the Spark shell to run

interactive queries

Using the Spark shell to run Spark

SQL queries

Using a standalone Scala program

Spark

Notebooks

Zeppelin – for

Scala users

Zupyter – for

Python users

Programming

Spark

2015 System

Human Error Rate 4%

Speech Recognition could reach human parity in the next 3 years

33

Microsoft 透過深度學習技術贏得 ImageNet 2015所有比賽項目冠軍

28.225.8

16.4

11.7

7.3 6.73.5

ILSVRC 2010NEC

America

ILSVRC 2011Xerox

ILSVRC 2012AlexNet

ILSVRC 2013Clarifi

ILSVRC 2014VGG

ILSVRC 2014GoogleNet

ILSVRC 2015MSRA

ResNet

ImageNet Classification top-5 error (%)

Microsoft had all 5 entries being the 1-st places this year: ImageNet

classification, ImageNet localization, ImageNet detection, COCO

detection, and COCO segmentation

CNTK At the Heart: Computational Networks

•A generalization of machine learning models that can be described as a series of computational steps.

• E.g., DNN, CNN, RNN, LSTM, DSSM, Seq2Sqe, Log-linear model

•Representation: • A list of computational nodes denoted as

n = {node name : operation name}

• The parent-children relationship describing the operands

{n : c1, · · · , cKn }• Kn is the number of children of node n. For leaf nodes Kn = 0.

• Order of the children matters: e.g., XY is different from YX

• Given the inputs (operands) the value of the node can be computed.

•Can flexibly describe deep learning models. • Adopted by many other popular tools as well

35

36

•A generalization of machine learning models that can be described as a series of computational steps.

• E.g., DNN, CNN, RNN, LSTM, DSSM, Log-linear model

•Representation: • A list of computational nodes denoted as

n = {node name : operation name}

• The parent-children relationship describing the operands

{n : c1, · · · , cKn }• Kn is the number of children of node n. For leaf nodes Kn = 0.

• Order of the children matters: e.g., XY is different from YX

• Given the inputs (operands) the value of the node can be computed.

•Can flexibly describe deep learning models. • Adopted by many other popular tools as well

“CNTK is production-ready: State-of-the-art accuracy, efficient, and scales to multi-GPU/multi-server.”

Theano only supports 1 GPU

Achieved with 1-bit gradient quantizationalgorithm

0

10000

20000

30000

40000

50000

60000

70000

80000

CNTK Theano TensorFlow Torch 7 Caffe

speed comparison (samples/second), higher = better[note: December 2015]

1 GPU 1 x 4 GPUs 2 x 4 GPUs (8 GPUs)

* TensorFlow add distributed compute support in April 2016

Micrsoft Reacher SLAWEKSMYL win in CIF 2016 byusing LSTM Neural Network

Powered by CNTK

CIF Competition 2016 – Final Results• Contestant 1 – Slawek Smyl (LSTM-based

NN on deseasonalized data)

• Contestant 2 – Slawek Smyl (weighted average of my 3 methods)

• Contestant 3 – prof. Sven Crone (Multilayer Perceptron with a thorough feature search)

• Contestant 4 - Mikhail Artyukhov (previous competition winner, ensemble models)

• Contestant 5 - Joerg Wichard, Bayer Healthcare AG (Adaptive Forecasting Strategy with Hybrid Ensemble Models)

• Contestant 6 – Slawek Smyl (LSTM-based NN)

CNTK Demo

CNTK Architecture

41

CNBuilder

LambdaCN

Description Use Build

ILearnerIDataReaderFeatures &

Labels Load Get data

IExecutionEngine

CPU/GPU

Task-specific

reader

SGD, AdaGrad,

etc.

Evaluate

Compute Gradient

(1) Kai Chen and Qiang Huo, “Scalable training of deep learning machines by incremental block training with intra-block

parallel optimization and blockwise model-update filtering”,

in Internal Conference on Acoustics, Speech and Signal Processing , March 2016, Shanghai, China.

CNTK is a powerful tool that supports CPU/GPU and runs under Windows/Linux

CNTK is extensible with the low-coupling modular design: adding new readers and new computation nodes is easy with a new reader design

Network definition language, macros, and model editing language (as well as Python and C++ bindings in the future) makes network design and modification easy

Compared to other tools CNTK has a great balance between efficiency, performance, and flexibility

microsoft.com/cognitive

Mahout Spark ML Azure ML R Server

Shared Service No No Yes No

Deployment Model PaaS PaaS PaaS IaaS

Extensibility High High Medium High

Deployment Complexity Medium High Low Medium

Cost High High Low High

Programming Languages Java/Scala Scala/Java/Python Python/R R

Algorithms Limited (growing) MLlib/scikit Many (scikit/CRAN) Many (CRAN)

Scalability High High Medium Medium

xgboost Vowpal Wabbit

Rattle

CNTK

*Copy

雲端隨選隨用 各式資料 快速上線服務 資料分享跟協同合作

開放 支援完整資料分析流程

https://gallery.cortanaintelligence.com/

唯一一家提供從資料匯入到產生行動及資料呈現完整的解決方案

ConnectR• High-speed & direct

connectors

Available for:• High-performance XDF

• SAS, SPSS, delimited & fixed format text data files

• Hadoop HDFS (text & XDF)

• Teradata Database & Aster

• EDWs and ADWs

• ODBC

ScaleR• Ready-to-Use high-performance

big data big analytics

• Fully-parallelized analytics

• Data prep & data distillation

• Descriptive statistics & statistical tests

• Range of predictive functions

• User tools for distributing customized R algorithms across nodes

DistributedR• Distributed computing framework

• Delivers cross-platform portability

Available on:

• Windows Servers

• Red Hat and SuSE Linux Servers

• Teradata Database

• Cloudera Hadoop

• Hortonworks Hadoop

• MapR Hadoop

R+CRAN• Open source R interpreter

• R 3.2.2

• Freely-available huge range of R algorithms

• Algorithms callable by RevoR

• 100% Compatible with existing R scripts, functions and packages

RevoR• Performance enhanced R

interpreter

• Based on open source R

• Adds high-performance math library to speed up linear algebra functions

R Open Microsoft R Server

DeployRDevelopR

Gradient Boosted Decision Trees

Naïve Bayes

Data import – Delimited, Fixed, SAS, SPSS,

OBDC

Variable creation & transformation

Recode variables

Factor variables

Missing value handling

Sort, Merge, Split

Aggregate by category (means, sums)

Min / Max, Mean, Median (approx.)

Quantiles (approx.)

Standard Deviation

Variance

Correlation

Covariance

Sum of Squares (cross product matrix for set

variables)

Pairwise Cross tabs

Risk Ratio & Odds Ratio

Cross-Tabulation of Data (standard tables & long

form)

Marginal Summaries of Cross Tabulations

Chi Square Test

Kendall Rank Correlation

Fisher’s Exact Test

Student’s t-Test

Subsample (observations & variables)

Random Sampling

Data Step Statistical Tests

Sampling

Descriptive Statistics Sum of Squares (cross product matrix for set

variables)

Multiple Linear Regression

Generalized Linear Models (GLM) exponential

family distributions: binomial, Gaussian, inverse

Gaussian, Poisson, Tweedie. Standard link

functions: cauchit, identity, log, logit, probit. User

defined distributions & link functions.

Covariance & Correlation Matrices

Logistic Regression

Classification & Regression Trees

Predictions/scoring for models

Residuals for all models

Predictive Models K-Means

Decision Trees

Decision Forests

Cluster Analysis

Classification

Simulation

Variable Selection

Stepwise Regression

Simulation (e.g. Monte Carlo)

Parallel Random Number Generation

Combination rxDataStep

rxExec

PEMA-R API Custom Algorithms

Additional Resources

• CNTK: • https://github.com/Microsoft/CNTK

• Contains all the source code and example setups

• You may understand better how CNTK works by reading the source code

• New features are added constantly

• How to contact:• CNTK team: ask a question on CNTK GitHub!

• Alexey: • Email: alexey.kamenev@microsoft.com

• : https://www.linkedin.com/in/alexeykamenev

59