industrial machine learning (sigkdd17)

SIGKDD 2017 (Halifax)Josh Bloom (GE Digital) @profjsb Industrial Machine Learning

Industrial Machine Learning

Josh Bloom VP Data & Analytics, GE Digital Professor, UC Berkeley @profjsb

SIGKDD 2017 (Halifax)

COPYRIGHT 2012-2017, WISE.IO INC.

‣ Brief Background/Introduction

‣ IML Challenges/Opportunities• Optimization metrics • Physics & Data-driven models• Interpretability / Regulation• Privacy & Security

Agenda


Teaching

‣ Python Bootcamps 200+ undergrad/grad

‣ Python for Data Science graduate course

‣Berkeley Institute for Data Science (BIDS)

Co-founder

Industry

‣ML Applications Company

Code / Repos

Q4’16

CTO, Co-founder Professor, UC Berkeley

Research

Gordon & Betty Moore Foundation

Data-Driven Investigator

‣ Automated Data-driven Discovery & Inference in the Time Domain

‣300+ refereed articles

wise.io

http://wise.io

Beyond “Smart” Thermostats, Fitbits, and Self-driving cars…

Industrial Internet of Things: 50b+ connected devices, 600ZB/yr (2020)


Preventative Maintenance Failure/Anomaly Detection

Assistive Diagnosis & Treatment Systems Optimization


False + True +

True – False –

Great movie Rec!

Didn’t see an ad I would have hated

!Spam in my

inbox!

Received a box of clothes I

detest

ML ImpactConsumer

False + True +

True – False –

Industrial

Avoided $200k unneeded per engine repair

Identified cancerous

tumor early

Missed a fatal gas pipeline

crack

Took a MegaWatt power

station offline unnecessarily

"


Implication for the Optimization Metric

‣ Higher accuracy More value

A rational case for Bias in IML

‣ Drive towards higher precision / Low FPR at Recall~1

Credit: Ratish Dalvi


https://www.slideshare.net/JuneAndrews/counter-intuitive-machine-learning-for-the-industrial-internet-of-things-77853709

Implication for the Optimization Metric

Industrial Setting Results in Choosing the Lower Bound of Positive Training Samples

Image Source: “ROC curve analysis” MedCalc.org

Optimize Cost Function with Infogain Loss Layer (Caffe):

Practical path: Fit noisy FN tail with (skewed normal/Beta), project lower bound

= CTN|TN|+CTP|TP|+CFN|FN|+CFP|FP|

= ε·|TN|+CTP|TP|+∞·|FN|+CFP|FP|

which classifier is best?depends...

9

IIoT Systems are Physical Systems

Image credits: MIT Courseware; US5074114


Physical Models Data-driven Models

✓ Captures a priori understanding of device/system. Little data required. ✓ Simulate/anticipate futures not yet seen in the real-world ✓ Parameters have physical meaning: interpretable

Do not learn from new data/outcomes Incomplete physics → imperfect model Refinement is a very manual process

requiring deep domain experts Difficult to incorporate non-physical

metadata

Often requires previous exemplars. Difficult to predict new failure modes

Uncertain generalizability Black boxy: Difficult to drive intuition/

improvements from model parameters.

✓ No explicit physical understanding required ✓ Naturally improves with more data/outcomes ✓Extensible to new data sources


Marrying Physics & Data Driven Models for IML

‣ Manual featurization - fits to physical models as input ‣ RL (AI gyms/supercomputer simulations) + transfer learning with updates in the real world

‣Modeling residuals of physical models

12 Shah, Dey, Lovett, and Kapoor

(a) Circle maneuver (b) Square maneuver

(c) Space-Time Plot for Circle (d) Space-Time Plot for Square

Fig. 4 Evaluating the differences between the simulated and the real-world flight. In top figures,the purple and the red lines depict the track from simulation and the real-world flights respectively.

factors such as integration errors, vehicle model approximations and mild randomwinds.

Sensor Models: Besides evaluating the entire simulation pipeline we also inves-tigated individual component models, namely the barometer (MEAS MS5611-01BA), the magnetometer (Honeywell HMC5883) and the IMU (InvenSense MPU6000). Note that the simulated GPS model is currently simplistic, thus, we only fo-cus on the three more complex sensor models. For each of the above sensors we usethe manufacture specified datasheets to set the parameters in the sensor models.

• IMU: We measured readings from the accelerometers and gyroscope as thevehicle was stationary and flying. We observed that while the characteristicswere similar when the vehicle was stationary (gyro: simulated variance 2.47e�7rad2/s2, real-world variance 6.71e�7 rad2/s2, accel.: simulated variance 1.78e�4m2/s4, real-world variance 1.93e�4 m2/s4), the observed variance for an in-flight vehicle was much higher than the simulated one (accel.: simulated 1.75e�3m2/s4 vs. real-world 9.46 m2/s4). This is likely in real-world the airframe vi-brates when the motors are running and that phenomenon is not yet modeled inAirSim.

• Barometer: We raised the sensor periodically between two fixed heights: groundlevel and then elevated to 178 cm (both in simulation and real-world). Figure 5(a)shows both the measurements (green is simulated, blue is real-world) and we ob-

Shah+1705.05065

eg., Christiano+1610.03518

‣Impart/impose/imbue physical constraints into models

• Computer vision: e.g., GVNN (s03 layer Euler,…)

• High energy physics: “QCD-Aware Recursive NN for Jet Physics” Louppe+ 1702.00748

Handa+1607.07405

https://arxiv.org/abs/1610.03518

SIGKDD 2017 (Halifax)Josh Bloom (GE Digital) @profjsb Industrial Machine LearningTime Time

Observations

Physical Model Residuals

Final Model Residuals

Physical Model

Model Physical Residuals (eg. LSTM)

pred

ict

t2t3

‣Modeling residuals of physical models

13


Applications Still Need to be Understood

“Monitoring the well ‘was a little hard that night’ because so many simultaneous operations were going on…”

Bloomberg - March 13, 2013


High

LowLow High

Inte

rpre

tabi

lity

Accuracy

Linear/Logistic Regression

Naive Bayes

Decision Trees

SVMs

Bagging

Boosting

Decision Forests

Neural Nets Deep Learning

Nearest Neighbors

Gaussian/Dirichlet

Processes

Splines

* on real-world data setsLasso

Warning

Unscientific &

opinionated!

instance-level feature

importances (ILFIs)


Anomaly detected last 1.1 hr

Temp 2.2σ higher than last 30 d

Pressure within range

Next Suggested Action • Set field service • Decrease rail speed

s3E-17-ab33Failure in next 30 days: unlikely

Surfacing the WHY, not just the WHAT

Engineer apps to Capture feedback (implicit & explicit) and outcomes → continual learning


2018: European Union regulations on algorithmic decision-making and a “right to explanation”

Consumer ML Industrial ML

Regulatory Oversite

Massive regulations/bodies (HIPAA, FAA, FDA, NTSB, NRC, …) but

For now, there is no standing/legal contemplation for continual learning models

Bryce Goodman, Seth Flaxman

17

Privacy & Security of Industrial Machine Learning Systems


Some Paths to Private & Secure IML

• “Privacy-preserving generative deep neural networks support clinical data sharing”

‣Differentially private GANs for as a surrogate for Data sharing

‣Enclave computation / homomorphic encryption

Bost+NDSS15, also Bos 10.1016/j.jbi.2014.04.003• “Machine Learning Classification over Encrypted Data”

• “CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy”Dowlin+MSR-TR-2016-3

‣Privacy Preserving Distributed (Deep) Learning

Beaulieu-Jones+17 https://doi.org/10.1101/159756

• Creating incentives for data sharing across adversaries (cf. Mike Jordan work) • Shokri & Shmatikov CCS15, Abadi+CCS16


• Secure Domain-Specific Approach for Prediction on and Transmission of Data with Compression and Privacy Preservation

D. Eads (MLConf, sub.)

JSB (62/508,246.)

Some Current Work

github.com/wiseio/paratext

• Data structure to collapsing hyperparameter optimization

• Memory-efficient data frame optimized for ML

• Testing tool to monitor ML workflows given changing data (stability)

https://github.com/wiseio/paratext


Academic Partnerships

Ray + Plasma Clipper Alegro Arx, …

Weld

New

Founding Partner


The Power of a 1% Gain in Efficiency

$27B$30B

$63B$66B

$90B

RailAviation

HealthcarePower

Oil & Gas

Source: “Industrial Internet Pushing Boundaries of Minds & Machines” GE, 2012



Josh Bloom VP Data & Analytics, GE Digital Professor, UC Berkeley @profjsb

SIGKDD 2017 (Halifax)

Thanks! (And yes, we’re hiring…)