big data berlin – automating decisions is the next frontier for big data
DESCRIPTION
Just collecting, storing and analyzing data is not enough. In order to benefit from it, you have to overcome organizational and human inertia and establish automated processes that use insights gained from your data. This presentation has been presented at http://dataconomy.com/28-august-2014-big-data-berlin/TRANSCRIPT
![Page 1: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/1.jpg)
A U T O M AT I N G D E C I S I O N S – T H E N E X T F R O N T I E R F O R B I G D ATA
L A R S T R I E L O F F – B L U E Y O N D E R – I ’ M @ T R I E L O F F O N T W I T T E R
![Page 2: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/2.jpg)
T H E F R O N T I E RW H A T N O B O D Y T E L L S Y O U A B O U T
![Page 3: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/3.jpg)
W H O I S U S I N G B I G D ATA T O D AY
![Page 4: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/4.jpg)
W H E R E B I G D ATA I S U S E D
Effective Use
Marketing
Finance
Everyone Else
S O U R C E : O W N , P O T E N T I A L LY B I A S E D O B S E R VAT I O N S
![Page 5: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/5.jpg)
W H E R E B I G D ATA I S U S E D
Potential Use
Marketing
Finance
Everyone Else
S O U R C E : O W N , P O T E N T I A L LY B I A S E D O B S E R VAT I O N S
![Page 6: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/6.jpg)
Better Decisions
Faster Data
More Data
T H R E E A P P R O A C H E S
![Page 7: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/7.jpg)
M O R E D ATAD I G I TA L M A R K E T I N G ’ S A P P R O A C H :
![Page 8: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/8.jpg)
FA S T E R D ATAF I N A N C E ’ S A P P R O A C H
![Page 9: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/9.jpg)
B U T W H AT A B O U T B E T T E R D E C I S I O N S ?
![Page 10: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/10.jpg)
Physiological
Safety
Love/Belonging
Esteem
Self-Actualization
![Page 11: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/11.jpg)
Collection
Storage
Analysis
Prediction
Decision
![Page 12: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/12.jpg)
Collection
Storage
Analysis
Prediction
Decision
Physiological
Safety
Love/Belonging
Esteem
Self-Actualization
![Page 13: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/13.jpg)
![Page 14: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/14.jpg)
“… all others bring data.” !
— W. E D W A R D S D E M I N G
![Page 15: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/15.jpg)
H O W D ATA - D R I V E N D E C I S I O N S S H O U L D W O R K
C O M P U T E R C O L L E C T S
C O M P U T E R S T O R E S
H U M A N A N A LY Z E S
H U M A N P R E D I C T S
H U M A N D E C I D E S
![Page 16: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/16.jpg)
H O W D ATA - D R I V E N D E C I S I O N S R E A L LY W O R K
C O M P U T E R C O L L E C T S
C O M P U T E R S T O R E S
H U M A N A N A LY Z E S
C O M M U N I C AT I O N B R E A K D O W N
H U M A N D E C I D E S
![Page 17: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/17.jpg)
C O M M U N I C AT I O N B R E A K D O W N
Communication Breakdown, It's always the same, I'm having a nervous breakdown, Drive me insane!
— L E D Z E P P E L I N
• Drill-down analysis … misunderstood or distorted• Metrics dashboards … contradictory and confusing• Monthly reports … ignored after two iterations• In-house analyst teams … overworked and powerless
![Page 18: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/18.jpg)
H O W D ATA - D R I V E N D E C I S I O N S R E A L LY W O R K
H T T P : / / D I L B E R T. C O M / S T R I P S / C O M I C / 2 0 0 7 - 0 5 - 1 6 /
![Page 19: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/19.jpg)
H O W D E C I S I O N S R E A L LY S H O U L D W O R K
C O M P U T E R C O L L E C T S
C O M P U T E R S T O R E S
C O M P U T E R A N A LY Z E S
C O M P U T E R P R E D I C T S
C O M P U T E R D E C I D E S
![Page 20: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/20.jpg)
99.9% of all business decisions can be automated
![Page 21: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/21.jpg)
H O W D E C I S I O N S A R E M A D E
Human Decisions
Business Rules
No Decision, Decision-by-default
![Page 22: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/22.jpg)
B U S I N E S S R U L E S = P R O G R A M M I N G
• Business rules are like programs – written by non-programmers
• Business rules can be contradictory, incomplete, and complex beyond comprehension
• Business rules have no built-in feedback mechanism: “It is the rule, because it is the rule”
![Page 23: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/23.jpg)
H U M A N D E C I S I O N M A K I N G
• • System 1: Fast, automatic, frequent, emotional, stereotypic, subconscious
• • System 2: Slow, effortful, infrequent, logical, calculating, conscious
D A N I E L K A H N E M A N N , T H I N K I N G FA S T A N D S L O W
![Page 24: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/24.jpg)
H O W T O D E C I D E FA S TF R E Q U E N T D E C I S I O N M A K I N G M E A N S FA S T D E C I S I O N M A K I N G , M E A N S U S I N G H E U R I S T I C S O R C O G N I T I V E B I A S E S
Anchoring effect
IKEA effect
Confirmation bias
Bandwagon effect
Substitution
Availability heuristic Texas Sharpshooter Fallacy
Rhyme as reason effect
Over-justification effect
Zero-risk bias
Framing effect
Illusory correlationSunk cost fallacy
Overconfidence
Outcome bias
Inattentional Blindness
Benjamin Franklin effect
Hindsight bias
Gambler’s fallacy
Anecdotal evidenceNegativity bias
Loss aversion
Backfire effect
![Page 25: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/25.jpg)
H O W C O M P U T E R S D E C I D E FA S TM A C H I N E L E A R N I N G O F F E R S A N A LT E R N A T I V E T O H U M A N C O G N I T I V E B I A S E S A N D C A N B E M A D E FA S T T H R O U G H B I G D A TA
K-Means Clustering
Naive BayesSupport Vector Machines Affinity Propagation
Least Angle Regression
Nearest Neighbors
Decision Trees
Markov Chain Monte Carlo
Spectral clustering
Restricted Bolzmann Machines
Logistic Regression
![Page 26: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/26.jpg)
Better decisions through predictive applications
![Page 27: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/27.jpg)
H O W P R E D I C T I V E A P P L I C AT I O N S W O R K
C O L L E C T & S T O R E
A N A LY Z E C O R R E L AT I O N S
B U I L D D E C I S I O N M O D E L
D E C I D E & T E S T O P T I M I Z E
![Page 28: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/28.jpg)
W H Y T E S T ?
– R A N D A L L M U N R O E
“Correlation doesn’t imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look
over there’”
![Page 29: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/29.jpg)
S T O R Y T I M E(Not safe for vegetarians)
![Page 30: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/30.jpg)
T H E G R O U N D B E E F D I L E M M A
![Page 31: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/31.jpg)
T H E G R O U N D B E E F D I L E M M A
Yesterday Today Tomorrow Next Delivery Next Day
In Stock Demand
![Page 32: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/32.jpg)
T H E G R O U N D B E E F D I L E M M A
• Order too much and you will have to throw meat away when it goes bad. You lose money and cows die in vain
• Order too little and you won’t serve all your potential customers. You lose money and customers stay hungry.
![Page 33: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/33.jpg)
A C C U R AT E LY P R E D I C T D E M A N DC H A L L E N G E # 1
![Page 34: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/34.jpg)
A U T O M AT E R E P L E N I S H M E N TC H A L L E N G E # 2
C O L L E C T S T O C K A N D
S A L E S
P R E D I C T D E M A N D
T R A D E O F F C O S T S
C R E AT E O R D E R S I N E R P
S Y S T E M
O P T I M I Z E & R E P E AT
![Page 35: Big Data Berlin – Automating Decisions is the Next Frontier for Big Data](https://reader034.vdocuments.site/reader034/viewer/2022042606/54b3bea04a79591d438b4580/html5/thumbnails/35.jpg)
@ B YA N A LY T I C S _ E N @ T R I E L O F F