from data to data driven - applications that will change your business
TRANSCRIPT
From Data to Data DrivenApplications That Will Change Your BusinessWith Matt Aslett/451 Research and Steven Noels/NGDATA
© 2014 by The 451 Group. All rights reserved
Big Data – Virtuous CircleIncreased use of interactive applications and data-generating machines
New commercial opportunities for analyzing previously ignored data
Increased desire to store and process all available data
More economically feasible to store and process previously ignored data
New infrastructureinvestments to support new data processing software
© 2014 by The 451 Group. All rights reserved
Important step on the path to influencing behavior via predictive analytics
For example ad/content serving, fraud detection, risk profiling, and customer retention
Growing Analytic Ambitions
REPORTING- What happened
ANALYSIS- Why did it happen?
PREDICTIVE- What will happen?
OPERATIONAL- What is happening now?
INFLUENCIAL- How to influence what
happens next?
PREDICTIVEANALYTICS
MACHINE LEARNING
OPERATIONAL INTELLIGENCE
© 2014 by The 451 Group. All rights reserved
Different Usage Patterns
REPORTING- What happened
ANALYSIS- Why did it happen?
Business analysts Historical data Manual processes People-driven dashboards
OPERATIONAL- What is happening now?
Business analysts/data scientists
Real-time data Data-driven dashboards
PREDICTIVE- What will happen?
Data scientists Historical data Automating processes Data-driven people
INFLUENCIAL- How to influence what
happens next?
Users/customers Automated processes Historical and
real-time data Data-driven applications
© 2013 by The 451 Group. All rights reserved
The desire to take advantage of that opportunity for competitive advantage led many enterprises to expand their data processing platforms
Taking advantage of Hadoop and MapReduce to enable the processing and analysis of multi-structured data
Where do we go from there? How to maximize the potential
investment in new platforms and move from reactive to proactive analytics and data-driven applications?
Expanding Data Platforms
STRUCTURED DATASTRUCTURED DATA
OPERATIONAL DATABASE
APPLICATIONS
DATA WAREHOUSE
AD HOC ANALYTICS
PRE-DEFINED REPORTING
MULTI-STRUCTURED DATA
HADOOP
PREDICTIVEANALYTICS
© 2013 by The 451 Group. All rights reserved
It does enable the economic storage and processing of data ill-suited to the cost and functional capabilities of traditional data management technologies
Hadoop requires a sophisticated skill set that was not prevalent within many organizations MapReduce programmers, HDFS administrators
The skillset has become more prevalent, and Hadoop itself easier to use Support for existing skills and tools (e.g. SQL analytics)
However, most initial projects were engineering led and limited to a discreet set of users ‘Data scientists’, specific departments/use-cases (e.g. search)
Net result: Hadoop offers a huge opportunity to store and process data that was previously ignored, but the true potential is not being realized
Hadoop is Not a Panacea
© 2013 by The 451 Group. All rights reserved
Hadoop’s Potential is Built on Flexibility
Hadoop can be used as a platform for: Reporting, Ad-hoc analytics Statistical analytics, Predictive analytics, Machine learning Graph and stream-based analytics
Initially, all programmed via the MapReduce batch process
HDFS
MAPREDUCE
HIVE PIG HBASE STORM GIRAPH MAHOUT
AD HOC ANALYTICS
PRE-DEFINED REPORTING
PREDICTIVEANALYTICS
MACHINE LEARNING
© 2013 by The 451 Group. All rights reserved
Hadoop’s Potential is Built on Flexibility
Next-generation Hadoop, with YARN and Spark (in-memory) enable: Multiple services to be run on data stored in HDFS, simultaneously By-passing the MapReduce batch process, enabling interactivity
HDFS
YARN
TEZ (optional)
HIVE PIG
HBASE
SQL-on-HADOOP,STORM, GIRAPH,OTHER
MAPREDUCE
TACHYON (developing)
SPARK
SHARKMLLIB (developing)
GRAPHX(developing)
SPARK STREAMING
SPARKR, SPORK,OTHER
AD HOC ANALYTICS
PRE-DEFINED REPORTING
PREDICTIVEANALYTICS
MACHINE LEARNING
MAHOUT
ITERATIVEANALYTICS
OPERATIONALINTELLIGENCE
© 2013 by The 451 Group. All rights reserved
Applications are Key to Fulfilling that Potential
HDFS
YARN
TEZ (optional)
HIVE PIG
HBASE
SQL-on-HADOOP,STORM, GIRAPH,OTHER
MAPREDUCESHARKMLLIB (developing)
GRAPHX(developing)
SPARK STREAMING
SPARKR, SPORK,OTHER
MAHOUT
APPLICATIONS
AD HOC ANALYTICS
PRE-DEFINED REPORTING
PREDICTIVEANALYTICS
MACHINE LEARNING
ITERATIVEANALYTICS
OPERATIONALINTELLIGENCE
TACHYON (developing)
SPARK
© 2013 by The 451 Group. All rights reserved
Putting it All Together
Despite all the focus on data platforms, it is the applications that deliver the value To the business To the user
And generate the real-time data, to be combined with historical data and new approaches to data processing to deliver data-driven
Delivering that requires not just Hadoop Real-time data processing Predictive analytics Machine learning Recommendation engine
But the underlying technology needs to serve not just business analysts and data scientists but data-driven applications that drive automated business processes
From Data to Data-Driven Delivering Value to the Business
12Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Defining Data-Driven Applications
Data-driven applications act on any volume of real-time data to actively guide and optimize business processes based on continuously growing insights on customer behavior, intent and preferences.
13Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Data-Driven Applications Allow You to…
Proactively Steer Processes
Go Beyond Insights
Have a Closed Feedback Loop
Benefit from Continuous Learning
Get Smarter Over Time
14Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Applying Data-Driven to the Customer Experience
Customer insight is limited to a subset of available data - Limiting relevance and timeliness of offers
Company& Customer
Activity
CustomerCRM
Systems
CustomerWeb & Mobile
Customer Channel Campaigns
CustomerService Desk
Social Data
Website &online apps
Mobile App Server
SMS
Broadcast
Offers
direct mail
ATM
web
Agent, IVR
mobile
chat
Relevance?Awareness?
Value?Timing?Clarity?
15Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Data-Driven – More than Big Data Graphs…
Enterprises are NOT looking for new presentations of their data anymore, but for RESULTS and for solutions and people to achieve them
16Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
I have a customer - what are the top 3 products he is likely to buy?
Data-Driven – Answering the Tougher Questions…
Which top hundred customers are likely to buy my product X today?
What is the best channel to connect with my customer, and when?
How can I turn around my most valuable potential churners?
17Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Pre-requisites to Deliver Data-Driven Customer Experiences
Preferences
Affinities
Context
Behavior
18Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
DNA
www
STB
Mobile
Network
App Phone
Call Centre
CDR
Shop
POS
Personalization
Acquisition
Journey
ARPU / CLTV
Marketing
Advertising
Churn
Fraud
www
STB
Mobile
Network
App Phone
Call Centre
Shop
POS
CDR
Personalization
Acquisition
Journey
ARPU / CLTV
Marketing
Advertising
Churn
Fraud
Towards a New Architecture
19Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Lily Real Time PersonalizationWithin the Current Enterprise Architecture – Improving existing BI landscape
Operational Systems
External Data
Contract/Product Data
Customer Operational Data
Reference Data
Reporting / Analytics
Enterprise BI and reporting
Enterprise Analytics Applications
Marketing and Social Data
Lily EnterpriseCustomer Interaction Data
Campaign Data
ERP/CRM Data
Data Warehouse Data
Service Desk
Customer CRM and IVR Systems
Real Time Integration
Data integration
BatchIntegration
ETL or
Lily API based
Lily Connector Based
Web and Mobile
Mobile Apps
Customer Website & Social
Channel / CampaignsMail
SMS
Broadcast
Mar
ketin
g Ca
mpa
ign
Mgt
Sales Office
Agent / Advisor
Feedback
20Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Big DataReal TimeAnalytics
Traditional Analytic Systems
What will happen?What do I need to do?
What has happened?Why?
Large VolumesMulti-structured Data
Real-Time SystemsBatch Systems
Sampled DataStructured Data
What is Different?
“My traditional BI environment will give the answers tomorrow of yesterday’s problem”
- CIO Fortune 50 Bank
21Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
Applied to Subscription-based BusinessesCustomer Lifetime Value (Inbound, Outbound, Risk)
• Personalization• Offer recommendations based on individual preferences• Customer Service assistance• Manage risk, fraud prevention, Reduce customer churn
• Personalization, Customer Services• Better targeting, micro campaigns, social leaders• Reduce customer churn, price sensitivity• Media and IPTV, audience measurement
• Personalization, Cross sell/upsell• Recommendations – ads, content, etc.• Audience measurement• Customer Services
Banks
Telecoms
Media and Publishers
22Copyright 2014 NGDATA®, Inc. Confidential – Distribution prohibited without permission
From Data to Data Driven Applications
Listen Bigger.
VO
LU
ME
Learn Faster.
SP
EED
Execute Smarter.
AN
SW
ER
S
DW/BI
Volumes of Data
Availability
Questions Answered
Zettabytes
Exaabytes
Petabytes
Terabytes
Gigabytes
DW/BI
Seconds
Minutes
Hours
Days
Weeks DW/BI
Unknown Unknowns
Known Unknowns
Known Knowns
Questions?
Go from Data to Data Driven Applications To learn more about: NGDATA and Lily go to NGDATA.com or [email protected] 451 Research – 451Research.com or [email protected]