predicting customer experience through hadoop and customer behavior graphs

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

Predicting Customer Experience through Hadoop and Customer Behavior Graphs with Hortonworks and Apigee

We do Hadoop.


Your speakers…

Sanjay Kumar General Manager, Telecom Hortonworks

Sanjeev Srivastav VP Data Strategy Apigee


Enhancing Customer Experience with Hadoop We Do Hadoop

Sanjay Kumar General Manager, Telecom Hortonworks


The New Landscape of the Telecom Industry

Service Providers

Social Media & Mobile:

Explosion of rich customer data through

Social Media and Mobile Apps for customer

sentiment & Interests

Customer Expectation:

With the cultural impact of web and mobile,

customers are expecting greater levels of service

and responsiveness

Competitive Differentiation:

As other service providers deliver similar levels of telecom service

and coverage, other areas of service levels

are needed

New Digital Ecosystem:

Greater value of Data on digital ecosystem for

Customers and Partners driving Data

Monetization

Internet Of Things: Explosion of data from IOT with benefits aligned with insight not correlated to

data volumes


Service Provider Focus

Service

Providers

Customer Experience Management -  Enhance End-to-end Experience of Customer -  Become Trusted Partner to Customer -  Awareness of customer’s needs when and where needed

New Business & Consumer Services -  New Digital & Infrastructure Services -  Data Monetization -  M2M, IoT, Analytics-as-a service

Network Optimization -  Move to Software Driven Networks -  Leverage Network Data Assets -  Self optimizing and provisioning


Traditional systems under pressure Challenges •  Constrains data to app •  Can’t manage new data •  Costly to Scale

Business Value

Clickstream

Geolocation

Web Data

Internet of Things

Docs, emails

Server logs

2012 2.8 Zettabytes

2020 40 Zettabytes

LAGGARDS

INDUSTRY LEADERS

1

2 New Data

ERP CRM SCM

New

Traditional


Tomorrow: A Data-Centric Model for Your Business

DATA-CENTRIC

Limitations: •  Multiple copies of data •  Difficult cross-system integration •  Upper-limit on data volumes

before harming performance

Advantages: •  One version of the data •  No need for cross-app integration •  System scales linearly

APP-CENTRIC

App1 App 2 App 3 App 4 App 5 App 6

App Centric will break down with x10, x100,x1000… Need to shift to Data Centric


Modern Data Architecture emerges to unify data & processing

Modern Data Architecture •  Enable applications to have access to

all your enterprise data through an efficient centralized platform

•  Supported with a centralized approach governance, security and operations

•  Versatile to handle any applications and datasets no matter the size or type

Clickstream Web & Social

Geoloca3on Sensor & Machine

Server Logs

Unstructured

SOU

RC

ES

Existing Systems

ERP CRM SCM

AN

ALY

TIC

S

Data Marts

Business Analytics

Visualization & Dashboards

AN

ALY

TIC

S

Applications Business Analytics

Visualization & Dashboards

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

HDFS (Hadoop Distributed File System)

YARN: Data Operating System

Interactive Real-Time Batch Partner ISV Batch Batch MPP EDW


HDP delivers a completely open data platform

Hortonworks Data Platform 2.3

Hortonworks Data Platform provides Hadoop for the Enterprise: a centralized architecture of core enterprise services, for any application and any data.

Completely Open

•  HDP incorporates every element required of an enterprise data platform: data storage, data access, governance, security, operations

•  All components are developed in open source and then rigorously tested, certified, and delivered as an integrated open source platform that’s easy to consume and use by the enterprise and ecosystem.

YARN: Data Operating System (Cluster Resource Management)

1 ° ° ° ° ° ° °

° ° ° ° ° ° ° °

Apa

che

Pig

° °

° °

° ° °

° ° °

HDFS (Hadoop Distributed File System)

GOVERNANCE BATCH, INTERACTIVE & REAL-TIME DATA ACCESS

Apache Falcon

Apa

che

Hiv

e C

asca

ding

A

pach

e H

Bas

e A

pach

e A

ccum

ulo

Apa

che

Sol

r A

pach

e S

park

Apa

che

Sto

rm

Apache Sqoop

Apache Flume

Apache Kafka

SECURITY

Apache Ranger

Apache Knox

Apache Falcon

OPERATIONS

Apache Ambari

Apache Zookeeper

Apache Oozie


Only HDP delivers a Centralized Architecture HDP is uniquely built around YARN serving as a data operating system that provides multi-tenant Resource Management, consistent Governance & Security and efficient Operations services across Hadoop applications.

Hortonworks Data Platform

YARN Data Operating System •  A centralized architecture of

consistent enterprise services for resource management, security, operations, and governance.

•  The versatility to support multiple applications and diverse workloads from batch to interactive to real-time, open source and commercial.

Key Benefits

•  Multiple applications on a shared data set with consistent levels of service: a multitenant data platform.

•  Provides a shared platform to enable new analytic applications.

•  Delivers maximum cost efficiency for cluster resource management. Fewer servers fewer nodes.

Storage

YARN: Data Operating System

Governance Security

Operations

Resource Management

Existing Applications

New Analytics

Partner Applications

Data Access: Batch, Interactive & Real-time


HDP: Any Data, Any Application, Anywhere

Any Application •  Deep integration with ecosystem

partners to extend existing investments and skills

•  Broadest set of applications through the stable of YARN-Ready applications

Any Data Deploy applications fueled by clickstream, sensor, social, mobile, geo-location, server log, and other new paradigm datasets with existing legacy datasets.

Anywhere Implement HDP naturally across the complete range of deployment options

Clickstream Web & Social

Geoloca3on Internet of Things

Server Logs

Files, emails ERP CRM SCM

hybrid

commodity appliance cloud

Over 70 Hortonworks Certified YARN Apps


Social Media

Sentiment

The View from the Customer

Call Center Interaction

Quality of

Service

Lifestyle & Interests

Clickstream

Geolocation

Web Data

Internet of Things

Docs, emails

Server logs

Streaming: Network Probes, Click Stream, Sensor, Location

Batch: Call Detail Records

On-Line: Customer Sentiment

Unstructured: Txt, Pictures, Video, Voice2Text


DELIVERY

The Destination: Data-Centric Operations

Clickstream

Geolocation

Web Data

Internet of Things

Docs, emails

Server logs

Streaming: Network Probes, Click Stream, Sensor, Location

Batch: Call Detail Records

On-Line: Customer Sentiment

Unstructured: Txt, Pictures, Video, Voice2Text

Personal Data Analysis & Customer Insight Services To Customer & Partners

Hadoop Distribution with Yarn: Allows central source of data across all mediums of ingestion and interaction

Existing & Legacy Systems can Contribute and Participate: May extend the life of existing and legacy systems from enriched data

New Applications interact with Data Lake, not each other: Next Generation Apps build around data and can deliver to customers and partners


Hadoop Driver: Enabling the data lake SC

ALE

SCOPE

Data Lake Definition •  Centralized Architecture

Multiple applications on a shared data set with consistent levels of service

•  Any App, Any Data Multiple applications accessing all data affording new insights and opportunities.

•  Unlocks ‘Systems of Insight’ Advanced algorithms and applications used to derive new value and optimize existing value.

Drivers: 1.  Cost Optimization 2.  Advanced Analytic Apps

Goal: •  Centralized Architecture •  Data-driven Business

DATA LAKE

Journey to the Data Lake with Hadoop

Systems of Insight


Journey to Enhanced Customer Experience

Real-time Event Action based on Customer Context & Location

Collection of Data for Customer Insights (Data Science)

Actionable Insights through Customer Models (BI)

Current State of the Customer (Dynamic Customer Profile - NPS)


HDFS Raw Event Storage

Customer Experience Journey – Data Lake for Collection

1 ° °

° ° °

° °

HBase Processed Event Storage

° ° °

° ° °

° ° N

°

Mul3tenant Processing: YARN

(Hadoop O

pera.ng System)

Metadata M

anagement HCatalog

Hive / Tez

(Interac.ve Query)

ISV

(YARN Apps, i.e. HPA / LASR)

Slider

(Always-‐on Services)

Message Queues

Log Files

Web Services

JMS

Update Data Lake

Network Probe Events

ODBC / JDBC

Rest API

Native API

External Customer Data References



1 ° °

° ° °

° °


° ° °

° ° °

° ° N

°


(Hadoop O

pera.ng System)

Metadata M

anagement HCatalog

Hive / Tez

(Interac.ve Query)

ISV


Slider


Message Queues

Log Files

Web Services

JMS

Update Data Lake


ODBC / JDBC

Rest API

Native API


Customer Experience Journey – Actionable Insights



1 ° °

° ° °

° °


° ° °

° ° °

° ° N

°


(Hadoop O

pera.ng System)

Metadata M

anagement HCatalog

Hive / Tez

(Interac.ve Query)

ISV


Slider


Message Queues

Log Files

Web Services

JMS

Update Data Lake


ODBC / JDBC

Rest API

Native API


State of the Customer -  Net Promoter Score -  Appetite for Info score -  Target Advertising Profile -  # Dropped calls today/month -  Sentiment / Churn Score

HBase /Accumulo Real-‐3me Serving

° ° °

° ° °

° ° N

In-‐Line Memory

(Spark)

Update Customer Profile and Scores

Customer Experience Journey – Current State of Customer



Customer Experience Journey – Real-time Event Interaction

1 ° °

° ° °

° °


° ° °

° ° °

° ° N

°


(Hadoop O

pera.ng System)

Metadata M

anagement HCatalog

Hive / Tez

(Interac.ve Query)

ISV


Slider


HBase /Accumulo Real-‐3me Serving

° ° °

° ° °

° ° N

Streaming Event Processor: Storm

Machine Learning

(Spark)

Indexing

(Lucene)

Rules Processing

(Drools)

In-‐Line Memory

(Spark) Message Queues

Log Files

Web Services

JMS

Enrich Events with Customer info

And Score Matrix

Update Data Lake

Real-time Intelligent Action -  Marketing Promotions -  Next Best Action -  Dynamic Network Provisioning


ODBC / JDBC

Rest API

Native API

Messaging Platrom

: KaLa Update Customer

Profile and Scores


State of the Customer -  Net Promoter Score -  Appetite for Info score -  Target Advertising Profile -  # Dropped calls today/month -  Sentiment / Churn Score


Hortonworks for Customer Experience

Functional Area

Solution Components / Use Case

Description Problem Addressed Business Benefits

Customer Experience Management

Central Data Lake for 360 Customer View

Visibility of customer household view across services and accounts through ingestion of account service and event data into a central Data Lake with views into granular customer’s service experience

-  Silo view of customer in different systems

-  Social media unstructured data does not fit into existing EDW

-  Incorporating customer call center discussions into customer view

-  Complete view into customer experience across all services

-  Reduction in Customer Churn

Dynamic Customer Profile

Summarized instant view of customer across service identifiers and customer key performance metrics and ‘net promoter scores’. Used for immediate view of customer profile

-  How to react to customer contact & events based on their experience

-  What is the customer’s experience level and context at this moment in time

-  Next Best Action based on customer’s experience with service provider (Retail /Call Center)

-  Greater targeted marketing/advertising

Real-time Next Best Action

Real-time streaming of network event data to identify customer location

-  How to determine next best action when and where they are most appropriate to customer

-  Marketing and CEM analysis is after the fact; need for real-time

-  Context sensitive promotions 10x customer acceptance

-  Improves customer experience levels and customer retention


Customer Experience Management & Marketing Journey

HDP Landing Zone

HDP DataLake

Real-‐.me Streaming HDP DataLake

Dynamic Customer Profile

360 Customer Awareness & Household

View

Loca3on Based CEM & Real-‐3me

customer response Next Best Ac3on

Customer Sen3ment

Customer Aware Loca3on Based Promo3ons

Mul3-‐channel Customer Scoring Models

Telco Customer Care using Hadoop & Customer Behavior Analytics

Sanjeev Srivastav / [email protected]

Agenda

The changing landscape in customer care •  Interactions across multiple channels •  Key experience management strategies

Data lakes for ongoing analysis and action Visualize and predict •  Sequences of activity across call center and operations data •  Next-best-actions, churn, etc. based on activity sequences

23

24

The Changing Landscape !in Customer Care!

24

Traditional models of customer behavior were largely based on

customer profiles.

New focus on behavior models.

25

Data Lakes!

25

Investments in storing customer engagement data across various

interaction channels

26

Visualize & Predict!

26

Granular customer activity across channels

Enterprises see interaction fragments

27

Email Analytics

Mobile Analytics

Web Analytics

Retail Analytics

Social Analytics

Call Center Analytics Time

Understand each customer’s journey

28

Identify common interactions and influences

29

Contextual and individualized actions

30

Customer journey insights

Individual and contextual actions

Real time

?

Use case – call center + operations data Predict the

customers likely to contact the

call center

Predict the likely reason of the

next call

Avert the next call

31

Use case – call center + operations data Predict the

customers likely to contact the

call center

Use sequence of events +

customer profile

Predict the likely reason of the

next call

Illustrate with individual customer

experience

Avert the next call

Act during call or after call

32

Use case – digital channel abandonment

Interaction sequences ending in abandonment of

a digital channel

Profile of customers and relevant metrics

Interactions after abandonment

Evaluate customer

preferences

Likelihood of abandonment

Influence engagement

with customers

33

Responding to trends: Key technology strategy

Data Lake for cross-channel experience

Journey analytics

Predictive analytics

Actionability §  Batch and real-time engagement §  APIs and interfaces with existing systems

§  Models for recommendations and targeting §  Next best actions

§  Customer journey visualization §  Time and sequence query interface

§  Raw data, not summarized or aggregates §  Time sequences preserved

Users Apps Developers Models + APIs Data + APIs Backend

End–to-end System

34

Apigee Insights Demo •  Customer interaction across various “channels”

–  Web site –  IVR –  Call center agents –  Retail stores

•  Visualize –  “Channel surfing” –  Customer satisfaction

•  Act –  Segmentation, based on customer experience

35

Visualize: repeated calls to a call center •  Calls center data •  Operations data •  Network data

36

IVR change 25

Escalation 226,359 (66,532 dropoff)

Network 9,340

Education 18,588 (6,119 dropoff)

Internal issue 9,121

Equipment issue 9,121

Internal issue 661

(blank) 576

Network 528

No sound 27

No pic 29

Education 107 (35 dropoff)

Equipment issue 765 (243 dropoff)

Visualize: Customer profiles with path analysis

37

Predict: using customer profiles and events

Event sequences

before t (training)

Prediction at time=t

Events after t (test)

38 Machine learning @scale

Ok to have small number of

historical events that correlate with predicted events

Apigee Insights platform and tools

Machine learning (ML)

Segment customers

Individualized actions

ML feedback §  APIs for customer engagement and recording customer responses

§  Engine for hyper-personalization; in-context recommendations

§  Tool for business analysts; data interfaces to business systems

§  Models of behavior (Apigee GRASP + 3rd party models) §  Flexible modeling time windows

Users Apps Developers Models + APIs Data + APIs Backend

End–to-end System

39

Next Steps…

Download the Hortonworks Sandbox

Learn Hadoop

Build Your Analytic App

Try Hadoop 2

More about Apigee & Hortonworks http://hortonworks.com/partner/apigee

Contact us: [email protected]


Q&A

predicting customer experience through hadoop and customer behavior graphs

Software

data centric

explosion of data

data volumes

enterprise data

new data costly

datacentric model

greater value of data

app centric