customer event hub – a modern customer 360° view with datastax enterprise (dse)
TRANSCRIPT
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Customer Event HubCustomer 360° view with DataStax Enterprise (DSE)
Guido Schmutz – 4.7.2017
@gschmutz guidoschmutz.wordpress.com
Guido Schmutz
Working at Trivadis for more than 20 yearsOracle ACE Director for Fusion Middleware and SOAConsultant, Trainer Software Architect for Java, Oracle, SOA andBig Data / Fast DataHead of Trivadis Architecture BoardTechnology Manager @ Trivadis
More than 30 years of software development experience
Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz
2
COPENHAGEN
MUNICH
LAUSANNEBERN
ZURICHBRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:CHF 5.0 million
Financially self-supporting andsustainably profitable
Experience from more than 1,900 projects per year at over 800customers
3
Agenda
4
1. Customer 360° View – Introduction2. Customer 360° View – Challenges3. Customer 360° View – DataStax Enterprise (Graph) to the rescue
Customer 360° View - Introduction
5
Why Customer 360° View?
“Get closer than ever to your
customers. So close that you tell
them what they need well before
they realize it themselves.”
Steve Jobs, Apple
q Enhance customer service
q Provides real-time personalization
q Opens doors to new applications
q Increases security
q Lowers operational costs
q Increases operational efficiency
6
Customer 360°: Experience Expectations
7
“The Amazon effect” – why can’t I do .... as easy as buying a product on Amazon
8
Each time a customer is exposed to an improved digital experience, their engagement expectations are reset to a new higher level.
Source: Forrester
Customer 360°: Experience Expectations
Consistent
Acrossallchannels,brandsanddevices
Personalized
Toreflectpreferencesandaspiration
Relevant
Inthemomenttocustomer’sneedsandexpectations
Contextualized
Topresentlocationandcircumstances
9
Customer 360 – Key Use Cases
• Customer micro segmentation
• Next Best Offer• Campaign Analytics• Geo-Location
Analytics• Recommendation
Models
• Churn Modeling & Prediction
• Rotational / Social Churn
• Customer Lifetime Value
• Sentiment Analytics• Price Elasticity
Modeling
• Proactive Care Dashboard
• Customer Lifetime Value
• Subscriber Analytics• QoS Analytics• Real-Time Alerts
Target Marketing & Personalization
Churn Prevention & Customer Retention Proactive Care
10
Customer 360° - What do I need to know?
Who are you?
Where are you?
What have you purchased?
What content do you prefer?
Who do you know?
What can you afford?
What is your value to the business?
How / why have you contacted us?
11
From Static to Dynamic, Real-Time Micro-Segmentation
AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade
Traditional Segmentation
AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade
Device HistoryOther products / servicesBundling preferencesOffer HistoryCampaign Adoption HistoryCall Center Tickets
LocationSocial Influence
Applications UsedContent Preferences
Usage DetailsRoaming Analysis
Travel Patterns
QoS HistoryHousehold Analysis
Lifetime ValueChurn Score
Clickstream InfoChannel Preferences
Survey
Real-Time Micro-Segmentation
12
From Static to Dynamic, Real-Time Micro-Segmentation
AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade
Traditional Segmentation
AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade
Device HistoryOther products / servicesBundling preferencesOffer HistoryCampaign Adoption HistoryCall Center Tickets
LocationSocial Influence
Applications UsedContent Preferences
Usage DetailsRoaming Analysis
Travel Patterns
QoS HistoryHousehold Analysis
Lifetime ValueChurn Score
Clickstream InfoChannel Preferences
Survey
Real-Time Micro-Segmentation
13
IndividualizationEngaging customers as a segment of one in real-time by listening, capturing, measuring, assessing, and addressing intent across every enterprise touchpoint.
Source: Forrester
Customer 360° View - Challenges
14
Customer 360° View - Traditional Flow Diagram
Enterprise Data Warehouse
ETL / Stored Procedures
Data Marts / AggregationsLocation
Social
Clickstream
Segmentation & ChurnAnalysis
BI Tools
Marketing Offers
Billing &Ordering
CRM / Profile
MarketingCampaigns
15
Customer 360° View - Traditional Flow Diagram
Enterprise Data Warehouse
ETL / Stored Procedures
Data Marts / AggregationsLocation
Social
Clickstream
Segmentation & ChurnAnalysis
BI Tools
Marketing Offers
Billing &Ordering
CRM / Profile
MarketingCampaigns
Limited Processing
Power
Does not model easily to traditional
database schema
Limited Processing
Power
Storage Scaling
very expensive
Based on sample /
limited dataLoss in Fidelity
Other / New Data Sources
High Voume
and Velocity
16
Key Challenges in Driving a Customer 360° ViewData Silos
New Data Sources Costs of Data Processing
Data Volumes
• Multiple Data Silos• Often store overlapping and
conflicting information
• Data growing rapidly• Internet of Things will add to
that substantially
• Semi/Un-Structured Data Sources
• Streaming / Real-time data• Critical for building a true
360° view
• Cost prohibitive• Cost of storing data in
relational database systems per year
Clickstream Location/GPS
Call centerRecords
Social Media
17
Customer 360° View: Why status quo won’t work?
• static version of the customer profile in their data warehouse• Mainly structured data• Only internal data• Only “important” data• Only limited history
• Activity data – clickstream data, content preferences, customer care logs are kept in siloes or not kept at all
Data Analyst
Data Analyst
Data Analyst
Data Analyst
Data Analyst
DetailedCustomerActivityDatasitsinsilos!
18
Customer Journey through multiple “Siloed Systems”
19
SocialMedia
CallCenter
ComplaintManagement
Marketing
Coupons
Warehouse CRM
Shipping
Billing
OrderProcessing
Web Application
Customer Journey through multiple “Siloed Systems”
20
SocialMedia
CallCenter
ComplaintManagement
Marketing
Coupons
Warehouse CRM
Shipping
Billing
OrderProcessing
Web Application
Customer 360° View – DataStaxEnterprise (Graph) to the rescue
21
Why using Graph for Customer 360° View
22
Traditional RDBMS• Multiple Data Locations => siloes• Not all information related• difficult to access all information and relate
to each other
Graph Database• Connect all customer-related information in
one central place• Support for analytics on graph• Performance & Scalability
PartnerIdFirstNameLastNameBirthDateProfileImageURL
IdCustomerIdWhenWhereTypeDescription
Contacts
AddressIdPartnerIdStreetStreetNrZipCodeCityCountry
IdNamePriceDescriptionImageURL
Products
CustomerIdProductId
Order
IdCustomerIdWhenWhereInteractionTypeDescription
Customer Service
IdProductIdCommentUserId
Product Reviews
IdPartnerIdTwitterIdFacebookId
Customer
Geo Point
Customer
Address
Product
ownerOf (since)
interestedIn (since, degree)
lives (since)
ActivityparticipatesIn Termuses
uses
Employee
interactsWith
reference
idname
idwhenwheretext
idfirstNamelastNamebirthDateprofileImageURL
usesidnameimageURL
idstreetstreetNrzipCodecity
idnamealternateNameslocationpopulationelevation
Country
neighbour
belongsTo
idnameiso2iso3isoNumericcapitalareapopulationcontinent
uses
ActivityType
idname
belongsTo
idname
at
knows(since)
mentions
uses
Customer 360° View - Example
23
Geo Point
Customer
Address
Product
ownerOf (since)
interestedIn (since, degree)
lives (since)
ActivityparticipatesIn Termuses
uses
Employee
interactsWith
reference
idname
idwhenwheretext
idfirstNamelastNamebirthDateprofileImageURL
usesidnameimageURL
idstreetstreetNrzipCodecity
idnamealternateNameslocationpopulationelevation
Country
neighbour
belongsTo
idnameiso2iso3isoNumericcapitalareapopulationcontinent
uses
ActivityType
idname
belongsTo
idname
at
knows(since)
mentions
uses
Customer 360° View - Example
24
Hadoop ClusterdHadoop ClusterDSE Cluster
Batch Data Ingestion into Customer Hub
Billing &Ordering
CRM / Profile
MarketingCampaigns
Location
Social
Clickstream
Sensor Data
WeatherData
MobileApps
Emails
v
File Import /SQL Import
Cassandra DSE Graph
25
highlatency
Event Stream
Batch Data Ingestion into Customer Hub
DSEGraphDSE
GraphLoaderRDBMS
GroovyScript
ClickStream
26
Geo Point
Customer
Address
Product
ownerOf (since)
interestedIn (since, degree)
lives (since)
ActivityparticipatesIn Termuses
uses
Employee
interactsWith
reference
idname
idwhenwheretext
idfirstNamelastNamebirthDateprofileImageURL
uses
idnameimageURL
idstreetstreetNrzipCodecity
idnamealternateNameslocationpopulationelevation
Country
neighbour
belongsTo
idnameiso2iso3isoNumericcapitalareapopulationcontinent
uses
ActivityType
idname
belongsTo
idname
at
knows(since)
Transformation CSV/JSON
PartnerIdFirstNameLastNameBirthDateProfileImageURL
IdCustomerIdWhenWhereTypeDescription
Contacts
AddressIdPartnerIdStreetStreetNrZipCodeCityCountry
IdNamePriceDescriptionImageURL
Products
CustomerIdProductId
Order
IdCustomerIdWhenWhereInteractionTypeDescription
Customer Service
IdProductIdCommentUserId
Product Reviews
IdPartnerIdTwitterIdFacebookId
Customer
Real-Time Insights?
Real-Time Data Ingestion into Customer Event Hub
Microservice Cluster
Microservice State
{}
API
Stream Processing Cluster
StreamProcessor
State
{}
API
Billing &Ordering
CRM / Profile
MarketingCampaigns
Location
Social
Clickstream
Sensor Data
WeatherData
MobileApps
File Import / SQL Import
EventStream
Event Hub
Event Hub
Event Hub
EventStream
EventStream
Hadoop ClusterdHadoop ClusterDSE Cluster
Cassandra DSE Graph
27lowlatency
Process native Event Streams
Tweet-to-Cassandra
Wide-Row/Search
Tweet
Graph
Tweet-to-Graph
CustomerReference
ClickStream ClickStream
Activity-to-Graph
DSECluster
28
Geo Point
Customer
Address
Product
ownerOf (since)
interestedIn (since, degree)
lives (since)
ActivityparticipatesIn Termuses
uses
Employee
interactsWith
reference
idname
idwhenwheretext
idfirstNamelastNamebirthDateprofileImageURL
uses
idnameimageURL
idstreetstreetNrzipCodecity
idnamealternateNameslocationpopulationelevation
Country
neighbour
belongsTo
idnameiso2iso3isoNumericcapitalareapopulationcontinent
uses
ActivityType
idname
belongsTo
idname
at
knows(since)
Event Hub Streaming AnalyticsSensor Storage
Single vs. Batch Insert
29
• One Event ends up in many modifications of vertex and edges
• many round-trips needed if done with single API calls
• Single “batched” API call using Groovy script provides 3 – 5x performance gains
DSEGraphActivity-to-GraphTweetUser
publishesTerm*uses*
DSEGraphActivity-to-Graph
…...
Geo Point
Customer
Address
Product
ownerOf (since)
interestedIn (since, degree)
lives (since)
ActivityparticipatesIn Termuses
uses
Employee
interactsWith
reference
idname
idwhenwheretext
idfirstNamelastNamebirthDateprofileImageURL
uses
idnameimageURL
idstreetstreetNrzipCodecity
idnamealternateNameslocationpopulationelevation
Country
neighbour
belongsTo
idnameiso2iso3isoNumericcapitalareapopulationcontinent
uses
ActivityType
idname
belongsTo
idname
at
knows(since)
Process CDC (Change Data Capture) Events
CDC
Customer-to-Cassandra
Wide-Row/Search
CustomerEvent
GraphCustomer-to-
Graph
CustomerCDC
AddressCDC
ContactCDC
Aggregate-to-Customer
DSECluster
30
Geo Point
Customer
Address
Product
ownerOf (since)
interestedIn (since, degree)
lives (since)
ActivityparticipatesIn Termuses
uses
Employee
interactsWith
reference
idname
idwhenwheretext
idfirstNamelastNamebirthDateprofileImageURL
uses
idnameimageURL
idstreetstreetNrzipCodecity
idnamealternateNameslocationpopulationelevation
Country
neighbour
belongsTo
idnameiso2iso3isoNumericcapitalareapopulationcontinent
uses
ActivityType
idname
belongsTo
idname
at
knows(since)
PartnerIdFirstNameLastNameBirthDateProfileImageURL
IdCustomerIdWhenWhereTypeDescription
Contacts
AddressIdPartnerIdStreetStreetNrZipCodeCityCountry
IdNamePriceDescriptionImageURL
Products
CustomerIdProductId
Order
IdCustomerIdWhenWhereInteractionTypeDescription
Customer Service
IdProductIdCommentUserId
Product Reviews
IdPartnerIdTwitterIdFacebookId
Customer
Event Hub Streaming AnalyticsSensor Storage
Real-Time Data Ingestion into Customer Event Hub
Microservice Cluster
Microservice State
{}
API
Stream Processing Cluster
StreamProcessor
State
{}
API
Billing &Ordering
CRM / Profile
MarketingCampaigns
Location
Social
Clickstream
Sensor Data
WeatherData
MobileApps
File Import / SQL Import
EventStream
Event Hub
Event Hub
Event Hub
EventStream
EventStream
Hadoop ClusterdHadoop ClusterDSE Cluster
Cassandra DSE Graph
31
How to implement an Event Hub? Apache Kafka to the rescue
• publish-subscribe messaging system
• Designed for processing high-volume, real time activity stream data (logs, metrics, social media, …)
• Stateless (passive) architecture, offset-based consumption
• Initially developed at LinkedIn, now part of Apache
• Peak Load on single cluster: 2 million messages/sec, 4.7 Gigabits/sec inbound, 15 Gigabits/sec outbound
Reliable Data Ingestion in Big Data/IoT
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Broker 1 Broker 2 Broker 3
ZookeeperEnsemble
Apache Kafka Connect
• Scalably and reliably streaming data between Apache Kafka and other data systems
• not an ETL framework• Pre-build connectors available for Data
Source and Data Sinks• JDBC (Source)• Cassandra (Source & Sink)• Oracle GoldenGate (Source)• MQTT (Source)• HDFS (Sink)• Elasticsearch (Sink)• MongoDB (Sink)
Reliable Data Ingestion in Big Data/IoT
Source: Confluent
Declarative Dataflow Definition & Execution
34
Apache NiFi StreamSets
Hadoop ClusterdHadoop ClusterDSE Cluster
Real-Time Data Ingestion into Customer Event Hub
OLA
P
Microservice Cluster
Microservice State
{}
API
File Import / SQL Import
OLT
P
ParallelProcessing
Cassandra
Cassandra
StreamProcessing
DSEFS
Billing &Ordering
CRM / Profile
MarketingCampaigns
Location
Social
Clickstream
Sensor Data
WeatherData
MobileApps
EventStream
Event Hub
Event Hub
Event Hub
EventStream
CassandraReplication
35
Hadoop ClusterdHadoop ClusterDSE Cluster
Real-Time Data Ingestion into Customer Event Hub
OLA
P
Microservice Cluster
Microservice State
{}
API
File Import / SQL Import
OLT
P
ParallelProcessing
Cassandra
Cassandra
StreamProcessing
DSEFS
Billing &Ordering
CRM / Profile
MarketingCampaigns
Location
Social
Clickstream
Sensor Data
WeatherData
MobileApps
EventStream
Event Hub
Event Hub
Event Hub
EventStream
CassandraReplication
36
Questions which can only be answered by Graph
37
Dependencies•Failurechains•Orderofoperation
Matching/CategorizingHighlightvariantofdependencies
ClusteringFindingthingscloselyrelatedtoeachother(friends,fraud)
Flow/CostFinddistributionproblems,efficiencies
SimilaritySimilarpathsorpatterns
Centrality,SearchWhichnodesarethemostconnectedorrelevant
Source: Expero
Questions which can only be answered by Graph -Visualize Customer 360
38
Source: Expero
Questions which can only be answered by Graph -Visualize Customer 360
39
KeyLines
Cytoscape
LinkuriousJS
SigmajsD3
Source: Expero
Questions which can only be answered by Graph -Visualize Customer 360
40
Keylines: http://keylines.com• powerful JavaScript toolkit for graph visualization• powerful visualizations features: Layouting, Filtering,
Combining & Grouping, Time Bar, Geospatial, …• compatible with any modern browser & any graph
database