sunz2013 vince morder
Post on 19-Oct-2014
372 views
DESCRIPTION
TRANSCRIPT
Vincent Morder
Analytics Manager, Loyalty New Zealand
Insights on Loyalty’s Approach
To Big Data
SUNZ February 2013
Loyalty Vision: To create, maintain and
motivate loyal customers for our Participants
End to End Marketing
4
History of data and analytics In the beginning….
• Data • Small, manual, fixed
• Systems • Static, relational
• Analytics • Sophistocated statistics and mathematics
• Retrospective
• Hand- crafted variables
• Static view
• Long time lags
5
History of data and analytics Over time….
• Data
• Bigger data, coming from various systems, more timely
• Systems
• Operational systems
• More storage required
• Relational databases
• Computing power increased
• Analytics
• Applying stats to business, banks, retail, telecommunications, etc…
• Same statistical principles applied, but now on hormones with software
• Sample size no longer an issue.
6
Today: Big Data
• Sheer magnitude of data captured by digital world:
• Web
• Social
• Free form text
• Retail Transactions
• Mobile device activity
• Software logs
• Cannot be managed by traditional data management tools
• Unstructured to a large degree
• Data is distributed, so difficult to perform traditional queries for analysis
• Applications and needs are for real time.
7
Big sets of Data
• Historically loyalty has a lot of data that is big
• Transactions
• SKU Transactions
• Campaign history
• But it has not been Big Data
• Here are some examples….
8
Transactions
• SKU transactions still fit into structures and can be managed
• But it’s big – over 1,000,000,000 items and 100,000 products.
• Challenged to make data manageable
• Models help to key dimensions to make actionable
• E.g., Clusters using Ideal Dimensions (Morder, SUNZ 2012)
• Boil down products and customers to 20 key dimensions.
9
Campaigns
• Campaign history still fit into structures and can be managed
• But it’s big – over 500,000,000 records and 000’s of campaigns.
• Models on campaign history have to boil down data to key dimensions to make actionable.
• E.g., SAS MO uses these models to optimise campaign performance.
• Boil down 70 response models (one per partner).
10
Where the world is heading
• Digital data will continue to grow faster than traditional data
• Digital representation of all our transactions and activity
• Retail environment craves more info to keep up with competition
• ……And our customers expect us to use it instantly
11
Big Data at Loyalty
• Last 12 months we have put a new web-server called Harry.
• Postgres System
• Captures web activity
• Integrates with Service Centre.
• Plans are for Harry expand to incorporate
• Core Fly Buys rewards, transactions, and points processing
• Smart phone app behaviour
• Real time recommendations
• … And this is Big Data.
12
CAP Theorem or Brewer’s Triangle
• Consistency: all clients always have the same view of the data
• Availability: each client can always read and write
• Partition Tolerance: operation continues despite physical network partitions.
13 (P)artition Tolerance
Tradeoffs • A and C means you will not have P
• C and P means you will not have A
• A and P means you will not have C
14
(P)artition Tolerance
A+C
A+P C+P
Pick Any Two
Big Data at Loyalty
• We have systems all around the CAP triangle for different purposes.
15
Riak Mongo
Data warehouse
Pick Any Two
(P)artition Tolerance
Postgres
Pick
Any
Two
Implications for Analytics
• Some data and models are same old style: static, historical looking. • One model or segmentation applied across all customers
• Still very important to have these.
• More and more models using data based on current activity.
Examples
• Real time decisions are rule- based and trigger off of observed activity
• Recommendation engines use a model for every individual • Recommendation based on individual product preferences
16
Models
• Preferences and associations to be updated as data comes in.
• Model structure needs to be able to handle being updated as data is updated.
• Models need to be able to deal with eventual consistency.
• Information may not be 100% complete.
• E.g., Poisson Models coefficients to be updated additively.
• Good for billions of record that need to be processed online, like movie recommendations. 1
7
Recommendation Engines
• Two main types of recommendation systems:
• Content-based
• Collaborative filtering,
• Batch vs. real time algorithms
• Push and pull through all our channels
18
Design of data
• Combine big historical string with new information as it comes in….
• Final Prepare has helped us ‘prepare’ for the future,
• Unfortunately it’s not final.
19
Slow moving models and characteristics
(FINAL PREPARE)
Fast moving new data
Going from Unstructured to Structured data
• Modeling and analysis still requires that data has a structure.
• Find and define your associations. Applies to: • Text mining
• Web logs activity
• Smart phone behaviour
• Consumer behaviour.
Two words: Map and Reduce
• Software functions to go from unstructured to structured
• Tell your IT guys to code for the established mappings.
20
Example - Hypothetical
• Data on a customer and his history of buying patterns – historical data on CA server • Discover that he is online and looking to find out about specials – data feed to AP system. • RFI indicates he is entering the store X – another data feed but to CP server. • CP server triggers for an algorithm to calculate the best offer for that person.
• Proactive message sent by CP server to offer a discount on complementary product. • Capture record of actual spend at checkout – another data feed to CA server • Feedback survey send after leave store – capture on AP server. • Capture record of spend and feedback to perform post campaign analysis and improve models on CA
system. • Report campaign results online to retailer on CA system • Better shopping experience for customer, greater commitment, spreads the word. • Better models, more accurate offers next time. Continual improvement. • Applications are limitless.
21
SAS Products on the Big Data Front
• Real Time Decision Manager
• Social Network Analytics
• Text Miner
• Visual Analytics
22
The Future of the Analytics Industry and of LNZ
• More applications, more digital presence
• Analytics are being led by technology and data
• Implications is that models are changing • Hybrid models: slow models + real time data
• One model per customer
• Analysts will be shifting their focus • Build new types of models and real time solutions
• Unstructured to structured
• Tracking of models will change.
• Lab 360 will be there to provide technology and services for both internal partners and external clients.
23