p 01 paw_methods_2017_10_30_v4
TRANSCRIPT
A METHOD TO AI MADNESS
Vishwa Kolla Head, Advanced Analytics
John Hancock Insurance
TOPICS
Background Framework Case Studies
2
BUILD ME A MODEL TO …
3
REDUCE COMPLAINTS
GROW WALLET-
SHARE
GROW CSAT
REDUCE CHURN
REDUCE COST TO
TARGET
GROW BOTTOM-LINE
GROW TOP-LINE
REDUCE COST TO
ACQUIRE
MODEL BUILD IS THE PATH OF LEAST RESISTANCE
4
Platforms
R
P H20
SPARK
TENSOR
FLOW
TBD
SUPERVISED
UN
SUPERVISED
NLTK NUM
PY
PAN
DAS
PLOT
LY
PLA
TFO
RM
S
ALG
OR
ITHM
S
PA
CK
AG
ES
PYO
DBC
CRY
PTO
PYPD
F
SCI
KIT
TOR
NAD
O
ZICT BAB
EL
BLA
ZE
A THOUGHTFUL APPROACH CAN YIELD BETTER OUTCOMES
BUSINESS
USE CASES DATA MATH
TECHNICAL
IMPL.
BUSINESS
IMPL.
FEED
BACK
5
TOPICS
Background Framework Case Studies
6
“A” MODEL BUILD FRAMEWORK
7
DATA TARGET CONSTRUCTION EVALUATION PERFORMANCE
SOURCES
DISTANCE FROM
SIGNAL
SAMPLING
METHOD
SAMPLE
SIZE
SIGNAL SIZE
PREDICTION
HORIZON
UNIT OF
ANALYSIS
ONE MODEL vs.
STRATIFIED
ONE MODEL vs.
SEVERAL MODELS
TARGET
DEFINITION
PRESENCE OR
ABSENCE
BLACK BOX vs.
CLEAR BOX
RECENCY
FREQUENCY
SEVERITY
FEATURE
SELECTION
MODELING
STRATEGY
MODEL
STRENGTH
EXPLANATORY vs.
IMPORTANCE
ACCURACY vs.
SENSITIVITY vs.
SPECIFICITY
ECONOMIES OF
SCOPE
MODEL
FIT
BAGGING
ENSEMBLE
SINGLE vs.
MULTIPLE STAGES
PREDICTION &
OPTIMIZATION
BOOSTING
TOPICS
Background Framework Case Studies
8
Business
9
IT (ALWAYS) STARTS WITH A BUSINESS PROBLEM
PROSPECTING NURTURE ACQUISITION
MARKET
SEGMENTS
CUSTOMER
SEGMENTS
LIKELY TO [*]
MEDIA
MIX
CHANNEL
SURVEY
ANALYTICS
CROSS / UP-
SELL
OCR
MISREP
LIKELIHOOD
MORTALITY
APS
SUMMARY
FLUIDLESS
SMOKER
LIKELIHOOD
MORBIDITY
CHURN
NEXT BEST
OFFER
CLAIM
LIKELI-
HOOD
JOURNEY
CLAIM
SEVERITY
NEXT BEST
ACTION
FRAUD
>>
TEXT
ANALYTICS
OPTIMIZE
NEXT LIKELY
ACTION
WELLNESS
IOT
ANALYTICS
NPS
ANOMALY
>>
10
FOCUS ON INCREMENTAL VALUE KEPT US GROUNDED
BUSINESS CASE OPTICAL REALIZABLE SHARED INCREMENTAL
IN PROSPECTING, TARGET OPTIMIZAITON IS A JOURNEY
12
… LOWER CUSTOMER TARGETING COSTS A SERIES OF OPTIMIZATION TARGETS …
Prospects
Leads
Apps
Issued
Placed
CPL
CPA
CPP
CP[*] CHANNEL
MIX
Data
13
PLANS ARE NOTHING ; PLANNING IS EVERYTHING
14
EDA USEABLE
USEFUL
DERIVATIVES
BI-VARIATE CROSSTAB PRINCOMP JOURNEY
A DATA STITCH IN TIME SAVES NINE
15
CLAIM TERMIN
ATION
CLAIM
ACTV. DEMOS
CALLS
INTERA
CTION
CLAIM
INIT.
CUSTOMER MONTH
FRAUD DETECTION
UNDERSTANDING DATA SAVES (NOT WASTES) TIME
16
Signal
Distribution
Pop. Incidence Rate
Skews Model Inclusion
Math
17
FLEXIBILITY IN TARGET DEFINITION IMPROVED ACTIONABILITY
18
2017 2014
Predict incidence
In next 3 years
2007
2017 2014
Predict incidence
3 years out
2007
Vs.
RIGHT SIZING SIGNAL CAN YIELD BETTER OUTCOMES
19
SIGNAL
DILUTION
SIGNAL
AMPLIFICATION
1%
99%
40%
60%
SIMPLE MODELS CAN HELP US EXPLAIN BIG DRIVERS
20
PREDICTORS
PRESENCE
RECENT
FREQUENT
SEVERE
QUANTIFICATION OF INFORMATION GAP IS A GOOD FIRST STEP
© Andrew Ng
INFORMATION GAP
WINNING
MODEL
CHALLENGING CHAMPIONS HELPS US UP THE ANTE
22
DATA TARGET
SOURCES
DISTANCE FROM
SIGNAL
SAMPLING
METHOD
SAMPLE
SIZE
SIGNAL SIZE
PREDICTION
HORIZON
UNIT OF
ANALYSIS
ONE MODEL vs.
STRATIFIED
ONE MODEL vs.
SEVERAL MODELS
TARGET
DEFINITION
METHODS
LINEAR
TREES
DEEP-
LEARNING
EVALUATION
MODEL
STRENGTH
EXPLANATORY vs.
IMPORTANCE
ACCURACY vs.
SENSITIVITY vs.
SPECIFICITY
ECONOMIES OF
SCOPE
MODEL
FIT
Technical Implementation
23
MULTI-STAGED MODELS PROVIDED IMPLEMENTATION FLEXIBILITY
24
9-1
0
1-8
1-7 8-10
Likely to
Qualify
Likely to
Respond
Sweet
Spot
DESIRED
SIGNAL
MODEL
MIS-
CLASSIFICATION
MODEL
EXCLUDE NOISE 1
INCLUDE MIS-CLASSIFIERS 2
STAGES
Business Implementation
25
A CULTURE OF MEASUREMENT, TEST AND LEARN IMPROVES VALUE
26
Measure
Test Learn
Build
CONTINUOUS
IMPROVEMENT