customers value optimization in digital marketing

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Customers Value Optimization in Digital MarketingGeorgios Theocharous (Adobe Research) Mohammad Ghavamzadeh (Adobe Research &

INRIA Lille)

Shie Mannor (Technion)


Adobe’s Marketing Cloud

2

Real-time web, social and mobile analytics

• Manage social content in social networks

• Listen and respond to customer conversations in real time

• Create social campaigns

Plan and execute orchestrated campaigns across all channels

Organize, manage, and deliver creative assets and other content across digital marketing channels (web site management)

Manage, forecast, and optimize your media mix to deliver peak return on your investment

Automated decision-making and targeting

7 out of 10 dollars transacted on the web pass through Adobe products


Outline

Problem

Need for lifetime optimization

Research challenges

Solutions

3


Problem

4


Problem: Sequential Decision Making under Uncertainty

5

MARKETINGAGENT

ACTION• Display offers

STATE OBSERVATIONS• User

Demographics

• Recency• Frequency• Monetary REWAR

D• Click

s

What is the optimal marketing strategy?


Problem: Algorithms

6

XDisp.offer

XDisp.offer

X Disp.offer

Visitor X (represented by behavioral and contextual features)

XDisp.offer

XDisp.offer

X Disp.offer

Myopic (State of the art ): Offers shown now are agnostic about the future

LTV (New solution): Offers shown now consider the impact on future offers

t0 (now) t1 t2


Need for LTV

7

3 722 329 visits in 90 days

1 867 916 visitors

28.53 % visitors are recurring

49.81 % of all visits are recurring visits


Need for LTV

8

41.70 % of all conversions are happening in a recurring visit

3.96 % of the buying visitors buy again


Challenge 1: Off-policy evaluation

Ideas:

Importance Sampling: But has high variance

Simulator: Hard to capture the true dynamics of a noisy and non-stationary world

9

A NEW POLICY ΠVALUE OF POLICY Π

REAL TRAJECTORIESFROM POLICY IN PRODUCTION Β


Challenge 2: Evaluating a simulator

Ideas:

Error in predicting next state

Performance statistics… similarity of expected rewards in real and simulated data (unknown or random behavior policy)

Bound the error

…

10

SIMULATORSCORE

REAL TRAJECTORIES


Challenge 3: Robust Optimization

Ideas:

Pessimistic solutions

11

ROBUST POLICY

TRAJECTORIES FROM MULTIPLEPERIOD SEGMENTS


Challenge 4: Scaling up

Ideas:

Hadoop

12

RL POLICY

REALTRAJECTORIES100’s features


Challenge 5: Online Versions

Ideas:

Turn FQI into an online or batch updates

13

RL POLICY UPDATEONE SAMPLE


Challenge 6: Learning the Right Representation

Ideas:

Dimensionality reduction

14

REALTRAJECTORIES1000’s features

TRAJECTORIES10’s features


Challenge 7: Policy Visualizations

Ideas:

Graph analysis

15

VISUALIZATIONTRAJECTORIES


Challenge 8: Progressively More Engaging Interactions

Ideas:

Activity learning

16

HIERARCHICAL MDPsTRAJECTORIES


Research challenges: Hierarchical Representations:

17

Home

ModelsCertified Pre-own

Innovations

ExploreBMW

BuildYourOwn

Dealer Locato

r

TestDriv

e

Sales & Progra

ms

Financial Services

MyBMW

Owners

Accessories

Search

OWNER

LEARN ABOUT BMW

FIND A CAR

PAYMENTOPTIONS

TEST DRIVE


Solutions: Data Sets Explored

Data set 1

zeros: 10256 (82%)

ones: 1575 (13%)

more than one success : 636 (5%)

at least one success: 2211 (18%)

number of episodes: 12467

number of interactions: 602755

Data set 2

zeros: 18472 (87%)

ones: 2218 (10%)





18

Data set 3

zeros: 1568 (65%)

ones: 446 (18%)






Solutions: Features

19

Features

Cumulative action There is one variable for each offer, which counts number of times each offer was shown

Visit time recency Time since last visit

Cumulative success

Sum of previous reward

Visit The number of visits so far

Success recency The last time there was positive reward

Longitude Geographic location [Degrees]

Latitude Geographic location [Degrees]

Day of week Any of the 7 days

User hour Any of the 24 hours

Local hour Any of the 24 hours

User hour type Any of weekday-free, weekday-busy, weekend

Operating system Any of unknown, windows, mac, linux

Interests There a finite number of interests for each company. Each interest is a variable, that gets a score according to the content of areas visited within the company websites

Demographics There are many variables in this category such as age, income, home value...


Solutions: Experimental Setup

20

SimulatorPolicy

Various LTV and myopic strategiesLearn to predict each feature for each action, using system Identification techniques

Time-series Data

(S,A,R,S’)TRAININ

GTESTING


Solutions: RL algorithms used

We use various state of the art batch RL algorithms. We computed optimally and myopic solutions:

Kernel Based RL on representative states

K-means RL

Fitted Q iteration (FQI )

FQI-sarsa

Random

21


Solutions: Simulator

22

1. Constant (e.g., a global interest)2. Remains the same (e.g., demographics)3. Constant increment (e.g., cum action counter)4. Same as another feature (e.g., interest)5. Increment with multiple of a future random variable

(e.g., cum success =cum success + reward)6. Counts binary events until reset (e.g., success

recency)7. Random variable (Visit time recency: sample

empirically)8. Everything else (predict with regression or

classification)


Results

Data set KBRL-RSoptimal

K-means –RLoptimal

FQIOptimal

FQI-sarsaoptimal

KBRL-RSmyopic

K-means-RLmyopic

FQImyopic

Random

1 0.054 0.06 0.007 0.015 0.037 0.03 0.044 0.033

2 0.038 0.03 0.04 0.04 0.036 0.02 0.026 0.033

3 0.003 0.003 0.003 0.004 0.003 0.002 0.002 0.002

20% improvement

With LTV users convert faster

With LTV users engage longer


Schedule

24

08:45 - 09:00 Introduction09:00 - 10:00 Craig Boutilier - University of Toronto (Invited Talk)10:00 - 10:20 Andres Munoz Medina - New York University10:20 - 10:40 Coffee Break (Poster Session)10:40 - 11:40 John Langford - Microsoft Research (Invited Talk) 11:40 - 12:00 Bruno Scherrer - INRIA Nancy

Lunch

14:00 - 15:00 Shie Mannor - Technion (Invited Talk)15:00 - 15:20 Mohammad Ghavamzadeh - Adobe Research & INRIA Lille15:20 - 15:40 Coffee Break (Poster Session)15:40 - 16:40 Esteban Arcaute - Walmart Labs (Invited Talk)16:40 - 17:00 Philip Thomas - University of Massachusetts Amherst

http://chercheurs.lille.inria.fr/~ghavamza/ICML-2014_LTV_Workshop/Invited_Talks.html


http://chercheurs.lille.inria.fr/~ghavamza/ICML-2014_LTV_Workshop/Papers.html










customers value optimization in digital marketing

Documents

adobe confidential

adobe systems incorporated

adobe products

social content

policy evaluationideas

social campaignsplan

social networks

recurring visits