![Page 1: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/1.jpg)
Markov Chains in R
Divyansh SaxenaData Scientist @Trainline
![Page 2: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/2.jpg)
“A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.”
2
![Page 3: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/3.jpg)
3
“A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.”
Deterministic vs Stochastic Model
Markov Property
![Page 4: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/4.jpg)
4
Deterministic Model Stochastic Model
In deterministic models, the output of the model is fully determined by the parameter values and the initial conditions.
Eg - Calculation to determine the return on a 5-year investment with an annual interest rate of 7%, compounded monthly.
The model is just the equation below
F = P(1 + r/m)Ym
Stochastic models possess some inherent randomness. The same set of parameter values and initial conditions will lead to an ensemble of different outputs.
Eg - A random walk process
Path A
Path B
Path C
![Page 5: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/5.jpg)
55
“A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.”
Deterministic vs Stochastic Model
Markov Property
![Page 6: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/6.jpg)
6
Markov Property
In probability theory and statistics, the term Markov property refers to the memoryless property of a stochastic process.
A stochastic process has the Markov Property if the conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not on the sequence of events that preceded it.
P(Xn = xn | Xn-1 = xn-1, . . . , X0 = x0) = P(Xn = xn | Xn-1 = xn-1)
![Page 7: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/7.jpg)
Marketing Attribution
![Page 8: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/8.jpg)
8
“In marketing, attribution is the identification of a set of user actions ("events" or "touchpoints") that contribute in some manner to a desired outcome, and then the
assignment of a value to each of these events.”
Display PPC SEO Conversion No ConversionDisplay PPC SEO No Conversion
![Page 9: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/9.jpg)
9
What is the contribution of each unique ad type towards conversion?
33%
33%
33%50% 50% 100%OR OR
SEOPPCDisplay
![Page 10: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/10.jpg)
10
Types of Attribution
Single Source Attribution
FractionalAttribution
Algorithmic / ProbabilisticAttribution
Last Click
First Click
Last Touch
Markov Chains
SurvivalModel
Recurrent Networks
![Page 11: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/11.jpg)
11
Let’s assume 3 customer journeys
Display PPC SEO Conversion No ConversionDisplay PPC SEO No Conversion
1 2 3
![Page 12: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/12.jpg)
Last Click Method
![Page 13: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/13.jpg)
12
Display PPC SEO Conversion
100%
0% 0%
![Page 14: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/14.jpg)
Markov Chain Method
![Page 15: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/15.jpg)
12
1 - Customer Journey 2 - Transformation 3 - Splitting for pairs
Display > PPC > SEO > Conversion
Display
PPC > SEO
(start) > Display > PPC > SEO > (conversion)(start) > Display, Display > PPC, PPC > SEO, SEO > (conversion)
(start) > Display > (null)
(start) > PPC > SEO > (null)
(start) > Display, Display > (null)
(start) > PPC, PPC > SEO, SEO > (null)
![Page 16: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/16.jpg)
Transition Matrix
from to probability total probability
(start) Display ⅓
(start) Display ⅓2/3
(start) PPC ⅓ 1/3
Display PPC ½
Display (null) ½
1/2
1/2
PPC SEO ½
PPC SEO ½ 1
SEO (conversion) ½
SEO (null) ½
1/2
1/2
13
![Page 17: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/17.jpg)
Markov Chain
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
14
![Page 18: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/18.jpg)
15
What is the total conversion probability?
![Page 19: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/19.jpg)
16
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
Total Conversion Probability = 66.7%
![Page 20: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/20.jpg)
17
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
Total Conversion Probability = 66.7% x 50%
![Page 21: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/21.jpg)
18
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
Total Conversion Probability = 66.7% x 50% x 100%
![Page 22: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/22.jpg)
19
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
Total Conversion Probability = 66.7% x 50% x 100% x 50%
![Page 23: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/23.jpg)
20
Total Conversion Probability = (66.7% x 50% x 100% x 50%) + (33.3%)
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
![Page 24: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/24.jpg)
21
Total Conversion Probability = (66.7% x 50% x 100% x 50%) + (33.3% x 100%)
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
![Page 25: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/25.jpg)
22
Total Conversion Probability = (66.7% x 50% x 100% x 50%) + (33.3% x 100% x 50%)
= (16.7% + 16.7%)
= 33.3%
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
![Page 26: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/26.jpg)
23
What is the contribution of each unique ad type towards conversion?
33%
33%
33%50% 50% 100%OR OR
SEOPPCDisplay
![Page 27: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/27.jpg)
Removal Effects
![Page 28: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/28.jpg)
25
Removal Effects
If we were to figure out what is the contribution of Display in our customer’s journey from start to end conversion, we will use the principle of removal effect.
Removal effect principle says that if we want to find the contribution of each channel in the customer journey, we can do so by removing each channel and see how many conversions are happening without that channel being in place.
![Page 29: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/29.jpg)
26
Removal Effect of Display
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
![Page 30: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/30.jpg)
27
Removal Effect of Display
(start)
PPC
(null)
(conversion) SEO
33.3%
100% 50%
50%
![Page 31: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/31.jpg)
28
Removal Effect of Display
(start)
PPC
(null)
(conversion) SEO
33.3%
100% 50%
50%
Conversion probability after removing Display = (33.3% x 100% x 50%)
= 16.7%
![Page 32: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/32.jpg)
29
Removal Effect of Display
(start)
PPC
(null)
(conversion) SEO
33.3%
100% 50%
50%
Conversion probability after removing Display = 16.7%
Removal Effect of Display = (33.3% - 16.7%) / 33.3%
= 50%
![Page 33: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/33.jpg)
30
Removal Effect of PPC
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
![Page 34: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/34.jpg)
31
Removal Effect of PPC
(start)
Display (null)
(conversion) SEO
66.7%
50%
50%
50%
![Page 35: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/35.jpg)
35
Removal Effect of PPC
(start)
Display (null)
(conversion) SEO
66.7%
50%
50%
50%
Conversion probability after removing PPC = 0%
Removal Effect of PPC = (33.3% - 0%) / 33.3%
= 100%
![Page 36: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/36.jpg)
36
Removal Effect of SEO
(start)
Display
PPC
(null)
(conversion) SEO
66.7%
33.3%
100% 50%
50%
50%
50%
![Page 37: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/37.jpg)
37
Removal Effect of SEO
(start)
Display
PPC
(null)
(conversion)
66.7%
33.3%
50%
50%
![Page 38: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/38.jpg)
38
Removal Effect of SEO
(start)
Display
PPC
(null)
(conversion)
66.7%
33.3%
50%
50%
Conversion probability after removing SEO = 0%
Removal Effect of SEO = (33.3% - 0%) / 33.3%
= 100%
![Page 39: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/39.jpg)
Removal Effect of each ad type / channel
50% 100% 100%
Display PPC SEO
39
![Page 40: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/40.jpg)
Contribution of Display
50% 100% 100%
Display PPC SEO
40
50%
Display
![Page 41: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/41.jpg)
Contribution of Display
50% 100% 100%
Display PPC SEO
41
50%
Display 20%
![Page 42: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/42.jpg)
Contribution of each ad type / channel using Markov Chain Method
20% 40% 40%
Display PPC SEO
42
![Page 43: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/43.jpg)
Results
![Page 44: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/44.jpg)
44
Display PPC SEO Conversion No ConversionDisplay PPC SEO No Conversion
Last Click Method Markov Chain Method
0% 0% 100%
Display PPC SEO
20% 40% 40%
Display PPC SEO
![Page 45: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/45.jpg)
Markov Chains in R
![Page 46: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/46.jpg)
46
Package in CRAN - ChannelAttribution
Two functions - Markov Model and Heuristic Model
Last TouchFirst Touch
![Page 47: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/47.jpg)
47
A customer converted after interacting with 4 channels / ad types.
![Page 48: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/48.jpg)
48
channel_9 > channel_6 > channel_0 > channel_1
![Page 49: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/49.jpg)
49
Markov Model in R
![Page 50: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/50.jpg)
50
Results TransitionMatrix
Removal Effects
![Page 51: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/51.jpg)
51
Results TransitionMatrix
Removal Effects
![Page 52: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/52.jpg)
52
Results TransitionMatrix
Removal Effects
![Page 53: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/53.jpg)
53
Results TransitionMatrix
Removal Effects
![Page 54: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/54.jpg)
54
Comparison of Markov Model result result with Heuristic Model result
![Page 55: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/55.jpg)
Conclusion
![Page 56: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/56.jpg)
56
Simple&
Easy to Interpret
![Page 57: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/57.jpg)
57
Simple&
Easy to InterpretPowerfullyPredictive
![Page 58: Markov Chains in R · conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f1625e51f5f1a58c13aab7c/html5/thumbnails/58.jpg)
58
Simple&
Easy to InterpretPowerfullyPredictive
Data - Driven