markov chain - basic conceptsricci/slidesmarkov.pdfmarkov chain: basic characteristics markov chain:...
TRANSCRIPT
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain
Basic Concepts
Laura Ricci
Dipartimento di Informatica
24 luglio 2012
PhD in Computer Science
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
P2P System Evolution: Brief Overview
Unstructured Approaches
random connectionssearch mechanisms: TTL-enhanced floodingexamples: Gnutella, Kazaa, BitTorrent,...pro: simplicity, cons: limited scalability
Structured Approaches
Distributed Hash Tables, Delaunay Overlayexamples: Kademlia, implementation in eMule/Bittorrentpro: routing efficiency, cons: consistency mainteinance inpresence of churn
Stochastic Approaches
Gossip, Random Walksimple, based on a well known mathematical theoryexploited also for distributed crawling of social networks
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
P2P System: Stochastic Approaches
unstructured/structured approaches presented in theCourse ”P2P Systems” of the Master Degree in ComputerScience
here we focus on stochastics appoches
mathematical background
Markov Chains, Random Walk
applications:
Distributed sampling for extimating network parametersEpidemic information diffusionDistributed computation af aggregate values
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Outline
1 Basic Definitions
2 Examples
3 It’s All Just Matrix Theory?
4 The Basic Theorem
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain: Basic Characteristics
Markov Chain: describes a system whose states changeover time
discrete time stochastic process
Changes are governed by a probability distribution.
The next state only depends upon the current system state
the path to the present state is not relevant
Class of random process useful in different areas
developments in theory and applications in recent decades.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain Specification
A sequence of Random Variables {X0, X1, . . . Xn}: Xi describesthe state of the system at time i
To specify a Markov Chain we need:
a finite or countable set of states S
S = {1, 2, . . . , N} for some finite Nthe value of the random variables Xi are taken from S
an initial distribution π0, π0(i) = P{X0 = i}: probabilitythat the Markov Chain starts out in state i
the probability transition rules
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
The Probability Matrix
The probability matrix P = (Pij) specifies the transitionrules
if the size of S is N , P is a N × N stochastic matrix
each entry is non negativethe sum of each row is 1
Pij is a conditional probability: defines the probability thatthe chain jumps to state j, at time n + 1, given that it is instate i at time n,
Pij = P (Xn+1 = j | Xn = i)
We assume Time Homogeneity: the probability does notdepend on the time n, but probability only on dependsstate i and j
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Time Homogeneity
”where I go next given that I am in state s at time x isequal to where I go next given that I am in the same states at time y 6= x”
every time the chain is in state s, the probability ofjumping to another state is the same
the same probability over time
we will assume time homogeneity in the following
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
The Markov Frog
A frog hopping among lily pads
State Space: S = {1, 2, 3}represent the pads
Initial Distribution:π0 = (1/2, 1/4, 1/4)
Probability Transition Matrix:
P =
0 1 01/3 0 2/31/3 1/3 1/3
defines the probabilities ofjumping from one state toanother one
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
The Markov Frog
the frog chooses its initial position X0 according to theinitial distribution π0
to this purpose, the frog can ask its computer to generate auniformly distributed random number U0 in the intervalI = [0, 1] and then taking
X0 =
1 if 0 ≤ U0 ≤ 1/2
2 if 1/2 < U0 ≤ 3/4
3 if 3/4 < U0 ≤ 1
for instance, if U0 = 0.8419 then X0 = 3
the frog starts on the third lily pad
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
The Markov Frog
Evolution of the Markov Chain: the frog chooses a lily padto jump
state after the first jump = value of the random variable X1
the frog starts from lily pad 3 so look at the probabilitydistribution in row 3 of P , namely (1/3,1/3,1/3)
again, generate a uniformly distributed random number U1
in the interval I = [0, 1] then takes
X1 =
1 if 0 ≤ U1 ≤ 1/3
2 if 1/3 < U1 ≤ 2/3
3 if 2/3 < U1 ≤ 1
if U1 = 0.1234, then X1 = 1
the frog jumps from the lily pad 3 to lily pad 1X1 = 1 there is no choice for the value of X2, it must be 2and so on....
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
The Markov Property
in the previous example
P{X3 = j | X2 = 2, X1 = 1, X0 = 3} = P{X3 = j | X2 = 2} ∀j
the only information relevant to the distribution to X3 is theinformation that X2 = 2
X0 = 3 and X1 = 1 may be ignored!
Definition
A stochastic process X0, X1,....satisfies the Markov Property ifP{Xn+1 = in+1 | Xn = in, Xn−1 = in−1, . . . , X0 = i0} =P{Xn+1 = in+1 | Xn = in}
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chains: Ehrenfest Chain
There is a total of 6 balls in two urns, 4 in the first and 2 in thesecond. We pick one of the 6 balls at random and move it tothe other urn.Xn number of balls in the first urn, after the nth move.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chains: Ehrenfest Chain
P(X0 = 4) = 1
P(X1 = j) =
4/6 j = 3
2/6 j = 5
0 otherwise
P(Xn+1 = l | Xn = j) =
j/6 l = j − 1
(6− j)/6 l = j + 1
0 otherwise
In 5 unites of time X0,. . . X5 might follow the following path:
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Ehrenfast Chain: Probability Transition Matrix
P =
0 1 0 0 0 0 01/6 0 5/6 0 0 0 00 2/6 0 4/6 0 0 00 0 3/6 0 3/6 0 00 0 0 4/6 0 2/6 00 0 0 0 5/6 0 1/60 0 0 0 0 1 0
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chains and Random Walks
given a graph and a starting point
select one of its neighbours at random, and move to thisneighbor
then select a neighbour of the new point point at random,and move to it, and so on . . .
the (random) sequence of points selected in this way is arandom walk on the graph.
there is a strict relation between Random Walk andMarkov chains
time-reversible Markov chains can be viewed as randomwalks on undirected graphswe will see this later . . .
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Random Walks and Complex Networks
what is the connection between Random Walk andP2P/Social Networks?
a simple way to analyse complex graph
gather information about all nodes and links at a singleserverexploit classical graph theoryunfeasible solution due to the huge dimension of the graphs
an alternative solution
compute a node sample through a random walkexploit the node sample to estimate network properties
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
what is the probability pnij that,given the chain in state i, it willbe in state j, n step after?
if we start on lily pad 3, what isthe probability of being on lilypad 1, after 2 steps?
p231 = p31p11+p32p21+p33p31 =
130 + 1
313 + 1
313=1
9+19=2
9
This is the dot product thirdrow-first column of P
this returns the 1,3-entry of theproduct of P with itself.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
The generalization of the previous observation leads to thefollowing theorem.
Theorem
Let P be the transition matrix of a Markov chain. The ij-thentry of the matrix Pn gives the probability that the Markovchain, starting in state si, will be in state sj after n steps.
The power of the transition matrix gives interesting informationabout the evolution of the process
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
P 1 =
pad1 pad2 pad3
pad1 0 1 0pad2 0.333 0 0.666pad3 0.333 0.333 0.333
P 2 =
pad1 pad2 pad3
pad1 0.333 0 0.666pad2 0.221 0.554 0.221pad3 0.221 0.443 0.332
P 3 =
pad1 pad2 pad3
pad1 0.258 0.295 0.442pad2 0.244 0.404 0.342pad3 0.244 0.392 0.355
P 4 =
pad1 pad2 pad3
pad1 0.246 0.368 0.371pad2 0.244 0.369 0.367pad3 0.245 0.369 0.367
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
P 5 =
pad1 pad2 pad3
pad1 0.241 0.363 0.362pad2 0.239 0.361 0.360pad3 0.240 0.361 0.361
P 6 =
pad1 pad2 pad3
pad1 0.231 0.249 0.248pad2 0.230 0.247 0.246pad3 0.230 0.247 0.247
P 7 =
pad1 pad2 pad3
pad1 0.213 0.322 0.321pad2 0.212 0.320 0.320pad3 0.212 0.321 0.320
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
The probabilities of the three lily pads are .2, .3, and .3 nomatter where the frog starts at the first step.
the long range predictions are independent from thestarting state
the columns are about identical because the chain forgetsthe initial state
but . . . this does not happen for all chains . . ..
we will show the conditions to be satisfied by the chain toguarantee this behaviour
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All just Matrix Theory?
consider π0 is a vector of N component defining, ∀ state i,the probability that the Markov chain is initially at i
π0(i) = P{X0 = i}, i=1 . . . N
πn(j) defines the probability that the Markov chain is at j,after n steps.
πn = {πn(1), . . . , πn(N)}
Theorem
Let P be the transition matrix of a Markov chain, and let π0 bethe probability vector which represents the starting distribution.Then the probability that the chain is in state si after n steps isthe ith entry of πn = π0P
n
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
πn(j) defines, ∀ state j, the probability that the Markovchain is at j, after n steps.
πn = {πn(1), . . . , πn(N))
thus, we have
π1 = π0Pπ2 = π1P = π0P
2
π3 = π2P = π0P3
in generalπn = π0P
n
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
let us supposeπ0 = (1/3, 1/3, 1/3)
we want to compute the statedistribution after 3 frog jumps
π3 = π0P 3 =
( 13, 1
3, 1
3)
0.258 0.295 0.4420.244 0.404 0.3420.244 0.392 0.355
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
It’s All Just Matrix Theory?
No, not just matrix theory...
In principle, matrix algebra can give us the answer to anyquestion about the probabilistic behaviour of a Markov chain,but this is not viable, in practice
Consider a P2P system
Markov chains describe random walk on P2P overlays
states corresponds to peertransition correspond to link overlays
the size of the matrix is huge, matrix algebra may be notexploited
The same for social graph analysis
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
A Random Walk on a Clock
a simplified clock with 6 number:0,1,2,3,4,5
from each state we can moveclockwise, counter clockwise, orstay in place with the sameprobability
the transition matrix is
P (i, j) =
1/3 if j=i-1 mod 61/3 if j=i1/3 if j=i+1 mod 6
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
A Random Walk on a Clock
suppose we start out atX0 = 2, that isπ0 = (0, 0, 1, 0, 0, 0)
π1 = (0, 1/3, 1/3, 1/3, 0, 0)
π2 = (1/9, 2/9, 1/3, 2/9, 1/9, 0)
π3 =(3/27, 6/27, 7/27, 6/27, 3/27, 2/27)
notice that the probability isspreading out away its initialconcentration on the initial state2
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
A Random Walk on a Clock
now guess what is the state of therandom walk at time 10000
an intuitive answer: X10000 isuniformly distributed over the 6states
this can be proven formally
πn → (16 ,16 ,
16 ,
16 ,
16 ,
16)
after 10000 steps, the randomwalk has forgotten that it startedout in state 2
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chains: The Basic Theorem
Theorem (Basic Limit Theorem)
Any irreducible, aperiodic, Markov chain defined on a set ofstate S and with a stochastic transition matrix P has a uniquestationary distribution π with all its component positive.Furthermore, let Pn be the n-th power of P , thenlimn→+∞ P
ni,j = π(j), ∀i, j ∈ S
to understand the theorem, we need to define the words
irreducible
aperiodic
stationary distribution
Before we shall see some examples where the limit does not exist
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Random Walk with Reflecting Boundary(Periodic)
Periodic chain with period 2
P =
0 1 0 0 00.5 0 0.5 0 00 0.5 0 0.5 00 0 0.5 0 0.50 0 0 1 0
P 2n =
0.25 0 0 0.5 0.250 0.5 0 0.5 0
0.25 0 0.5 0 0.250 0.5 0 0.5 0
0.25 0 0.5 0 0.25
P 2n+1 =
0 0.5 0 0.5 0
0.25 0 0.5 0 0.250 0.5 0 0.5 0.25 0 0.5 0 0.250 0.5 0 0.5 0
This means at every state i , if Pn(i, i) > 0, then n is even.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Random Walk With Absorbing Boundary
P =
1 0 0 0 0
0.5 0 0.5 0 00 0.5 0 0.5 00 0 0.5 0 0.50 0 0 1 0
Pn =
1 0 0 0 0
0.75 0 0 0 0.250.5 0 0 0 0.50.25 0 0 0 0.75
0 0 0 0 0
rows are not identical, i.e, the chain does not forget the initialstate
states 1, 2, 3 are transient: pni,1= pni,2= pni,3 = 0 ∀i, i.e. afterpassing a large amount of time the chain will stop visiting states1, 2, 3.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Reducible Chains
P =
1/2 1/2 0 0 01/6 5/6 0 0 00 0 3/4 1/4 00 0 1/8 2/3 5/240 0 0 1/6 1/6
Pn =
0.25 0.75 0 0 00.25 0.75 0 0 0
0 0 0.182 0.364 0.4550 0 0.182 0.364 0.4550 0 0.182 .364 0.455
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Stationary Distribution
a distribution which satisfies
π = πP
every finite state Markov chain has at least one stationarydistribution
A trivial example with infinitely many stationarydistribution: when P is the identity matrix, in which caseall distributions are stationary
The stationary distribution is the one obtained bycomputing the power of the transition matrix
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain Irreducibility
State j is reachable from state i if probability to go from ito j in n ≥ 0 steps is greater then 0
A subset X of the state space S is closed if pij = 0, ∀i ∈ Sand j /∈ S
A closed set of states is irreducible if any state j ∈ S isreachable from any state i ∈ S
A Markov Chain is irreducible if the state space S isirreducible
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain Irreducibility
Irreducible Markov Chain
Reducible Markov Chain
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain Periodicity
Definition
Given a Markov chain X0,X1,. . ., the period pi of a state i isthe greatest common divisor (gcd)
pi = {gcd n such that Pn(i, i) > 0}
Definition
An irreducible Markov chain is aperiodic if its period is 1, andperiodic otherwise.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain Periodicity
If the period of state i is 6= 1, the chain returns to state iat regular times,
If the period of state i is 1, the chain returns to state i canoccur at irregular times
A simple sufficient (but not necessary) condition for anirreducible chain to be aperiodic is that there exist a state isuch that P (i, i) > 0
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Markov Chain Recurrence
Hitting Time: Tij = min{k > 0 : X0 = i,Xk = j}
Recurrence Time: is the first time that the chain returns tostate i
Let ρi the probability that the chain will come back tostate i, given it starts from state i
A stete is recurrent if ρ = 1, otherwise it is transient.
Basic Definitions Examples It’s All Just Matrix Theory? The Basic Theorem
Recurrent and Transient States