Hello all, I am studying Markov chains in my math class, and I have some extra study problems that my professor provided but no solutions given. I would appreciate it if anyone can check over my work and kindly provide guidance if I'm understanding the problem correctly. I am having some trouble with the last 2 parts trying to solve the systems of equations for the long-term pmf, because all I get is 0. Any help with checking my work is greatly appreciated!
Problem:
Approach:
(a) The transient states are 1 and 2, since a path exists where if you leave, there is no way to get back to that state.
(b) The recurrent states are 0 and 3, once leaving these states, there is always a way back
(c) I believe states 1 and 2 are periodic with period 2, always requires an even number of paths to get back to original state
(d) The absorbing states are 0 and 3, since you cannot leave that state, i.e. the [imath] P_{ii} = 1 [/imath], where i = 0 or 3
(e) The probability is 0, since you cannot get back to state 1 in just 1 step.
(f) The probability is 0.5 from the graph, with just 1 step to get from state 1 to state 0.
(g)
I believe the the transition matrix [imath]P[/imath] is below.
[math]P=\begin{bmatrix} 1& 0 & 0 & 0\\ 0.5 & 0 & 0.5 & 0\\ 0 & 2/3 & 0 & 1/3\\ 0 & 0 & 0 & 1\\ \end{bmatrix}[/math]
The question is wanting the probability [math]P(X_k = j | x_0 = i) [/math] of going from state i to j in k steps
[math]P(x_6 = 3 | x_0 = 1) = (P^6)_{1,3}[/math] or the (1,3) element of [imath] P^6 [/imath]. Using a matrix power calculator, I find [math]P(x_6 = 3 | x_0 = 1) = 13/54 [/math]. Is there a simpler way to compute by hand, or just exploit powers of 2?
(h) I'm somewhat struggling to find the long-term pmf because when I solve [imath] \pi = \pi P [/imath], where [imath] \pi = [\pi_0 \pi_1 \pi_2 \pi_3 ] [/imath], I get 4 equations where I can trivially find that [imath] \pi_1 = \pi_2= 0 [/imath] . If I use a Markov calculator, I also find [imath] \pi_1 = 0 [/imath]
(i) Similarly using a calculator, I find the long-term pmf as [imath] \pi_0 = 0.75 [/imath]
Problem:
Approach:
(a) The transient states are 1 and 2, since a path exists where if you leave, there is no way to get back to that state.
(b) The recurrent states are 0 and 3, once leaving these states, there is always a way back
(c) I believe states 1 and 2 are periodic with period 2, always requires an even number of paths to get back to original state
(d) The absorbing states are 0 and 3, since you cannot leave that state, i.e. the [imath] P_{ii} = 1 [/imath], where i = 0 or 3
(e) The probability is 0, since you cannot get back to state 1 in just 1 step.
(f) The probability is 0.5 from the graph, with just 1 step to get from state 1 to state 0.
(g)
I believe the the transition matrix [imath]P[/imath] is below.
[math]P=\begin{bmatrix} 1& 0 & 0 & 0\\ 0.5 & 0 & 0.5 & 0\\ 0 & 2/3 & 0 & 1/3\\ 0 & 0 & 0 & 1\\ \end{bmatrix}[/math]
The question is wanting the probability [math]P(X_k = j | x_0 = i) [/math] of going from state i to j in k steps
[math]P(x_6 = 3 | x_0 = 1) = (P^6)_{1,3}[/math] or the (1,3) element of [imath] P^6 [/imath]. Using a matrix power calculator, I find [math]P(x_6 = 3 | x_0 = 1) = 13/54 [/math]. Is there a simpler way to compute by hand, or just exploit powers of 2?
(h) I'm somewhat struggling to find the long-term pmf because when I solve [imath] \pi = \pi P [/imath], where [imath] \pi = [\pi_0 \pi_1 \pi_2 \pi_3 ] [/imath], I get 4 equations where I can trivially find that [imath] \pi_1 = \pi_2= 0 [/imath] . If I use a Markov calculator, I also find [imath] \pi_1 = 0 [/imath]
(i) Similarly using a calculator, I find the long-term pmf as [imath] \pi_0 = 0.75 [/imath]
Last edited:
