Consider the Markov chain below:
(a) Suppose the chain is started in state i and goes through n transitions; let vi(n) be the expected number of transitions (out of the total of n) until the chain enters the trapping state, state 1. Find an expression for v(n) = (v1(n),v2(n), v3(n))T in terms of v(n - 1) (take v1(n) = 0 for all n). Hint: View the system as a Markov reward system; what is the value of r?
(b) Find the numerical value of limn→∞ v(n). Interpret the meaning of the elements vi in the solution of (4.32).
(c) Give a direct argument why (4.32) provides the solution directly to the expected time from each state to enter the trapping state.
Text Book: Stochastic Processes: Theory for Applications By Robert G. Gallager.