Consider an irreducible Markov chain that is positive recurrent. Recall the technique used to find the expected first-passage time from j to i, i.e., Tji, in Section 4.5. The state i was turned into a trapping state by turning all the transitions out of i into a single transition Pii = 1. Here, in order to preserve the positive recurrence, we instead move all transitions out of state i into the single transition Pij = 1.
(a) Use Figure 6.2 to illustrate that the above strategy can turn an irreducible chain into a reducible chain. Also explain why states i and j are still positive recurrent and still in the same class.
(b) Let {π ∗ ; k ≥ 0} be the steady-state probabilities for the positive-recurrent class in the modified Markov chain. Show that the expected first passage time T∗ from j to i in the modified Markov chain is (1/π ∗) - 1.
(c) Show that the expected first passage time from j to i is the same in the modified and unmodified chains.
(d) Show by example that after the modification above, two states i and j that were not positive recurrent before the modification can become positive recurrent and the above technique can again be used to find the expected first-passage time.
Text Book: Stochastic Processes: Theory for Applications By Robert G. Gallager.