<s>[INST] Steady state probabilities of divergent Markov matrices I have a problem for which the Markov matrix turns out to be the following:
$$P = \begin{pmatrix}
                    0 & 0.5 & 0 & 0.5\\
                    0.5 & 0 & 0.5 & 0 \\
                    0 & 0.5 & 0 & 0.5\\
                    0.5 & 0 & 0.5 & 0\\
                  \end{pmatrix}$$
This matrix has eigenvalues $-1$ and $1$. Hence, it does not converge to some matrix $A$ when raised to some power $n$. However, there exists a solution to $Pv = v$. This is basically, the eigenvector corresponding to the eigenvalue of $1$. Is $v$ the steady state probabilites?
 [/INST] This Markov chain is periodic with period $2$, meaning that for any initial state $i$, the limiting probability $\lim_{n\to\infty}\mathbb P(X_n=j\mid X_0=i)$ does not exist. However, $\{X_n\}$ is irreducible, positive recurrent, and has finitely many states, so a unique stationary distribution $\pi$ exists. Since $P$ is doubly stochastic (both the rows and the columns sum to one), it follows that $\pi$ is the uniform distribution over $0,1,2,3$, i.e. $\pi_0=\pi_1=\pi_2=\pi_3=\frac14$. 
Now, because $\{X_n\}$ is periodic with period $2$, the limits of $P^{2n}$ and $P^{2n+1}$ as $n\to\infty$ exist. In particular, 
$$
P^{2n} = \left(
\begin{array}{cccc}
 \frac{1}{2} & 0 & \frac{1}{2} & 0 \\
 0 & \frac{1}{2} & 0 & \frac{1}{2} \\
 \frac{1}{2} & 0 & \frac{1}{2} & 0 \\
 0 & \frac{1}{2} & 0 & \frac{1}{2} \\
\end{array}
\right),\quad P^{2n+1} = P
$$
for all $n$, so 
$$
\lim_{n\to\infty} P^{2n} = \left(
\begin{array}{cccc}
 \frac{1}{2} & 0 & \frac{1}{2} & 0 \\
 0 & \frac{1}{2} & 0 & \frac{1}{2} \\
 \frac{1}{2} & 0 & \frac{1}{2} & 0 \\
 0 & \frac{1}{2} & 0 & \frac{1}{2} \\
\end{array}
\right),\quad \lim_{n\to\infty}P^{2n+1} = P.
$$
</s>