sub-stochastic. Therefore, we can estimate the entropy of this distribution as: $$\label{eq:disc_ent}
\setlength{\abovedisplayskip}{3pt}
  {\mathbb{H}}(d_{\alpha, \gamma, \pi_\theta}) \approx -\frac{1}{T}\sum_{t=0}^T\log \tilde p(S_t).$$

#### Entropy of the *stationary* state distribution ${\mathbb{H}}(d_{1, \pi_{\theta}})$ : 

For the average reward case with infinite horizon MDPs, we can similarly compute the entropy of the *stationary* state distribution. The stationary distribution $d_{1, \pi_\theta}$ is a solution of the following fixed point equation satisfying $
    d_{1, \pi_\theta} = P_{\pi_\theta}^\intercal d_{1, \pi_\theta},
$ where $P_{\pi_\theta}$ is the transition probability matrix corresponding to policy $\pi_\theta$. In practice, this is the long term state distribution under policy $\pi_\theta$, which is denoted as $p(s)$. In infinite horizon problems, the stationary state distribution is indicative of the majority of the states visited under the policy. We expect the stationary state distribution to change slowly, as we adapt the policy parameters (especially for a stochastic policy). Hence, we assume that the states have mixed, as we learn the policy over several iterations. In practice, instead of adding a mixing time specifically, we can use different time-scales for learning the policy and estimating the stationary state distribution. The entropy of the stationary state distribution can therefore be computed as : $$\label{eq:stationary_ent}
\setlength{\abovedisplayskip}{3pt}
{\mathbb{H}}(d_{1, \pi_\theta}) \stackrel{(a)}{=} -\sum_{s 