the policy parameters $\theta$ into the latent space $Z$ given by $q_{\phi}(Z\mid \theta)$ with a Gaussian prior over the policy parameters $\theta$. During implementation we feed the parameters of the last two layers of the policy network, assuming the previous layers extract the relevant state features and the last two layers map the obtained features to a distribution over actions. Hence $\theta$ only comprises of the weights of these last two layers ensuring computation tractability. We take this approach since the discounted future state distribution is a function of the policy parameters $\theta$. Our overall gradient objective with the regularized update is therefore given by $\tilde{J}(\theta) = \expectation_{\pi_{\theta}} \Bigl[\sum_{t=0}^{\infty} \gamma^{t} r(S_t, A_t) - \lambda  \mathcal{L_{\gamma}}(\phi, \theta) \Bigr]$, where $\mathcal{L_{\gamma}}(\phi, \theta)$ directly depends on the policy parameters $\theta$. This gives the regularized policy gradient update with the entropy of the discounted future state distribution, for stochastic policies as :

$$\setlength{\abovedisplayskip}{1pt}
\label{eq:overall_gradient_update}
    \nabla_{\theta} \tilde{J}(\theta) = \expectation_{\pi_{\theta}} \Bigl[ \nabla_{\theta} \log \pi(A_t \mid S_t) Q^{\pi}(S_t, A_t) -\lambda \nabla_{\theta} \mathcal{L}_{\gamma}(\phi, \theta).  \Bigr],\text{where $\mathcal L_{\gamma}(\phi,\theta)=(1-\gamma)\gamma^t\mathcal L (\phi,\theta)$}$$

Experiments
===========

In this section, we demonstrate our approach based on entropy regularization with the normalized discounted weighting of states, also known as the discounted future state distribution. Our 