policy gradient objective, where for the *discounted* case, with stochastic policies [@sutton], we have $$\setlength{\abovedisplayskip}{3pt}
\label{eq:obj_discounted}
\tilde{J}(\theta) = \expectation_{\pi_{\theta}} \Bigl[\sum_{t=0}^{\infty} \gamma^{t} r(S_t, A_t) - \lambda \log  d_{\alpha, \gamma, \pi_{\theta}}(s_t) \Bigm | S_0 \Bigr]$$ The objective in the *stationary* case can be obtained by substituting the $d_{\alpha, \gamma, \pi_{\theta}}(s_t)$ with $d_{1, \pi_{\theta}}(s_t)$ in . The neural density estimator is independently parametrized by $\phi$, and is a function that maps the policy parameters $\theta$ to a state distribution. The loss function for this density estimator is the KL divergence between $ \texttt{KL}(q_{\phi}(Z \mid \theta) || p(Z|\theta))$. The training objective for our density estimator in the *stationary* case is given by : $$\setlength{\abovedisplayskip}{3pt}
\label{eq:vae_lower_bound}
    \mathcal{L_{\gamma}}(\phi, \theta) = (1 - \gamma) \gamma^{k} \expectation_{q_{\phi}(Z| \theta)} \big[\log p_{\phi}(S|\theta)  \big] - KL\big(q_{\phi}(Z|\theta) || p(\theta)\big)$$

Equation  gives the expression for the loss function for training the state density estimator (which is the variational inference lower bound loss for estimating $log(p(s))$,i.e.,ELBO [@DBLP:journals/corr/KingmaW13]. Here $\theta$ are the parameters of the policy network $\pi_{\theta}$, $\phi$ are the parameters of the density estimator. The novelty of our approach is that the density estimator takes as input the parameters of the policy network directly (similar to hypernetworks [@krueger2017bayesian; @ha2016hypernetworks]). The encoder then maps 