we define an MDP as a tuple $(\mathcal{S}, \mathcal{A}, P, r)$, where[^1]: $\mathcal{S}$ is the state space, $P: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$ is the transition probability matrix/kernel of that outputs a distribution over states given a state-action pair, $r: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ is the reward function that maps state-action pairs to real numbers. In this work, we focus on policy based methods. We use parametrized policies $\pi_\theta$, where the parameters are $\theta \in \Theta$ and $\Theta$ is a compact, convex set. The performance of any such policy is given by $    J(\theta) = \expectation_{A_t \sim \pi_{\theta}, S_{t+1} \sim P(S_t, A_t)} \Bigl[\sum_{t=0}^{\infty} \gamma^{t} r(S_t, A_t)  \Bigm | S_0 \Bigr]$, where $\gamma \in (0,1)$ is the discount factor and $S_0$ is the initial state with a distribution $P_{0}$. The objective of the agent is to find a parameter $\theta^*$ that maximizes this performance i.e., $\theta^* \in \arg\max J(\theta)$. The usual approach to find such a $\theta^*$ is to use a stochastic gradient ascent based iteration: $\theta_{k+1} = \theta_k + \alpha_k \grad_\theta{J(\theta_k)}$, where $\grad_\theta{J(\theta)}$ can be obtained using the policy gradient theorem [@sutton; @dpg] or one of its variants. It has been shown in literature that this iteration converges almost 