the widely used primal-dual method, where the Lagrangian is iteratively minimized with respect to the primal variable and maximized with respect to the dual. This minimization/maximization can be solved by computing gradient descent steps with respect to $\theta$ and gradient ascent steps with respect to $\lambda$. The gradient of the Lagrangian with respect to $\theta$ takes the form $$\begin{aligned}
    \nabla_\theta L(\theta, \lambda) &= \mathbb{E} [\nabla_\theta \mathcal{L}( {s}, {a}, \phi_\theta)] +\sum_{i=1}^P \lambda_i \mathbb{E} [\nabla_\theta g_i( {s}, {a}, \phi_\theta)], \end{aligned}$$ and the gradient of the Lagrangian with respect to the multiplier $\lambda$ yields $$\nabla_\lambda L(\theta, \lambda) = \mathbb{E} [g( {s}, {a}, \phi_\theta)].$$ Each iteration must be followed by a projection of $\lambda$ onto the positive orthant to make sure it is non negative. Given a static distribution of states and actions to optimize this model, the algorithm is summarized in Algorithm \[alg:model\_learning\].

\[alg:model\_learning\] $\lambda \leftarrow \lambda_0$ , set dual variable to an initial value $\theta \leftarrow \theta_0$ $\theta$

Note that gradient ascent and descent steps can be modified to include momentum or include more complicated algorithms such as ADAM [@kingma2014adam]. In the next section we show that the sufficiently accurate learning formulated in has almost zero duality gap which motivates the use 