enough neural network, the gap between the optimal solution with no function approximation and the optimal solution to the parameterized dual problem can be made as small as desired. While this does not mean that the optimal parameterized dual problem can be solved, it does motivate the use of the primal dual method. A sample training curve is shown in Fig. \[fig:model\_learning\].

![**Model training.** The orange curve shows the value of the objective function, $l$. Teal curve shows the value of the constraint function, $g$. Dark blue curve shows the dual variable, $\lambda$. The objective and constraint functions are smoothed and shown in a darker color. The dotted red line shows the $y=0$ line and the constraint curve must go below that for the problem to have a feasible solution. The curves have different values and are normalized so that they can be displayed on one graph.[]{data-label="fig:model_learning"}](images/training_curve.png)

The primal and dual variables interact in a very natural manner. As the constraints are violated, the dual variable increases in magnitude which heavily penalizes constraint violations. In order to drive the constraints into the feasible region, the primal loss may suffer. However, in the end, when the constraints are satisfied, the dual 