flexibility is not a feature of many end-to-end systems. While there exists some recent work that attempts to differentiate through the optimization procedure itself [@amos2018differentiable], it can be hard to train.

Model Refinement
----------------

After a model is learned, it can be improved such that it has higher accuracy on states that the controller may visit more. The simple method of using a static dataset collected from such a controller does not work due to the fact that once a new model is trained, the controller will choose different actions as it is dependent on the model itself. Thus, the distribution of states the model was trained to minimize is no longer the same distribution that the controller will induce. This is a problem that also plagues Imitation Learning. One solution is to use the DAgger algorithm [@ross2011reduction]. It involves an iterative process where a model is trained on a distribution of states induced by the controller. Then, the controller based on the new model is used to collect more data. The procedure is described in Algorithm \[alg:dagger\].

\[alg:dagger\] $\theta \leftarrow \theta_0$ $D \leftarrow \{\}$ $\theta$

Experiments {#sec:experiments}
===========

To demonstrate the effectiveness of our approach, we consider a Ball-Paddle 