the reference ${\boldsymbol{r}}_k$ follows ${\boldsymbol{r}}_{k+1}={\boldsymbol{f}}_{\text{ref}}({\boldsymbol{r}}_k)$ (respectively $\dot{{\boldsymbol{r}}}(t)={\boldsymbol{f}}_{\text{ref}}({\boldsymbol{r}}(t))$ in the continuous-time case).

However, for various applications, this assumption does not hold (e.g. vehicles that follow a road, robots in sophisticated human-machine collaboration or specific time-varying sequences in process engineering). Consequently, present exo-system-based methods are not suited for these types of applications. This is because the learned parameters (i.e. Q-function or value function approximations and associated control laws) are corresponding to the system dynamics and reference dynamics ${\boldsymbol{f}}_{\text{ref}}$ and thus need to be re-learned as soon as any other reference $\tilde{{\boldsymbol{f}}}_{\text{ref}}\neq{\boldsymbol{f}}_{\text{ref}}$ should be followed. A method that tries to cope with this challenge is the multiple-model approach in [@Kiumarsi.2015], where an adaptive self-organizing map detects changes and switches between various learned models. However, in this approach new sub-models need to be trained whenever the reference dynamics ${\boldsymbol{f}}_{\text{ref}}$ changes.

An RL method that tracks reference trajectories not necessarily resulting from a time-invariant exo-system ${\boldsymbol{f}}_{\text{ref}}$ (or multiple switched models [@Kiumarsi.2015]) has not yet been proposed. Therefore, in contrast to [@Dierks.2009; @Kiumarsi.2014; @Kiumarsi.2015b; @Kopf.2018; @Luo.2016; @Lv.2017; @Modares.2014; @Modares.2016; @Mu.2017; @Mu.2017b; @Zhang.2017; @Bernhard.2018; @Kiumarsi.2015], our work provides for the first time an RL method allowing to track an arbitrary reference trajectory on a moving horizon.

Our 