arbitrary reference that is given on a moving horizon of length $N$ without knowledge of the system matrices ${\boldsymbol{A}}$ and ${\boldsymbol{B}}$. If the system dynamics is not expected to change over time, learning might be stopped after the first complete value iteration based on $M$ data tuples has been performed. This can be done by stopping Algorithm \[algorithm\] after line 11. In practice, learning might be enabled again whenever the Bellman error increases which is an indicator for suboptimal weights ${\boldsymbol{w}}$.

apply $\tilde{{\boldsymbol{u}}}_k^*$ to the system $i=0, {\boldsymbol{w}}^{(i)}={\boldsymbol{w}}$ policy evaluation: ${\boldsymbol{w}}^{(i+1)}$ policy improvement: ${\boldsymbol{L}}^{(i+1)}$ $i = i+1$ ${\boldsymbol{w}}={\boldsymbol{w}}^{(i)}$

The iterative procedure of policy evaluation and policy improvement in Algorithm \[algorithm\] is a value iteration (cf. [@Sutton.2018]). This is due to the definition of ${\epsilon}_k$ in , where $\hat{Q}_{1}$ relies on ${\boldsymbol{w}}^{(i)}$ and $\hat{Q}_{0}$ on ${\boldsymbol{w}}^{(i+1)}$ (cf. [@Lewis.2009]).

Just as regular Q-learning [@Watkins.1992], our algorithm belongs to the off-policy RL methods [@Sutton.2018] as the behavior policy $\tilde{{\boldsymbol{u}}}_k^*=\hat{{\boldsymbol{u}}}_k^*+{\boldsymbol{\xi}}$ is followed while the agent learns the Q-function belonging to the target policy $\hat{{\boldsymbol{u}}}_k^*$.

Convergence Analysis of the Learning Algorithm {#convergence}
----------------------------------------------

In this section, we will provide convergence proofs, i.e. show that the estimated reference-dependent Q-function $\hat{Q}_{0}$ converges to the underlying Q-function $Q_{0}$ 