given ${\boldsymbol{x}}_k$ and ${\boldsymbol{r}}_k, {\boldsymbol{r}}_{k+1}, \dots, {\boldsymbol{r}}_{k+N}$.

Extended Q-Function for Reference Tracking {#qfunc_section}
==========================================

Our idea is to define a new reference-dependent Q-function in contrast to the commonly used Q-function (i.e. we define a state-action-*reference* function rather than a state-action function). This Q-function is constructed such that its minimizing control input constitutes a solution to Problem \[problem1\].

In this section, we introduce the reference-dependent Q-function and derive its analytical solution. This provides important insights in how to parametrize the Q-function in Section \[section\_learning\], which is the basic ingredient for a convergent reinforcement learning algorithm capable of learning the optimal tracking solution.

We begin with the general case of a *finite* optimization horizon $K$, later we let $K\rightarrow \infty$. The notation seems to be a little bit clumsy but is of high importance, since optimization horizon $K$ and moving horizon of the reference $N$ should not be confused. Furthermore, the notation which control input is plugged into the one-step cost and Q-functions will be useful and we need to distinguish between the actual time step $k$ which the system is in and the time step $\kappa$ on the optimization horizon $K$.

\[def:Q\] Our proposed reference-dependent Q-function is defined as $$\begin{aligned}
    \tensor*[^K]{\!Q}{_{\kappa}}&=c_{k+\kappa}+\gamma\sum_{i=k+\kappa+1}^{k+K}\gamma^{i-(k+\kappa+1)}\left.c_i\right|_{{\boldsymbol{u}}_i^*} 