{\boldsymbol{h}}_{r_2r_2}    & \cdots & {\boldsymbol{h}}_{r_{2}r_{K}} \\ 
        {\boldsymbol{h}}_{r_3x}        & {\boldsymbol{h}}_{r_3u}      & {\boldsymbol{0}}                 & {\boldsymbol{0}}                 & {\boldsymbol{h}}_{r_3r_2}    & \cdots & {\boldsymbol{h}}_{r_3r_{K}}     \\
        \,\vdots                & \,\vdots          & \,\vdots          & \,\vdots          & \,\vdots          & \ddots & \,\vdots             \\ 
        {\boldsymbol{h}}_{r_{K}x}  & {\boldsymbol{h}}_{r_{K}u}    &{\boldsymbol{0}}                  & {\boldsymbol{0}}                 & {\boldsymbol{h}}_{r_{K}r_2}& \cdots & {\boldsymbol{h}}_{r_{K}r_{K}} 
\end{bmatrix*}}.
    \end{aligned}$$ The exact values of ${\boldsymbol{H}}_K$ follow from the subsequent proof.

The proof is of rather technical nature and given in \[append:notations\].

For $K\rightarrow \infty$ let ${\boldsymbol{H}}$ be the northwestern $\left((N+2)n+m\right)\times\left((N+2)n+m\right)$-submatrix of ${\boldsymbol{H}}_K$. Then, $$\begin{aligned}
Q_{0} &= \lim_{K\rightarrow\infty}\tensor*[^K]{\!Q}{_{0}}=\frac{1}{2}{\boldsymbol{z}}_k^\intercal{\boldsymbol{H}}{\boldsymbol{z}}_k\label{eq:H_N_steps}\end{aligned}$$ follows, where ${\boldsymbol{z}}_k={\begin{bmatrix*}[r]{\boldsymbol{x}}_k^\intercal & {\boldsymbol{u}}_k^\intercal & {\boldsymbol{r}}_k^\intercal & \cdots & {\boldsymbol{r}}_{k+N}^\intercal
\end{bmatrix*}}^\intercal$, as ${{\boldsymbol{r}}_i = {\boldsymbol{0}}}$, $\forall i>k+N$.

Thus, the Q-function for the LQ tracking problem is quadratic w.r.t. the state ${\boldsymbol{x}}_k$, the control input ${\boldsymbol{u}}_k$ and the reference ${\boldsymbol{r}}_k, \dots, {\boldsymbol{r}}_{k+N}$ and furthermore completely parametrized by ${\boldsymbol{H}}$. This obviously is a generalization of the well known structure for reference free RL-procedures [@Bradtke.1994; @Landelius.1997]. In addition, ${\boldsymbol{H}}$ is not only quadratic but has a specific structure, which is a new result and allows an efficient parametrization for the Q-learning based algorithm in the following.

Q-Learning Based Tracking {#section_learning}
=========================

We use the new Q-function for reference tracking on a moving horizon where the system matrices ${\boldsymbol{A}}$ 