${\boldsymbol{r}}_k, {\boldsymbol{r}}_{k+1}, \dots, {\boldsymbol{r}}_{k+N}$, in each state ${\boldsymbol{x}}_k$ at time step $k$, find the control ${\boldsymbol{u}}_k^*$ minimizing the reference-dependent Q-function $Q_{0}$, where $$\begin{aligned}
    Q_{0} = \lim_{K\rightarrow\infty}\tensor*[^K]{\!Q}{_{0}}
    \end{aligned}$$ and apply it to the system whose matrices ${\boldsymbol{A}}$ and ${\boldsymbol{B}}$ are unknown.

We will proceed in two steps. First, it is assumed that the system matrices ${\boldsymbol{A}}$ and ${\boldsymbol{B}}$ are known. Later on, this assumption will be dropped and an iterative solution based on a temporal difference error will be introduced. We will now propose the analytical solution of $\tensor*[^K]{\!Q}{_{0}}$ in Theorem \[Theo:Q\].

\[Theo:Q\] For $K\geq N$, the Q-function $\tensor*[^K]{\!Q}{_{0}}$ (cf. Definition \[def:Q\]) with the objective function is given by $$\begin{aligned}
    \tensor*[^K]{\!Q}{_{0}} = \frac{1}{2}{\begin{bmatrix*}[r]{\boldsymbol{x}}_k^\intercal & {\boldsymbol{u}}_k^\intercal & {\boldsymbol{r}}_k^\intercal & \cdots & {\boldsymbol{r}}_{k+N}^\intercal & {\boldsymbol{0}}_{\vphantom{k+N}}^\intercal
\end{bmatrix*}}{\boldsymbol{H}}_K{\begin{bmatrix*}[c]{\boldsymbol{x}}_k\\{\boldsymbol{u}}_k\\{\boldsymbol{r}}_k\\ \vdots \\ {\boldsymbol{r}}_{k+N}\\ {\boldsymbol{0}}
\end{bmatrix*}},\label{eq:Q0_mit_H}
    \end{aligned}$$ where ${\boldsymbol{H}}_K={\boldsymbol{H}}_K^\intercal\in\mathbb{R}^{\left((K+2)n+m\right)\times\left((K+2)n+m\right)}$ with $$\begin{aligned}
    {\boldsymbol{H}}_K = {\begin{bmatrix*}[l]{\boldsymbol{h}}_{xx}      & {\boldsymbol{h}}_{xu}        & {\boldsymbol{h}}_{xr_0}      & {\boldsymbol{h}}_{xr_1}      & {\boldsymbol{h}}_{xr_2}      & \cdots & {\boldsymbol{h}}_{xr_{K}}   \\ \label{eq:FormDerQ-Fkt_2}
        {\boldsymbol{h}}_{ux}          & {\boldsymbol{h}}_{uu}        & {\boldsymbol{0}}                 & {\boldsymbol{h}}_{ur_1}      & {\boldsymbol{h}}_{ur_2}      & \cdots & {\boldsymbol{h}}_{ur_{K}}   \\
        {\boldsymbol{h}}_{r_0x}        & {\boldsymbol{0}}                 & {\boldsymbol{h}}_{r_0r_0}    & {\boldsymbol{0}}                 & {\boldsymbol{0}}                 & \cdots & {\boldsymbol{0}}                \\
        {\boldsymbol{h}}_{r_1x}        & {\boldsymbol{h}}_{r_1u}      & {\boldsymbol{0}}                 & {\boldsymbol{h}}_{r_1r_1}    & {\boldsymbol{0}}                 & \cdots & {\boldsymbol{0}}                \\
        {\boldsymbol{h}}_{r_2x}        & {\boldsymbol{h}}_{r_2u}      & {\boldsymbol{0}}                 & {\boldsymbol{0}}                 & 