and ${\boldsymbol{B}}$ are unknown. Thus, our aim is to determine ${\boldsymbol{H}}$ (cf. ) by observations of states and rewards.

The optimal control ${\boldsymbol{u}}_k^*$, which is equivalent to for $\kappa=0$ and $K\rightarrow\infty$, can be expressed directly by means of ${\boldsymbol{H}}$ and is given by Corollary \[corollary:uoptH\]. Here, ensures that ${\boldsymbol{u}}_k^*$ in fact minimizes the Q-function $Q_{0}$.

\[corollary:uoptH\] With Lemma \[lemmauQisuJ\] and Theorem \[Theo:Q\], the optimal control at time step $k$ is given by $$\begin{aligned}
\label{eq:uoptH}
    {\boldsymbol{u}}_{k}^* = -{\boldsymbol{h}}_{uu}^{-1}{\begin{bmatrix*}[r]{\boldsymbol{h}}_{ux} & {\boldsymbol{h}}_{ur_1}  & \cdots & {\boldsymbol{h}}_{ur_{N}}
\end{bmatrix*}} {\begin{bmatrix*}[c]{\boldsymbol{x}}_{k} \\ {\boldsymbol{r}}_{k+1} \\ \vdots \\ {\boldsymbol{r}}_{k+N}
\end{bmatrix*}}.
    \end{aligned}$$

Hence, if the Q-function $Q_{0}$ is known by means of the matrix ${\boldsymbol{H}}$, the optimal control directly results from .

In contrast to usual controllers learned by Reinforcement Learning, our control law explicitly depends on the reference values ${\boldsymbol{r}}_{k+1},\dots,{\boldsymbol{r}}_{k+N}$ and is therefore able to generalize to arbitrary reference trajectories on the horizon $N$.

Parametrization of the Reference-dependent Q-function
-----------------------------------------------------

In order to learn the Q-function, we parametrize $Q_{0}$ and perform a value iteration on the resulting squared Bellman-like temporal difference (TD) error in order to estimate the Q-function parameters as well as the corresponding optimal control law. Let the estimated Q-function be parametrized by means of a 