weight vector ${\boldsymbol{w}}$ does not depend on a specific reference we intend to follow.

Online Learning Algorithm
-------------------------

In this section, we propose our value iteration based algorithm in order to learn the weights ${\boldsymbol{w}}$ online by minimizing the squared temporal difference error. Let $$\begin{aligned}
\label{eq:Qhat1}
\hat{Q}_{1}&={\boldsymbol{w}}^\intercal{\boldsymbol{\phi}}\left({\boldsymbol{x}}_{k+1}, {\boldsymbol{u}}_{k+1}, {\boldsymbol{r}}_{k+1}, \dots, {\boldsymbol{r}}_{k+N}, {\boldsymbol{0}}\right)\nonumber \\&={\boldsymbol{w}}^\intercal{\boldsymbol{\phi}}\left({\boldsymbol{z}}_{k+1}\right),\end{aligned}$$ where ${\boldsymbol{z}}_{k+1}={\begin{bmatrix*}[r]{\boldsymbol{x}}_{k+1}^\intercal & {\boldsymbol{u}}_{k+1}^\intercal & {\boldsymbol{r}}_{k+1}^\intercal & \cdots & {\boldsymbol{r}}_{k+N}^\intercal & {\boldsymbol{0}}_{\vphantom{k+N}}^\intercal
\end{bmatrix*}}^\intercal$.

The estimated optimal control input $\hat{{\boldsymbol{u}}}_{k+\kappa}^*$ based on the estimated Q-function $\left.\hat{Q}_{\kappa}\right|_{{\boldsymbol{u}}_{k+\kappa}}$ is defined as $$\begin{aligned}
\label{eq:opt_u_estimated}
\hat{{\boldsymbol{u}}}_{k+\kappa}^*&=\operatorname*{arg\,min}_{{\boldsymbol{u}}_{k+\kappa}}\left.\hat{Q}_{\kappa}\right|_{{\boldsymbol{u}}_{k+\kappa}}\nonumber \\ &={\boldsymbol{L}}{\begin{bmatrix*}[r]{\boldsymbol{x}}_{k+\kappa}^\intercal&{\boldsymbol{r}}_{k+\kappa+1}^\intercal & \dots & {\boldsymbol{r}}_{k+\kappa+N}^\intercal
\end{bmatrix*}}^\intercal,\end{aligned}$$ where ${\boldsymbol{L}}=-\hat{{\boldsymbol{h}}}_{uu}^{-1}{\begin{bmatrix*}[r]\hat{{\boldsymbol{h}}}_{ux} & \hat{{\boldsymbol{h}}}_{ur_1}  & \cdots & \hat{{\boldsymbol{h}}}_{ur_{N}}
\end{bmatrix*}}$ (cf. ). Here, let $\hat{{\boldsymbol{H}}}$ be the estimated matrix ${\boldsymbol{H}}$ based on ${\boldsymbol{w}}$ and $\hat{{\boldsymbol{h}}}$ denote the submatrices of $\hat{{\boldsymbol{H}}}$ as in . Furthermore, $\hat{{\boldsymbol{u}}}_{k+\kappa}^*={\boldsymbol{u}}_{k+\kappa}^*$ results if ${\boldsymbol{w}}={\boldsymbol{w}}^*$ according to Lemma \[lemma:Qexakt\] and the optimal control law ${\boldsymbol{L}}^*$ results if $\hat{{\boldsymbol{H}}}={\boldsymbol{H}}$.

Then, based on the Bellman-like equation , the TD error $\epsilon_k$ [@Sutton.1988] is given in Definition \[def:TD\]. If ${{\boldsymbol{w}}={\boldsymbol{w}}^*}$, $\epsilon_k$ would vanish.

\[def:TD\] The temporal difference error $\epsilon_k$, i.e. the approximation error of the Bellman-like equation due to the deviation of the weight estimate ${\boldsymbol{w}}$ from ${\boldsymbol{w}}^*$ is defined as $$\begin{aligned}
\label{eq:td}
    \epsilon_k&=c_k+\gamma\left.\hat{Q}_{1}\right|_{\hat{{\boldsymbol{u}}}_{k+1}^*}-\hat{Q}_{0}\nonumber \\ &=c_k+\gamma{\boldsymbol{w}}^\intercal \phi\left({\boldsymbol{z}}_{k+1}^*\right)-{\boldsymbol{w}}^\intercal \phi\left({\boldsymbol{z}}_{k}\right).
    \end{aligned}$$ Here, 