excitation condition $$\begin{aligned}
\label{rankcondition}
\operatorname{rank}\left({{\boldsymbol{\Phi}}^\intercal}{\boldsymbol{\Phi}}\right) = L\end{aligned}$$ is satisfied, ${\boldsymbol{w}}^{(i+1)}$ minimizing exists, is unique and given by $$\begin{aligned}
\label{eq:LS}
{\boldsymbol{w}}^{(i+1)}&=\left({{\boldsymbol{\Phi}}^\intercal}{\boldsymbol{\Phi}}\right)^{-1}{{\boldsymbol{\Phi}}^\intercal}{\boldsymbol{c}}\end{aligned}$$ (cf. [@Astrom.1995 Theorem 2.1]).

Then, the *policy improvement* step is based on the new weight estimate ${\boldsymbol{w}}^{(i+1)}$ and its corresponding ${\boldsymbol{H}}^{(i+1)}$ and results in $$\begin{aligned}
\label{eq:policy_improvement}
{\boldsymbol{L}}^{(i+1)}=-{{\boldsymbol{h}}_{uu}^{(i+1)}}^{-1}{\begin{bmatrix*}[r]{\boldsymbol{h}}_{ux}^{(i+1)} & {\boldsymbol{h}}_{ur_1}^{(i+1)}  & \cdots & {\boldsymbol{h}}_{ur_{N}^{(i+1)}}
\end{bmatrix*}}\end{aligned}$$ (cf. Corollary \[corollary:uoptH\]).

With the time step still being fixed at $k$, this iteration is performed until the change in ${\boldsymbol{w}}$ stays below a given threshold $e_{{\boldsymbol{w}}}$. Although we would like to evaluate the Q-function corresponding to the target policy $\hat{{\boldsymbol{u}}}_k^*$ (note that ${\boldsymbol{z}}_{k+1}^{*(i)}$ is used in ), we have not yet discussed how to choose the behavior policy, i.e. the control ${\boldsymbol{u}}_k$ that is applied to the system and appears in $c_k$ and ${\boldsymbol{z}}_k$ (cf. ). During the learning process is active, let $$\begin{aligned}
\label{utilde}
{\boldsymbol{u}}_k=\tilde{{\boldsymbol{u}}}_k^*=\hat{{\boldsymbol{u}}}_k^*+{\boldsymbol{\xi}}.\end{aligned}$$ Here, the Gaussian noise ${\boldsymbol{\xi}} \sim \mathcal{N}_{m}\left({\boldsymbol{0}},{\boldsymbol{\Sigma}}\right)$ serves as exploration noise as persistent excitation is required for convergence (cf. , [@Narendra.1987; @Astrom.1995; @Lewis.2009]). Furthermore, if the reference to track is smooth, additional excitation noise should be added to the reference in order to satisfy condition . The complete algorithm is shown in Algorithm \[algorithm\] and learns to track an 