${\boldsymbol{z}}_{k+1}^* = \left.{\boldsymbol{z}}_{k+1}\right|_{{\boldsymbol{u}}_{k+1}=\hat{{\boldsymbol{u}}}_{k+1}^*}$.

In order to improve the estimated reference-dependent Q-function $\hat{Q}_{0}$ as well as the resulting estimated optimal control $\hat{{\boldsymbol{u}}}_k^*$, we employ a value iteration procedure. This iteration consists of a *policy evaluation* which updates the weight estimate ${\boldsymbol{w}}^{(i)}$ representing the Q-function and a *policy improvement* step, where, based on the updated Q-function weight ${\boldsymbol{w}}^{(i)}$ and the corresponding matrix ${\boldsymbol{H}}^{(i)}$, the control law ${\boldsymbol{L}}^{(i)}$ is adapted according to .

To evaluate $\epsilon_k$ in , $\hat{{\boldsymbol{u}}}_{k+1}^*$ is required. However, as the optimal weight ${\boldsymbol{w}}^*$ is unknown a priori, we initialize ${\boldsymbol{w}}^{(0)}={\boldsymbol{0}}$ and the estimated optimal control by $\hat{{\boldsymbol{u}}}_{k+1}^{*(0)} = {\boldsymbol{0}}$ which is achieved by setting ${\boldsymbol{L}}^{(0)}={\boldsymbol{0}}$.

In the *policy evaluation* step, the aim is to find an updated ${\boldsymbol{w}}^{(i+1)}$ such that $\left({\epsilon}_k^{(i)}\right)^2$ is minimized, where $$\begin{aligned}
\label{eq:td_VI}
{\epsilon}_k^{(i)} &= c_k + \gamma{{\boldsymbol{w}}^{(i)}}^\intercal{\boldsymbol{\phi}}\left({\boldsymbol{z}}_{k+1}^{*{(i)}}\right)-{{\boldsymbol{w}}^{(i+1)}}^\intercal{\boldsymbol{\phi}}\left({\boldsymbol{z}}_{k}\right)\end{aligned}$$ in analogy to . In accordance with Lemma \[lemma:Qexakt\], ${\boldsymbol{w}}\in\mathbb{R}^L$ follows. Thus, ${\epsilon}_k^{(i)}$ needs to be considered at $M\geq L$ time steps in order to perform a least-squares update, where $M$ is the number of samples used for the policy evaluation. Then, ${\boldsymbol{w}}^{(i+1)}$ results from $$\begin{aligned}
\label{eq:PE}
{\boldsymbol{w}}^{(i+1)}=\operatorname*{arg\,min}_{{\boldsymbol{w}}^{(i+1)}}\sum_{j=k-M+1}^{k}\left({\epsilon}_j^{(i)}\right)^2.\end{aligned}$$ Now, we define $$\begin{aligned}
\label{eq:Phimatrix}
{\boldsymbol{\Phi}}&={\begin{bmatrix*}[c]{\boldsymbol{\phi}}\left({\boldsymbol{z}}_{k-M+1}\right)& \dots & {\boldsymbol{\phi}}\left({\boldsymbol{z}}_{k}\right)
\end{bmatrix*}}^\intercal\end{aligned}$$ and $$\begin{aligned}
\label{eq:cmatrix}
{\boldsymbol{c}}={\begin{bmatrix*}[c]c_{k-M+1}+\gamma{{\boldsymbol{w}}^{(i)}}^\intercal\phi\left({\boldsymbol{z}}_{k-M+2}^{*(i)}\right)\\ \vdots \\ c_k+\gamma{{\boldsymbol{w}}^{(i)}}^\intercal\phi\left({\boldsymbol{z}}_{k+1}^{*(i)}\right)
\end{bmatrix*}}.\end{aligned}$$ If the 