shown. For the regulation case, boundedness was shown in [@Landelius.1997 Lemma B.1.2]. In contrast to that, we consider the specific structure of the iteration in the tracking case (cf. ) and prove boundedness of ${\boldsymbol{H}}^{(i)}$ for the more generalized tracking formulation. For reasons of self-consistency, the complete proof is given.

\[lemma:upper\_bounded\] Let $\left({\boldsymbol{A}}, {\boldsymbol{B}}\right)$ be controllable and ${\boldsymbol{H}}^{(i)}$ be the sequence with ${\boldsymbol{H}}^{(0)}={\boldsymbol{0}}$. Then, there exists ${\boldsymbol{Y}}$ such that ${\boldsymbol{0}}\preceq{\boldsymbol{H}}^{(i)}\preceq{\boldsymbol{Y}}$.

See \[append:upper\_bounded\].

We will now get to the main result of our convergence analysis and show that the proposed value iteration converges to the optimal weight vector ${\boldsymbol{w}}^*$ and the optimal control law ${\boldsymbol{L}}^*$ in the tracking case described by Problem \[problem1\].

Let ${\boldsymbol{R}}^\intercal = {\boldsymbol{R}} \succ {\boldsymbol{0}}$, ${\boldsymbol{Q}}^\intercal = {\boldsymbol{Q}} \succeq {\boldsymbol{0}}$, $({\boldsymbol{A}}, {\boldsymbol{B}})$ controllable and ${\boldsymbol{w}}^{(0)}={\boldsymbol{0}}$, i.e. ${\boldsymbol{H}}^{(0)} = {\boldsymbol{0}}$. Then, iterating between and yields ${\boldsymbol{H}}^{(i)}\rightarrow{\boldsymbol{H}}$, i.e. ${\boldsymbol{w}}^{(i)}\rightarrow{\boldsymbol{w}}^*$ as well as ${\boldsymbol{L}}^{(i)}\rightarrow{\boldsymbol{L}}^*$.

According to Lemma \[lemma:VI\_matrix\_iteration\], the value iteration is equivalent to iterating on ${\boldsymbol{H}}^{(i)}$ (cf. ). With ${\boldsymbol{Z}}^{(0)}={\boldsymbol{H}}^{(0)}$ and ${\boldsymbol{Z}}^{(i+1)}=F\left({\boldsymbol{Z}}^{(i)}, {\boldsymbol{L}}\left({\boldsymbol{H}}^{(i+1)}\right)\right)$, Lemma \[lemma:monotonically\_increasing\] yields ${\boldsymbol{0}}\preceq{\boldsymbol{H}}^{(i)}\preceq{\boldsymbol{Z}}^{(i)}$. With ${\boldsymbol{H}}^{(0)}={\boldsymbol{0}}$ follows ${\boldsymbol{H}}^{(1)}={\boldsymbol{G}}\succeq{\boldsymbol{0}}$ and hence ${\boldsymbol{H}}^{(1)}-{\boldsymbol{Z}}^{(0)}\succeq{\boldsymbol{0}}$. The proof is drawn by induction: Assume the induction hypothesis ${\boldsymbol{H}}^{(i)}-{\boldsymbol{Z}}^{(i-1)}\succeq{\boldsymbol{0}}$. Then, $$\begin{aligned}
    &{\boldsymbol{H}}^{(i+1)}-{\boldsymbol{Z}}^{(i)}\nonumber \\ &=\gamma {\boldsymbol{M}}\left({\boldsymbol{L}}\left({\boldsymbol{H}}^{(i)}\right)\right)^\intercal\left({\boldsymbol{H}}^{(i)}-{\boldsymbol{Z}}^{(i-1)}\right){\boldsymbol{M}}\left({\boldsymbol{L}}\left({\boldsymbol{H}}^{(i)}\right)\right)\nonumber \\&\succeq{\boldsymbol{0}}.
    \end{aligned}$$ This implies ${\boldsymbol{0}}\preceq{\boldsymbol{H}}^{(i)}\preceq{\boldsymbol{Z}}^{(i)}\preceq{\boldsymbol{H}}^{(i+1)}$.

As the matrix 