and that ${{\boldsymbol{w}}^{(i)}\rightarrow {\boldsymbol{w}}^*}$, i.e. ${\boldsymbol{H}}^{(i)}\rightarrow {\boldsymbol{H}}$ as $i\rightarrow \infty$. This also implies that ${\boldsymbol{L}}^{(i)}\rightarrow {\boldsymbol{L}}^*$, i.e. the value iteration converges to the optimal control law.

Our convergence analysis is structured as follows. First, we prove that the value iteration (i.e. iterating between and ) is equivalent to a matrix sequence on ${\boldsymbol{H}}^{(i)}$. Second, we prove that this sequence of matrices is upper bounded in the sense that ${\boldsymbol{0}}\preceq{\boldsymbol{H}}^{(i)}\preceq {\boldsymbol{Y}}$ while ${\boldsymbol{H}}^{(i)}$ is monotonically increasing, i.e. ${\boldsymbol{0}}\preceq{\boldsymbol{H}}^{(i)}\preceq{\boldsymbol{H}}^{(i+1)}$. Hence, the sequence converges. Finally, we show that the converged sequence fulfills the Bellman equation and the corresponding control law is optimal.

The following Lemma \[lemma:VI\_matrix\_iteration\] extends [@AlTamimi.2007 Lemma 1] to the tracking case and shows that our proposed value iteration is equivalent to a matrix sequence on ${\boldsymbol{H}}^{(i)}$.

\[lemma:VI\_matrix\_iteration\] Let ${\boldsymbol{H}}^{(0)} = {\boldsymbol{0}}$, ${\boldsymbol{R}}^\intercal = {\boldsymbol{R}} \succ {\boldsymbol{0}}$, ${\boldsymbol{Q}}^\intercal = {\boldsymbol{Q}} \succeq {\boldsymbol{0}}$ and $({\boldsymbol{A}}, {\boldsymbol{B}})$ controllable. The value iteration described by and is equivalent to the iteration $$\begin{aligned}
\label{eq:iteration_Hi1}
    {\boldsymbol{H}}^{(i+1)}={\boldsymbol{G}}+\gamma{\boldsymbol{M}}\!\left({\boldsymbol{L}}^{(i)}\right)^\intercal {\boldsymbol{H}}^{(i)}{\boldsymbol{M}}\!\left({\boldsymbol{L}}^{(i)}\right),
    \end{aligned}$$ where $$\begin{aligned}
    {\boldsymbol{G}}={\begin{bmatrix*}[r]{\boldsymbol{Q}}&{\boldsymbol{0}}&-{\boldsymbol{Q}}&{\boldsymbol{0}}\\ {\boldsymbol{0}}&{\boldsymbol{R}}&{\boldsymbol{0}}&{\boldsymbol{0}}\\ -{\boldsymbol{Q}}&{\boldsymbol{0}}&{\boldsymbol{Q}}&{\boldsymbol{0}}\\ {\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}
\end{bmatrix*}}
    \end{aligned}$$ and $$\begin{aligned}
\label{eq:M}
    {\boldsymbol{M}}\!\left({\boldsymbol{L}}^{(i)}\right)&={\begin{bmatrix*}[c]{\boldsymbol{A}}&{\boldsymbol{B}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&\cdots&{\boldsymbol{0}}\\
        {\boldsymbol{L}}_x^{(i)}{\boldsymbol{A}}&{\boldsymbol{L}}_x^{(i)}{\boldsymbol{B}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{L}}_1^{(i)}&\cdots&{\boldsymbol{L}}_{N-1}^{(i)}\\
        {\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{I}}_n&{\boldsymbol{0}}&\cdots&{\boldsymbol{0}}\\
        {\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{I}}_n&\cdots&{\boldsymbol{0}}\\
        \vdots&\vdots&\vdots&\vdots&\vdots&\ddots&\vdots\\
        {\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&\cdots&{\boldsymbol{I}}_n\\
        {\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&{\boldsymbol{0}}&\cdots&{\boldsymbol{0}}
\end{bmatrix*}}
    \end{aligned}$$ with ${\boldsymbol{L}}^{(i)}={\begin{bmatrix*}[r]{\boldsymbol{L}}^{(i)}_x&{\boldsymbol{L}}^{(i)}_1&\cdots&{\boldsymbol{L}}^{(i)}_N
\end{bmatrix*}}={\boldsymbol{L}}\!\left({\boldsymbol{H}}^{(i)}\right)$ depending on ${\boldsymbol{H}}^{(i)}$ as in .

See \[append:VI\_matrix\_iteration\].

The following technical Lemma \[lemma:hilfslemma\] is required later 