$\displaystyle=\underbrace{\begin{bmatrix}{\mathbf{0}}&{\mathbf{0}}&{\mathbf{0}}&{\mathbf{0}}&{\mathbf{0}}\\\
\bm{D}_{2,1}&{\mathbf{0}}&{\mathbf{0}}&{\mathbf{0}}&{\mathbf{0}}\\\
\bm{D}_{3,1}&\bm{D}_{3,2}&{\mathbf{0}}&{\mathbf{0}}&{\mathbf{0}}\\\
\vdots&\ddots&\ddots&\ddots&\\\
\bm{D}_{T,1}&\cdots&\bm{D}_{T,T-2}&\bm{D}_{T,T-1}&{\mathbf{0}}\\\
\end{bmatrix}}_{{\mathcal{D}}}\underbrace{\begin{bmatrix}\bm{a}^{1}\\\
\bm{a}^{2}\\\ \bm{a}^{3}\\\ \vdots\\\ \bm{a}^{T}\\\ \end{bmatrix}}_{\bm{a}}.$

In the above equation, the matrix ${\mathcal{M}}$ is the same as the one
defined in Lemma C.1, i.e.,
${\mathcal{M}}=\bm{I}_{pT}+{\mathcal{D}}(\bm{I}_{T}\otimes\tfrac{\bm{X}^{\top}\bm{X}}{n})-{\mathcal{J}}$
for ${\mathcal{D}},{\mathcal{J}}$ in (2.10). Solving the linear system (C.11)
for $\dot{\bm{b}}^{t}$, we obtain

$\displaystyle\dot{\bm{b}}^{t}=\operatorname{\mathbf{vec}}(\dot{\bm{B}}\bm{e}_{t})=$
$\displaystyle(\bm{e}_{t}^{\top}\otimes\bm{I}_{p})\operatorname{\mathbf{vec}}(\dot{\bm{B}})$
by (B.5) $\displaystyle=$
$\displaystyle(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}\bm{a}$
$\displaystyle\mbox{by \eqref{eq:linear-system}}.$

By definition of $\bm{a}\in{\mathbb{R}}^{pT}$ and $\bm{a}^{t}$ in (C.10), this
completes the proof of (C.2) by taking
$(\dot{\bm{\varepsilon}},\dot{\bm{X}})=({\mathbf{0}}_{n},\bm{e}_{i}\bm{e}_{j}^{\top})$
and of (C.3) by taking
$(\dot{\bm{\varepsilon}},\dot{\bm{X}})=(\bm{e}_{l},{\mathbf{0}}_{n\times p})$.
∎

##  Appendix D Change of variables

For the linear model $\bm{y}=\bm{X}\bm{b}^{*}+{\bm{\varepsilon}}$, its design
matrix $\bm{X}$ may not be isotropic. However, we can always consider the
following change of variables:

(D.1)
$\displaystyle\bm{X}\rightsquigarrow\bm{X}\bm{\Sigma}^{-1/2}:=\bm{G},\quad\bm{b}^{*}\rightsquigarrow\bm{\Sigma}^{1/2}\bm{b}^{*}:=\bm{\theta}^{*}.$

This way, the original linear model can be rewritten as

(D.2) $\displaystyle\bm{y}=\bm{G}\bm{\theta}^{*}+{\bm{\varepsilon}},$

where $\bm{G}$ is the design matrix with i.i.d. entries from
$\mathsf{N}(0,1)$, and $\bm{\theta}^{*}$ is the new unknown coefficient
vector. For the linear model (D.2) with isotropic design matrix $\bm{G}$, we
use ${\widehat{\bm{\theta}}}^{1},...,{\widehat{\bm{\theta}}}^{T}$ to denote
the first $T$ iterates of an iterative algorithm detailed in next paragraph,
where ${\widehat{\bm{\theta}}}^{t}$ are constructed such that
${\widehat{\bm{\theta}}}^{t}=\bm{\Sigma}^{1/2}{\widehat{\bm{b}}}^{t}$. Similar
to the definition of $\bm{F},\bm{H}$ in (C.1), for the linear model (D.2), we
define

$\displaystyle\bm{F}^{*}=[\bm{y}-\bm{G}{\widehat{\bm{\theta}}}^{1},...,\bm{y}-\bm{G}{\widehat{\bm{\theta}}}^{T}],\quad\bm{H}^{*}=[{\widehat{\bm{\theta}}}^{1}-\bm{\theta}^{*},...,{\widehat{\bm{\theta}}}^{T}-\bm{\theta}^{*}].$

Denoting with a superscript∗ the quantities after the change of variable, we
have $\bm{F}^{*}=\bm{F}$ and $\bm{H}^{*}=\bm{\Sigma}^{1/2}\bm{H}$.

We now describe the iteration to generate the iterates
${\widehat{\bm{\theta}}}^{1},...,{\widehat{\bm{\theta}}}^{T}$ that guarantees
${\widehat{\bm{\theta}}}^{t}=\bm{\Sigma}^{1/2}{\widehat{\bm{b}}}^{t}$. We
start with
${\widehat{\bm{\theta}}}^{1}=\bm{\Sigma}^{1/2}{\widehat{\bm{b}}}^{1}$, and
generate ${\widehat{\bm{\theta}}}^{t}$ for $t\geq 2$ by the recursion:
${\widehat{\bm{\theta}}}^{t}={\widetilde{\bm{g}}}_{t}({\widehat{\bm{\theta}}}^{t-1},{\widetilde{\bm{v}}}^{t-1},{\widehat{\bm{\theta}}}^{t-2},{\widetilde{\bm{v}}}^{t-2})$,
where
${\widetilde{\bm{v}}}^{t}=\frac{1}{n}\bm{G}^{\top}(\bm{y}-\bm{G}{\widehat{\bm{\theta}}}^{t})$,
and the iteration function ${\widetilde{\bm{g}}}_{t}$ is defined as

(D.3)
$\displaystyle{\widetilde{\bm{g}}}_{t}\bigl{(}{\widehat{\bm{\theta}}}^{t-1},\dots,{\widehat{\bm{\theta}}}^{1},~{}~{}{\widetilde{\bm{v}}}^{t-1},\dots\widetilde{\bm{v}}^{1}\bigr{)}$
$\displaystyle=\bm{\Sigma}^{1/2}$
$\displaystyle\bm{g}_{t}\Bigl{(}\bm{\Sigma}^{-1/2}{\widehat{\bm{\theta}}}^{t-1},\dots\bm{\Sigma}^{-1/2}{\widehat{\bm{\theta}}}^{1},~{}~{}\bm{\Sigma}^{1/2}{\widetilde{\bm{v}}}^{t-1},\dots\bm{\Sigma}^{1/2}{\widetilde{\bm{v}}}^{1}\Bigr{)}.$

Since $\bm{g}_{t}$ is $\zeta$-Lipschitz continuous from Assumption 2.1, we
have ${\widetilde{\bm{g}}}_{t}$ is $\zeta\kappa$-Lipschitz continuous using
that
$\lVert\bm{\Sigma}\rVert_{\rm{op}}\leq\lVert\bm{\Sigma}\rVert_{\rm{op}}\lVert\bm{\Sigma}^{-1}\rVert_{\rm{op}}\leq\kappa$
from Assumption 2.2. By construction,
${\widehat{\bm{\theta}}}^{s}=\bm{\Sigma}^{1/2}{\widehat{\bm{b}}}^{s}$ for all
$s\leq t-1$ implies
${\widehat{\bm{\theta}}}^{t}=\bm{\Sigma}^{1/2}{\widehat{\bm{b}}}^{t}$, so by
induction the relation holds for all $t\geq 1$. For the iteration function
${\widetilde{\bm{g}}}_{t}$ in (D.3), similarly to (2.9), we define the
derivative matrices with respect to each argument as

(D.4)
$\bm{J}_{t,s}^{*}=\pdv{{\widetilde{\bm{g}}}_{t}}{\bm{\theta}^{s}}\Bigl{(}{\widehat{\bm{\theta}}}^{t-1},\dots,{\widehat{\bm{\theta}}}^{1},{\widetilde{\bm{v}}}^{t-1},\dots\widetilde{\bm{v}}^{1}\Bigr{)},\qquad\bm{D}_{t,s}^{*}=\pdv{{\widetilde{\bm{g}}}_{t}}{{\widetilde{\bm{v}}}^{s}}\Bigl{(}{\widehat{\bm{\theta}}}^{t-1},\dots,{\widehat{\bm{\theta}}}^{1},{\widetilde{\bm{v}}}^{t-1},\dots\widetilde{\bm{v}}^{1}\Bigr{)}$

for each $s\leq t-1$. Furthermore, we have by chain rule,

(D.5) $\displaystyle\bm{J}_{t,s}^{*}$
$\displaystyle=\pdv{{\widetilde{\bm{g}}}_{t}}{{\widehat{\bm{\theta}}}^{s}}=\bm{\Sigma}^{1/2}\pdv{\bm{g}_{t}}{{\widehat{\bm{b}}}^{s}}\bm{\Sigma}^{-1/2}=\bm{\Sigma}^{1/2}\bm{J}_{t,s}\bm{\Sigma}^{-1/2},$
$\displaystyle\bm{D}_{t,t-1}^{*}$
$\displaystyle=\pdv{{\widetilde{\bm{g}}}_{t}}{{\widetilde{\bm{v}}}^{s}}=\bm{\Sigma}^{1/2}\pdv{\bm{g}_{t}}{\bm{v}^{s}}\bm{\Sigma}^{1/2}=\bm{\Sigma}^{1/2}\bm{D}_{t,s}\bm{\Sigma}^{1/2}.$

We may apply Lemma C.1 to ${\widetilde{\bm{g}}}_{t}$, $\bm{\theta}_{t}$,
$\bm{G}$ to obtain with
$\mathcal{M}^{*}=\bm{I}_{pT}+\mathcal{D}^{*}(\bm{I}_{T}\otimes\bm{G}^{\top}\bm{G}/n)-\mathcal{J}^{*}$

$\displaystyle\pdv{{\widehat{\bm{\theta}}}^{t}}{g_{ik}}$
$\displaystyle=n^{-1}(\bm{e}_{t}\otimes\bm{I}_{p})({\mathcal{M}}^{*})^{-1}{\mathcal{D}}^{*}\sum_{j=1}^{p}\Bigl{[}\Bigl{(}(\bm{F}^{*})^{\top}\bm{e}_{i}\Bigr{)}\otimes\bm{e}_{k}-\Bigl{(}(\bm{H}^{*})^{\top}\bm{e}_{k}\Bigr{)}\otimes(\bm{G}^{\top}\bm{e}_{i})\Bigr{]}$

where
${\mathcal{D}}^{*}=\sum_{t=2}^{T}\sum_{s=1}^{t-1}(\bm{e}_{t}\bm{e}_{s}^{\top})\otimes\bm{D}_{t,s}^{*}$
and
${\mathcal{J}}^{*}=\sum_{t=2}^{T}\sum_{s=1}^{t-1}(\bm{e}_{t}\bm{e}_{s}^{\top})\otimes\bm{J}_{t,s}^{*}$
as in (2.10). Note that the relationships

(D.6)
${\mathcal{M}}^{*}=(\bm{I}_{T}\otimes\bm{\Sigma}^{1/2}){\mathcal{M}}(\bm{I}_{T}\otimes\bm{\Sigma}^{-1/2}),\qquad{\mathcal{D}}^{*}=(\bm{I}_{T}\otimes\bm{\Sigma}^{1/2}){\mathcal{D}}(\bm{I}_{T}\otimes\bm{\Sigma}^{1/2})$

hold due to (D.5). Now we verify that $\smash{\widehat{\mathbf{A}}}$ is
unchanged by the change of variable, that is,
$\smash{\widehat{\mathbf{A}}}=\smash{\widehat{\mathbf{A}}}^{*}$. By definition
of $\smash{\widehat{\mathbf{A}}}^{*}$ (as in (2.12)),

$\displaystyle\smash{\widehat{\mathbf{A}}}^{*}\stackrel{{\scriptstyle\text{\tiny
def}}}{{=}}\frac{1}{n}\sum_{i=1}^{n}(\bm{I}_{T}\otimes\bm{e}_{i}^{\top}\bm{G})({\mathcal{M}}^{*})^{-1}{\mathcal{D}}^{*}(\bm{I}_{T}\otimes\bm{G}^{\top}\bm{e}_{i})=\frac{1}{n}\sum_{i=1}^{n}(\bm{I}_{T}\otimes(\bm{e}_{i}^{\top}\bm{X})){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{I}_{T}\otimes(\bm{X}^{\top}\bm{e}_{i}))$

using (D.6) for the second equality. The rightmost quantity is exactly
$\smash{\widehat{\mathbf{A}}}$, so
$\smash{\widehat{\mathbf{A}}}=\smash{\widehat{\mathbf{A}}}^{*}$ stays the same
after the change of variable.

In summary, we present the relevant quantities for both the original model
(1.1) and the new model (D.2) in Table 1.

Model | $\bm{y}=\bm{X}\bm{b}^{*}+{\bm{\varepsilon}}$ | $\bm{y}=\bm{G}\bm{\theta}^{*}+{\bm{\varepsilon}}$ | Relationship
---|---|---|---
Design matrix | $\bm{X}$ | $\bm{G}$ | $\bm{X}=\bm{G}\bm{\Sigma}^{1/2}$
Covariance matrix | $\bm{\Sigma}$ | $\bm{\Sigma}^{*}=\bm{I}_{p}$ | $\bm{\Sigma}=\bm{\Sigma}\bm{\Sigma}^{*}$
Coefficient vector | $\bm{b}^{*}$ | $\bm{\theta}^{*}$ | $\bm{b}^{*}=\bm{\Sigma}^{-1/2}\bm{\theta}^{*}$
$t$-th iterate | ${\widehat{\bm{b}}}^{t}$ | ${\widehat{\bm{\theta}}}^{t}$ | ${\widehat{\bm{b}}}^{t}=\bm{\Sigma}^{-1/2}{\widehat{\bm{\theta}}}^{t}$
Error matrix | $\bm{H}=\sum_{t}({\widehat{\bm{b}}}^{t}-\bm{b}^{*})\bm{e}_{t}^{\top}$ | $\bm{H}^{*}=\sum_{t}({\widehat{\bm{\theta}}}^{t}-\bm{\theta}^{*})\bm{e}_{t}^{\top}$ | $\bm{H}=\bm{\Sigma}^{-1/2}\bm{H}^{*}$
Residual matrix | $\bm{F}=\sum_{t}(\bm{y}-\bm{X}{\widehat{\bm{b}}}^{t})\bm{e}_{t}^{\top}$ | $\bm{F}^{*}=\sum_{t}(\bm{y}-\bm{G}{\widehat{\bm{\theta}}}^{t})\bm{e}_{t}^{\top}$ | $\bm{F}=\bm{F}^{*}$
Memory matrix | $\smash{\widehat{\mathbf{A}}}$ | $\smash{\widehat{\mathbf{A}}}^{*}$ | $\smash{\widehat{\mathbf{A}}}=\smash{\widehat{\mathbf{A}}}^{*}$
Table 1: Change of variables.

The benefit of this change of variable is that, the transformed model (D.2)
has isotropic design matrix $\bm{G}$, and the following quantities stay the
same:

$\bm{F}=\bm{F}^{*},\quad\smash{\widehat{\mathbf{A}}}=\smash{\widehat{\mathbf{A}}}^{*},\quad\bm{\Sigma}^{1/2}\bm{H}=\bm{H}^{*}.$

Therefore, Theorems 2.1 and 2.2 hold for general $\bm{\Sigma}$ if they hold
for $\bm{\Sigma}=\bm{I}_{p}$. The only change is the Lipschitz constant of the
function $\bm{g}_{t}$ changes to the Lipschitz constant of
${\widetilde{\bm{g}}}_{t}$, with an extra factor of $\kappa$ by Assumption 2.2
as we argued below (D.3). Throughout the subsequent proofs of Theorems 2.1 and
2.2, we will assume $\bm{\Sigma}=\bm{I}_{p}$ without loss of generality. The
proof for general $\bm{\Sigma}$ follows by changing the Lipschitz constant
from $\zeta$ to $\kappa\zeta$.

##  Appendix E Useful intermediate results

###  E.1 Deterministic operator norm bound

######  Lemma E.1.

Under Assumption 2.1, for ${\mathcal{J}},{\mathcal{D}}$ in (2.10) we have

(E.1)
$\max\\{\lVert{\mathcal{D}}\rVert_{\rm{op}},\lVert{\mathcal{J}}\rVert_{\rm{op}}\\}\leq
T\zeta.$

###### Proof of Lemma E.1.

By Assumption 2.1 the function $\bm{g}_{t}$ is $\zeta$-Lipschitz, thus the
block row of size $p\times pT$ corresponding to $\bm{g}_{t}$ in
${\mathcal{D}}$ and ${\mathcal{J}}$ is bounded in operator norm by $\zeta$.
Since it has $T$ such row-blocks of size $p\times pT$, we obtain the result by
the triangle inequality. ∎

######  Lemma E.2.

With ${\mathcal{M}}$ as in Lemma C.1 and
$\bm{N}=n^{-1}(\bm{I}_{T}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{I}_{T}\otimes\bm{X}^{\top})$,

(E.2) $\|{\mathcal{M}}^{-1}\|_{\rm op}\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T-1}\text{ and
}\|\bm{N}\|_{\rm op}\leq C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}$

where $C(\zeta,T)$ is a constant depending on $\zeta$ and $T$ only.

###### Proof of Lemma E.2.

By definition,
${\mathcal{M}}=\bm{I}_{pT}-[{\mathcal{J}}-{\mathcal{D}}(\bm{I}_{T}\otimes\tfrac{\bm{X}^{\top}\bm{X}}{n})]$
and $(\bm{I}_{pT}-\mathcal{M})^{T}={\mathbf{0}}_{pT\times pT}$ since
$\bm{I}_{pT}-\mathcal{M}$ is lower triangular with zero diagonal blocks of
size $p\times p$. Using the matrix identity
$(\bm{I}-\bm{A})^{-1}=\sum_{k=0}^{\infty}\bm{A}^{k}=\sum_{k=1}^{T-1}$ for any
matrix $\bm{A}$ with $\bm{A}^{T}=0$, we have
${\mathcal{M}}^{-1}=\sum_{k=0}^{T-1}[{\mathcal{J}}-{\mathcal{D}}(\bm{I}_{T}\otimes\tfrac{\bm{X}^{\top}\bm{X}}{n})]^{k}$.
Furthermore for each $k=0,...,T_{1}$,

$\displaystyle\|{\mathcal{J}}-{\mathcal{D}}(\bm{I}_{T}\otimes\tfrac{\bm{X}^{\top}\bm{X}}{n})\|^{k}_{\rm
op}$
$\displaystyle\leq(\lVert{\mathcal{J}}\rVert_{\rm{op}}+\|{\mathcal{D}}\|_{\rm
op}\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{k}$ by (B.4)
$\displaystyle\leq[(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)\zeta T]^{k}$
$\displaystyle\mbox{by \eqref{eq:opnorm-D-J}}.$

This provides $\|\mathcal{M}\|_{\rm op}\leq T(1+\zeta
T)^{T-1}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T-1}$. The bound on
$\lVert\bm{N}\rVert_{\rm{op}}$ follows from $\|\bm{N}\|_{\rm
op}\leq\|{\mathcal{M}}^{-1}\|_{\rm op}\|{\mathcal{D}}\|_{\rm
op}\lVert\bm{X}\rVert_{\rm{op}}^{2}/n$ and the previous bound. ∎

######  Lemma E.3.

Under Assumption 2.1, we have

(E.3) $\displaystyle\|\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n\|_{\rm op}$
$\displaystyle\leq C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T},$
(E.4) $\displaystyle\|(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}\|_{\rm
op}$ $\displaystyle\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T^{2}}.$

###### Proof of Lemma E.3.

Since
$\smash{\widehat{\mathbf{A}}}=\sum_{i=1}^{n}(\bm{I}_{T}\otimes\bm{e}_{i}^{\top})\bm{N}(\bm{I}_{T}\otimes\bm{e}_{i})$
we have for any $\bm{u},\bm{u}\in{\mathbb{R}}^{T}$ that
$|\bm{u}^{\top}\smash{\widehat{\mathbf{A}}}\bm{v}|=|\sum_{i=1}^{n}(\bm{u}\otimes\bm{e}_{i})^{\top}\bm{N}(\bm{v}\otimes\bm{e}_{i})|\leq\|\bm{N}\|_{\rm
op}\sum_{i=1}^{n}\|\bm{u}\otimes\bm{e}_{i}\|\|\bm{v}\otimes\bm{e}_{i}\|\leq
n\|\bm{N}\|_{\rm op}$ using (B.1) and (B.4). The bound on the operator norm of
$\bm{N}$ from Lemma E.2 gives (E.3).

Since $\smash{\widehat{\mathbf{A}}}$ is a lower triangular matrix, we know
$\smash{\widehat{\mathbf{A}}}$ is a nilpotent matrix with
$\smash{\widehat{\mathbf{A}}}^{k}=0$ if $k\geq T$. Consequently
$(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}=\sum_{k=0}^{\infty}(\smash{\widehat{\mathbf{A}}}/n)^{k}=\sum_{k=0}^{T-1}(\smash{\widehat{\mathbf{A}}}/n)^{k}$,
hence

$\displaystyle\|(\bm{I}_{T}-\frac{\smash{\widehat{\mathbf{A}}}}{n})^{-1}\|_{\rm
op}$
$\displaystyle\leq\sum_{k=0}^{T-1}\|\frac{\smash{\widehat{\mathbf{A}}}}{n}\|_{\rm
op}^{k}\leq
C(\zeta,T)\sum_{k=0}^{T-1}\Bigl{(}1+\frac{\lVert\bm{X}\rVert_{\rm{op}}^{2}}{n}\Bigr{)}^{Tk}\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T^{2}}.$

This finishes the proof. ∎

######  Lemma E.4.

Under Assumption 2.1, let
${\widehat{\bm{C}}}=\sum_{j=1}^{p}(\bm{I}_{T}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{I}_{T}\otimes\bm{e}_{j})$,
we have

(E.5) $\displaystyle\|\bm{I}_{T}+{\widehat{\bm{C}}}/n\|_{\rm op}$
$\displaystyle\leq
C(\zeta,T,\gamma)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T},$ (E.6)
$\displaystyle\|(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{-1}\|_{\rm op}$
$\displaystyle\leq
C(\zeta,T,\gamma)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T^{2}}.$

###### Proof of Lemma E.4.

By triangular inequality, we have

$\displaystyle\lVert{\widehat{\bm{C}}}\rVert_{\rm{op}}/n$ $\displaystyle\leq
n^{-1}\sum_{j=1}^{p}\lVert(\bm{I}_{T}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{I}_{T}\otimes\bm{e}_{j})\rVert_{\rm{op}}$
$\displaystyle\leq
p/n\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}$
$\displaystyle\leq
C(\zeta,T,\gamma)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T-1}$
$\displaystyle\mbox{by \lx@cref{creftypepluralcap~refnum}{lem:opnorm-M-N}
and~\lx@cref{refnum}{lem:opnorm-D-J}}.$

This directly implies (E.5) by the triangle inequality. Since
${\widehat{\bm{C}}}$ is lower triangular by its definition, the same argument
in the proof of Lemma E.3 gives (E.6). ∎

######  Lemma E.5.

Under Assumption 2.1, we have

(E.7)
$\displaystyle\big{\|}\pdv{\bm{F}\bm{e}_{t}}{{\bm{\varepsilon}}}\big{\|}_{\rm
op}=\big{\|}\bm{I}_{n}-(\bm{e}_{t}\otimes\bm{I}_{n})^{\top}\bm{N}({\mathbf{1}}_{T}\otimes\bm{I}_{n})\|_{\rm
op}$ $\displaystyle\leq C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T},$
(E.8)
$\displaystyle\big{\|}\pdv{\bm{H}\bm{e}_{t}}{{\bm{\varepsilon}}}\big{\|}_{\rm
op}=\big{\|}\frac{1}{n}(\bm{e}_{t}\otimes\bm{I}_{p})^{\top}{\mathcal{M}}^{-1}{\mathcal{D}}({\mathbf{1}}_{T}\otimes\bm{X}^{\top})\big{\|}_{\rm
op}$
$\displaystyle\leq\frac{C(\zeta,T)}{\sqrt{n}}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}.$

###### Proof of Lemma E.5.

By Corollary C.2, we have

$\pdv{\bm{F}\bm{e}_{t}}{{\bm{\varepsilon}}}=\bm{I}_{n}-n^{-1}(\bm{e}_{t}^{\top}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}({\mathbf{1}}_{T}\otimes\bm{X}^{\top})=\bm{I}_{n}-(\bm{e}_{t}\otimes\bm{I}_{n})^{\top}\bm{N}({\mathbf{1}}_{T}\otimes\bm{I}_{n})$

so the bound for $\bm{F}$ follows from (E.2) and (B.4). The bound for $\bm{H}$
follows from (C.3) and the same argument with the operator norm bounds (E.1)
and (E.2) for ${\mathcal{M}}$. ∎

######  Lemma E.6.

Assume Assumption 2.1 holds. Let
$\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\in{\mathbb{R}}^{nT\times
np}$ be the Jacobian of the mapping
${\mathbb{R}}^{np}\to{\mathbb{R}}^{n}:\operatorname{\mathbf{vec}}(\bm{X})\mapsto\operatorname{\mathbf{vec}}(\bm{F})$.
Then we have

(E.9)
$\displaystyle\big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\big{\|}_{\rm
op}$ $\displaystyle\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}(\lVert\bm{H}\rVert_{\rm{op}}+\lVert\bm{F}\rVert_{\rm{op}}/\sqrt{n}),$
(E.10)
$\displaystyle\big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\big{\|}_{\rm
F}$ $\displaystyle\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}(\lVert\bm{H}\otimes\bm{I}_{n}\rVert_{\rm{F}}+\lVert\bm{F}\otimes\bm{I}_{p}\rVert_{\rm{F}}/\sqrt{n})$
$\displaystyle=C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}(\lVert\bm{H}\rVert_{\rm{F}}\sqrt{n}+\lVert\bm{F}\rVert_{\rm{F}}\sqrt{p/n}).$

###### Proof of Lemma E.6.

With $\bm{F}=\sum_{lt}\bm{e}_{l}F_{lt}\bm{e}_{t}^{\top}$ we have
$\operatorname{\mathbf{vec}}(\bm{F})=\sum_{lt}(\bm{e}_{t}\otimes\bm{e}_{l})F_{lt}$
by (B.5) and so that

$\displaystyle\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}$
$\displaystyle=\sum_{l=1}^{n}\sum_{t=1}^{T}\sum_{i=1}^{n}\sum_{j=1}^{p}(\bm{e}_{t}\otimes\bm{e}_{l})\pdv{F_{lt}}{x_{ij}}(\bm{e}_{j}^{\top}\otimes\bm{e}_{i}^{\top})=\sum_{lt,ij}(\bm{e}_{t}\otimes\bm{e}_{l})(D_{ij}^{lt}+\Delta_{ij}^{lt})(\bm{e}_{i}^{\top}\otimes\bm{e}_{j}^{\top})$

where $D_{ij}^{lt}$ and $\Delta_{ij}^{lt}$ are defined in (C.5) and (C.6).
Since permuting the ordering of canonical basis vectors does not change the
operator norm and the Frobenius norm, we find by (C.5) and the definition of
$\bm{N}$ in Lemma E.2 that

(E.11)
$\Big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\Big{\|}_{?}\leq\Big{\|}\bm{N}(\bm{H}^{\top}\otimes\bm{I}_{n})\Big{\|}_{?}+\Big{\|}n^{-1}(\bm{I}_{T}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}\bm{F}^{\top}\otimes\bm{I}_{p}\bigr{)}\Big{\|}_{?}$

where $\|\cdot\|_{?}$ is either the operator norm or the Frobenius norm. The
property (B.4) of the Kronecker product and the operator norm bounds (E.2),
(E.1) provide the two inequalities. For the last equality, we use
$\|\bm{H}\otimes\bm{I}_{n}\|_{\rm F}=\|\bm{H}\|_{\rm F}\sqrt{n}$ and similarly
for $\bm{F}^{\top}\otimes\bm{I}_{p}$ by (B.4). ∎

######  Lemma E.7.

Assume Assumption 2.1 holds. Let
$\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{\operatorname{\mathbf{vec}}(\bm{X})}\in{\mathbb{R}}^{n\times
np}$ be the Jacobian of the mapping
${\mathbb{R}}^{np}\to{\mathbb{R}}^{n}:\operatorname{\mathbf{vec}}(\bm{X})\mapsto\operatorname{\mathbf{vec}}(\bm{H})$.
Then we have

(E.12)
$\displaystyle\big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{\operatorname{\mathbf{vec}}(\bm{X})}\big{\|}_{\rm
op}$ $\displaystyle\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}\Bigl{(}\lVert\bm{H}\rVert_{\rm{op}}+\lVert\bm{F}\rVert_{\rm{op}}/\sqrt{n}\Bigr{)}\frac{1}{\sqrt{n}},$
(E.13)
$\displaystyle\big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{\operatorname{\mathbf{vec}}(\bm{X})}\big{\|}_{\rm
F}$ $\displaystyle\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}\Bigl{(}\lVert\bm{F}^{\top}\otimes\bm{I}_{p}\rVert_{\rm{F}}+\sqrt{n}\lVert\bm{H}\otimes\bm{I}_{n}\rVert_{\rm{F}}\Bigr{)}\frac{1}{n}$
$\displaystyle=C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}\bigl{(}\lVert\bm{F}n^{-1/2}\rVert_{\rm{F}}\sqrt{\gamma}+\lVert\bm{H}\rVert_{\rm{F}}\bigr{)}.$

###### Proof of Lemma E.7.

By the same argument as in Lemma E.6, given (C.2),

$\Big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{\operatorname{\mathbf{vec}}(\bm{X})}\Big{\|}_{?}\leq\Big{\|}n^{-1}{\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{I}_{p})\Big{\|}_{?}+\Big{\|}n^{-1}{\mathcal{M}}^{-1}{\mathcal{D}}(\bm{I}_{T}\otimes\bm{X}^{\top})(\bm{H}^{\top}\otimes\bm{I}_{n})\Big{\|}_{?}$

where $\|\cdot\|_{?}$ is either the operator norm or the Frobenius norm. The
property (B.4) of the Kronecker product and the operator norm bounds (E.2),
(E.1) provide the two inequalities. For the last equality, we use
$\|\bm{F}^{\top}\otimes\bm{I}_{p}\|_{\rm F}=\|\bm{F}\|_{\rm F}\sqrt{p}$ and
similarly for $\bm{H}^{\top}\otimes\bm{I}_{n}$ by (B.4). ∎

###  E.2 Moment bounds

######  Lemma E.8 (Moment bound of $\bm{X}$).

Under Assumptions 2.2 and 2.4, the following inequality holds for any
positive, finite integer $k$:

${\mathbb{E}}[\|\bm{X}\bm{\Sigma}^{-1/2}/\sqrt{n}\|_{\rm op}^{k}]\leq
C(\gamma,k).$

###### Proof of Lemma E.8.

By (Davidson and Szarek, 2001, Theorem II.13), there exists a random variable
$z\sim\mathsf{N}(0,1)$ s.t. $\|\bm{X}\bm{\Sigma}^{-1/2}\|_{\rm
op}\leq\sqrt{n}+\sqrt{p}+z$ almost surely. Thus,

${\mathbb{E}}[\|\bm{X}\bm{\Sigma}^{-1/2}/\sqrt{n}\|_{\rm
op}^{k}]\leq{\mathbb{E}}[(1+\sqrt{p/n}+z)^{k}]\leq C(\gamma,k).$

The last inequality follows from $p/n\leq\gamma$ and the fact that
${\mathbb{E}}[z^{k}]$ is bounded for any finite $k$ if $z\sim\mathsf{N}(0,1)$.
∎

######  Lemma E.9 (Moment bound of ${\bm{\varepsilon}}$).

Under Assumption 2.3, for any positive finite integer $k$, there exists a
constant $C(k)$ depending only on $k$ such that
${\mathbb{E}}[(\lVert{\bm{\varepsilon}}\rVert^{2}/n)^{k}]\leq
C(k)(\sigma^{2})^{k}.$

This is a known bound on the finite moment of the $\chi^{2}_{n}$ distribution.
Alternatively, since ${\bm{\varepsilon}}/\sigma$ has a same distribution as
any column of $\bm{X}\bm{\Sigma}^{-1/2}$, we have
${\mathbb{E}}[(\|{\bm{\varepsilon}}\|^{2}/\sigma^{2})^{k}]\leq{\mathbb{E}}[\|\bm{X}\bm{\Sigma}^{-1/2}\|_{\rm
op}^{2k}]$ so Lemma E.9 follows from Lemma E.8 with $\gamma=1$.

######  Lemma E.10 (Moment bounds of $\bm{H}$ and $\bm{F}$).

If Assumptions 2.2, 2.3, 2.4 and 2.1 are fulfilled then

$\displaystyle{\mathbb{E}}[\|\bm{\Sigma}^{1/2}\bm{H}\|_{\rm
F}^{2}]\vee{\mathbb{E}}[\|\bm{\Sigma}^{1/2}\bm{H}\|_{\rm
F}^{4}]^{1/2}\vee{\mathbb{E}}[\|\bm{\Sigma}^{1/2}\bm{H}\|_{\rm F}^{8}]^{1/4}$
$\displaystyle\leq C(\zeta,T,\gamma,\kappa){\rm var}(y_{1}),$
$\displaystyle{\mathbb{E}}[\|\bm{F}\|_{\rm
F}^{2}/n]\vee{\mathbb{E}}[\|\bm{F}\|_{\rm
F}^{4}/n^{2}]^{1/2}\vee{\mathbb{E}}[\|\bm{F}\|_{\rm F}^{8}/n^{4}]^{1/4}$
$\displaystyle\leq C(\zeta,T,\gamma,\kappa){\rm var}(y_{1}).$

where $y_{1}$ is the first entry of the response vector and ${\rm
var}(y_{1})=\lVert\bm{\Sigma}^{1/2}\bm{b}^{*}\rVert^{2}+\sigma^{2}$.
Consequently, by compactness, we can extract a subsequence of regression
problems such that

(E.14) ${\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\to\bm{K},\text{ and
}{\mathbb{E}}[\bm{H}^{\top}\bm{\Sigma}\bm{H}+\bm{S}]\to\bar{\bm{K}},$

where $\bm{K}$ and $\bar{\bm{K}}$ are two positive semi-definite deterministic
matrices.

###### Proof of Lemma E.10.

Without loss of generality, we assume $\bm{\Sigma}=\bm{I}_{p}$. Otherwise, if
$\bm{\Sigma}\neq\bm{I}_{p}$, we apply the change of variable (D.1), the
desired result follows.

Using the fact that $\|\bm{H}\|_{\rm
F}^{2}=\sum_{t=1}^{T}\|\bm{H}\bm{e}_{t}\|^{2}$ and $\|\bm{F}\|_{\rm
F}^{2}=\sum_{t=1}^{T}\|\bm{F}\bm{e}_{t}\|^{2}$, it suffices to bound the
moments of $\|\bm{H}\bm{e}_{t}\|$ and $\|\bm{F}\bm{e}_{t}\|$ for each
$t\in[T]$.

Let
$a_{s}=\max\\{\|[{\widehat{\bm{b}}}^{s}\mid{\widehat{\bm{b}}}^{s-1}\mid\dots\mid{\widehat{\bm{b}}}^{1}]\|_{\rm
F},n^{-1/2}\|[\bm{y}-\bm{X}{\widehat{\bm{b}}}^{s}\mid\bm{y}-\bm{X}{\widehat{\bm{b}}}^{s-1}\mid\bm{y}-\bm{X}{\widehat{\bm{b}}}^{1}]\|_{\rm
F}\\}$. By definition of ${\widehat{\bm{b}}}^{t}$ in (2.8), using
$\bm{g}_{t}({\mathbf{0}})={\mathbf{0}}$ and Assumption 2.1 we have

$\displaystyle\lVert{\widehat{\bm{b}}}^{t}-{\mathbf{0}}\rVert=\lVert{\widehat{\bm{b}}}^{t}-\bm{g}_{t}({\mathbf{0}})\rVert$
$\displaystyle\leq\zeta\|\bigl{[}{\widehat{\bm{b}}}^{t-1}\mid{\widehat{\bm{b}}}^{t-2}\mid\dots\mid{\widehat{\bm{b}}}^{1}\mid\bm{v}^{t-1}\mid\dots\mid\bm{v}^{1}\bigr{]}\|_{\rm
F}$
$\displaystyle\leq\zeta\|\bigl{[}{\widehat{\bm{b}}}^{t-1}\mid{\widehat{\bm{b}}}^{t-2}\mid\dots\mid{\widehat{\bm{b}}}^{1}\bigr{]}\|_{\rm
F}+\zeta\|\bigl{[}\bm{v}^{t-1}\mid\dots\mid\bm{v}^{1}\bigr{]}\|_{\rm F}$
$\displaystyle\leq\zeta(a_{t-1}+\|\bm{X}n^{-1/2}\|_{\rm op}a_{t-1})$

and
$\|\bm{y}-\bm{X}{\widehat{\bm{b}}}^{t}\|\leq\|\bm{y}\|+\|\bm{X}n^{-1/2}\|_{\rm
op}\|{\widehat{\bm{b}}}^{t}\|$. Since
$\bm{y}=\bm{y}-\bm{X}{\widehat{\bm{b}}}^{1}$ since
${\widehat{\bm{b}}}^{1}={\mathbf{0}}$, we also have
$\|\bm{y}\|/\sqrt{n}=a_{1}\leq a_{t-1}$ and
$\|\bm{y}-\bm{X}{\widehat{\bm{b}}}^{t}\|\leq\sqrt{n}a_{t-1}C(\zeta)(1+\|\bm{X}n^{-1/2}\|_{\rm
op}^{2})$. This proves

(E.15) $a_{t}\leq C(\zeta)(1+\|\bm{X}n^{-1/2}\|_{\rm
op}^{2})a_{t-1},\quad\text{ and }\quad a_{T}\leq
C(\zeta,T)(1+\|\bm{X}n^{-1/2}\|_{\rm op}^{2})^{T}a_{1}$

by induction, where $a_{1}=\|\bm{y}\|/\sqrt{n}$. By the triangle inequality,
$\|\bm{H}\|_{\rm F}\leq\sqrt{T}\|\bm{b}^{*}\|+a_{T}$, so we have established

${\rm var}(y_{1})^{-1/2}\Bigl{(}\|\bm{H}\|_{\rm F}+\|\bm{F}\|_{\rm
F}/\sqrt{n}\Bigr{)}\leq C(\zeta,T)(1+\|\bm{X}n^{-1/2}\|_{\rm
op}^{2})^{T}\Bigl{(}\frac{\|\bm{y}\|}{\sqrt{n}{\rm
var}(y_{1})^{1/2}}+1\Bigr{)}.$

The moments of order 2, 4 and 8 (and any other finite moment) of the right-
hand side are bounded by $C(\zeta,T,\gamma)$ by Lemmas E.8 and E.9. ∎

###  E.3 Frobenius norm bounds on Jacobians

######  Lemma E.11 (Frobenius norm bound of $\bm{F}$ w.r.t. $\bm{X}$).

Under Assumptions 2.1 and 2.4,

$\frac{1}{n}\big{\|}\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\big{\|}_{\rm
F}^{2}=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{F}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\leq
C(\zeta,T)\Bigl{(}1+\frac{1}{n}\lVert\bm{X}\rVert_{\rm{op}}^{2}\Bigr{)}^{2T}\Bigl{(}\|\bm{H}\|_{\rm
F}^{2}+\gamma\|\bm{F}\|_{\rm F}^{2}/n\Bigr{)},$

Furthermore, if Assumptions 2.3 and 2.2 hold with $\bm{\Sigma}=\bm{I}_{p}$
then

${\mathbb{E}}\Bigl{[}\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{F}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Bigr{]}\leq{\mathbb{E}}\Bigl{[}\Bigl{(}\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{F}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Bigr{)}^{2}\Bigr{]}^{1/2}\leq
C(\zeta,T,\gamma){\rm var}(y_{1}).$

###### Proof of Lemma E.11.

The first line is proved in (E.10). For the second line, we use the Cauchy-
Schwarz inequality with the moment bounds from Lemmas E.10 and E.8. ∎

######  Lemma E.12 (Frobenius norm bound of $\bm{H}$ w.r.t. $\bm{X}$).

Under Assumptions 2.1 and 2.4,

$\displaystyle\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{H}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\leq\gamma
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}(\lVert\bm{F}\rVert_{\rm{F}}^{2}/n+\lVert\bm{H}\rVert_{\rm{F}}^{2}).$

In addition, if Assumptions 2.3 and 2.2 hold with $\bm{\Sigma}=\bm{I}_{p}$
then

$\displaystyle{\mathbb{E}}\Bigl{[}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{H}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Bigr{]}\leq{\mathbb{E}}\Bigl{[}\Bigl{(}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{H}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Bigr{)}^{2}\Bigr{]}^{1/2}\leq~{}$
$\displaystyle C(\zeta,T,\gamma){\rm var}(y_{1}).$

###### Proof of Lemma E.12.

The same argument as for (E.10) would provide the desired bound. We provide an
alternative argument to showcase another means to control such quantities. By
the expression of $\pdv{{\widehat{\bm{b}}}^{t}}{x_{ij}}$ in (C.2), we have
$\pdv{\bm{e}_{k}^{\top}\bm{H}\bm{e}_{t}}{x_{ij}}=\pdv{\bm{e}_{k}^{\top}({\widehat{\bm{b}}}^{t}-\bm{b}^{*})}{x_{ij}}=\pdv{\bm{e}_{k}^{\top}{\widehat{\bm{b}}}^{t}}{x_{ij}}$
so that

$\displaystyle\pdv{\bm{e}_{k}^{\top}\bm{H}\bm{e}_{t}}{x_{ij}}$
$\displaystyle=n^{-1}\bm{e}_{k}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}[((\bm{F}^{\top}\bm{e}_{i})\otimes\bm{e}_{j})-((\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i}))]$
by (C.2)
$\displaystyle=n^{-1}(\bm{e}_{t}^{\top}\otimes\bm{e}_{k}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}[(\bm{F}^{\top}\otimes\bm{I}_{p})(\bm{e}_{i}\otimes\bm{e}_{j})-(\bm{H}^{\top}\otimes\bm{X}^{\top})(\bm{e}_{j}\otimes\bm{e}_{i})]$
$\displaystyle\mbox{by \eqref{eq:kronecker-mix-product}}.$

Therefore, using $(a+b)^{2}\leq 2a^{2}+2b^{2}$,

$\displaystyle\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{H}}{x_{ij}}\right\rVert_{\rm{F}}^{2}=\sum_{ij,kt}\Bigl{(}\pdv{\bm{e}_{k}^{\top}\bm{H}\bm{e}_{t}}{x_{ij}}\Bigr{)}^{2}$
$\displaystyle\leq
2n^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{I}_{p})\rVert_{\rm{F}}^{2}+2n^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}(\bm{H}^{\top}\otimes\bm{X}^{\top})\rVert_{\rm{F}}^{2}$
$\displaystyle\leq
2n^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}^{2}\lVert\bm{F}^{\top}\otimes\bm{I}_{p}\rVert_{\rm{F}}^{2}+2n^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}^{2}\lVert\bm{H}^{\top}\otimes\bm{X}^{\top}\rVert_{\rm{F}}^{2}$
$\displaystyle=2pn^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}^{2}\lVert\bm{F}\rVert_{\rm{F}}^{2}+2n^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}^{2}\lVert\bm{H}\rVert_{\rm{F}}^{2}\lVert\bm{X}\rVert_{\rm{F}}^{2}$
by (B.4) $\displaystyle\leq
2pn^{-2}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}^{2}(\lVert\bm{F}\rVert_{\rm{F}}^{2}+\lVert\bm{H}\rVert_{\rm{F}}^{2}\lVert\bm{X}\rVert_{\rm{op}}^{2})$
by $\lVert\bm{X}\rVert_{\rm{F}}\leq\sqrt{p}\lVert\bm{X}\rVert_{\rm{op}}$
$\displaystyle\leq
2pn^{-2}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}^{2}\lVert{\mathcal{D}}\rVert_{\rm{op}}^{2}(\lVert\bm{F}\rVert_{\rm{F}}^{2}+\lVert\bm{H}\rVert_{\rm{F}}^{2}\lVert\bm{X}\rVert_{\rm{op}}^{2})$
submultiplicativity of $\|\cdot\|_{\rm op}$ $\displaystyle\leq\gamma
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}(\lVert\bm{F}\rVert_{\rm{F}}^{2}/n+\lVert\bm{H}\rVert_{\rm{F}}^{2})$
$\displaystyle\mbox{by \eqref{eq:opnorm-D-J} and \eqref{eq:opnorm-M-N}}.$

For the upper bound on the moments, we use the Cauchy-Schwarz inequality with
the moment bounds from Lemmas E.10 and E.8 as for the proof of Lemma E.11. ∎

######  Lemma E.13.

Under Assumptions 2.2, 2.4, 2.1 and 2.3, we have

(E.16)
$\displaystyle{\mathbb{E}}[\|\bm{F}^{\top}\bm{F}/n-{\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\|_{\rm
F}^{2}]$ $\displaystyle\leq C(\gamma,\zeta,T,\kappa){\rm var}(y_{1})^{2}/n,$
(E.17)
$\displaystyle{\mathbb{E}}[\|\bm{H}^{\top}\bm{\Sigma}\bm{H}-{\mathbb{E}}[\bm{H}^{\top}\bm{\Sigma}\bm{H}]\|_{\rm
F}^{2}]$ $\displaystyle\leq C(\gamma,\zeta,T,\kappa){\rm var}(y_{1})^{2}/n.$

Consequently, if ${\rm var}(y_{1})$ is bounded from above by a constant, by
Markov’s inequality,

(E.18)
$\bm{F}^{\top}\bm{F}/n-{\mathbb{E}}(\bm{F}^{\top}\bm{F}/n)=O_{P}(n^{-1/2})\text{
and
}\bm{H}^{\top}\bm{\Sigma}\bm{H}-{\mathbb{E}}(\bm{H}^{\top}\bm{\Sigma}\bm{H})=O_{P}(n^{-1/2}).$

###### Proof of Lemma E.13.

By the change of variable argument in Lemma E.10, it suffices to prove the
results under $\bm{\Sigma}=\bm{I}_{p}.$ We view $\bm{F}$ as a function of
$(\bm{X},{\bm{\varepsilon}})$. By the Gaussian Poincaré inequality applied to
$\bm{e}_{t}^{\top}\bm{F}^{\top}\bm{F}\bm{e}_{s}$ for each $t,s\in[T]$, we find

(E.19) ${\rm
var}\Bigl{(}(\bm{F}\bm{e}_{t})^{\top}\bm{F}\bm{e}_{s}\Bigr{)}\leq{\mathbb{E}}\sum_{i=1}^{n}\sigma^{2}\Big{(}\pdv{(\bm{e}_{t}^{\top}\bm{F}^{\top}\bm{F}\bm{e}_{s})}{\varepsilon_{i}}\Big{)}^{2}+{\mathbb{E}}\sum_{i=1}^{n}\sum_{j=1}^{p}\Big{(}\pdv{(\bm{e}_{t}^{\top}\bm{F}^{\top}\bm{F}\bm{e}_{s})}{x_{ij}}\Big{)}^{2}$

Let $\partial$ denote either $\partial/\partial x_{ij}$ or
$\partial/\partial\epsilon_{i}$. Using the product rule
$\partial(\bm{e}_{t}^{\top}\bm{F}^{\top}\bm{F}\bm{e}_{s}))=\bm{e}_{t}^{\top}(\partial\bm{F})^{\top}\bm{F}\bm{e}_{s}+\bm{e}_{t}^{\top}\bm{F}^{\top}(\partial\bm{F})\bm{e}_{s}$
as well as $(a+b)^{2}\leq 2a^{2}+2b^{2}$ and summing over all $s\in[T]$ and
all $t\in[T]$,

$\displaystyle\sum_{s=1}^{T}\sum_{t=1}^{T}{\rm
var}\Bigl{(}(\bm{F}\bm{e}_{t})^{\top}\bm{F}\bm{e}_{s}\Bigr{)}$
$\displaystyle\leq
4{\mathbb{E}}\sum_{s=1}^{T}\sum_{t=1}^{T}\sum_{i=1}^{n}\Bigl{(}\sigma^{2}\Bigl{(}\bm{e}_{s}^{\top}\bm{F}^{\top}\frac{\partial\bm{F}\bm{e}_{t}}{\partial\epsilon_{i}}\Bigr{)}^{2}+\sum_{j=1}^{p}\Bigl{(}\bm{e}_{s}^{\top}\bm{F}^{\top}\frac{\partial\bm{F}\bm{e}_{t}}{\partial
x_{ij}}\Bigr{)}^{2}\Bigr{)}$ (E.20)
$\displaystyle=4{\mathbb{E}}\Bigl{[}\sum_{i=1}^{n}\Bigl{(}\sigma^{2}\|\bm{F}^{\top}\frac{\partial\bm{F}}{\partial\epsilon_{i}}\|_{\rm
F}^{2}+\sum_{j=1}^{p}\|\bm{F}^{\top}\frac{\partial\bm{F}}{\partial
x_{ij}}\|_{\rm F}^{2}\Bigr{)}\Bigr{]}$

where we used $\sum_{s=1}^{T}\bm{e}_{s}\bm{e}_{s}^{\top}=\bm{I}_{T}$ and
similarly for the sum over $t\in[T]$. We rewrite the above using the
vectorization operator: $\|\bm{F}^{\top}\tfrac{\partial}{\partial
x_{ij}}\bm{F}\|_{\rm
F}^{2}=\|(\bm{I}_{T}\otimes\bm{F}^{\top})\operatorname{\mathbf{vec}}(\tfrac{\partial}{\partial
x_{ij}}\bm{F})\|^{2}$, which is also the squared norm of the $(i,j)$-th column
of
$(\bm{I}_{T}\otimes\bm{F}^{\top})\frac{\partial\operatorname{\mathbf{vec}}\bm{F}}{\partial\operatorname{\mathbf{vec}}\bm{X}}$,
so that

(E.21)
$\sum_{i=1}^{n}\sum_{j=1}^{p}\|\bm{F}^{\top}\frac{\partial\bm{F}}{\partial
x_{ij}}\|_{\rm
F}^{2}=\Big{\|}(\bm{I}_{T}\otimes\bm{F}^{\top})\frac{\partial\operatorname{\mathbf{vec}}(\bm{F})}{\partial\operatorname{\mathbf{vec}}(\bm{X})}\Big{\|}_{\rm
F}^{2}\leq T\|\bm{F}\|_{\rm
F}^{2}\Big{\|}\frac{\partial\operatorname{\mathbf{vec}}(\bm{F})}{\partial\operatorname{\mathbf{vec}}(\bm{X})}\Big{\|}_{\rm
op}^{2}$

using $\|\bm{M}\bm{M}^{\prime}\|_{\rm F}\leq\|\bm{M}\|_{\rm
F}\|\bm{M}^{\prime}\|_{\rm op}$ for the inequality. By the same argument,

(E.22)
$\sum_{i=1}^{n}\|\bm{F}^{\top}\frac{\partial\bm{F}}{\partial\epsilon_{i}}\|_{\rm
F}^{2}=\Big{\|}(\bm{I}_{T}\otimes\bm{F}^{\top})\frac{\partial\operatorname{\mathbf{vec}}(\bm{F})}{\partial{\bm{\varepsilon}}}\Big{\|}_{\rm
F}^{2}\leq T\|\bm{F}\|_{\rm
F}^{2}\Big{\|}\frac{\partial\operatorname{\mathbf{vec}}(\bm{F})}{\partial{\bm{\varepsilon}}}\Big{\|}_{\rm
op}^{2}.$

By (E.9) and Equation E.7, both previous displays are bounded from above by

$C(\zeta,T,\gamma)\|\bm{F}\|_{\rm F}^{2}\max\\{\sigma^{2},\|\bm{H}\|_{\rm
F}^{2}+\|\bm{F}\|_{\rm F}^{2}/n\\}(1+\|\bm{X}\|_{\rm op}^{2}/n)^{T}.$

Using the Cauchy-Schwarz inequality to leverage the moment bounds (E.8) and
(E.10), we find that (E.20) is bounded from above by $C(\zeta,T,\gamma)n$ and
the proof of (E.16) is complete.

By exactly the same argument, (E.19), (E.20), (E.22) and (E.21) hold with
$\bm{F}$ replaced by $\bm{H}$. We use the upper bounds (E.12) and (E.8) to
control the operator norm of
$\frac{\partial\operatorname{\mathbf{vec}}(\bm{F})}{\partial\operatorname{\mathbf{vec}}(\bm{X})}$
and
$\frac{\partial\operatorname{\mathbf{vec}}(\bm{F})}{\partial{\bm{\varepsilon}}}$,
and the moment bounds (E.8) and (E.10) to obtain (E.17). ∎

##  Appendix F Proof of Theorem 2.2

Throughout this proof, we assume $\bm{\Sigma}=\bm{I}_{p}$, and the proof for
general $\bm{\Sigma}$ follows the same line of argument by changing $\zeta$ to
$\kappa\zeta$, thanks to the change of variables in Appendix D.

Before stating the proof, we recall a few defintions from (C.1):

$\bm{H}=[{\widehat{\bm{b}}}^{1}-\bm{b}^{*},\ldots,{\widehat{\bm{b}}}^{T}-\bm{b}^{*}]\in{\mathbb{R}}^{p\times
T},\quad\bm{F}=[\bm{y}-\bm{X}{\widehat{\bm{b}}}^{1},\ldots,\bm{y}-\bm{X}{\widehat{\bm{b}}}^{T}]\in{\mathbb{R}}^{n\times
T}.$

We first derive the upper bound of ${\rm var}(r_{tt^{\prime}})$. By definition
of $r_{tt^{\prime}}$ and $\bm{H}$, we know $r_{tt^{\prime}}$ is the
$(t,t^{\prime})$ entry of $\bm{H}^{\top}\bm{H}$. Thus,

$\displaystyle{\rm var}(r_{tt^{\prime}})$ $\displaystyle=$
$\displaystyle{\mathbb{E}}[(r_{tt^{\prime}}-{\mathbb{E}}[r_{tt^{\prime}}])^{2}]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}\big{[}\big{(}[\bm{H}^{\top}\bm{H}]_{t,t^{\prime}}-{\mathbb{E}}[[\bm{H}^{\top}\bm{H}]_{t,t^{\prime}}]\big{)}^{2}\big{]}$
$\displaystyle\leq$
$\displaystyle{\mathbb{E}}\big{[}\lVert\bm{H}^{\top}\bm{H}-{\mathbb{E}}[\bm{H}^{\top}\bm{H}]\rVert_{\rm{F}}^{2}\big{]}$
$\displaystyle\leq$ $\displaystyle n^{-1}C(\gamma,\zeta,T,\kappa){\rm
var}(y_{1})^{2}$ $\displaystyle\mbox{by \lx@cref{creftypecap~refnum}{lem:var-
F-H}}.$

It thus remains to show
${\mathbb{E}}[|r_{tt^{\prime}}-\hat{r}_{tt^{\prime}}|]\leq
n^{-1/2}C(\gamma,\zeta,T,\kappa){\rm var}(y_{1})$. Define
$\bm{S}=\sigma^{2}{\mathbf{1}}_{T}{\mathbf{1}}_{T}^{\top}\in{\mathbb{R}}^{T\times
T}$, then we have

$\displaystyle{\mathbb{E}}[|r_{tt^{\prime}}-\hat{r}_{tt^{\prime}}|]=$
$\displaystyle{\mathbb{E}}\Big{[}[\bm{H}^{\top}\bm{H}+\bm{S}]_{t,t^{\prime}}-[(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{-1}\bm{F}^{\top}\bm{F}/n(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}^{\top})^{-1}]_{t,t^{\prime}}\Big{]}$
$\displaystyle\leq$
$\displaystyle{\mathbb{E}}\Big{[}\big{\lVert}\bm{H}^{\top}\bm{H}+\bm{S}-(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{-1}\bm{F}^{\top}\bm{F}/n(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}^{\top})^{-1}\big{\rVert}_{\rm{F}}\Big{]}.$

So it suffices to show

(F.1)
${\mathbb{E}}[\lVert\bm{H}^{\top}\bm{H}+\bm{S}-(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{-1}\bm{F}^{\top}\bm{F}/n(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}^{\top})^{-1}\rVert_{\rm{F}}]\leq
n^{-1/2}C(\zeta,T,\gamma){\rm var}(y_{1}).$

To this end, we define

(F.2)
${\widehat{\bm{C}}}=\sum_{j=1}^{p}(\bm{I}_{T}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{I}_{T}\otimes\bm{e}_{j})\in{\mathbb{R}}^{T\times
T},$

where
${\mathcal{M}}=\bm{I}_{pT}+{\mathcal{D}}(\bm{I}_{T}\otimes\tfrac{\bm{X}^{\top}\bm{X}}{n})-{\mathcal{J}}$
as in Lemma C.1. We also define the matrices
$\bm{Q}_{1},\bm{Q}_{2}\in{\mathbb{R}}^{T\times T}$ that are bounded in
Propositions F.1 and F.2 below:

$\displaystyle\bm{Q}_{1}$
$\displaystyle=n^{-1/2}\big{[}\bm{F}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}-(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})(\bm{H}^{\top}\bm{H}+\bm{S})\big{]},$
$\displaystyle\bm{Q}_{2}$
$\displaystyle=n^{-1/2}\big{[}n(\bm{H}^{\top}\bm{H}+\bm{S})-(\bm{I}_{T}+{\widehat{\bm{C}}}/n)\bm{F}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}\big{]}.$

######  Proposition F.1 (Proof is given in Section F.1).

Let Assumptions 2.1, 2.3, 2.2 and 2.4 be fulfilled and
$\bm{\Sigma}=\bm{I}_{p}$, then we have ${\mathbb{E}}[\|\bm{Q}_{1}\|_{\rm
F}^{2}]\leq C(\zeta,T,\gamma){\rm var}(y_{1})^{2}.$ As a consequence, by
Jensen’s inequality, ${\mathbb{E}}[\|\bm{Q}_{1}\|_{\rm F}]\leq
C(\zeta,T,\gamma){\rm var}(y_{1}).$

######  Proposition F.2 (Proof is given in Section F.2).

Let Assumptions 2.1, 2.3, 2.2 and 2.4 be fulfilled and
$\bm{\Sigma}=\bm{I}_{p}$, then
${\mathbb{E}}[\left\lVert\bm{Q}_{2}\right\rVert_{\rm{F}}]\leq
C(\zeta,T,\gamma){\rm var}(y_{1}).$

Now we are ready to prove Equation F.1 using the above two propositions. For
brevity, let
$\bm{V}=[-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}]^{\top}\in{\mathbb{R}}^{(p+1)\times
T}$ and the lower triangular matrices
$\bm{L}=\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n$ and
$\bm{T}=\bm{I}_{T}+{\widehat{\bm{C}}}/n$. With this notation,

(F.3) $\displaystyle\bm{Q}_{1}$
$\displaystyle=\sqrt{n}[(\bm{F}^{\top}\bm{F}/n)\bm{T}^{\top}-\bm{L}\bm{V}^{\top}\bm{V}],$
(F.4) $\displaystyle\bm{Q}_{2}$
$\displaystyle=\sqrt{n}[\bm{V}^{\top}\bm{V}-\bm{T}(\bm{F}^{\top}\bm{F}/n)\bm{T}^{\top}].$

By expanding the expressions of $\bm{Q}_{1}$ and $\bm{Q}_{2}$ in (F.3)-(F.4),
we have by simple algebra

$\displaystyle
n^{-1/2}\bigl{[}\bm{Q}_{1}^{\top}(\bm{L}^{\top})^{-1}+(\bm{L}^{-1}\bm{Q}_{1}+\bm{Q}_{2})(\bm{T}^{\top})^{-1}(\bm{L}^{\top})^{-1}\bigr{]}$
$\displaystyle=\begin{cases}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{T}({\bm{F}^{\top}\bm{F}}/{n})(\bm{L}^{\top})^{-1}}-\bm{V}^{\top}\bm{V}\\\
+\bm{L}^{-1}({\bm{F}^{\top}\bm{F}}/{n})(\bm{L}^{\top})^{-1}-{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\bm{V}^{\top}\bm{V}(\bm{T}^{\top})^{-1}(\bm{L}^{\top})^{-1}}\\\
+{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\bm{V}^{\top}\bm{V}(\bm{T}^{\top})^{-1}(\bm{L}^{\top})^{-1}}-{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{T}(\bm{F}^{\top}\bm{F}/n)(\bm{L}^{\top})^{-1}}\end{cases}$
$\displaystyle=\bm{L}^{-1}(\bm{F}^{\top}\bm{F}/n)(\bm{L}^{\top})^{-1}-\bm{V}^{\top}\bm{V}$
(F.5)
$\displaystyle=(\bm{I}_{T}-\tfrac{1}{n}\smash{\widehat{\mathbf{A}}})^{-1}(\bm{F}^{\top}\bm{F}/n)(\bm{I}_{T}-\tfrac{1}{n}\smash{\widehat{\mathbf{A}}}^{\top})^{-1}-(\bm{H}^{\top}\bm{H}+\bm{S})$

as all terms except two cancel out. Consequently by the triangle inequality,

$\sqrt{n}\lVert\bm{L}^{-1}(\bm{F}^{\top}\bm{F}/n)(\bm{L}^{\top})^{-1}-\bm{V}^{\top}\bm{V}\rVert_{\rm{F}}\leq\max\\{1,\|\bm{L}^{-1}\|_{\rm
op}^{3},\|\bm{T}\|_{\rm op}^{3}\\}(2\|\bm{Q}_{1}\|_{\rm F}+\|\bm{Q}_{2}\|_{\rm
F}).$

Let $\Omega=\\{\bm{X}\in{\mathbb{R}}^{n\times
p}:\lVert\bm{X}\rVert_{\rm{op}}/\sqrt{n}\leq 2+\sqrt{\gamma}\\}$. Under
Assumption 2.2 and here $\bm{\Sigma}=\bm{I}_{p}$, (Davidson and Szarek, 2001,
Theorem II.13) implies that ${\mathbb{P}}(\Omega)\geq 1-e^{-n}$. Then in the
event $\Omega$, we have by Lemmas E.3 and E.4,

(F.6) $\displaystyle\max\\{\|\bm{L}\|_{\rm op},\|\bm{L}^{-1}\|_{\rm
op},\|\bm{T}\|_{\rm op},\|\bm{T}^{-1}\|_{\rm op}\\}\leq C(\zeta,T,\gamma),$

so that by the previous display and Propositions F.1 and F.2,

$\displaystyle{\mathbb{E}}[\mathbb{I}(\Omega)\sqrt{n}\|\bm{L}^{-1}(\bm{F}^{\top}\bm{F}/n)(\bm{L}^{\top})^{-1}-\bm{V}^{\top}\bm{V}\|_{\rm
F}]$ $\displaystyle\leq C(T,\zeta,\gamma){\mathbb{E}}[\|\bm{Q}_{1}\|_{\rm
F}+\|\bm{Q}_{2}\|_{\rm F}]$ (F.7) $\displaystyle\leq C(T,\zeta,\gamma){\rm
var}(y_{1}).$

On the other hand, the same expectation with $\Omega^{c}$ is exponentially
small due to

$\displaystyle{\mathbb{E}}[{\mathbb{I}}(\Omega^{c})\|\eqref{eq:to_bound_FTF_LHTHLT}\|_{\rm
F}]$
$\displaystyle\leq{\mathbb{P}}(\Omega^{c})^{1/2}{\mathbb{E}}[\\{(1+\|\bm{L}\|_{\rm
op}^{2})(\|\bm{F}\|_{\rm F}^{2}/n+\|\bm{V}\|^{2}_{\rm F})\\}^{2}]^{1/2}$ (C.
Schwarz)
$\displaystyle\leq{\mathbb{P}}(\Omega^{c})^{1/2}{\mathbb{E}}[(1+\|\bm{L}\|_{\rm
op}^{2})^{4}]^{1/4}{\mathbb{E}}[(\|\bm{F}\|_{\rm F}^{2}/n+\|\bm{V}\|^{2}_{\rm
F})^{4}]^{1/4}$ (C. Schwarz) (F.8) $\displaystyle\leq
e^{-n/2}C(\zeta,T,\gamma){\rm var}(y_{1})$

thanks to Lemmas E.3, E.4 and E.8 and ${\mathbb{P}}(\Omega)\leq e^{-n}$ for
the last line. This completes the proof of Theorem 2.2 for
$\bm{\Sigma}=\bm{I}_{p}$.

For $\bm{\Sigma}\neq\bm{I}_{p}$, we apply the change of variables argument
presented in Appendix D to achieve the desired result. In this context, the
constant $C$ is dependent on $\zeta,T,\gamma,\kappa$. This concludes the proof
of Theorem 2.2.

In passing, let us mention that we also have by definitions of $\bm{Q}_{1}$
and $\bm{Q}_{2}$ that

(F.9)
$[\bm{L}^{-1}\bm{Q}_{1}(\bm{T}^{\top})^{-1}+\bm{Q}_{2}(\bm{T}^{\top})^{-1}](\bm{L}^{-1}-\bm{T})^{\top}=\sqrt{n}(\bm{L}^{-1}-\bm{T})(\bm{F}^{\top}\bm{F}/n)(\bm{L}^{-1}-\bm{T})^{\top}$

holds. By the same argument as in (F.7)-(F.8), that the right-hand side of the
previous display is bounded as

(F.10)
$\sqrt{n}{\mathbb{E}}[\|(\bm{L}^{-1}-\bm{T})\bm{F}^{\top}n^{-1/2}\|_{\rm
F}^{2}]\leq C(T,\zeta,\gamma){\rm var}(y_{1}).$

###  F.1 Proofs of Proposition F.1

We frist write

$\bm{F}^{\top}\bm{F}=\bm{F}^{\top}\Bigl{[}\bm{X},\frac{{\bm{\varepsilon}}}{\sigma}\Bigr{]}\bigl{[}-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}\bigr{]}^{\top}.$

Applying (Tan et al., 2022, Lemma E.10) to
$\bm{U}=\bm{F}\in{\mathbb{R}}^{n\times T}$,
$\bm{Z}=[\bm{X},{\bm{\varepsilon}}/\sigma]\in{\mathbb{R}}^{n\times(p+1)}$, and
$\bm{V}=\bigl{[}-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}\bigr{]}^{\top}\in{\mathbb{R}}^{(p+1)\times
T}$ gives

(F.11)
$\displaystyle{\mathbb{E}}\Big{[}\Big{\|}\bm{U}^{\top}\bm{Z}\bm{V}-\sum_{j=1}^{p+1}\sum_{i=1}^{n}\frac{\partial}{\partial
z_{ij}}\Bigl{(}\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V}\Bigr{)}\Big{\|}_{\rm
F}^{2}\Big{]}$ (F.12) $\displaystyle\leq~{}$
$\displaystyle{\mathbb{E}}[\left\lVert\bm{U}\right\rVert_{\rm{F}}^{2}\left\lVert\bm{V}\right\rVert_{\rm{F}}^{2}]+{\mathbb{E}}\sum_{i=1}^{n}\sum_{j=1}^{p+1}\Big{[}2\left\lVert\bm{V}\right\rVert_{\rm{F}}^{2}\left\lVert\frac{\partial\bm{U}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}+2\left\lVert\bm{U}\right\rVert_{\rm{F}}^{2}\left\lVert\frac{\partial\bm{V}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}.$

To prove Proposition F.1, we need the following two lemmas.

######  Lemma F.3 (Proof is given on Section F.3).

Let the assumptions of Proposition F.1 be fulfilled. For $\bm{U}=\bm{F}$,
$\bm{Z}=[\bm{X},{\bm{\varepsilon}}/\sigma]$, and
$\bm{V}=\bigl{[}-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}\bigr{]}^{\top}$, we have

$\displaystyle\bm{U}^{\top}\bm{Z}\bm{V}-\sum_{j=1}^{p+1}\sum_{i=1}^{n}\frac{\partial}{\partial
z_{ij}}\Bigl{(}\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V}\Bigr{)}=\sqrt{n}\bm{Q}_{1}+{\rm
Rem}_{1},$

where ${\rm Rem}_{1}$ is a $T\times T$ matrix satisfying
${\mathbb{E}}[\lVert{\rm Rem}_{1}\rVert_{\rm{F}}^{2}]\leq
nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}.$

######  Lemma F.4 (Proof is given on Section F.3).

Let the assumptions of Proposition F.1 be fulfilled, then

$\eqref{UZV-RHS}\leq nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}.$

Now we prove Proposition F.1 using the above two lemmas. According to Lemma
F.3, we have
$\sqrt{n}\bm{Q}_{1}=\bm{U}^{\top}\bm{Z}\bm{V}-\sum_{j=1}^{p+1}\sum_{i=1}^{n}\frac{\partial}{\partial
z_{ij}}(\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V})-{\rm Rem}_{1}$. By
triangular inequality, we obtain

$\displaystyle n{\mathbb{E}}[\lVert\bm{Q}_{1}\rVert_{\rm{F}}^{2}]$
$\displaystyle\leq$ $\displaystyle
2{\mathbb{E}}\Big{[}\lVert\bm{U}^{\top}\bm{Z}\bm{V}-\sum_{j=1}^{p+1}\sum_{i=1}^{n}\frac{\partial}{\partial
z_{ij}}(\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V})\rVert_{\rm{F}}^{2}\Big{]}+2{\mathbb{E}}[\lVert{\rm
Rem}_{1}\rVert_{\rm{F}}^{2}]$ $\displaystyle\leq$ $\displaystyle
2\Bigl{(}\eqref{UZV-RHS}+{\mathbb{E}}[\lVert{\rm
Rem}_{1}\rVert_{\rm{F}}^{2}]\Bigr{)}$ $\displaystyle\leq$ $\displaystyle
nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}$ by Lemmas F.3 and F.4.

This completes the proof of Proposition F.1.

###  F.2 Proofs of Proposition F.2

We first state a useful lemma, which is an extension of Lemma E.12 in Tan et
al. (2022) to allow $\lVert\bm{U}\rVert_{\rm{F}}\geq 1$ and
$\lVert\bm{V}\rVert_{\rm{F}}\geq 1$.

######  Lemma F.5 (Proof is given on Section F.3).

Let $\bm{U},\bm{V}:{\mathbb{R}}^{n\times p}\to{\mathbb{R}}^{n\times T}$ be two
locally Lipschitz functions of $\bm{Z}$ with i.i.d. $\mathsf{N}(0,1)$ entries.
Provided the following expectations are finite, we have

$\displaystyle{\mathbb{E}}\Bigl{[}\Big{\lVert}p\bm{U}^{\top}\bm{V}-\sum_{j=1}^{p}\Bigl{(}\sum_{i=1}^{n}\partial_{ij}\bm{U}^{\top}\bm{e}_{i}-\bm{U}^{\top}\bm{Z}\bm{e}_{j}\Bigr{)}\Bigl{(}\sum_{i=1}^{n}\partial_{ij}\bm{e}_{i}^{\top}\bm{V}-\bm{e}_{j}^{\top}\bm{Z}^{\top}\bm{V}\Bigr{)}\Big{\rVert}_{\rm{F}}\Bigr{]}$
$\displaystyle\leq~{}$
$\displaystyle(1+2\sqrt{p})\bigl{(}{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\lVert\bm{U}\rVert^{4}_{\partial}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert^{4}_{\partial}]^{1/2}\bigr{)},$

where $\partial_{ij}\bm{U}=\partial\bm{U}/\partial z_{ij}$ and
$\|\bm{U}\|_{\partial}=(\sum_{i=1}^{n}\sum_{j=1}^{p}\|\partial_{ij}\bm{U}\|_{\rm
F}^{2})^{1/2}$.

To apply Lemma F.5, we consider the following mapping:

${\mathbb{R}}^{(p+1)\times n}\to{\mathbb{R}}^{(p+1)\times
T}:\bm{Z}^{\top}\mapsto\bm{V},$

where $\bm{Z}=[\bm{X},\frac{{\bm{\varepsilon}}}{\sigma}]$ and
$\bm{V}=[-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}]^{\top}$. Applying Lemma F.5 to
$\bm{U}=\bm{V}=[-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}]^{\top}$ and the
Gaussian matrix $\bm{Z}^{\top}$, we have

(F.13)
$\displaystyle{\mathbb{E}}\Bigl{[}\Big{\lVert}n\bm{V}^{\top}\bm{V}-\sum_{i=1}^{n}\Bigl{(}\sum_{j=1}^{p+1}\pdv{\bm{V}^{\top}}{z_{ij}}\bm{e}_{j}-\bm{V}^{\top}\bm{Z}^{\top}\bm{e}_{i}\Bigr{)}\Bigl{(}\sum_{j=1}^{p+1}\pdv{\bm{e}_{j}^{\top}\bm{V}}{z_{ij}}-\bm{e}_{i}^{\top}\bm{Z}\bm{V}\Bigr{)}\Big{\rVert}_{\rm{F}}\Bigr{]}$
$\displaystyle\leq{}$ $\displaystyle
2(1+2\sqrt{n})\bigl{(}{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert^{4}_{\partial}]^{1/2}\bigr{)},$

where
$\lVert\bm{V}\rVert_{\partial}:=(\sum_{i=1}^{n}\sum_{j=1}^{p+1}\lVert\pdv{\bm{V}}{z_{ij}}\rVert_{\rm{F}}^{2})^{1/2}$.
The desired bound of ${\mathbb{E}}[\lVert\bm{Q}_{2}\rVert_{\rm{F}}]$ then
follows from the subsequent two lemmas.

######  Lemma F.6 (Proof is given on Section F.3).

Let the assumptions of Proposition F.2 be fulfilled. For
$\bm{Z}=[\bm{X},\frac{{\bm{\varepsilon}}}{\sigma}]$ and
$\bm{V}=[-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}]^{\top}$, we have

(F.14) $\displaystyle
n\bm{V}^{\top}\bm{V}-\sum_{i=1}^{n}\Bigl{(}\sum_{j=1}^{p+1}\pdv{\bm{V}^{\top}}{z_{ij}}\bm{e}_{j}-\bm{V}^{\top}\bm{Z}^{\top}\bm{e}_{i}\Bigr{)}\Bigl{(}\sum_{j=1}^{p+1}\pdv{\bm{e}_{j}^{\top}\bm{V}}{z_{ij}}-\bm{e}_{i}^{\top}\bm{Z}\bm{V}\Bigr{)}=\sqrt{n}\bm{Q}_{2}-{\rm
Rem}_{2},$

where ${\rm Rem}_{2}$ is a $T\times T$ matrix satisfying
${\mathbb{E}}[\lVert{\rm
Rem}_{2}\rVert_{\rm{F}}]\leq\sqrt{n}C(\zeta,T,\gamma){\rm var}(y_{1})$.

######  Lemma F.7 (Proof is given on Section F.3).

Under the same conditions of Proposition F.2, for
$\bm{V}=[-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}]^{\top}$, we have

$\displaystyle
2(1+2\sqrt{n})\bigl{(}{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert^{4}_{\partial}]^{1/2}\bigr{)}$
$\displaystyle\leq\sqrt{n}C(\zeta,T,\gamma){\rm var}(y_{1}).$

Now we are ready to prove Proposition F.2. By the triangle inequality, we have

$\displaystyle{\mathbb{E}}[\lVert\sqrt{n}\bm{Q}_{2}\rVert_{\rm{F}}]$
$\displaystyle\leq$ $\displaystyle{\mathbb{E}}[\lVert{\rm
Rem}_{2}\rVert_{\rm{F}}]+2(1+2\sqrt{n})\bigl{(}{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert^{4}_{\partial}]^{1/2}\bigr{)}$
by (F.13) and (F.14) $\displaystyle\leq$
$\displaystyle\sqrt{n}C(\zeta,T,\gamma){\rm var}(y_{1})$
$\displaystyle\mbox{by \lx@cref{creftypepluralcap~refnum}{lem:A2-Rem}
and~\lx@cref{refnum}{lem:A2-RHS}}.$

This finishes the proof of Proposition F.2.

###  F.3 Proofs of supporting lemmas

###### Proof of Lemma F.3.

First, by definitions of $\bm{U},\bm{Z},\bm{V}$ in Lemma F.3, we have

$\bm{U}^{\top}\bm{Z}\bm{V}=\bm{F}^{\top}\bm{F}.$

Next, by product rule and spliting the summation over $j$ into two parts:
$j\in[p]$ and $j=p+1$, we have

$\displaystyle\sum_{j=1}^{p+1}\sum_{i=1}^{n}\frac{\partial}{\partial
z_{ij}}\Bigl{(}\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V}\Bigr{)}$
$\displaystyle=$
$\displaystyle\sum_{j=1}^{p+1}\sum_{i=1}^{n}\Bigl{(}\frac{\partial\bm{U}^{\top}}{\partial
z_{ij}}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V}+\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial\bm{V}}{\partial
z_{ij}}\Bigr{)}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{p}\sum_{i=1}^{n}\Bigl{(}\frac{\partial\bm{F}^{\top}}{\partial
x_{ij}}\bm{e}_{i}\bm{e}_{j}^{\top}(-\bm{H})\Bigr{)}+\sum_{i=1}^{n}\pdv{\bm{F}^{\top}}{\epsilon_{i}/\sigma}\bm{e}_{i}\sigma{\mathbf{1}}_{T}^{\top}-\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial\bm{H}}{\partial
x_{ij}}$ (F.15) $\displaystyle=$
$\displaystyle-\sum_{j=1}^{p}\sum_{i=1}^{n}\Bigl{(}\frac{\partial\bm{F}^{\top}}{\partial
x_{ij}}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{H}\Bigr{)}+\sigma^{2}\sum_{i=1}^{n}\pdv{\bm{F}^{\top}}{\epsilon_{i}}\bm{e}_{i}{\mathbf{1}}_{T}^{\top}-\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial\bm{H}}{\partial
x_{ij}}.$

For the first term in (F.15), we have its transpose is

$\displaystyle-\bm{H}^{\top}\sum_{j=1}^{p}\sum_{i=1}^{n}\bm{e}_{j}\bm{e}_{i}^{\top}\frac{\partial\bm{F}}{\partial
x_{ij}}=\bm{H}^{\top}\bm{H}(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{\top}-{\rm
Rem}_{1,1}$

by (F.16). For the second term in (F.15), we have its transpose is

$\displaystyle\sigma^{2}{\mathbf{1}}_{T}\sum_{i=1}^{n}\bm{e}_{i}^{\top}\pdv{\bm{F}}{\epsilon_{i}}$
$\displaystyle=$
$\displaystyle\sigma^{2}{\mathbf{1}}_{T}\sum_{i=1}^{n}\sum_{t=1}^{T}\bm{e}_{i}^{\top}\pdv{\bm{F}\bm{e}_{t}}{\epsilon_{i}}\bm{e}_{t}^{\top}$
$\displaystyle=$
$\displaystyle\sigma^{2}{\mathbf{1}}_{T}\sum_{t=1}^{T}\operatorname{Tr}\Big{(}\pdv{\bm{F}\bm{e}_{t}}{{\bm{\varepsilon}}}\Big{)}\bm{e}_{t}^{\top}$
$\displaystyle=$
$\displaystyle\sigma^{2}{\mathbf{1}}_{T}\sum_{t=1}^{T}\bm{e}_{t}^{\top}(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}){\mathbf{1}}_{T}\bm{e}_{t}^{\top}$
by (C.8) $\displaystyle=$
$\displaystyle\sigma^{2}{\mathbf{1}}_{T}{\mathbf{1}}_{T}^{\top}(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{\top}$
$\displaystyle=$
$\displaystyle\bm{S}(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{\top}.$

For the third term in (F.15), we have by (F.17),

$\displaystyle\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial\bm{H}}{\partial
x_{ij}}=\bm{F}^{\top}\bm{F}{\widehat{\bm{C}}}^{\top}/n+{\rm Rem}_{1,2}.$

Combining the three terms in (F.15), we have

$\displaystyle\eqref{abc}=(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})(\bm{H}^{\top}\bm{H}+\bm{S})-\bm{F}^{\top}\bm{F}{\widehat{\bm{C}}}^{\top}/n-{\rm
Rem}_{1,1}^{\top}-{\rm Rem}_{1,2}.$

It follows that

$\displaystyle\bm{U}^{\top}\bm{Z}\bm{V}-\sum_{j=1}^{p+1}\sum_{i=1}^{n}\frac{\partial}{\partial
z_{ij}}\Bigl{(}\bm{U}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\bm{V}\Bigr{)}$
$\displaystyle=$
$\displaystyle\bm{F}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}-(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})(\bm{H}^{\top}\bm{H}+\bm{S})+({\rm
Rem}_{1,1}^{\top}+{\rm Rem}_{1,2})$ $\displaystyle=$
$\displaystyle\sqrt{n}\bm{Q}_{1}+{\rm Rem}_{1},$

where ${\rm Rem}_{1}=({\rm Rem}_{1,1}^{\top}+{\rm Rem}_{1,2})$, and ${\rm
Rem}_{1,1}$ and ${\rm Rem}_{1,2}$ are defined in Lemma F.8.

It remains to bound ${\mathbb{E}}[\lVert{\rm Rem}_{1}\rVert_{\rm{F}}^{2}]$. By
the triangle inequality,

$\displaystyle{\mathbb{E}}[\lVert{\rm Rem}_{1}\rVert_{\rm{F}}^{2}]\leq$
$\displaystyle 2{\mathbb{E}}[\lVert{\rm
Rem}_{1,1}\rVert_{\rm{F}}^{2}]+2{\mathbb{E}}[\lVert{\rm
Rem}_{1,2}\rVert_{\rm{F}}^{2}]$ $\displaystyle\leq$ $\displaystyle
nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}$ by (F.20).

This concludes the proof of Lemma F.3. ∎

###### Proof of Lemma F.4.

Let us recall (F.12) here for convenience:

$\displaystyle\eqref{UZV-
RHS}={\mathbb{E}}\Bigl{[}\left\lVert\bm{U}\right\rVert_{\rm{F}}^{2}\left\lVert\bm{V}\right\rVert_{\rm{F}}^{2}\Bigr{]}+2{\mathbb{E}}\Big{[}\left\lVert\bm{V}\right\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\sum_{j=1}^{p+1}\left\lVert\frac{\partial\bm{U}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}+2{\mathbb{E}}\Big{[}\left\lVert\bm{U}\right\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\sum_{j=1}^{p+1}\left\lVert\frac{\partial\bm{V}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}.$

For the first term in (F.12), since
$\lVert\bm{V}\rVert_{\rm{F}}^{2}=\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2}$,
we have

$\displaystyle{\mathbb{E}}[\left\lVert\bm{U}\right\rVert_{\rm{F}}^{2}\left\lVert\bm{V}\right\rVert_{\rm{F}}^{2}]=$
$\displaystyle{\mathbb{E}}[\lVert\bm{F}\rVert_{\rm{F}}^{2}(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})]$
$\displaystyle\leq$
$\displaystyle{\mathbb{E}}[\lVert\bm{F}\rVert_{\rm{F}}^{4}]^{1/2}{\mathbb{E}}[\lVert\bm{H}\rVert_{\rm{F}}^{4}+T^{2}\sigma^{4}]^{1/2}$
by C. Schwarz $\displaystyle\leq$ $\displaystyle nC(\zeta,T,\gamma){\rm
var}(y_{1})^{2}$ by Lemma E.10.

For the second term in (F.12), we have

$\displaystyle{\mathbb{E}}\Big{[}\left\lVert\bm{V}\right\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\sum_{j=1}^{p+1}\left\lVert\frac{\partial\bm{U}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}$ $\displaystyle=$
$\displaystyle{\mathbb{E}}\Big{[}(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})\Big{(}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\frac{\partial\bm{F}}{\partial
x_{ij}}\right\rVert_{\rm{F}}^{2}+\sigma^{2}\sum_{i=1}^{n}\left\lVert\pdv{\bm{F}}{\epsilon_{i}}\right\rVert_{\rm{F}}^{2}\Big{)}\Big{]}$
$\displaystyle=$
$\displaystyle{\mathbb{E}}\Big{[}(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})\Big{(}\left\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\right\rVert_{\rm{F}}^{2}+\sigma^{2}\left\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{{\bm{\varepsilon}}}\right\rVert_{\rm{F}}^{2}\Big{)}\Big{]}$
$\displaystyle\leq$
$\displaystyle{\mathbb{E}}\Big{[}(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})\Big{(}\left\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{\operatorname{\mathbf{vec}}(\bm{X})}\right\rVert_{\rm{F}}^{2}+\sigma^{2}\left\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{F})}{{\bm{\varepsilon}}}\right\rVert_{\rm{F}}^{2}\Big{)}\Big{]}$
$\displaystyle\leq$ $\displaystyle
C(\zeta,T){\mathbb{E}}[(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}(\|\bm{H}\|_{\rm
F}^{2}+\gamma\|\bm{F}\|_{\rm F}^{2}/n)]$ by Lemma E.11
$\displaystyle+n\sigma^{2}C(\zeta,T){\mathbb{E}}[(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}]$
by (E.7) $\displaystyle\leq$ $\displaystyle nC(\zeta,T,\gamma){\rm
var}(y_{1})^{2}$ by Lemmas E.10 and E.8.

For the third term in (F.12), since $\bm{V}=\begin{bmatrix}-\bm{H}\\\
\sigma{\mathbf{1}}_{T}^{\top}\end{bmatrix}$, we have

$\displaystyle{\mathbb{E}}\Big{[}\left\lVert\bm{U}\right\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\sum_{j=1}^{p+1}\left\lVert\frac{\partial\bm{V}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}$ $\displaystyle=$
$\displaystyle{\mathbb{E}}\Big{[}\lVert\bm{F}\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\sum_{j=1}^{p+1}\left\lVert\frac{\partial\bm{H}}{\partial
z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}$ by $\pdv{{\mathbf{1}}_{T}}{z_{ij}}=0$
$\displaystyle=$
$\displaystyle{\mathbb{E}}\Big{[}\lVert\bm{F}\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\frac{\partial\bm{H}}{\partial
x_{ij}}\right\rVert_{\rm{F}}^{2}+\sigma^{2}\lVert\bm{F}\rVert_{\rm{F}}^{2}\sum_{i=1}^{n}\left\lVert\pdv{\bm{H}}{\epsilon_{i}}\right\rVert_{\rm{F}}^{2}\Big{]}$
$\displaystyle=$
$\displaystyle{\mathbb{E}}\Big{[}\lVert\bm{F}\rVert_{\rm{F}}^{2}\left\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{\operatorname{\mathbf{vec}}(\bm{X})}\right\rVert_{\rm{F}}^{2}+\sigma^{2}\lVert\bm{F}\rVert_{\rm{F}}^{2}\left\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{{\bm{\varepsilon}}}\right\rVert_{\rm{F}}^{2}\Big{]}$
$\displaystyle\leq$ $\displaystyle
C(\zeta,T){\mathbb{E}}\Big{[}\lVert\bm{F}\rVert_{\rm{F}}^{2}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}\bigl{(}\lVert\bm{F}n^{-1/2}\rVert_{\rm{F}}\sqrt{\gamma}+\lVert\bm{H}\rVert_{\rm{F}}\bigr{)}^{2}\Big{]}$
by (E.13)
$\displaystyle+C(\zeta,T){\mathbb{E}}\Big{[}\sigma^{2}\lVert\bm{F}\rVert_{\rm{F}}^{2}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}\Big{]}$
by (E.8) $\displaystyle\leq$ $\displaystyle nC(\zeta,T,\gamma){\rm
var}(y_{1})^{2}$ by Lemmas E.10 and E.8.

Combining the above three bounds gives the desired bound for (F.12). This
concludes the proof. ∎

######  Lemma F.8 (Proof is given on Section F.3).

Under the same conditions of Proposition F.1, we have

(F.16)
$\displaystyle\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{e}_{j}\bm{e}_{i}^{\top}\frac{\partial\bm{F}}{\partial
x_{ij}}$
$\displaystyle=-\bm{H}^{\top}\bm{H}(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}^{\top})+{\rm
Rem}_{1,1},$ (F.17)
$\displaystyle\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial\bm{H}}{\partial
x_{ij}}$ $\displaystyle=\bm{F}^{\top}\bm{F}{\widehat{\bm{C}}}^{\top}/n+{\rm
Rem}_{1,2},$

where

(F.18) $\displaystyle{\rm Rem}_{1,1}$
$\displaystyle=\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{e}_{j}\sum_{t}\Delta_{ij}^{it}\bm{e}_{t}^{\top},$
(F.19) $\displaystyle{\rm Rem}_{1,2}$
$\displaystyle=-n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\bm{e}_{t}^{\top}.$

In addition, we have

(F.20) $\displaystyle{\mathbb{E}}[\lVert{\rm Rem}_{1,1}\rVert_{\rm{F}}^{2}]$
$\displaystyle\leq nC(\zeta,T,\gamma){\rm var}(y_{1})^{2},$
$\displaystyle{\mathbb{E}}[\lVert{\rm Rem}_{1,2}\rVert_{\rm{F}}^{2}]$
$\displaystyle\leq nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}.$

###### Proof of Lemma F.5.

For each $j\in[p]$, let ${\mathbb{E}}_{j}[\cdot]$ denote the conditional
expectation ${\mathbb{E}}[\cdot|\\{\bm{Z}\bm{e}_{k},k\neq j\\}]$. The left-
hand side of the desired inequality can be rewritten as

$\displaystyle{\mathbb{E}}\Big{[}\big{\lVert}p\bm{U}^{\top}\bm{V}-\sum_{j=1}^{p}({\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}-\bm{L}^{\top})\bm{e}_{j}\bm{e}_{j}^{\top}(\bm{Z}^{\top}{\mathbb{E}}_{j}[\bm{V}]-\hat{\bm{L}})\big{\rVert}_{\rm{F}}\Big{]}$

with $\bm{L}\in{\mathbb{R}}^{p\times T}$ defined by
$\bm{L}^{\top}\bm{e}_{j}={\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}-\bm{U}^{\top}\bm{Z}\bm{e}_{j}+\sum_{i=1}^{n}\partial_{ij}\bm{U}^{\top}\bm{e}_{i}$
and $\hat{\bm{L}}$ defined similarly with $\bm{U}$ replaced by $\bm{V}$. We
develop the terms in the sum over $j$ as follows:

$\displaystyle
p\bm{U}^{\top}\bm{V}-\sum_{j}({\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}-\bm{L}^{\top})\bm{e}_{j}\bm{e}_{j}^{\top}(\bm{Z}^{\top}{\mathbb{E}}_{j}[\bm{V}]-\hat{\bm{L}})$
(F.21) $\displaystyle=\quad$
$\displaystyle\sum_{j}\Bigl{(}\bm{U}^{\top}\bm{V}-{\mathbb{E}}_{j}[\bm{U}]^{\top}{\mathbb{E}}_{j}[\bm{V}]\Bigr{)}$
(F.22)
$\displaystyle+\sum_{j}\Bigl{(}{\mathbb{E}}_{j}[\bm{U}]^{\top}{\mathbb{E}}_{j}[\bm{V}]-{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\bm{e}_{j}^{\top}\bm{Z}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Bigr{)}$
(F.23) $\displaystyle-\bm{L}^{\top}\hat{\bm{L}}$ (F.24)
$\displaystyle+\sum_{j}\Bigl{(}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\bm{e}_{j}^{\top}\hat{\bm{L}}\Bigr{)}+\Bigl{(}\bm{L}^{\top}\bm{e}_{j}\bm{e}_{j}^{\top}\bm{Z}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Bigr{)}.$

The following proof is dedicated to bounding the expectation of Frobenius norm
of each term in (F.21)-(F.24). We now bound (F.23), (F.24), (F.21), and (F.22)
in order.

For (F.23). We have by the Cauchy-Schwarz inequality
${\mathbb{E}}\bigl{[}\|\bm{L}^{\top}\hat{\bm{L}}\|_{\rm
F}\bigr{]}\leq{\mathbb{E}}\bigl{[}\|\bm{L}\|_{\rm
F}^{2}\bigr{]}^{\frac{1}{2}}{\mathbb{E}}\bigl{[}\|\hat{\bm{L}}\|_{\rm
F}^{2}\bigr{]}^{\frac{1}{2}}$. For a fixed $j\in[p]$ and $t\in[T]$,

$\displaystyle{\mathbb{E}}[(\bm{e}_{j}^{\top}\bm{L}\bm{e}_{t})^{2}]$
$\displaystyle\leq\sum_{i=1}^{n}{\mathbb{E}}[(\bm{e}_{i}^{\top}({\mathbb{E}}_{j}[\bm{U}]-\bm{U})\bm{e}_{t})^{2}]+{\mathbb{E}}\sum_{i=1}^{n}\sum_{l=1}^{n}\Bigl{(}\frac{\bm{e}_{i}^{\top}\partial\bm{U}\bm{e}_{t}}{\partial
z_{lj}}\Bigr{)}^{2}$ $\displaystyle\leq
2{\mathbb{E}}\sum_{i=1}^{n}\sum_{l=1}^{n}\Bigl{(}\frac{\bm{e}_{i}^{\top}\partial\bm{U}\bm{e}_{t}}{\partial
z_{lj}}\Bigr{)}^{2},$

where the two inequalities are due to the second-order stein inequality in
Bellec and Zhang (2021), and Gaussian-Poincaré inequality in (Boucheron et
al., 2013, Theorem 3.20), respectively. Summing over $j\in[p]$ and $t\in[T]$
we obtain ${\mathbb{E}}[\|\bm{L}\|_{\rm F}^{2}]\leq
2{\mathbb{E}}\sum_{lj}\|\partial_{lj}\bm{U}\|_{\rm
F}^{2}=2{\mathbb{E}}[\|\bm{U}\|_{\partial}^{2}]$. Combined with the same bound
for $\hat{\bm{L}}$, we obtain

${\mathbb{E}}[\|\eqref{eq:Xi-hat-Xi-term}\|_{\rm F}]\leq
2{\mathbb{E}}[\|\bm{U}\|^{2}_{\partial}]^{1/2}{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]^{1/2}.$

For (F.24). We focus on the first term in (F.24); the similar bound applies to
the second term by exchanging the role of $\bm{U}$ and $\bm{V}$. For the first
term, we have

$\displaystyle{\mathbb{E}}\Bigl{[}\|\sum_{j}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\bm{e}_{j}^{\top}\hat{\bm{L}}\|_{\rm
F}\Bigr{]}$
$\displaystyle\leq\sum_{j}{\mathbb{E}}\Bigl{[}\|{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\|_{2}\|\bm{e}_{j}^{\top}\hat{\bm{L}}\|_{2}\Bigr{]}$
$\displaystyle\leq{\mathbb{E}}[\sum_{j}\|{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\|_{2}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\sum_{j}\|\bm{e}_{j}^{\top}\hat{\bm{L}}\|_{2}^{2}]^{\frac{1}{2}}$
$\displaystyle\leq(p{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}])^{\frac{1}{2}}{\mathbb{E}}[\|\hat{\bm{L}}\|_{\rm
F}^{2}]^{\frac{1}{2}}$
$\displaystyle\leq(p{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}])^{\frac{1}{2}}\sqrt{2}{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]^{1/2},$

where we used that $\|\bm{a}\bm{b}^{\top}\|_{\rm
F}=\|\bm{a}\|_{2}\|\bm{b}\|_{2}$ for two vectors $\bm{a},\bm{b}$, the Cauchy-
Schwarz inequality,
${\mathbb{E}}[\|\bm{A}\bm{z}_{j}\|_{2}^{2}|\bm{A}]=\|\bm{A}\|_{\rm F}^{2}$ if
matrix $\bm{A}$ is independent of $\bm{z}_{j}\sim\mathsf{N}(0,\bm{I}_{n})$
(set $\bm{z}_{j}=\bm{Z}\bm{e}_{j}$), and Jensen’s inequality. By symmetry of
the two terms in (F.24), we have

$\displaystyle{\mathbb{E}}\|\eqref{eq:last-two-terms}\|_{\rm
F}\leq\sqrt{2p}{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]^{1/2}+\sqrt{2p}{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\|\bm{U}\|_{\partial}^{2}]^{1/2}.$

For (F.21). We decompose (F.21) as
$\sum_{j}\bm{U}^{\top}(\bm{V}-{\mathbb{E}}_{j}[\bm{V}])+\sum_{j}(\bm{U}-{\mathbb{E}}_{j}[\bm{U}])^{\top}{\mathbb{E}}_{j}[\bm{V}]$.
We focus on the left term; similar bound will apply to the second term by
exchanging the roles of $\bm{V}$ and $\bm{U}$. For the first term, we have by
the submultiplicativity of the Frobenius norm and the Cauchy-Schwarz
inequality

$\displaystyle{\mathbb{E}}[\|\bm{U}^{\top}\sum_{j}(\bm{V}-{\mathbb{E}}_{j}[\bm{V}])\|_{\rm
F}]$
$\displaystyle\leq{\mathbb{E}}[\sum_{j}\lVert\bm{U}\rVert_{\rm{F}}\|\bm{V}-{\mathbb{E}}_{j}[\bm{V}]\|_{\rm
F}]$
$\displaystyle\leq{\mathbb{E}}[p\lVert\bm{U}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\sum_{j}\|\bm{V}-{\mathbb{E}}_{j}[\bm{V}]\|_{\rm
F}^{2}]^{\frac{1}{2}}.$

By the Gaussian Poincaré inequality applied $p$ times,
${\mathbb{E}}[\sum_{j}\|\bm{V}-{\mathbb{E}}_{j}[\bm{V}]\|_{\rm
F}^{2}]\leq{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]$, so that the previous
display is bounded from above by
$\sqrt{p}{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\lVert\bm{V}\rVert^{2}_{\partial}]^{\frac{1}{2}}$
. Using the same argument, we have

$\displaystyle{\mathbb{E}}[\lVert\sum_{j}(\bm{U}-{\mathbb{E}}_{j}[\bm{U}])^{\top}{\mathbb{E}}_{j}[\bm{V}]\rVert_{\rm{F}}]\leq\sqrt{p}{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\lVert\bm{U}\rVert^{2}_{\partial}]^{\frac{1}{2}}.$

Hence,

$\displaystyle{\mathbb{E}}[\|\eqref{eq:first-term-swap-trick}\|_{\rm
F}]\leq\sqrt{p}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{2}]^{1/2}{\mathbb{E}}[\lVert\bm{V}\rVert^{2}_{\partial}]^{\frac{1}{2}}+\sqrt{p}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{2}]^{1/2}{\mathbb{E}}[\lVert\bm{U}\rVert^{2}_{\partial}]^{\frac{1}{2}}.$

For (F.22). We first use ${\mathbb{E}}[\|\eqref{eq:difficult-term}\|_{\rm
F}]\leq{\mathbb{E}}[\|\eqref{eq:difficult-term}\|_{\rm F}^{2}]^{\frac{1}{2}}$
by Jensen’s inequality and now proceed to bound
${\mathbb{E}}\|\eqref{eq:difficult-term}\|_{\rm F}^{2}$. We have

$\|\eqref{eq:difficult-term}\|_{\rm
F}^{2}=\|\sum_{j}{\mathbb{E}}_{j}[\bm{U}]^{\top}{\mathbb{E}}_{j}[\bm{V}]-{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\bm{e}_{j}^{\top}\bm{Z}^{\top}{\mathbb{E}}_{j}[\bm{V}]\|_{\rm
F}^{2}=\sum_{j,k}\operatorname{Tr}[\bm{M}_{j}^{\top}\bm{M}_{k}],$

where
$\bm{M}_{j}={\mathbb{E}}_{j}[\bm{U}]^{\top}{\mathbb{E}}_{j}[\bm{V}]-{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{Z}\bm{e}_{j}\bm{e}_{j}^{\top}\bm{Z}^{\top}{\mathbb{E}}_{j}[\bm{V}]$.
For the summation $\sum_{j,k}$, we split it into two cases $\sum_{j=k}$ and
$\sum_{j\neq k}$. We first bound ${\mathbb{E}}[\sum_{j}\|\bm{M}_{j}\|_{\rm
F}^{2}]$. Since the variance of
$\bm{a}^{\top}\bm{b}-\bm{a}^{\top}\bm{g}\bm{g}^{\top}\bm{b}$ for standard
normal $\bm{g}\sim\mathsf{N}(0,I_{p})$ is
$2\|(\bm{a}\bm{b}^{\top}+\bm{b}\bm{a}^{\top})/2\|_{\rm F}^{2}\leq
2\|\bm{a}\|_{2}^{2}\|\bm{b}\|_{2}^{2}$, applying this variance bound on each
pair of coordinates $(t,t^{\prime})\in[T]\times[T]$ gives
$\sum_{j}\|\bm{M}_{j}\|_{\rm
F}^{2}\leq\sum_{j}2\|{\mathbb{E}}_{j}[\bm{U}]\|_{\rm
F}^{2}\|{\mathbb{E}}_{j}[\bm{V}]\|_{\rm F}^{2}$. Hence, using the Cauchy-
Schwarz inequality and Jensen’s inequality, we have

$\displaystyle{\mathbb{E}}[\sum_{j}\|\bm{M}_{j}\|_{\rm F}^{2}]\leq
2\sum_{j}{\mathbb{E}}[\|{\mathbb{E}}_{j}[\bm{U}]\|_{\rm
F}^{2}\|{\mathbb{E}}_{j}[\bm{V}]\|_{\rm F}^{2}]\leq
2p{\mathbb{E}}[\|\bm{U}\|_{\rm F}^{4}]^{1/2}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/2}.$

We now bound $\sum_{j\neq k}\operatorname{Tr}[\bm{M}_{j}^{\top}\bm{M}_{k}]$.
Setting $\bm{z}_{j}=\bm{Z}\bm{e}_{j}\sim\mathsf{N}(0,\bm{I}_{n})$ for every
$j\in[p]$, we will use many times the identity

(F.25)
${\mathbb{E}}[(\bm{z}_{j}^{\top}f(\bm{Z})-\sum_{i}\partial_{ij}f(\bm{Z})^{\top}\bm{e}_{i})g(\bm{Z})={\mathbb{E}}[\sum_{i}f(\bm{Z})^{\top}\bm{e}_{i}\partial_{ij}g(\bm{Z})]$

which follows from Stein’s formula for $f:{\mathbb{R}}^{n\times
p}\to{\mathbb{R}}^{n}$ and $g:{\mathbb{R}}^{n\times p}\to{\mathbb{R}}$. With
$f^{tt^{\prime}}(\bm{Z})=(\bm{z}_{j}^{\top}{\mathbb{E}}_{j}[\bm{U}]\bm{e}_{t^{\prime}}){\mathbb{E}}_{j}[\bm{V}]\bm{e}_{t}$
and $g^{tt^{\prime}}(\bm{Z})=\bm{e}_{t^{\prime}}^{\top}\bm{M}_{k}\bm{e}_{t}$,
we find

$\displaystyle{\mathbb{E}}\operatorname{Tr}[\bm{M}_{j}^{\top}\bm{M}_{k}]={\mathbb{E}}\operatorname{Tr}[\bm{M}_{j}^{\top}\sum_{t^{\prime}}\bm{e}_{t^{\prime}}\bm{e}_{t^{\prime}}^{\top}\bm{M}_{k}]={\mathbb{E}}[\sum_{tt^{\prime}}\bm{e}_{t}^{\top}\bm{M}_{j}^{\top}\bm{e}_{t^{\prime}}\bm{e}_{t^{\prime}}^{\top}\bm{M}_{k}\bm{e}_{t}]$
$\displaystyle={\mathbb{E}}\sum_{tt^{\prime}}\Bigl{(}\bm{z}_{j}^{\top}f^{tt^{\prime}}(\bm{Z})-\sum_{i}\bm{e}_{i}^{\top}\partial_{ij}f^{tt^{\prime}}(\bm{Z})\Bigr{)}g^{tt^{\prime}}(\bm{Z})$
$\displaystyle={\mathbb{E}}\sum_{tt^{\prime}}\sum_{i}\bm{e}_{i}^{\top}f^{tt^{\prime}}(\bm{Z})\partial_{ij}g^{tt^{\prime}}(\bm{Z}).$

where
$g_{tt^{\prime}}(\bm{Z})=(\bm{e}_{t}^{\top}{\mathbb{E}}_{k}\bm{V}^{\top}{\mathbb{E}}_{k}\bm{V}\bm{e}_{t}^{\prime}-\bm{e}_{t}^{\top}{\mathbb{E}}_{k}\bm{U}^{\top}\bm{z}_{k}\bm{z}_{k}^{\top}{\mathbb{E}}_{k}\bm{V}\bm{e}_{t}^{\prime})$
and

$\partial_{ij}g_{tt^{\prime}}=\partial_{ij}\bm{e}_{t^{\prime}}^{\top}\bm{M}_{k}\bm{e}_{t}=\bm{e}_{t^{\prime}}^{\top}\partial_{ij}[{\mathbb{E}}_{k}\bm{U}^{\top}{\mathbb{E}}_{k}\bm{V}]\bm{e}_{t}-\bm{z}_{k}^{\top}\partial_{ij}[{\mathbb{E}}_{k}\bm{U}\bm{e}_{t^{\prime}}\bm{e}_{t}^{\top}{\mathbb{E}}_{k}\bm{U}^{\top}]\bm{z}_{k}.$

Now define
$\tilde{f}^{tt^{\prime}}(\bm{Z})=\partial_{ij}[{\mathbb{E}}_{k}\bm{U}\bm{e}_{t^{\prime}}\bm{e}_{t}^{\top}{\mathbb{E}}_{k}\bm{U}^{\top}]\bm{z}_{k}$
and
$\tilde{g}^{tt^{\prime}}(\bm{Z})=\sum_{i}\bm{e}_{i}^{\top}f^{tt^{\prime}}(\bm{Z})$.
By definition of $\tilde{f}^{tt^{\prime}}(\bm{Z})$, the previous display is
equal to
$\bm{z}_{k}^{\top}\tilde{f}^{tt^{\prime}}(\bm{Z})-\sum_{l}\partial_{lk}\bm{e}_{l}^{\top}\tilde{f}^{tt^{\prime}}(\bm{Z})$.
We apply (F.25) again with respect to $\bm{z}_{k}$, so that

$\displaystyle{\mathbb{E}}\operatorname{Tr}[\bm{M}_{j}^{\top}\bm{M}_{k}]$
$\displaystyle=\sum_{il,tt^{\prime}}\bm{e}_{i}^{\top}\partial_{lk}[f^{tt^{\prime}}(\bm{Z})]\bm{e}_{l}^{\top}\tilde{f}^{tt^{\prime}}(\bm{Z})$
$\displaystyle=\sum_{il,tt^{\prime}}\Bigl{(}\bm{e}_{i}^{\top}\partial_{lk}\Bigl{[}{\mathbb{E}}_{j}[\bm{V}]\bm{e}_{t}\bm{e}_{t^{\prime}}^{\top}{\mathbb{E}}_{j}[\bm{U}]^{\top}\Bigr{]}\bm{z}_{j}\Bigr{)}\Bigl{(}\bm{e}_{l}^{\top}\partial_{ij}\Bigl{[}{\mathbb{E}}_{k}[\bm{U}]\bm{e}_{t^{\prime}}\bm{e}_{t}^{\top}{\mathbb{E}}_{k}[\bm{V}]^{\top}\Bigr{]}\bm{z}_{k}\Bigr{)}.$

To remove the indices $t,t^{\prime}$, we rewrite the above using
$\sum_{t}\bm{e}_{t}\bm{e}_{t}^{\top}=I_{T}$ and
$\sum_{t^{\prime}}\bm{e}_{t^{\prime}}\bm{e}_{t^{\prime}}^{\top}=I_{T}$ so that
it equals

${\mathbb{E}}\sum_{il}\operatorname{Tr}\Bigl{\\{}\partial_{lk}\Bigl{[}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{z}_{j}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Bigr{]}\partial_{ij}\Bigl{[}{\mathbb{E}}_{k}[\bm{V}]^{\top}\bm{z}_{k}\bm{e}_{l}^{\top}{\mathbb{E}}_{k}[\bm{U}]\Bigr{]}\Bigr{\\}}.$

Summing over $j,k$, using
$\operatorname{Tr}[\bm{A}^{\top}\bm{B}]\leq\|\bm{A}\|_{\rm F}\|\bm{B}\|_{\rm
F})$ and the Cauchy-Schwarz inequality, the above is bounded from above by

(F.26)
$\displaystyle\Bigl{\\{}{\mathbb{E}}\sum_{jk,il}\Big{\|}\partial_{lk}\Bigl{[}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{z}_{j}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Bigr{]}\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}\Bigl{\\{}{\mathbb{E}}\sum_{jk,il}\Big{\|}\partial_{ij}\Bigl{[}{\mathbb{E}}_{k}[\bm{V}]^{\top}\bm{z}_{k}\bm{e}_{l}^{\top}{\mathbb{E}}_{k}[\bm{U}]\Bigr{]}\Big{\|}_{\rm
F}^{2}\Bigr{\\}}^{\frac{1}{2}}.$

At this point the two factors are symmetric, with $(\bm{V},\bm{U})$ in the
left factor replaced with $(\bm{U},\bm{V})$ on the right factor. We focus on
the left factor; similar bound will apply to the right one by exchanging the
roles of $\bm{V}$ and $\bm{U}$. If $\bm{z}_{j}$ is independent of matrices
$\bm{A}^{(q)}$, then
${\mathbb{E}}_{j}[\|\sum_{q=1}^{n}(\bm{e}_{q}^{\top}\bm{z}_{j})\bm{A}^{(q)}\|_{\rm
F}^{2}]=\sum_{q=1}^{n}\|\bm{A}^{(q)}\|_{\rm F}^{2}$. So that with
$A^{(q)}=\partial_{lk}[{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{e}_{q}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{U}]]$,
the first factor in the above display is equal to

$\displaystyle\quad~{}\Bigl{\\{}{\mathbb{E}}\sum_{jk,ilq}\Big{\|}\partial_{lk}\Bigl{(}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{e}_{q}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Bigr{)}\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}$
$\displaystyle\stackrel{{\scriptstyle\text{(i)}}}{{=}}\Bigl{\\{}{\mathbb{E}}\sum_{jk,ilq}\Big{\|}\partial_{lk}\Bigl{(}{\mathbb{E}}_{j}[\bm{U}]^{\top}\Bigr{)}\bm{e}_{q}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{V}]+{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{e}_{q}\bm{e}_{i}^{\top}\partial_{lk}\Bigl{(}{\mathbb{E}}_{j}[\bm{V}]\Bigr{)}\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}$
$\displaystyle\stackrel{{\scriptstyle\text{(ii)}}}{{\leq}}\Bigl{\\{}{\mathbb{E}}\sum_{jk,ilq}\Big{\|}\partial_{lk}\Bigl{(}{\mathbb{E}}_{j}[\bm{U}]^{\top}\Bigr{)}\bm{e}_{q}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}+\Bigl{\\{}{\mathbb{E}}\sum_{jk,ilq}\Big{\|}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{e}_{q}\bm{e}_{i}^{\top}\partial_{lk}\Bigl{(}{\mathbb{E}}_{j}[\bm{V}]\Bigr{)}\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}$
$\displaystyle\stackrel{{\scriptstyle\text{(iii)}}}{{=}}\Bigl{\\{}{\mathbb{E}}\sum_{jk,ilq}\Big{\|}{\mathbb{E}}_{j}[\partial_{lk}\bm{U}]^{\top}\bm{e}_{q}\Big{\|}_{2}^{2}\Big{\|}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\bm{V}]\Big{\|}_{2}^{2}\Bigl{\\}}^{\frac{1}{2}}+\Bigl{\\{}{\mathbb{E}}\sum_{jk,ilq}\Big{\|}{\mathbb{E}}_{j}[\bm{U}]^{\top}\bm{e}_{q}\Big{\|}_{2}^{2}\Big{\|}\bm{e}_{i}^{\top}{\mathbb{E}}_{j}[\partial_{lk}\bm{V}]\Big{\|}_{2}^{2}\Bigl{\\}}^{\frac{1}{2}}$
$\displaystyle\stackrel{{\scriptstyle\text{(iv)}}}{{=}}\Bigl{\\{}{\mathbb{E}}\sum_{jk,l}\Big{\|}{\mathbb{E}}_{j}[\partial_{lk}\bm{U}]^{\top}\Big{\|}_{\rm
F}^{2}\Big{\|}{\mathbb{E}}_{j}[\bm{V}]\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}+\Bigl{\\{}{\mathbb{E}}\sum_{jk,l}\Big{\|}{\mathbb{E}}_{j}[\bm{U}]^{\top}\Big{\|}_{\rm
F}^{2}\Big{\|}{\mathbb{E}}_{j}[\partial_{lk}\bm{V}]\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}},$

where (i) is the chain rule, (ii) the triangle inequality, (iii) holds
provided that the order of the derivation $\partial_{lk}$ and the expectation
sign ${\mathbb{E}}_{j}$ can be switched (thanks to dominant convergence
theorem) and using $\|\bm{a}\bm{b}^{\top}\|_{\rm
F}^{2}=\|\bm{a}\|_{2}^{2}\|\bm{b}\|_{2}^{2}$ for vectors $\bm{a},\bm{b}$, and
(iv) holds using $\sum_{i}\|\bm{A}\bm{e}_{i}\|_{2}^{2}=\|\bm{A}\|_{\rm
F}^{2}=\sum_{q}\|\bm{A}\bm{e}_{q}\|_{2}^{2}$ for a matrix $\bm{A}$ with $n$
columns. Finally, by Jensen’s inequality, the above display is bounded by

$\Bigl{\\{}{\mathbb{E}}\sum_{k,l}\Big{\|}\partial_{lk}\bm{U}\Big{\|}_{\rm
F}^{2}\sum_{j}\Big{\|}{\mathbb{E}}_{j}[\bm{V}]\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}+\Bigl{\\{}{\mathbb{E}}\sum_{k,l}\Big{\|}\partial_{lk}\bm{V}\Big{\|}_{\rm
F}^{2}\sum_{j}\Big{\|}{\mathbb{E}}_{j}[\bm{U}]\Big{\|}_{\rm
F}^{2}\Bigl{\\}}^{\frac{1}{2}}.$

By the Cauchy-Schwarz inequality, the first term in the above display is
bounded by

$\displaystyle\Bigl{\\{}\sum_{j}\sqrt{{\mathbb{E}}\Bigl{[}\bigl{(}\sum_{k,l}\|\partial_{lk}\bm{U}\|_{\rm
F}^{2}\bigr{)}^{2}\Bigr{]}}\sqrt{{\mathbb{E}}\|{\mathbb{E}}_{j}[\bm{V}]\|_{\rm
F}^{4}}\Bigr{\\}}^{1/2}$ $\displaystyle\leq~{}$
$\displaystyle\Bigl{\\{}p\sqrt{{\mathbb{E}}\Bigl{[}\bigl{(}\sum_{k,l}\|\partial_{lk}\bm{U}\|_{\rm
F}^{2}\bigr{)}^{2}\Bigr{]}}\sqrt{{\mathbb{E}}\|\bm{V}\|_{\rm
F}^{4}}\Bigr{\\}}^{1/2}$ $\displaystyle=~{}$
$\displaystyle\sqrt{p}{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/4},$

where we used $\|{\mathbb{E}}_{j}[\bm{V}]\|_{\rm
F}^{4}\leq{\mathbb{E}}_{j}[\|\bm{V}\|_{\rm F}^{4}]$ by Jensen’s inequality.
Hence,

$\displaystyle\eqref{eq:chi-a}$
$\displaystyle\leq\big{(}\sqrt{p}{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/4}+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/4}\big{)}^{2}$ $\displaystyle\leq
2p{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/2}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/2}+2p{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/2}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/2},$

where we uses $(a+b)^{2}\leq 2a^{2}+2b^{2}$.

Combining the above bounds of
$\sum_{j=k}\operatorname{Tr}[\bm{M}_{j}^{\top}\bm{M}_{k}]$ and $\sum_{j\neq
k}\operatorname{Tr}[\bm{M}_{j}^{\top}\bm{M}_{k}]$, we have

$\displaystyle{\mathbb{E}}[\|\eqref{eq:difficult-term}\|_{\rm F}]$
$\displaystyle\leq$ $\displaystyle{\mathbb{E}}[\|\eqref{eq:difficult-
term}\|_{\rm F}^{2}]^{\frac{1}{2}}$ $\displaystyle\leq$
$\displaystyle\big{(}2p{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/2}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/2}+2p{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/2}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/2}+2p{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/2}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/2}\big{)}^{1/2}$ $\displaystyle\leq$
$\displaystyle\sqrt{2p}\big{(}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/4}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/4}+{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/4}+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/4}\big{)},$

where the last inequality uses $(a+b+c)^{1/2}\leq a^{1/2}+b^{1/2}+c^{1/2}$ for
non-negative $a$, $b$, and $c$.

Now, combining the bounds on the terms (F.21)-(F.22)-(F.23)-(F.24) with the
triangle inequality, the expectation in the lemma statement can be bounded
from above by

$\displaystyle{\mathbb{E}}\Bigl{[}\Big{\lVert}p\bm{U}^{\top}\bm{V}-\sum_{j=1}^{p}\Bigl{(}\sum_{i=1}^{n}\partial_{ij}\bm{U}^{\top}\bm{e}_{i}-\bm{U}^{\top}\bm{Z}\bm{e}_{j}\Bigr{)}\Bigl{(}\sum_{i=1}^{n}\partial_{ij}\bm{e}_{i}^{\top}\bm{V}-\bm{e}_{j}^{\top}\bm{Z}^{\top}\bm{V}\Bigr{)}\Big{\rVert}_{\rm{F}}\Bigr{]}$
$\displaystyle\leq$ $\displaystyle
2{\mathbb{E}}[\|\bm{U}\|_{\partial}^{2}]^{1/2}{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]^{1/2}$
$\displaystyle+(1+\sqrt{2})\sqrt{p}\bigl{(}{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{2}]^{\frac{1}{2}}{\mathbb{E}}[\|\bm{U}\|_{\partial}^{2}]^{1/2}\bigr{)}$
$\displaystyle+\sqrt{2p}\big{(}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/4}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/4}+{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/4}+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/4}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/4}\big{)}$ $\displaystyle\leq$
$\displaystyle{\mathbb{E}}[\|\bm{U}\|_{\partial}^{2}]+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]$
$\displaystyle+(1+\sqrt{2})\sqrt{p}/2\bigl{(}{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}]+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{2}]+{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{2}]+{\mathbb{E}}[\|\bm{U}\|_{\partial}^{2}]\bigr{)}$
$\displaystyle+\sqrt{2p}/2\big{(}{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{V}\|_{\rm
F}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{U}\|_{\rm
F}^{4}]^{1/2}\big{)}$ $\displaystyle\leq$
$\displaystyle(1+2\sqrt{p})\bigl{(}{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{V}\|_{\partial}^{4}]^{1/2}+{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}+{\mathbb{E}}[\|\bm{U}\|_{\partial}^{4}]^{1/2}\bigr{)},$

where the second inequality uses $ab\leq(a^{2}+b^{2})/2$ and the last
inequality uses

${\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{2}]^{1/2}\leq{\mathbb{E}}[\lVert\bm{U}\rVert_{\rm{F}}^{4}]^{1/4}\text{
and
}{\mathbb{E}}[\lVert\bm{U}\rVert^{2}_{\partial}]^{1/2}\leq{\mathbb{E}}[\lVert\bm{U}\rVert^{4}_{\partial}]^{1/4}$

from Jensen’s inequality. This completes the proof of Lemma F.5.

∎

###### Proof of Lemma F.6.

For notation simplicity, we define

$\displaystyle\bm{\rho}_{i}$
$\displaystyle=\sum_{j=1}^{p+1}\pdv{\bm{V}^{\top}}{z_{ij}}\bm{e}_{j}-\bm{V}^{\top}\bm{Z}^{\top}\bm{e}_{i}.$

Then the left-hand side of (F.14) can be written as
$n\bm{V}^{\top}\bm{V}-\sum_{i=1}^{n}\bm{\rho}_{i}\bm{\rho}_{i}^{\top}$. Since
$\bm{V}=[-\bm{H}^{\top},\sigma{\mathbf{1}}_{T}]^{\top}$, we have
$\bm{V}^{\top}\bm{V}=\bm{H}^{\top}\bm{H}+\bm{S}$. We need the following lemma
for this proof.

######  Lemma F.9 (Proof is given on Section F.3).

We have

(F.27)
$\bm{\rho}_{i}^{\top}=\sum_{j=1}^{p+1}\pdv{\bm{e}_{j}^{\top}\bm{V}}{z_{ij}}-\bm{e}_{i}^{\top}\bm{Z}\bm{V}=-\bm{e}_{i}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}+\bm{e}_{i}^{\top}\bm{\Xi},$

where
$\bm{\Xi}=n^{-1}\sum_{j=1}^{p}\bm{X}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes\bm{I}_{p}\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j})$
and it satisfies

$\lVert\bm{\Xi}\rVert_{\rm{op}}\leq
C(\zeta,T,\gamma)\lVert\bm{H}\rVert_{\rm{F}}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}.$

By Lemma F.9,

$\displaystyle
n\bm{V}^{\top}\bm{V}-\sum_{i=1}^{n}\bm{\rho}_{i}\bm{\rho}_{i}^{\top}$
$\displaystyle=$ $\displaystyle
n(\bm{H}^{\top}\bm{H}+\bm{S})-\big{(}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)\bm{F}^{\top}+\bm{\Xi}^{\top}\big{)}\big{(}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}+\bm{\Xi}\big{)}$
$\displaystyle=$ $\displaystyle
n(\bm{H}^{\top}\bm{H}+\bm{S})-(\bm{I}_{T}+{\widehat{\bm{C}}}/n)\bm{F}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}$
$\displaystyle-\underbrace{\bm{\Xi}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}-(\bm{I}_{T}+{\widehat{\bm{C}}}/n)\bm{F}^{\top}\bm{\Xi}-\bm{\Xi}^{\top}\bm{\Xi}}_{{\rm
Rem}_{2}}$ $\displaystyle=$ $\displaystyle\sqrt{n}\bm{Q}_{2}-{\rm Rem}_{2}.$

The desired bound for ${\mathbb{E}}[\lVert{\rm Rem}_{2}\rVert_{\rm{F}}]$ then
follows from the Cauchy-Schwarz inequality, Lemmas E.10, E.8 and E.4 and the
operator norm bound of $\bm{\Xi}$ from Lemma F.9. This completes the proof of
Lemma F.6. ∎

###### Proof of Lemma F.7.

We need to bound ${\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}$ and
${\mathbb{E}}[\lVert\bm{V}\rVert_{\partial}^{4}]^{1/2}$. For the first term,
since
$\lVert\bm{V}\rVert_{\rm{F}}^{2}=\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2}$,
we have by Lemma E.10,

$\displaystyle{\mathbb{E}}[\lVert\bm{V}\rVert_{\rm{F}}^{4}]^{1/2}={\mathbb{E}}[(\lVert\bm{H}\rVert_{\rm{F}}^{2}+T\sigma^{2})^{2}]^{1/2}\leq
2{\mathbb{E}}[\lVert\bm{H}\rVert_{\rm{F}}^{4}]^{1/2}+2T\sigma^{2}\leq
C(\zeta,T,\gamma){\rm var}(y_{1}).$

For the second term, we have

$\displaystyle\lVert\bm{V}\rVert_{\partial}^{2}=\sum_{i=1}^{n}\sum_{j=1}^{p+1}\lVert\pdv{\bm{V}}{z_{ij}}\rVert_{\rm{F}}^{2}=\sum_{i=1}^{n}\sum_{j=1}^{p}\lVert\pdv{\bm{H}}{x_{ij}}\rVert_{\rm{F}}^{2}=\lVert\pdv{\operatorname{\mathbf{vec}}(\bm{H})}{\operatorname{\mathbf{vec}}(\bm{X})}\rVert_{\rm{F}}.$

Applying (E.13), and the moment bound of $\bm{F},\bm{H},\bm{X}$ in Lemmas E.10
and E.8, we have

$\displaystyle{\mathbb{E}}[\lVert\bm{V}\rVert_{\partial}^{4}]^{1/2}\leq
C(\zeta,T,\gamma){\rm var}(y_{1}).$

Combining the above two bounds, we have the desired inequality. This completes
the proof of Lemma F.7. ∎

###### Proof of Lemma F.8.

By Corollary C.2, we have $\frac{\partial F_{lt}}{\partial
x_{ij}}=D_{ij}^{lt}+\Delta_{ij}^{lt}$, where

$\displaystyle D_{ij}^{lt}$
$\displaystyle=-(\bm{e}_{t}^{\top}\otimes\bm{e}_{l}^{\top})(\bm{I}_{nT}-\bm{N})((\bm{H}^{\top}\bm{e}_{j})\otimes\bm{e}_{i}),$
$\displaystyle\Delta_{ij}^{lt}$
$\displaystyle=-n^{-1}(\bm{e}_{t}^{\top}\otimes\bm{e}_{l}^{\top})(\bm{I}_{T}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}\bm{F}^{\top}\otimes\bm{I}_{p}\bigr{)}(\bm{e}_{i}\otimes\bm{e}_{j}).$

We first prove (F.16). Since $\bm{e}_{i}^{\top}\frac{\partial\bm{F}}{\partial
x_{ij}}=\sum_{t}\pdv{F_{it}}{x_{ij}}\bm{e}_{t}^{\top}=\sum_{t}(D_{ij}^{it}+\Delta_{ij}^{it})\bm{e}_{t}^{\top}$,
we have

$\displaystyle\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{e}_{j}\bm{e}_{i}^{\top}\frac{\partial\bm{F}}{\partial
x_{ij}}$ $\displaystyle=~{}$
$\displaystyle\bm{H}^{\top}\sum_{ij}\bm{e}_{j}\sum_{t=1}^{T}(D_{ij}^{it}+\Delta_{ij}^{it})\bm{e}_{t}^{\top}$
$\displaystyle=~{}$
$\displaystyle\bm{H}^{\top}\sum_{ij}\bm{e}_{j}\sum_{t}D_{ij}^{it}\bm{e}_{t}^{\top}+\underbrace{\bm{H}^{\top}\sum_{ij}\bm{e}_{j}\sum_{t}\Delta_{ij}^{it}\bm{e}_{t}^{\top}}_{{\rm
Rem}_{1,1}}.$

The first term in the last line can be simplified as below,

$\displaystyle\bm{H}^{\top}\sum_{j=1}^{p}\sum_{i=1}^{n}\bm{e}_{j}\sum_{t=1}^{T}D_{ij}^{it}\bm{e}_{t}^{\top}$
$\displaystyle=~{}$
$\displaystyle-\bm{H}^{\top}\sum_{j=1}^{p}\sum_{i=1}^{n}\bm{e}_{j}\sum_{t=1}^{T}(\bm{e}_{j}^{\top}\bm{H}\otimes\bm{e}_{i}^{\top})(\bm{I}_{nT}-\bm{N}^{\top})(\bm{e}_{t}\otimes\bm{e}_{i})\bm{e}_{t}^{\top}$
by (C.5) $\displaystyle=~{}$
$\displaystyle-\bm{H}^{\top}\bm{H}\Bigl{[}\sum_{i=1}^{n}(\bm{I}_{T}\otimes\bm{e}_{i}^{\top})(\bm{I}_{nT}-\bm{N}^{\top})(\bm{I}_{T}\otimes\bm{e}_{i})\Bigr{]}$
$\displaystyle=~{}$
$\displaystyle-\bm{H}^{\top}\bm{H}(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}^{\top})$
by (2.12).

Next, we prove (F.17).

$\displaystyle\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial\bm{H}}{\partial
x_{ij}}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}\frac{\partial{\widehat{\bm{b}}}^{t}}{\partial
x_{ij}}\bm{e}_{t}^{\top}$ $\displaystyle=$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}\Bigl{(}(\bm{F}^{\top}\bm{e}_{i})\otimes\bm{e}_{j}\Bigr{)}\bm{e}_{t}^{\top}+{\rm
Rem}_{1,2}$ by (C.2) $\displaystyle=$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\Big{[}(\bm{e}_{t}^{\top}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\Bigl{(}(\bm{F}^{\top}\bm{e}_{i})\otimes\bm{e}_{j}\Bigr{)}\Big{]}^{\top}\bm{e}_{t}^{\top}+{\rm
Rem}_{1,2}$ $\displaystyle=$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\Bigl{(}(\bm{e}_{i}^{\top}\bm{F})\otimes\bm{e}_{j}^{\top}\Bigr{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{e}_{t}\otimes\bm{e}_{j})\bm{e}_{t}^{\top}+{\rm
Rem}_{1,2}$ by (B.1) $\displaystyle=$ $\displaystyle
n^{-1}\bm{F}^{\top}\bm{F}\sum_{j=1}^{p}(\bm{I}_{T}\otimes\bm{e}_{j}^{\top})({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j})+{\rm
Rem}_{1,2}$ by (B.1) $\displaystyle=$
$\displaystyle\bm{F}^{\top}\bm{F}{\widehat{\bm{C}}}^{\top}/n+{\rm Rem}_{1,2}$
$\displaystyle\mbox{by \eqref{hbC}}.$

It remains to bound ${\mathbb{E}}[\lVert{\rm Rem}_{1,1}\rVert_{\rm{F}}^{2}]$
and ${\mathbb{E}}[\lVert{\rm Rem}_{1,2}\rVert_{\rm{F}}^{2}]$. By expression of
${\rm Rem}_{1,1}$ in (F.18), we have

$\displaystyle\quad{\rm Rem}_{1,1}$
$\displaystyle=\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\bm{e}_{j}\sum_{t=1}^{T}\Delta_{ij}^{it}\bm{e}_{t}^{\top},$
$\displaystyle=-n^{-1}\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{e}_{j}(\bm{e}_{t}^{\top}\otimes\bm{e}_{i}^{\top})(\bm{I}_{T}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}\bm{F}^{\top}\otimes\bm{I}_{p}\bigr{)}(\bm{e}_{i}\otimes\bm{e}_{j})\bm{e}_{t}^{\top}$
by (C.6)
$\displaystyle=-n^{-1}\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{e}_{j}\big{[}(\bm{e}_{t}^{\top}\otimes\bm{e}_{i}^{\top})(\bm{I}_{T}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}\bm{F}^{\top}\otimes\bm{I}_{p}\bigr{)}(\bm{e}_{i}\otimes\bm{e}_{j})\big{]}^{\top}\bm{e}_{t}^{\top}$
$\displaystyle=-n^{-1}\bm{H}^{\top}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{e}_{j}(\bm{e}_{i}^{\top}\otimes\bm{e}_{j}^{\top})(\bm{F}\otimes\bm{I}_{p})({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{X}^{\top})(\bm{e}_{t}\otimes\bm{e}_{i})\bm{e}_{t}^{\top}$
$\displaystyle=-n^{-1}\bm{H}^{\top}\sum_{i=1}^{n}(\bm{e}_{i}^{\top}\otimes\bm{I}_{p})(\bm{F}\otimes\bm{I}_{p})({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{X}^{\top})(\bm{I}_{T}\otimes\bm{e}_{i})$
by (B.1)
$\displaystyle=-n^{-1}\bm{H}^{\top}\sum_{i=1}^{n}((\bm{e}_{i}^{\top}\bm{F})\otimes\bm{I}_{p})({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes(\bm{X}^{\top}\bm{e}_{i}))$
by (B.1).

Using the fact that
$\lVert\bm{M}\rVert_{\rm{F}}\leq\sqrt{T}\lVert\bm{M}\rVert_{\rm{op}}$ for
$\bm{M}\in{\mathbb{R}}^{T\times T}$ and the triangular inequality, we have

$\displaystyle\left\lVert{\rm Rem}_{1,1}\right\rVert_{\rm{F}}$
$\displaystyle\leq$
$\displaystyle\sqrt{T}n^{-1}\lVert\bm{H}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}\sum_{i=1}^{n}\lVert(\bm{e}_{i}^{\top}\bm{F})\otimes\bm{I}_{p}\rVert_{\rm{op}}\lVert\bm{I}_{T}\otimes(\bm{X}^{\top}\bm{e}_{i})\rVert_{\rm{op}}$
$\displaystyle\leq$
$\displaystyle\sqrt{T}n^{-1}\lVert\bm{H}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}{\mathcal{D}}\rVert_{\rm{op}}\big{[}\sum_{i=1}^{n}\lVert(\bm{e}_{i}^{\top}\bm{F})\otimes\bm{I}_{p}\rVert_{\rm{op}}^{2}\big{]}^{1/2}\big{[}\sum_{i=1}^{n}\lVert\bm{I}_{T}\otimes(\bm{X}^{\top}\bm{e}_{i})\rVert_{\rm{op}}^{2}\big{]}^{1/2}$
$\displaystyle\leq$
$\displaystyle\sqrt{T}n^{-1}\lVert\bm{H}\rVert_{\rm{F}}\lVert\bm{F}\rVert_{\rm{F}}\lVert\bm{X}\rVert_{\rm{F}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}$
$\displaystyle\leq$
$\displaystyle\sqrt{T}n^{-1/2}\lVert\bm{H}\rVert_{\rm{F}}\lVert\bm{F}\rVert_{\rm{F}}\lVert\bm{X}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}$
by $\lVert\bm{X}\rVert_{\rm{F}}\leq\sqrt{n}\lVert\bm{X}\rVert_{\rm{op}}$
$\displaystyle\leq$ $\displaystyle
C(\zeta,T,\gamma)\lVert\bm{F}\rVert_{\rm{F}}\lVert\bm{H}\rVert_{\rm{F}}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}$
$\displaystyle\mbox{by \lx@cref{creftypepluralcap~refnum}{lem:opnorm-M-N}
and~\lx@cref{refnum}{lem:opnorm-D-J}}.$

It follows by the Cauchy-Schwarz inequality and Lemmas E.10 and E.8 that

$\displaystyle{\mathbb{E}}[\lVert{\rm Rem}_{1,1}\rVert_{\rm{F}}^{2}]\leq
nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}.$

We next bound ${\mathbb{E}}[\lVert{\rm Rem}_{1,2}\rVert_{\rm{F}}^{2}]$. By the
expression of ${\rm Rem}_{1,2}$ in (F.19), we have

$\displaystyle-{\rm Rem}_{1,2}$
$\displaystyle=n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\bm{e}_{j}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\bm{e}_{t}^{\top}$
$\displaystyle=n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\Big{[}(\bm{e}_{t}^{\top}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\Big{]}^{\top}\bm{e}_{t}^{\top}$
$\displaystyle=n^{-1}\sum_{i=1}^{n}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{F}^{\top}\bm{e}_{i}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes(\bm{e}_{i}^{\top}\bm{X})\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{e}_{t}\otimes\bm{e}_{j})\bm{e}_{t}^{\top}$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes(\bm{F}^{\top}\bm{X})\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j}).$

Using the same argument that we used to bound ${\mathbb{E}}[\lVert{\rm
Rem}_{1,1}\rVert_{\rm{F}}^{2}]$, we have

$\displaystyle{\mathbb{E}}[\lVert{\rm Rem}_{1,2}\rVert_{\rm{F}}^{2}]\leq
nC(\zeta,T,\gamma){\rm var}(y_{1})^{2}.$

This completes the proof of Lemma F.8. ∎

###### Proof of Lemma F.9.

By the definition of $\bm{V}$, we have

$\displaystyle\sum_{j=1}^{p+1}\pdv{\bm{e}_{j}^{\top}\bm{V}}{z_{ij}}$
$\displaystyle=$
$\displaystyle-\sum_{j=1}^{p}\pdv{\bm{e}_{j}^{\top}\bm{H}}{x_{ij}}$
$\displaystyle=$
$\displaystyle-\sum_{j=1}^{p}\sum_{t=1}^{T}\pdv{\bm{e}_{j}^{\top}{\widehat{\bm{b}}}^{t}}{x_{ij}}\bm{e}_{t}^{\top}$
$\displaystyle=$
$\displaystyle-n^{-1}\sum_{j=1}^{p}\sum_{t=1}^{T}(\bm{e}_{t}^{\top}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{F}^{\top}\bm{e}_{i})\otimes\bm{e}_{j}\bigr{)}\bm{e}_{t}^{\top}+\bm{\Delta}_{i}$
by (C.2) $\displaystyle=$
$\displaystyle-n^{-1}\sum_{j=1}^{p}\sum_{t=1}^{T}[\bm{e}_{t}(\bm{e}_{t}^{\top}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{F}^{\top}\bm{e}_{i})\otimes\bm{e}_{j}\bigr{)}]^{\top}+\bm{\Delta}_{i}$
$\displaystyle=$
$\displaystyle-n^{-1}\sum_{j=1}^{p}[(\bm{I}_{T}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}\bm{I}_{T}\otimes\bm{e}_{j}\bigr{)}\bm{F}^{\top}\bm{e}_{i}]^{\top}+\bm{\Delta}_{i}$
$\displaystyle=$
$\displaystyle-\bm{e}_{i}^{\top}\bm{F}{\widehat{\bm{C}}}^{\top}/n+\bm{\Delta}_{i},$

where
$\bm{\Delta}_{i}=n^{-1}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{e}_{j}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\bm{e}_{t}^{\top}.$

We claim that

(F.28) $\bm{\Delta}_{i}=\bm{e}_{i}^{\top}\bm{\Xi}.$

Therefore, using $\bm{Z}\bm{V}=\bm{F}$, we have

$\displaystyle\bm{\rho}_{i}^{\top}$
$\displaystyle=\sum_{j=1}^{p+1}\pdv{\bm{e}_{j}^{\top}\bm{V}}{z_{ij}}-\bm{e}_{i}^{\top}\bm{Z}\bm{V}$
$\displaystyle=-\bm{e}_{i}^{\top}\bm{F}{\widehat{\bm{C}}}^{\top}/n+\bm{\Delta}_{i}-\bm{e}_{i}^{\top}\bm{F}$
$\displaystyle=-\bm{e}_{i}^{\top}\bm{F}(\bm{I}_{T}+{\widehat{\bm{C}}}/n)^{\top}+\bm{e}_{i}^{\top}\bm{\Xi}.$

Now we prove the claim (F.28). By definition,

$\displaystyle\bm{\Delta}_{i}$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\sum_{t=1}^{T}\bm{e}_{j}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{I}_{p}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\bm{e}_{t}^{\top}$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\sum_{t=1}^{T}(\bm{e}_{t}^{\top}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\bm{e}_{t}^{\top}$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\sum_{t=1}^{T}\big{[}\bm{e}_{t}(\bm{e}_{t}^{\top}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\big{]}^{\top}$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\big{[}(\bm{I}_{T}\otimes\bm{e}_{j}^{\top}){\mathcal{M}}^{-1}{\mathcal{D}}\bigl{(}(\bm{H}^{\top}\bm{e}_{j})\otimes(\bm{X}^{\top}\bm{e}_{i})\bigr{)}\big{]}^{\top}$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes(\bm{e}_{i}^{\top}\bm{X})\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j})$
$\displaystyle=n^{-1}\sum_{j=1}^{p}\bm{e}_{i}^{\top}\bm{X}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes\bm{I}_{p}\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j})$
$\displaystyle=\bm{e}_{i}^{\top}\bm{\Xi}.$

It remains to bound $\lVert\bm{\Xi}\rVert_{\rm{op}}$. By definition of
$\bm{\Xi}$, we have

$\displaystyle\lVert\bm{\Xi}\rVert_{\rm{op}}$ $\displaystyle=$ $\displaystyle
n^{-1}\lVert\sum_{j=1}^{p}\bm{X}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes\bm{I}_{p}\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j})\rVert_{\rm{op}}$
$\displaystyle\leq$ $\displaystyle
n^{-1}\sum_{j=1}^{p}\lVert\bm{X}\big{(}(\bm{e}_{j}^{\top}\bm{H})\otimes\bm{I}_{p}\big{)}({\mathcal{M}}^{-1}{\mathcal{D}})^{\top}(\bm{I}_{T}\otimes\bm{e}_{j})\rVert_{\rm{op}}$
$\displaystyle\leq$ $\displaystyle
n^{-1}\lVert\bm{X}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}\sum_{j=1}^{p}\lVert(\bm{e}_{j}^{\top}\bm{H})\otimes\bm{I}_{p}\rVert_{\rm{op}}\lVert\bm{I}_{T}\otimes\bm{e}_{j}\rVert_{\rm{op}}$
$\displaystyle\leq$ $\displaystyle
n^{-1}\lVert\bm{X}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}[\sum_{j=1}^{p}\lVert(\bm{e}_{j}^{\top}\bm{H})\otimes\bm{I}_{p}\rVert_{\rm{op}}^{2}]^{1/2}[\sum_{j}\lVert\bm{I}_{T}\otimes\bm{e}_{j}\rVert_{\rm{op}}^{2}]^{1/2}$
$\displaystyle\leq$ $\displaystyle
n^{-1}\lVert\bm{X}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}\lVert\bm{H}\rVert_{\rm{F}}\sqrt{Tp}$
$\displaystyle\leq$ $\displaystyle
C(\zeta,T,\gamma)\lVert\bm{H}\rVert_{\rm{F}}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}$
by Lemmas E.2 and E.1.

This completes the proof of Lemma F.9. ∎

##  Appendix G Proof of Theorem 2.1

###### Proof of Theorem 2.1.

By definition, we know $\hat{r}_{t}=\hat{r}_{t,t}$ and $r_{t}=r_{t,t}$. Thus
the result of this theorem follows directly from the result of Theorem 2.2. ∎

##  Appendix H Proof of Corollary 2.3

###### Proof of Corollary 2.3.

By the definition of $r_{t}$ in (2.14), it suffices to show that

$\displaystyle{\mathbb{P}}\Bigl{(}r_{\hat{t}}-\min_{s\in[T]}r_{s}\geq\frac{{\rm
var}(y_{1})}{n^{1/2-c}}\Bigr{)}\leq\frac{C(\zeta,\gamma,T,\kappa)}{n^{c}}.$

To see this, we have

$\displaystyle{\mathbb{P}}\Bigl{(}r_{\hat{t}}-\min_{s\in[T]}r_{s}\geq\frac{{\rm
var}(y_{1})}{n^{1/2-c}}\Bigr{)}$ $\displaystyle\leq$
$\displaystyle\frac{n^{1/2-c}}{{\rm
var}(y_{1})}{\mathbb{E}}\bigl{[}r_{\hat{t}}-\min_{s\in[T]}r_{s}\bigr{]}$
Markov’s inequality $\displaystyle\leq$ $\displaystyle\frac{n^{1/2-c}}{{\rm
var}(y_{1})}\Bigl{(}{\mathbb{E}}[|r_{\hat{t}}-\hat{r}_{\hat{t}}|]+{\mathbb{E}}[|\hat{r}_{\hat{t}}-\min_{s\in[T]}r_{s}|]\Bigr{)}$
triangle inequality $\displaystyle=$ $\displaystyle\frac{n^{1/2-c}}{{\rm
var}(y_{1})}\Bigl{(}{\mathbb{E}}[|r_{\hat{t}}-\hat{r}_{\hat{t}}|]+{\mathbb{E}}[|\min_{s\in[T]}\hat{r}_{s}-\min_{s\in[T]}r_{s}|]\Bigr{)}$
by definition of $\hat{t}$ $\displaystyle\leq$
$\displaystyle\frac{n^{1/2-c}}{{\rm
var}(y_{1})}\Bigl{(}{\mathbb{E}}[|r_{\hat{t}}-\hat{r}_{\hat{t}}|]+{\mathbb{E}}[\max_{s\in[T]}|\hat{r}_{s}-r_{s}|]\Bigr{)}$
by $|\min_{s}a_{s}-\min_{s}b_{s}|\leq\max_{s}|a_{s}-b_{s}|$
$\displaystyle\leq$ $\displaystyle\frac{n^{1/2-c}}{{\rm
var}(y_{1})}\Bigl{(}{\mathbb{E}}[|r_{\hat{t}}-\hat{r}_{\hat{t}}|]+\max_{s\in[T]}{\mathbb{E}}[|\hat{r}_{s}-r_{s}|]\Bigr{)}$
Jensen’s inequality $\displaystyle\leq$
$\displaystyle\frac{C(\zeta,\gamma,T,\kappa)}{n^{c}}$ $\displaystyle\mbox{by
\lx@cref{creftypecap~refnum}{thm:rt}}.$

This concludes the proof. ∎

##  Appendix I Proof of Theorem 2.4

Note that for this proof, we will consider general $\bm{\Sigma}$ directly. We
first present two lemmas that will be useful.

######  Lemma I.1.

Under the same condition of Theorem 2.4. For ${\rm
Rem}_{j}=\sum_{i=1}^{n}\sum_{t=1}^{T}\Delta_{ij}^{it}\bm{e}_{t}^{\top}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$,
we have

$\displaystyle{\mathbb{E}}\big{[}\sum_{j=1}^{p}\|{\rm
Rem}_{j}\|^{2}\big{]}\leq nC(\zeta,T,\kappa,\gamma){\rm var}(y_{1}).$

###### Proof of Lemma I.1.

By definition, ${\rm Rem}_{j}$ is a row vector in ${\mathbb{R}}^{T}$, and its
$t$-th entry is

$\displaystyle{\rm Rem}_{jt}$
$\displaystyle=\sum_{i=1}^{n}\Delta_{ij}^{it}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$
$\displaystyle=-n^{-1}\sum_{i=1}^{n}(\bm{e}_{t}^{\top}\otimes(\bm{e}_{i}^{\top}\bm{X})){\mathcal{M}}^{-1}{\mathcal{D}}((\bm{F}^{\top}\bm{e}_{i})\otimes\bm{e}_{j})\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$
by (C.6)
$\displaystyle=-n^{-1}\sum_{i=1}^{n}\bm{e}_{i}^{\top}(\bm{e}_{t}^{\top}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{e}_{j})\bm{e}_{i}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$
by (B.1)
$\displaystyle=-n^{-1}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}\operatorname{Tr}\bigl{(}(\bm{e}_{t}^{\top}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{e}_{j})\bigr{)}.$

For the trace in the last line above, we have for each $j\in[p],t\in[T]$, the
following holds,

Since
$(\bm{F}^{\top}\otimes\bm{e}_{j})=(\bm{I}_{T}\otimes\bm{e}_{j})\bm{F}^{\top}$,
whose rank is at most $T$, we have

$\displaystyle\quad\operatorname{Tr}\bigl{(}(\bm{e}_{t}^{\top}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{e}_{j})\bigr{)}$
$\displaystyle\leq
T\lVert(\bm{e}_{t}^{\top}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{e}_{j})\rVert_{\rm{op}}$
$\displaystyle\leq
T\lVert(\bm{e}_{t}^{\top}\otimes\bm{X})\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}\lVert(\bm{F}^{\top}\otimes\bm{e}_{j})\rVert_{\rm{op}}$
submultiplicativity of $\|\cdot\|_{\rm op}$
$\displaystyle=\sqrt{T}\lVert\bm{X}\rVert_{\rm{op}}\lVert{\mathcal{M}}^{-1}\rVert_{\rm{op}}\lVert{\mathcal{D}}\rVert_{\rm{op}}\lVert\bm{F}\rVert_{\rm{op}}$
by (B.4) $\displaystyle\leq
C(\zeta,T)\lVert\bm{X}\rVert_{\rm{op}}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T-1}\lVert\bm{F}\rVert_{\rm{F}}$
by (E.1) and (E.2) $\displaystyle\leq
C(\zeta,T)\sqrt{n}(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}\lVert\bm{F}\rVert_{\rm{F}}.$
by
$\lVert\bm{X}\rVert_{\rm{op}}/\sqrt{n}\leq(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)/2$

Thus, using
$\max_{j\in[p]}\tfrac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|^{2}}\leq\lVert\bm{\Sigma}\rVert_{\rm{op}}$
and above inequality for the trace, we have

$\displaystyle\quad\sum_{j=1}^{p}\|{\rm Rem}_{j}\|^{2}$
$\displaystyle=\sum_{j=1}^{p}\sum_{t=1}^{T}({\rm Rem}_{jt})^{2}$
$\displaystyle\leq
n^{-2}\lVert\bm{\Sigma}\rVert_{\rm{op}}\sum_{j=1}^{p}\sum_{t=1}^{T}\operatorname{Tr}\bigl{(}(\bm{e}_{t}^{\top}\otimes\bm{X}){\mathcal{M}}^{-1}{\mathcal{D}}(\bm{F}^{\top}\otimes\bm{e}_{j})\bigr{)}^{2}$
$\displaystyle\leq
n^{-1}p\lVert\bm{\Sigma}\rVert_{\rm{op}}C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}\lVert\bm{F}\rVert_{\rm{F}}^{2}$
$\displaystyle\leq\gamma\kappa
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}\lVert\bm{F}\rVert_{\rm{F}}^{2}.$
by $p/n\leq\gamma$ and $\lVert\bm{\Sigma}\rVert_{\rm{op}}\leq\kappa$ from
Assumption 2.2

Therefore, Taking expectation using the Cauchy-Schwarz inequality and Lemmas
E.8 and E.10, we have

$\displaystyle{\mathbb{E}}[\sum_{j=1}^{p}\|{\rm Rem}_{j}\|^{2}]\leq
nC(\zeta,T,\kappa,\gamma){\rm var}(y_{1}).$

This completes the proof of Lemma I.1. ∎

######  Lemma I.2.

Under Assumptions 2.2, 2.3, 2.4 and 2.1, we have

$\displaystyle
n^{-1}{\mathbb{E}}\Big{[}\sum_{t=1}^{T}\sum_{i=1}^{n}\sum_{j=1}^{p}(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1}\bm{e}_{j})^{-1}\left\lVert\pdv{\bm{F}\bm{e}_{t}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}\leq
C(\zeta,T,\kappa,\gamma){\rm var}(y_{1}).$

###### Proof of Lemma I.2.

By Lemma E.6 the mapping ${\mathbb{R}}^{n\times
p}\to{\mathbb{R}}^{n}:\bm{X}\mapsto\bm{F}\bm{e}_{t}$ is Lipschitz. Since the
Frobenius norm of the Jacobian of an $L_{*}$-Lipschitz function valued in
${\mathbb{R}}^{n}$ is at most $L_{*}\sqrt{n}$, we have the Frobenius norm of
the Jacobian of the mapping ${\mathbb{R}}^{n\times
p}\to{\mathbb{R}}^{n}:\bm{X}\mapsto\bm{F}\bm{e}_{t}$ is bounded by

$\displaystyle\sqrt{n}C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T}(\lVert\bm{H}\rVert_{\rm{F}}+\lVert\bm{F}\rVert_{\rm{F}}/\sqrt{n}).$

Therefore, using
$\max_{j\in[p]}(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1}\bm{e}_{j})^{-1}\leq(\lambda_{\min}(\bm{\Sigma}^{-1}))^{-1}=\lVert\bm{\Sigma}\rVert_{\rm{op}}$,
we have

$\displaystyle\quad
n^{-1}\sum_{t=1}^{T}\sum_{i=1}^{n}\sum_{j=1}^{p}(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1}\bm{e}_{j})^{-1}\left\lVert\pdv{\bm{F}\bm{e}_{t}}{x_{ij}}\right\rVert_{\rm{F}}^{2}$
$\displaystyle\leq
n^{-1}\lVert\bm{\Sigma}\rVert_{\rm{op}}\sum_{t=1}^{T}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{F}\bm{e}_{t}}{x_{ij}}\right\rVert_{\rm{F}}^{2}$
$\displaystyle\leq
n^{-1}\lVert\bm{\Sigma}\rVert_{\rm{op}}\sum_{t=1}^{T}\left\lVert\pdv{\bm{F}\bm{e}_{t}}{\operatorname{\mathbf{vec}}(\bm{X})}\right\rVert_{\rm{F}}^{2}$
$\displaystyle\leq\lVert\bm{\Sigma}\rVert_{\rm{op}}C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{2T}(\lVert\bm{H}\rVert_{\rm{F}}+\lVert\bm{F}\rVert_{\rm{F}}/\sqrt{n})^{2}.$

Taking expectations, using the Cauchy-Schwarz inequality and Lemmas E.8 and
E.10, we have

$\displaystyle\quad
n^{-1}{\mathbb{E}}\Big{[}\sum_{t=1}^{T}\sum_{i=1}^{n}\sum_{j=1}^{p}(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1}\bm{e}_{j})^{-1}\left\lVert\pdv{\bm{F}\bm{e}_{t}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}$
$\displaystyle\leq C(\zeta,T,\kappa,\gamma){\rm var}(y_{1}).$

∎

###### Proof of Theorem 2.4.

For any $j\in[p]$, $\bm{e}_{j}\in{\mathbb{R}}^{p}$, let
$\bm{z}_{j}=\bm{X}\bm{\Sigma}^{-1}\bm{e}_{j}/\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|$,
then $\bm{z}_{j}\sim\mathsf{N}({\mathbf{0}},\bm{I}_{n})$. Let
$\bm{F}(\bm{z}_{j})=\bm{Y}-\bm{X}{\widehat{\bm{B}}}\in{\mathbb{R}}^{n\times
T}$. Since

$\bm{X}=\bm{X}(\bm{I}_{p}-\frac{\bm{\Sigma}^{-1}\bm{e}_{j}\bm{e}_{j}^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|^{2}})+\bm{X}\frac{\bm{\Sigma}^{-1}\bm{e}_{j}\bm{e}_{j}^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|^{2}}:=\bm{X}\bm{\Pi}_{j}+\bm{z}_{j}\frac{\bm{e}_{j}^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|},$

where
$\bm{\Pi}_{j}=\bm{I}_{p}-\frac{\bm{\Sigma}^{-1}\bm{e}_{j}\bm{e}_{j}^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|^{2}}$.

We now show $\operatorname{\mathbf{vec}}(\bm{X}\bm{\Pi}_{j})$ is independent
of $\bm{z}_{j}$. Let $\bm{G}=\bm{X}\bm{\Sigma}^{-1/2}$, we have

$\displaystyle\operatorname{\mathbf{vec}}(\bm{X}\bm{\Pi}_{j})=\operatorname{\mathbf{vec}}(\bm{G}\bm{\Sigma}^{1/2}\bm{\Pi}_{j})=(\bm{\Sigma}^{1/2}\bm{\Pi}_{j}\otimes\bm{I}_{n})\operatorname{\mathbf{vec}}(\bm{G})$

and

$\displaystyle\bm{z}_{j}=\bm{G}\bm{\Sigma}^{-1/2}\bm{e}_{j}/\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|=\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1/2}\otimes\bm{I}_{n})\operatorname{\mathbf{vec}}(\bm{G}).$

Since $\operatorname{\mathbf{vec}}(\bm{G})$ is a standard Gaussian vector in
${\mathbb{R}}^{np}$, the above two vectors
$\operatorname{\mathbf{vec}}(\bm{X}\bm{\Pi}_{j})$ and $\bm{z}_{j}$ are two
Gaussian vectors. It suffices to show that
$\operatorname{\mathbf{vec}}(\bm{X}\bm{\Pi}_{j})$ and $\bm{z}_{j}$ are
uncorrelated, which can be shown by verifying that

$(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1/2}\otimes\bm{I}_{n})(\bm{\Sigma}^{1/2}\bm{\Pi}_{j}\otimes\bm{I}_{n})=(\bm{e}_{j}^{\top}\bm{\Pi}_{j}\otimes\bm{I}_{n})={\mathbf{0}}$

thanks to $\bm{\Pi}_{j}^{\top}\bm{e}_{j}={\mathbf{0}}$ from definition of
$\bm{\Pi}_{j}$.

Applying (Tan and Bellec, 2024, Lemma S5.3) to the mapping:
$\bm{z}_{j}\mapsto\bm{F}(\bm{z}_{j})/\sqrt{n}$, using
$z_{ij}=\bm{e}_{i}^{\top}\bm{z}_{j}$, we have

(I.1)
${\mathbb{E}}\Big{[}\Big{\lVert}\frac{\bm{z}_{j}^{\top}\bm{F}(\bm{z}_{j})}{\sqrt{n}}-\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\pdv{\bm{e}_{i}^{\top}\bm{F}(\bm{z}_{j})}{z_{ij}}-\frac{\bm{z}_{j}^{\top}\bm{F}(\tilde{\bm{z}}_{j})}{\sqrt{n}}\Big{\rVert}^{2}\Big{]}\leq\frac{3}{n}{\mathbb{E}}\sum_{i=1}^{n}\left\lVert\pdv{\bm{F}(\bm{z}_{j})}{z_{ij}}\right\rVert_{\rm{F}}^{2},$

where $\tilde{\bm{z}}_{j}\sim\mathsf{N}(\mathbf{0},\bm{I}_{n})$ is independent
of $(\bm{X},\bm{y})$.

We want to compute the derivative of $\bm{F}(\bm{z}_{j})$ with respect to
$z_{ij}$. Since
$\bm{X}=\bm{X}\bm{\Pi}_{j}+\bm{z}_{j}\frac{\bm{e}_{j}^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$.
Conditional on $\bm{X}\bm{\Pi}_{j}$ (that is, with $\bm{X}\bm{\Pi}_{j}$ held
fixed), we have
$\pdv{\bm{F}}{z_{ij}}=\pdv{\bm{F}}{x_{ij}}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$.
According to the expression of $\pdv{F_{lt}}{x_{ij}}$ in Corollary C.2, we
have

$\displaystyle\sum_{i=1}^{n}\pdv{\bm{e}_{i}^{\top}\bm{F}}{z_{ij}}$
$\displaystyle=\sum_{i=1}^{n}\pdv{\bm{e}_{i}^{\top}\bm{F}}{x_{ij}}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$
$\displaystyle=\sum_{i=1}^{n}\sum_{t=1}^{T}(D_{ij}^{it}+\Delta_{ij}^{it})\bm{e}_{t}^{\top}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$
(I.2)
$\displaystyle=\frac{-\bm{e}_{j}^{\top}({\widehat{\bm{B}}}-\bm{B}^{*})(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}+{\rm
Rem}_{j}$ $\displaystyle\mbox{by \eqref{eq:sum-D}},$

where ${\rm
Rem}_{j}=\sum_{i=1}^{n}\sum_{t=1}^{T}\Delta_{ij}^{it}\bm{e}_{t}^{\top}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$.
Define $\bm{w}_{j}\in{\mathbb{R}}^{T}$ by

$\bm{w}_{j}^{\top}\stackrel{{\scriptstyle\text{\tiny
def}}}{{=}}\frac{\bm{e}_{j}^{\top}\bm{\Sigma}^{-1}\bm{X}^{\top}(\bm{Y}-\bm{X}{\widehat{\bm{B}}})+\bm{e}_{j}^{\top}({\widehat{\bm{B}}}-\bm{B}^{*})(n\bm{I}_{T}-\smash{\widehat{\mathbf{A}}})^{\top}}{\sqrt{n}\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}-\frac{\bm{z}_{j}^{\top}\bm{F}(\tilde{\bm{z}}_{j})}{\sqrt{n}}.$

Combining (I.1) and (I.2), by the triangle inequality,

$\displaystyle\sum_{j=1}^{p}{\mathbb{E}}\Big{[}\Big{\|}\bm{w}_{j}^{\top}\Big{\|}^{2}\Big{]}$
$\displaystyle=\sum_{j=1}^{p}{\mathbb{E}}\Big{[}\Big{\|}\frac{\bm{z}_{j}^{\top}\bm{F}(\bm{z}_{j})}{\sqrt{n}}-\sum_{i}\pdv{\bm{e}_{i}^{\top}\bm{F}(\bm{z}_{j})}{z_{ij}}/\sqrt{n}-\frac{\bm{z}_{j}^{\top}\bm{F}(\tilde{\bm{z}}_{j})}{\sqrt{n}}-\frac{{\rm
Rem}_{j}}{\sqrt{n}}\Big{\|}^{2}\Big{]}$
$\displaystyle\leq\frac{2}{n}{\mathbb{E}}\Big{[}\sum_{j=1}^{p}\|{\rm
Rem}_{j}\|^{2}\Big{]}+\frac{6}{n}{\mathbb{E}}\Big{[}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{F}(\bm{z}_{j})}{z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}.$

We now bound the two terms in the last line. For the first term, it is bounded
from above by $C(\zeta,T,\kappa,\gamma){\rm var}(y_{1})$ thanks to Lemma I.1.
For the second term, we first rewrite the derivative with respect to $z_{ij}$
using chain rule of differentiation. Recall
$\bm{X}=\bm{X}\bm{\Pi}_{j}+\bm{z}_{j}\frac{\bm{e}_{j}^{\top}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$.
Conditional on $\bm{X}\bm{\Pi}_{j}$, we have
$\pdv{\bm{F}}{z_{ij}}=\pdv{\bm{F}}{x_{ij}}\frac{1}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}$.
Thus,

$\displaystyle\frac{1}{n}{\mathbb{E}}\Big{[}\sum_{i=1}^{n}\sum_{j=1}^{p}\left\lVert\pdv{\bm{F}(\bm{z}_{j})}{z_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}=\frac{1}{n}{\mathbb{E}}\Big{[}\sum_{t=1}^{T}\sum_{i=1}^{n}\sum_{j=1}^{p}(\bm{e}_{j}^{\top}\bm{\Sigma}^{-1}\bm{e}_{j})^{-1}\left\lVert\pdv{\bm{F}\bm{e}_{t}}{x_{ij}}\right\rVert_{\rm{F}}^{2}\Big{]}.$

According to Lemma I.2, the last line is bounded by
$C(\zeta,T,\kappa,\gamma)(\|\bm{\Sigma}^{1/2}b^{*}\|+\sigma^{2})$. In summary,
$\sum_{j=1}^{p}{\mathbb{E}}[\|\bm{w}_{j}^{\top}\|^{2}]\leq
C(\zeta,T,\kappa,\gamma){\rm var}(y_{1}).$ Let

$\bm{L}_{n}=(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}.$

Left multiplying $(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}$ inside the
$\ell_{2}$ norm of $\bm{w}_{j}$, we have

$\sum_{j=1}^{p}{\mathbb{E}}\Big{[}\Big{\|}\frac{\bm{L}_{n}(\bm{Y}-\bm{X}{\widehat{\bm{B}}})^{\top}\bm{X}\bm{\Sigma}^{-1}\bm{e}_{j}+n({\widehat{\bm{B}}}-\bm{B}^{*})^{\top}\bm{e}_{j}}{\sqrt{n}\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}-\frac{\bm{L}_{n}\bm{F}(\tilde{\bm{z}}_{j})^{\top}\bm{z}_{j}}{\sqrt{n}}\Big{\|}^{2}\Big{]}=\sum_{j=1}^{p}{\mathbb{E}}\Big{[}\Big{\|}\bm{L}_{n}\bm{w}_{j}\Big{\|}^{2}\Big{]}.$

Since $\|(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}\|_{\rm op}\leq
C(\zeta,T)(1+\lVert\bm{X}\rVert_{\rm{op}}^{2}/n)^{T^{2}}$ from Lemma E.3,
define the event $\Omega=\\{\bm{X}\in{\mathbb{R}}^{n\times
p}:\lVert\bm{X}\rVert_{\rm{op}}/\sqrt{n}\leq 2+\sqrt{\gamma}\\}$ as before, so
that

$\sum_{j=1}^{p}{\mathbb{E}}[\|\bm{L}_{n}\bm{w}_{j}\|^{2}]\leq\sum_{j=1}^{p}C(\zeta,T,\gamma){\mathbb{E}}\Bigl{[}\|\mathbb{I}\\{\Omega\\}\bm{w}_{j}\|^{2}\Bigr{]}+\sum_{j=1}^{p}{\mathbb{E}}\Bigl{[}\|\mathbb{I}\\{\Omega^{c}\\}\bm{L}_{n}\bm{w}_{j}\|^{2}\Bigr{]}.$

For the first term with $\mathbb{I}\\{\Omega\\}$, the previously established
bound $\sum_{j=1}^{p}{\mathbb{E}}[\|\bm{w}_{j}\|^{2}]\leq
C(\zeta,T,\kappa,\gamma){\rm var}(y_{1})$ bounds from above the first term.
Each summand in the second term is exponentially small, using
${\mathbb{E}}[\mathbb{I}\\{\Omega^{c}\\}\|\bm{w}_{j}\|^{2}]\leq{\mathbb{P}}(\Omega^{c})^{1/2}{\mathbb{E}}[\|\bm{w}_{j}\|^{4}]^{1/2}$
with ${\mathbb{P}}(\Omega^{c})^{1/2}\leq e^{-n/4}$, while
${\mathbb{E}}[\|\bm{w}_{j}\|^{4}]^{1/2}/{\rm var}(y_{1})$ is at most
polynomial in $(n,p)$ with multiplicate constant $C(\zeta,T,\kappa,\gamma)$
thanks to several applications of the Cauchy-Schwarz inequality to combine the
moment bounds in Lemmas E.8 and E.10 and the equality in distribution
$\bm{F}(\tilde{\bm{z}}_{j})=^{d}\bm{Y}-\bm{X}{\widehat{\bm{B}}}$.

We wish to show that for the 2-Wasserstein distance,

(I.3)
$\max_{j\in[p]}W_{2}\Bigl{(}\mathsf{Law}\bigl{(}\frac{\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}}{\sqrt{n}}\bigr{)},~{}~{}\mathsf{N}\bigl{(}\mathbf{0}_{T},\bm{S}+{\mathbb{E}}[\bm{H}^{\top}\bm{\Sigma}\bm{H}]\bigr{)}\Bigr{)}\to
0.$

By Lemma E.10, we may extract a subsequence of regression problems such that
there exists psd matrices $\bm{K},\bar{\bm{K}}\in{\mathbb{R}}^{T\times T}$
such that (E.14) holds. We will not adapt a particular notation for the
extracted subsequence, but keep in mind that we are now working along this
subsequence and that $n\to+\infty$ with $n$ being restricted to this
subsequence.

By (E.16) and (E.17),

(I.4) ${\mathbb{E}}[\|\bm{F}^{\top}\bm{F}/n-\bm{K}\|_{\rm F}^{2}]\to
0,\qquad{\mathbb{E}}[\|\bm{H}^{\top}\bm{\Sigma}\bm{H}+\bm{S}-\bar{\bm{K}}\|_{\rm
F}^{2}]\to 0.$

Note that Theorem 2.2 further shows that

(I.5)
${\mathbb{E}}[\|\bm{L}_{n}(\bm{F}^{\top}\bm{F}/n)\bm{L}_{n}^{\top}-\bar{\bm{K}}\|_{\rm
F}]\to 0.$

Consider the LDLT decompositions $\bm{K}=\bm{L}\bm{D}\bm{L}^{\top}$ and
$\bar{\bm{K}}=\bar{\bm{L}}\bar{\bm{D}}\bar{\bm{L}}^{\top}$ for the psd
matrices $\bm{K}$ and $\bar{\bm{K}}$, so that $\bm{D},\bar{\bm{D}}$ are
diagonal with non-negative entries and $\bm{L},\bar{\bm{L}}$ are lower
triangular with all diagonal entries equal to 1. Since Lemma E.3 gives
$\|\bm{L}_{n}\|_{\rm op}=O_{P}(1)$,

(I.6)
$\bm{L}_{n}\bm{K}\bm{L}_{n}^{\top}-\bar{\bm{K}}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0,\qquad\bar{\bm{L}}^{-1}\bm{L}_{n}\bm{L}\bm{D}-\bar{\bm{D}}\bar{\bm{L}}^{\top}(\bm{L}_{n}^{\top})^{-1}(\bm{L}^{\top})^{-1}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0.$

On the right, we have the difference of a lower triangular matrix with
diagonal $\bm{D}$ and an upper triangular matrix with diagonal $\bar{\bm{D}}$.
Thus the convergence to 0 in probability gives $\bm{D}=\bar{\bm{D}}$ and
$\bar{\bm{L}}^{-1}\bm{L}_{n}\bm{L}\bm{D}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}\bm{D}$.
We also have
$\bar{\bm{L}}^{-1}\bm{L}_{n}\bm{L}\sqrt{\bm{D}}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}\sqrt{\bm{D}}$
by multiplying on the right by the pseudo-inverse of $\sqrt{\bm{D}}$. Now let
$\bm{L}_{\infty}=\bar{\bm{L}}\bm{L}^{-1}$, so that

$\displaystyle(\bm{L}_{n}-\bm{L}_{\infty})\bm{L}\sqrt{\bm{D}}$
$\displaystyle=\bar{\bm{L}}\bar{\bm{L}}^{-1}(\bm{L}_{n}-\bm{L}_{\infty})\bm{L}\sqrt{\bm{D}}$
$\displaystyle=\bar{\bm{L}}(\bar{\bm{L}}^{-1}\bm{L}_{n}\bm{L}\sqrt{\bm{D}}-\sqrt{\bm{D}}){\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0.$

By the continuous mapping theorem,
$(\bm{L}_{n}-\bm{L}_{\infty})\bm{K}(\bm{L}_{n}-\bm{L}_{\infty})^{\top}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0$,
and since Lemma E.3 gives $\|\bm{L}_{n}\|_{\rm op}=O_{P}(1)$,

(I.7)
$(\bm{L}_{n}-\bm{L}_{\infty})(\bm{F}^{\top}\bm{F}/n)(\bm{L}_{n}-\bm{L}_{\infty})^{\top}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0,\qquad\|(\bm{L}_{n}-\bm{L}_{\infty})\bm{F}^{\top}/\sqrt{n}\|{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0.$

Although we have not shown that
$\bm{L}_{n}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}\bm{L}_{\infty}$, this
means that we can still replace $\bm{L}_{n}$ by the deterministic
$\bm{L}_{\infty}$ up to when multiplied on the right by
$\bm{F}^{\top}/\sqrt{n}$. The previous display also holds with $\bm{F}$
replaced by $\bm{F}({\widetilde{\bm{z}}}_{j})$, uniformly over $j\in[p]$ since

(I.8)
$\displaystyle{\mathbb{E}}\|(\bm{L}_{n}-\bm{L}_{\infty})\tfrac{\bm{F}^{\top}\bm{F}}{n}(\bm{L}_{n}-\bm{L}_{\infty})^{\top}-(\bm{L}_{n}-\bm{L}_{\infty})\tfrac{\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{F}({\widetilde{\bm{z}}}_{j})}{n}(\bm{L}_{n}-\bm{L}_{\infty})^{\top}\|_{\rm
op}$ $\displaystyle\leq{\mathbb{E}}[\|\bm{L}_{n}-\bm{L}_{\infty}\|_{\rm
op}^{2}\|\bm{F}^{\top}\bm{F}/n-\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{F}({\widetilde{\bm{z}}}_{j})/n\|_{\rm
op}]$ $\displaystyle\leq{\mathbb{E}}[\|\bm{L}_{n}-\bm{L}_{\infty}\|_{\rm
op}^{4}]^{1/2}2{\mathbb{E}}[\|\bm{F}^{\top}\bm{F}/n-{\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\|_{\rm
op}^{2}]\to 0$

thanks to the Cauchy-Schwarz inequality and the equality in distribution
$\bm{F}=\bm{F}({\widetilde{\bm{z}}}_{j})$ for the last inequality. The last
line converges to 0 thanks to Lemma E.3 to show that
${\mathbb{E}}[\|\bm{L}_{n}-\bm{L}_{\infty}\|_{\rm op}^{4}]^{1/2}$ is bounded,
and thanks to (E.16) which gives
${\mathbb{E}}[\|\bm{F}^{\top}\bm{F}/n-{\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\|_{\rm
op}^{2}]\to 0$. This gives that
$\|\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})/\sqrt{n}\|_{\rm
op}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0$ uniformly over $j\in[p]$, so
that

$\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}=\bm{L}_{\infty}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}+o_{P}(1).$

Since $\bm{L}_{\infty}$ is deterministic, the distribution
$\bm{L}_{\infty}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}$ is
independent of $j$ and is the same as the distribution of
$\bm{L}_{\infty}\bm{F}^{\top}\bm{g}/\sqrt{n}$ where
$\bm{g}\sim\mathsf{N}(\mathbf{0},\bm{I}_{n})$ is independent of $\bm{F}$. For
any $\bm{w}\in{\mathbb{R}}^{T}$, the characteristic function of
$\bm{L}_{\infty}\bm{F}^{\top}\bm{g}/\sqrt{n}$ evaluated at $\bm{w}$ is

$\displaystyle{\mathbb{E}}[\exp(\sqrt{-1}\bm{w}^{\top}\bm{F}^{\top}\bm{g}/\sqrt{n})]$
$\displaystyle={\mathbb{E}}\big{[}{\mathbb{E}}[\exp(\sqrt{-1}\bm{w}^{\top}\bm{L}_{\infty}\bm{F}^{\top}\bm{g}/\sqrt{n})\mid\bm{g}]\big{]}$
$\displaystyle={\mathbb{E}}\big{[}\exp(-\tfrac{1}{2n}\bm{w}^{\top}\bm{L}_{\infty}\bm{F}^{\top}\bm{F}\bm{L}_{\infty}^{\top}\bm{w})\big{]}.$

This converges to $\exp(-\frac{1}{2}\bm{w}^{\top}\bar{\bm{K}}\bm{w})$ by (I.7)
and (I.5). We have established the weak convergence
$\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}{\smash{\xrightarrow{~{}\mathrm{d}~{}}}}\mathsf{N}(\mathbf{0},\bar{\bm{K}})$
uniformly over $j\in[p]$.

To prove convergence in 2-Wasserstein distance to
$\mathsf{N}(\mathbf{0},\bar{\bm{K}})$, it is enough to establish convergence
in distribution and convergence in the second moment (Villani et al., 2009,
Def. 6.8 and Theorem 6.9). Since we have already established convergence in
distribution, it is enough to prove

(I.9)
${\mathbb{E}}[\|\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}\|^{2}]\to\operatorname{Tr}\bar{\bm{K}}={\mathbb{E}}[\|\mathsf{N}(\mathbf{0},\bar{\bm{K}})\|^{2}].$

Since $\bm{F}$ is equal in distribution to $\bm{F}({\widetilde{\bm{z}}}_{j})$,
and ${\widetilde{\bm{z}}}_{j}$ is independent of
$\bm{F}({\widetilde{\bm{z}}}_{j})$, we have

${\mathbb{E}}[\|\bm{L}_{\infty}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}\|^{2}]=\operatorname{Tr}[\bm{L}_{\infty}{\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\bm{L}_{\infty}^{\top}]\to\operatorname{Tr}[\bm{L}_{\infty}\bm{K}\bm{L}_{\infty}^{\top}]=\operatorname{Tr}\bar{\bm{K}}.$

where we used that $\bm{D}=\bar{\bm{D}}$ and
$\bm{L}_{\infty}=\bar{\bm{L}}\bm{L}^{-1}$ for the last inequality. By the
triangle inequality in $L^{2}({\mathbb{R}}^{T})$,

$\Big{|}{\mathbb{E}}\Bigl{[}\|\bm{L}_{\infty}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}\|^{2}\Bigr{]}^{\frac{1}{2}}-{\mathbb{E}}\Bigl{[}\|\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}\|^{2}\Bigr{]}^{\frac{1}{2}}\Big{|}\leq{\mathbb{E}}\Bigl{[}\|(\bm{L}_{n}-\bm{L}_{\infty})\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}/\sqrt{n}\|^{2}\Bigr{]}^{\frac{1}{2}}.$

Let ${\widetilde{\bm{Q}}}\in{\mathbb{R}}^{n\times n}$ be the orthogonal
projector on the image of $\bm{F}({\widetilde{\bm{z}}}_{j})$, which is of rank
at most $T$ and such that
$\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}{\widetilde{\bm{Q}}}=\bm{F}({\widetilde{\bm{z}}}_{j})$.
Let $\chi^{2}_{T}=\|{\widetilde{\bm{Q}}}\bm{z}_{j}\|^{2}$ which has chi-square
distribution with at most $T$ degrees of freedom. We may bound from above the
integrand in the right-hand side by
$\|\bm{L}_{n}-\bm{L}_{\infty}\bm{F}({\widetilde{\bm{z}}}_{j})/\sqrt{n}\|^{2}\chi^{2}_{T}$.
Since $T$ is bounded, $\chi^{2}_{T}=O_{P}(1)$ and
$\|\bm{L}_{n}-\bm{L}_{\infty}\bm{F}({\widetilde{\bm{z}}}_{j})/\sqrt{n}\|{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0$
by (I.8). Since the integrand has finite second moment by Lemmas E.10 and E.3
and ${\mathbb{E}}[(\chi^{2}_{T})^{4}]<\infty$, it is uniformly integrable so
the last display converges to 0.

We have established convergence in distribution and in the second moment,
hence by (Villani et al., 2009, Def. 6.8 and Theorem 6.9),

(I.10)
$\max_{j\in[p]}W_{2}\Bigl{(}\mathsf{Law}\Bigl{(}\frac{\bm{L}_{n}\bm{F}({\widetilde{\bm{z}}}_{j})^{\top}\bm{z}_{j}}{\sqrt{n}}\Bigr{)},~{}~{}\mathsf{N}(0,\bar{\bm{K}})\Bigr{)}\to
0.$

Since $\bm{S}+{\mathbb{E}}[\bm{H}^{\top}\bm{\Sigma}\bm{H}]\to\bar{\bm{K}}$ and
$\bm{C}\mapsto\mathsf{N}(\mathbf{0},\bm{C})$ is continuous in 2-Wasserstein
distance, so that
$W_{2}(\mathsf{N}({\mathbf{0}},\bar{\bm{K}}),\bm{S}+{\mathbb{E}}[\bm{H}^{\top}\bm{\Sigma}\bm{H}])\to
0$. Combined with (I.10) and (Villani et al., 2009, Corollary 6.11), we obtain
(I.3) along any subsequence such that (E.14) holds.

Actually, the convergence (I.3) holds without extracting a subsequence for the
following reason. Since both distributions inside $W_{2}$ in (I.3) have
uniformly bounded second (thanks to Lemmas E.10 and E.3), the $W_{2}$ distance
in (I.3) is uniformly bounded. To prove (I.3), it is sufficient to prove that
0 is the only limit point of the sequence in the left-hand side of (I.3). For
any subsequence such that the left-hand side of (I.3) converges to a limit
$\nu$, we may extract further a subsequence such that (E.14) and the above
argument holds, showing that the limit point must be $\nu=0$. Since the
bounded sequence in the left-hand side of (I.3) has a unique limit point equal
to 0, the convergence (I.3) holds. ∎

##  Appendix J Proof of Corollary 2.5

###### Proof of Corollary 2.5.

Let $J_{n,p}\subset[p]$ be the set of $j\in[p]$ such that

(J.1)
${\mathbb{E}}\Bigl{[}\Bigl{(}\sqrt{\frac{n}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|^{2}}}\Bigl{(}{\widehat{\bm{b}}}^{t,\rm{debias}}_{j}-\bm{b}^{*}_{j}\Bigr{)}-\bm{\zeta}_{jt}\Bigr{)}^{2}\Bigr{]}\leq\frac{1}{a_{p}}C(\zeta,T,\kappa,\gamma){\rm
var}(y_{1})$

for the same constant $C(\zeta,T,\kappa,\gamma)$ as in (2.23). Bounding the
left-hand side of (2.23) from below by the sum indexed over $[p]\setminus
J_{n,p}$, we get $\frac{1}{a_{p}}|[p]\setminus J_{n,p}|\leq 1$. On the one
hand,
$W_{2}(\mathsf{Law}(\bm{\zeta}_{jt}),\mathsf{N}(0,{\mathbb{E}}[r_{t}]))\to 0$
by (2.22), uniformly over $j\in[p]$. On the other hand, (J.1) provides a
vanishing upper bound on the $W_{2}$ distance between the law of
$\bm{\zeta}_{jt}$ and the law of
$\frac{\sqrt{n}}{\|\bm{\Sigma}^{-1/2}\bm{e}_{j}\|}({\widehat{\bm{b}}}^{t,\rm{debias}}_{j}-\bm{b}^{*}_{j})$,
uniformly over $j\in J_{n,p}$. The triangle inequality for the 2-Wasserstein
distance completes the proof of (2.24). ∎

##  Appendix K Asymptotic normality and state evolution in the separable case

######  Theorem K.1.

Let Assumptions 2.2, 2.1, 2.3 and 2.4 be fulfilled with
$\bm{\Sigma}=\bm{I}_{p}$, assume ${\rm var}(y_{1})\leq v_{0}$, and assume that
$T,v_{0}$ are fixed constant as $n,p\to+\infty$. Then there exists
deterministic weights $\bar{w}_{s,t}$ (possibly depending on $n,p$) such that

(K.1)
$\max_{t=1,...,T}\Bigl{(}\frac{1}{n}\|\sum_{s=1}^{t}(\hat{w}_{t,s}-\bar{w}_{t,s})(\bm{y}-\bm{X}{\widehat{\bm{b}}}^{s})\|^{2}\Bigr{)}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0.$

Given these deterministic weights $\bar{w}_{t,s}$, define the debiased
estimate $\bar{\bm{b}}^{T,\rm{debias}}\in{\mathbb{R}}^{p}$ by
$\bar{\bm{b}}_{j}^{T,\rm{debias}}={\widehat{\bm{b}}}^{T}_{j}+n^{-1}(\bm{X}\bm{e}_{j})^{\top}\sum_{s=1}^{T}\bar{w}_{T,s}(\bm{y}-\bm{X}{\widehat{\bm{b}}}^{s})$.

Let $(\eta_{j})_{j\in[p]}$ be $\zeta$-Lipschitz functions. Choose the $T+1$-th
nonlinear function $\bm{g}_{T+1}$ in (2.8) at $t=T+1$ by applying the
functions $\eta_{j}$ componentwise to $\bar{\bm{b}}_{j}^{T,\rm{debias}}$,
i.e.,
${\widehat{\bm{b}}}^{T+1}_{j}=\eta_{j}(\bar{\bm{b}}_{j}^{T,\rm{debias}}).$
Then the following asymptotic recursion holds between ${\mathbb{E}}[r_{T}]$
and ${\mathbb{E}}[r_{T+1}]$,

(K.2)
$\displaystyle\Bigl{(}\sigma^{2}+{\mathbb{E}}_{Z_{j}\sim\mathsf{N}(0,{\mathbb{E}}[r_{t}])}\Bigl{[}\sum_{j=1}^{p}\Bigl{\\{}\eta_{j}\bigl{(}\bm{b}_{j}^{*}+\frac{Z_{j}}{\sqrt{n}}\bigr{)}-\bm{b}_{j}^{*}\Bigr{\\}}^{2}\Bigr{]}\Bigr{)}^{1/2}-{\mathbb{E}}\Bigl{[}r_{T+1}\Bigr{]}^{1/2}$
$\displaystyle\to 0$

as $n,p\to+\infty$, as well as
$(\sigma^{2}+{\mathbb{E}}[\sum_{j=1}^{p}\\{\eta_{j}({\widehat{\bm{b}}}^{t,\rm{debias}}_{j})-\bm{b}_{j}^{*}\\}^{2}])^{1/2}-{\mathbb{E}}[r_{T+1}]^{1/2}\to
0$.

###### Proof of Theorem K.1.

Without loss of generality, assume that the sequence of regression problems
and data $(\bm{X},\bm{y})$ is defined on the same probability space
$(\Omega,\mathcal{A},\mathbb{P})$. We have already proved in Lemmas E.10, E.3
and E.13 and Theorem 2.2 that the event $\Omega_{n}$ defined as

(K.3) $\displaystyle\|(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}\|_{\rm
op}+\|\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n\|_{\rm op}\leq
C(\zeta,T,\gamma,v_{0}),$ (K.4)
$\displaystyle\|(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}\bm{F}^{\top}\bm{F}/n(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}-(\sigma^{2}{\mathbf{1}}_{T}{\mathbf{1}}_{T}^{\top}+\bm{H}^{\top}\bm{H})\|_{\rm
F}\leq n^{-0.49}$ (K.5)
$\displaystyle\|\bm{F}^{\top}\bm{F}/n-{\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\|_{\rm
F}+\|\bm{H}^{\top}\bm{H}-{\mathbb{E}}[\bm{H}^{\top}\bm{H}]\|_{\rm F}\leq
n^{-0.49}$

for a large enough constant $C(\zeta,T,\gamma,v_{0})$ has probability
approaching one. In particular, this event $\Omega_{n}\in\mathcal{A}$ is non-
empty and contains some outcome $\omega_{n}$. For each $s,t\leq T$, for a
given regression problem in the sequence (indexed by $n$), we may define the
weight $\bar{w}_{t,s}$ as the random variable
$\hat{w}_{t,s}=\bm{e}_{t}^{\top}(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}\bm{e}_{s}$
taken as the outcome $\omega_{n}$, so that $\bar{w}_{t,s}$ is deterministic.
Let us emphasize that this weight implicitly depends on $n$, i.e., it depends
on $n,p,\bm{b}^{*},(\bm{g}_{t})_{t\leq T}$ and all other parameters that are
allowed to change as $n,p\to+\infty$. Since $\omega_{n}$ is an outcome in
$\Omega_{n}$, the above inequalities give that
$\bar{\bm{W}}=(\bar{w}_{t,s})_{t\in T,s\in[T]}$ satisfies

(K.6)
$\|\bar{\bm{W}}{\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\bar{\bm{W}}^{\top}-(\sigma^{2}{\mathbf{1}}_{T}{\mathbf{1}}_{T}^{\top}+{\mathbb{E}}[\bm{H}^{\top}\bm{H}])\|_{\rm
F}\leq C(\zeta,T,\gamma,v_{0})n^{-0.49}$

in the sense of convergence of a deterministic sequence. Let us now show that
(K.1) holds. Denote by
${\widehat{\bm{W}}}=(\bm{I}_{T}-\smash{\widehat{\mathbf{A}}}/n)^{-1}$ so that
(K.1) is equivalent to
$\|(\bar{\bm{W}}-{\widehat{\bm{W}}})\bm{F}^{\top}/\sqrt{n}\|_{\rm
F}^{2}{\smash{\xrightarrow{~{}\mathrm{p}~{}}}}0$. Since convergence in
probability to 0 for the sequence $U_{n}$ is equivalent to convergence of
${\mathbb{E}}[1\wedge|U_{n}|]\to 0$, the convergence in probability (K.1) is
equivalent to $u_{n}\stackrel{{\scriptstyle\text{\tiny
def}}}{{=}}{\mathbb{E}}[1\wedge\|(\bar{\bm{W}}-{\widehat{\bm{W}}})\bm{F}^{\top}/\sqrt{n}\|_{\rm
F}^{2}]\to 0$. Consider a converging subsequence $(u_{n_{k}})$ of
$(u_{n})_{n\geq 1}$ with limit point $u_{\infty}\in[0,1]$. By (E.14), we may
further extract a subsequence $(u_{n_{k_{m}}})$, such that in this nested
subsequence $u_{n_{k_{m}}}\to u_{\infty}$ and
${\mathbb{E}}[\bm{F}^{\top}\bm{F}/n]\to\bm{K}$ both holds. In the event
$\Omega_{n}$, (K.6) is also satisfied for ${\widehat{\bm{W}}}$, hence in this
subsequence and in $\Omega_{n}$,

(K.7)