its concentration will follow from Lipschitz concentration results along the lines of the marginal concentration (Lemma <ref>) and conditional characterization (Lemma <ref>) or standard Gaussian concentration of Lipschitz functions.
Some care is needed to track the correct dependence on $p$ and $n_k$ when extending these results to $\< \ba^{(1)},\ba^{(2)}\>$.
The following lemma provides the required extension.

Consider two random vectors $\ba^{(1)},\ba^{(2)} \in \reals^n$, constants $C,c> 0$ and $\sC(\epsilon),\sc(\epsilon): \reals_{>0} \rightarrow \reals_{>0}$, $p \geq 1$, $M,K > 0$, and $\bS \in \SS_+^2$, which satisfy the following concentration guarantee:

* There exist $M$-Lipschitz functions $\bphi^{(1)},\bphi^{(2)}: \reals^n \rightarrow \reals$
such that for Gaussian vectors $\bxi^{(1)},\bxi^{(2)} \sim \normal(0,\bS \otimes \id_n) $ and any $1$-Lipschitz functions $\phi: \reals^n\rightarrow \reals$, $\phi': (\reals^n)^2\rightarrow \reals$,

\begin{equation}\label{eq:generic-concentration}
\begin{gathered}
\P\Big(
\Big|
\phi(\ba^{(k)})
\E\big[
\phi\big(
\bphi^{(k)}(\bxi^{(k)})
\big)
\big]
\Big|
C \epsilon
\Big)
\leq \sC(\epsilon)e^{-\sc(\epsilon)p}
\;\; \text{for $k = 1,2$},
\\
\P\Big(
\Big|
\phi'(\ba^{(1)},\ba^{(2)})
\E\big[
\phi'\big(
\bphi^{(1)}(\bxi^{(1)}),
\bphi^{(2)}(\bxi^{(2)})
\big)
\big]
\Big|
K \epsilon
\Big)
\leq \sC(\epsilon)e^{-\sc(\epsilon)p},
\end{gathered}
\end{equation}

\begin{equation}\label{eq:E-bound}
\E[\|\bphi^{(i)}(\bxi^{(i)})\|_2^2] \leq C,\quad i = 1,2.
\end{equation}

* The parameters above satisfy $M \leq CK/\sqrt{p}$ and the singular values of $\bS_\xi$ are bounded by $C$.

Then there exist $C',c'> 0$ and $\sC'(\epsilon),\sc'(\epsilon): \reals_{>0} \rightarrow \reals_{>0}$ depending only on $C,c> 0$ and $\sC(\epsilon),\sc(\epsilon): \reals_{>0} \rightarrow \reals_{>0}$
such that for any $\epsilon < c'$ we have

\begin{equation}
\P\Big(
\Big|
\<\ba^{(1)},\ba^{(2)}\>
\E\big[
\big\<
\bphi^{(1)}(\bxi^{(1)}),
\bphi^{(2)}(\bxi^{(2)})
\big\>
\big]
\Big|
K \epsilon
\Big)
\leq \sC'(\epsilon)e^{-\sc'(\epsilon)p}.
\end{equation}

In all applications of Lemma <ref>, we will take $C,c,\sC(\epsilon),\sc(\epsilon)$ to be $\cPregr,\cPmodel$, and regression-method dependent constants,
and $K,M$ to be constants which possibly depend also on $p/n_k$.

The function $\<\ba^{(1)},\ba^{(2)}\>$ is $2C$-Lipschitz on $\{\| \ba^{(1)}\|_2,\| \ba^{(2)}\|_2 \leq 2C \}$.
Define $\overline \bt(\ba^{(1)},\ba^{(2)})$ to be a $2C$-Lipschitz extension of $\<\ba^{(1)},\ba^{(2)}\>$ which agrees with $\<\ba^{(1)},\ba^{(2)}\>$ on $\{\| \ba^{(1)}\|_2,\| \ba^{(2)}\|_2 \leq 2C\}$.
By the first line of Eq. (<ref>),
we have with probability at least $1 - 2Ce^{-cp}$ that
$\max\{\| \ba^{(1)} \|_2,\| \ba^{(1)} \|_2\} \leq 2C$.
Thus, by the second line of Eq. (<ref>), with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p} - 2Ce^{-cp}$

\begin{equation}
\| \<\ba^{(1)},\ba^{(2)}\> - \E[\overline \bt(\bphi^{(1)},\bphi^{(2)})] \|_{\sF}
\| \<\ba^{(1)},\ba^{(2)}\> - \E[\overline \bt(\bphi^{(1)},\bphi^{(2)})] \|_{\sF}
\leq
2CK \epsilon.
\end{equation}

Moreover, becuase $\overline \bt(0,0) = 0$ and is $2C$-Lipschitz,
we have $\big|\overline \bt(\bphi^{(1)},\bphi^{(2)})\big| \leq 2C (\|\bphi^{(1)}\|_2 + \|\bphi^{(2)}\|_2) $.
By Cauchy-Schwartz, we have $|\< \bphi^{(1)},\bphi^{(2)}\>| \leq \| \bphi^{(1)} \|_2 \| \bphi^{(2)}\|_2$.

\begin{equation}
\big| \overline \bt(\bphi^{(1)},\bphi^{(2)}) - \<\bphi^{(1)},\bphi^{(2)}\> \big|
\leq
\big(2C(\| \bphi^{(1)} \|_2 + \|\bphi^{(2)} \|_2) + \| \bphi^{(1)} \|_2 \|\bphi^{(2)} \|_2\big)
\indic{\max\{\| \bphi^{(1)} \|_2, \|\bphi^{(2)} \|_2\} > 2C}.
\end{equation}

By sub-Gaussian concentration, using Eq. (<ref>) and that the singular values of $\bS$ are bounded above by $C$,
we have
$\P(\| \bphi^{(i)} \|_2 > 2C + t) \leq e^{-(C+t)^2/(2CM^2)}$,

\begin{equation}
\begin{aligned}
\big| \E\big[\overline \bt(\bphi^{(1)},\bphi^{(2)})] - \E\big[\<\bphi^{(1)},\bphi^{(2)}\> \big] \big|
&\leq \int_0^\infty (4C(C+t) + (C+t)^2) e^{-(C+t)^2/(CM^2)}\de t
\leq C' M
\\
&\leq \int_0^\infty C'(1+(t/M)^2) e^{- \big(2(t/M) + (t/M)^2\big)/C }\de t
\leq C' M .
\end{aligned}
\end{equation}

where $C'$ depends only on $C$ and changes at each appearance, and in the second inequality we have used $M \leq C$ to conclude that $(4C(C+t) + (C+t)^2) e^{-C/M^2} \leq C'(1+(t/M)^2)$ and $-2t/M^2 \leq - 2 t/(CM)$.
We conclude that

\begin{equation}
\P\Big(
\Big|
\<\ba^{(1)},\ba^{(2)}\>
\E\big[
\big\<
\bphi^{(1)}(\bxi^{(1)}),
\bphi^{(2)}(\bxi^{(2)})
\big\>
\big]
\Big|
C'M + K\epsilon
\Big)
\leq
\sC(\epsilon)e^{-\sc(\epsilon)p} + 2Ce^{-cp},
\end{equation}

For $K\epsilon \leq C'M$, we have $e^{1-C^2p\epsilon^2/(C'(M/K)\sqrt{p})^2} > 1$,

\begin{equation}
\P\Big(
\Big|
\<\ba^{(1)},\ba^{(2)}\>
\E\big[
\big\<
\bphi^{(1)}(\bxi^{(1)}),
\bphi^{(2)}(\bxi^{(2)})
\big\>
\big]
\Big|
\Big)
\leq
\sC(\epsilon)e^{-\sc(\epsilon)p} + 2Ce^{-cp} + e^{1-(K/M)^2\epsilon^2/C'^2},
\end{equation}

because the bound becomes trivial for $K\epsilon \leq C'M$ and is implied by the previous display for $K\epsilon > C'M$.
Because $(K/M)^2 \geq p/C^2$,
the result follows.

§ EMPIRICAL DEGREES-OF-FREEDOM: PROOF OF THEOREM <REF>(III) AND LEMMA <REF>(II)

We extend Theorem <ref>(i) and (ii) to the case that $\hat \btheta_k^\de$ and $\hat \be_k^\de$ are computed using $\hat \df_k$ in place of $\df_k$ (recall $\hat \df_k$ defined in Eq. (<ref>)),
and make the similar extension for Lemma <ref>(i).
To distinguish between $\hat \btheta_k^\de,\hat \be_k^\de$ which are computed using $\hat \df_k$ from those computed using $\df_k$,
in this section we denote by $\hat \btheta_k^{\de,\mathrm{emp}},\hat \be_{k}^{\de,\mathrm{emp}}$
the fully empirical debiased estimates which use $\hat \df_k$.
As in the rest of the paper, the notation $\hat \btheta_k^\de,\hat \be_k^\de$ will denote the quantities computed using $\df_k$, as in Eqs. (<ref>) and (<ref>).

Our main tool is the following.

Assume  and .

There exist $\cPmodel$, $\cPregr$ and regression method-dependent $c' > 0$ and $\sC,\sc: \reals_{>0} \rightarrow \reals_{>0}$ such that

\begin{equation}
\begin{gathered}
\Big|
\frac{\hat \df_k}{p} - \frac{\df_k}{p}
\Big| < \epsilon.
\end{gathered}
\end{equation}

There exist $\cPmodel$, $\cPregr$ and regression method-dependent $c' > 0$ and $\sC,\sc: \reals_{>0} \rightarrow \reals_{>0}$ such that

\begin{equation}
\begin{gathered}
\| \hat \btheta_k^{\de,\mathrm{emp}} - \hat \btheta_k^\de \|_2 < \frac{p}{n_k}\,\epsilon,
\qquad\qquad
\frac{\| \hat \be_{k,\cI_l}^{\de,\mathrm{emp}} - \hat \be_{k,\cI_l}^{\de,\mathrm{emp}} \|_2 }{\sqrt{n_l}}
\leq \frac{p}{n_k}\,\epsilon.
\end{gathered}
\end{equation}

In the case of least-squares, $\hat \df_k = p = \df_k$, and there is nothing to prove.

In the case of the Lasso, this follows from Theorem 8 of [15],
where we recall that for the lasso we assume in  that $c < n_k / p < C$, so that $n$ and $p$ may be interchanged at the cost of changing constants.

For ridge-regression, the bound follows from Theorem 3.7 of [29], as we now describe.
By Eq. (<ref>),
we have that $\hat \df_k/p = 1 - \sqrt{\frac{p}{n_k}}\,\lambda_k \Tr\big(\frac1p\big(\frac1{n_k}\bX_{\cI_k}^\top \bX_{\cI_k} + \sqrt{\frac{p}{n_k}}\,\lambda_k\id_p\big)^{-1}\big)$.
The trace on the right-hand side is the resolvent of $\bX_{\cI_k}^\top \bX_{\cI_k}/n_k$ evaluated at $-\lambda_k \sqrt{p/n_k}$ and normlized by $1/p$.
Theorem 3.7 of [29] implies that this resolvent has fluctations $O_p(1/\sqrt{n_k})$ from a deterministic quantity defined by Eqs. (3.5) and (2.1) of that paper, where $O_p$ hides only $\cPregr$ and $\cPmodel$-dependent constants.
We compare these equations to the second line of the fixed-point equations Eqs. (<ref>), which in the case of ridge regression do not depend on $\tau$.
Some algebra shows that the deterministic quantity given by [29] corresponds to $\zeta_k/(\sqrt{p/n_k}\,\lambda_k)$.
Thus, we conclude that $\hat \df_k/p$ has fluctuations $O_p(\sqrt{p}/n_k) = O_p(1/\sqrt{p})$ around $\df_k$.

We now show that the concentration occurs with exponentially (in $p$) high probability.
By Weyl's inequality, the singular values of $\bX_{\cI_k}/\sqrt{n_k}$ are $1$-Lipschitz in $\bX_{\cI_k}/\sqrt{n_k}$ with respect to Frobenius norm.
Because $x \mapsto (x^2 + \lambda_k \sqrt{p/n_k})^{-1}$ is $C (n_k/p)^{3/4} $-Lipschitz,
we see that $\hat \df_k/p$ is $C \sqrt{p/n_k} \lambda_k \,(n_k/p)^{3/4} \leq C(n_k/p)^{1/4} \leq C\sqrt{n_k/p} $-Lipschitz in the $\bX_{\cI_k}/\sqrt{n_k}$ with respect to Frobenius norm.
By Gaussian concentration of Lipschitz functions,
we conclude that with probability at least $1 - Ce^{-cp\epsilon^2}$ we have $\big| \hat \df_k / p - \E[\hat \df_k/p]\big| < \epsilon$.
Combined with the fluctuation bound in the previous paragraph, we conclude that $\big|\E[\hat \df_k/p] - \df_k/p\big| \leq C/\sqrt{p}$.
By adjusting constants so that the bound becomes trivial for $\epsilon \leq C/\sqrt{p}$,
we conclude Lemma <ref> in the case of ridge regression.

\begin{equation}
\| \hat \btheta_k^{\de,\mathrm{emp}} - \hat \btheta_k^\de \|_2
\leq
\frac{\| \bX_{\cI_k} \bSigma^{-1} \|_{\mathrm{op}}}{\sqrt{n_k}}
\frac{\| \hat \be_{k,\cI_k} \|_2}{\sqrt{n_k}}
\Big|
\frac1{1 - \hat \df_k/n_k} - \frac1{1 - \df_k/n_k}
\Big|.
\end{equation}

By Eq. (<ref>) and because $\hat \be_{k,\cI_k^c} = 0$,
$\hat \be_k^{\de,\mathrm{emp}}$ and $\hat \be_k^{\de}$ differ only on the indices in $\cI_k$.
one can write $\hat \be_{k,\cI_k}^\de = \hat \be_{k,\cI_k}/(1 - \df_k/n_k)$
and $\hat \be_{k,\cI_k}^{\de,\mathrm{emp}} = \hat \be_{k,\cI_k}/(1 - \hat \df_k/n_k)$.

\begin{equation}
\frac{\| \hat \be_{k,\cI_2}^{\de,\mathrm{emp}} - \hat \be_{k,\cI_2}^{\de,\mathrm{emp}} \|_2 }{\sqrt{n_2}}
\frac{\| \hat \be_{k,\cI_k \cap \cI_2}^{\de,\mathrm{emp}} - \hat \be_{k,\cI_k \cap \cI_2}^{\de,\mathrm{emp}} \|_2 }{\sqrt{n_2}}
\leq
\frac{\| \hat \be_{k, \cI_2}\|_2}{\sqrt{n_2}}
\Big|
\frac1{1 - \hat \df_k/n_k} - \frac1{1 - \df_k/n_k}
\Big|.
\end{equation}

where we have used in the inequality that $\| \hat \be_{k,\cI_k \cap \cI_2}\|_2 \leq \| \hat \be_{k,\cI_2}\|_2$.

We have that ,
$\| \bX_{\cI_k} \bSigma^{-1} \|_{\mathrm{op}}/\sqrt{n_k} < C$ and $\| \hat \be_{k,\cI_k}\|_2 / \sqrt{n_k} < C$,
where in the first bound we use
<cit.> and that the singular values of $\bSigma$ are bouned below and above by $c$ and $C$,
and in the second bound we use the marginal characterization (Lemma <ref>) and that $\E[\|\hat \be_{k,\cI_k}^f\|_2]/\sqrt{n_k} \leq \E[\|\hat \be_{k,\cI_k}^f\|_2^2]^{1/2}/\sqrt{n_k} \leq C$.
By Lemma <ref> and using that $1 - \df_k/n_k = \zeta_k > c$ by Eq. (<ref>) and Lemma <ref>,
we conclude that , $\big|(1 - \hat\df_k/n_k)^{-1} - (1 - \df_k/n_k)^{-1}\big| < (p/n_k)\epsilon$.
We conclude Lemma <ref>(ii) by combining these bounds with the previous two displays.

Using Lemma <ref>, Theorem <ref>(iii) follows from Theorem <ref>(i) and (ii) and Lemma <ref>(ii) follows from Lemma <ref>(i).

§ THE GOOD MARGINAL CHARACTERIZATION EVENT: PROOF OF LEMMA <REF>

First, consider the concentration of $\bT\Big(\frac{\be_{k,\cI_k}}{\sqrt{n_k}},\frac{\bxi_{h,\cI_k}}{\sqrt{n_k}}\Big) $.
For the first diagonal entry, there is nothing to show because it is a function of $\be_k$.
The conditional expectation of the second diagonal entry is $1$ for all realization of $\be_k$, because $\bxi_h$ is independent of $\be_k$.
Because $\bxi_{h,\cI_k}$ is a Gaussian vector with variance $\id_{n_k}$,
by Gaussian concentration of Lipschitz functions Eq. (<ref>) is satisfied with $\ba^{(1)} = \ba^{(2)} = \bxi_{h,\cI_k}/\sqrt{n_k}$, $\bxi^{(1)},\bxi^{(2)} = \bxi_{h,\cI_k}$, $\bphi^{(1)}(\ba) = \bphi^{(2)}(\ba) = \ba / \sqrt{n_k}$, and $K = \sqrt{p/n_k}$.
We see that $M = 1/\sqrt{n_k} \leq C / \sqrt{p}$ and $\bS_\xi$ has bounded singular values.
Moreover, $\E[\|\bxi_{h,\cI_k}\|_2^2]/n_k = 1$, so Eq. (<ref>) is satisfied.
Thus, Lemma <ref> implies the desired concentration of the second diagonal entry.

Now consider the off-diagonal entries.
Because $\be_{k,\cI_k}$ is a Gaussian vector with variance $\tau_{e_k}^2\id_{n_k}$, and $\tau_{e_k}^2 \leq C$,
by Gaussian concentration Eq. (<ref>) is satisfied with $\ba^{(1)} = \ba^{(2)} = \be_{k,\cI_k}/\sqrt{n_k}$, $\bxi^{(1)},\bxi^{(2)} = \be_{k,\cI_k}$, $\bphi^{(1)}(\ba) = \bphi^{(2)}(\ba) = \ba / \sqrt{n_k}$, and $K = \sqrt{p/n_k}$.
We see that $M = 1/\sqrt{n_k} \leq C / \sqrt{p}$ and $\bS_\xi$ has bounded singular values.
Moreover, $\E[\|\be_{k,\cI_k}\|_2^2]/n_k = \tau_{e_k}^2 \leq C$, so Eq. (<ref>) is satisfied.
Thus, we may apply Lemma <ref>.
We conclude that , $\big| \| \hat \be_{k,\cI_k} \|_2^2 / n_k - \tau_{e_k}^2\big| < \sqrt{p/n_k}\,\epsilon$.
with probability at least $1 - Ce^{-cp}$, $\| \be_{k,\cI_k}\|_2^2 / n_k \leq \tau_{e_k}^2 + \sqrt{p/n_k} \leq C$, where the second inequality uses , .
For such $\be_{k,\cI_k}$, the distribution of $\< \be_{k,\cI_k} , \xi_{h,\cI_k} \> / n_k$ conditional on $\be_{k,\cI_k}$ is Gaussian with variance $C^2 / n_k$.
Then, the desired concentration of the off diagonal entries occurs by Gaussian concentration and the fact that $\| \be_{k,\cI_k}\|_2^2 / n_k \leq C$ with high-probability.

Second, consider the concentration of $\| \be_{k,\cI_k} \|_2 / \sqrt{n_k}$.
The previous paragraphs shows that , $\big| \| \hat \be_{k,\cI_k} \|_2^2 / n_k - \tau_{e_k}^2\big| < \sqrt{p/n_k}\,\epsilon$. Because by assumption  $\tau_{e_k}^2 > c$,
taking square roots gives the desired concentration of $\| \be_{k,\cI_k} \|_2 / \sqrt{n_k}$.

Third, consider the concentration of $\bT\Big(\sqrt{\frac{n_k}{p}}\,\bv_k^f,\frac1{\sqrt{p}}\bxi_g\Big)$.
By Eqs. (<ref>), (<ref>), and because proximal operators are $1$-Lispchitz [36],
the vectors $\sqrt{n_k/p}\, \bv_k^f, \bxi_g^f / \sqrt{p}$ are $(\sqrt{n_k/p}\,\tau_{g_k} \vee 1/\sqrt{p})$-Lipschitz functions of $\bxi_g$,
and by Lemma <ref>, we have $\sqrt{n_k/p}\,\tau_{g_k} \leq C/\sqrt{p}$.
Thus, we may take $M = C/\sqrt{p}$ in Lemma <ref>.
By Gaussian concentration, we get Eq. (<ref>) for $K = 1$.
Finally, by Eq. (<ref>), Eq. (<ref>) is satisfied,
whence we may apply Lemma <ref> and get the desired concentration.

Next, we establish the concentration of $\bar \Omega_k(\bv_k^f)$.
In the case of least-squares, $\bar \Omega_k(\bv_k^f) = 0$ always,
and there is nothing to show.
In the case of ridge-regression,
we apply Lemma <ref>.
Recall that $\bar \Omega_k(\bv_k^f) = \sqrt{p/n_k}\,(\lambda/2)\|\hat \btheta_k^f\|_2^2$.
By Eqs. (<ref>) and (<ref>), because proximal operators are 1-Lipschitz [36] and $\bSigma$ has singular values bounded below by $c$ and $\tau_{g_k} \leq C / \sqrt{n_k}$ by Lemma <ref>,
we have that $\hat \btheta_k^f$ is $M= 1 / \sqrt{n_k}$-Lipschitz in $\bg_k^f / \tau_{g_k}$, which is standard Gaussian.
Moreover, by the marginal characterization (Lemma <ref>),
Eq. (<ref>) holds for $K = \sqrt{p/n_k}$.
By  and Lemma <ref>, we have $\E[\|\btheta_k + \bSigma^{-1/2}\bv_k^f\|_2^2] \leq 2\|\btheta_k\|_2^2 + 2 C\E[\|\bv_k^f\|_2^2] \leq C(2 + p/n_k) \leq C$.
Thus, using that $\omega_k = \E[\bar \Omega_k(\bv_k^f)]$ (see Eq. (<ref>)),
we my apply Lemma <ref> to get the required concentration of $\bar \Omega_k(\bv_k^f)$, where we use additionally that $\lambda < C$ by .
In the case of the $\alpha$-smoothed Lasso,
we have that $\bar \Omega_k(\bv_k^f) = \lambda \| \btheta_k + \bSigma^{-1/2} \bv_k^f \|_1/\sqrt{n_k}$ is
$C \lambda \tau_{g_k} \sqrt{p/n_1} \leq C \sqrt{p}/n_k$-Lipschitz in $\bxi_g$.
The required concentration of $\bar \Omega_k(\bv_k^f)$ follows by Gaussian concentration of Lipschitz functions.

Because $\bv_k^f$ is $\tau_{g_k} \leq C/\sqrt{n_k}$-Lipschitz in $\bxi_g$,
Gaussian concentration of Lipschitz functions implies that $\bv_k^f \in E_v(\Delta/2)$ with the desired high-probability.
Because $\bu_k^f$ is $\zeta_k \tau_{h_k} \leq  C\sqrt{p/n_k}$-Lipschitz function of in $\bxi_h$ conditional on $\be_1,\be_2$,
Gaussian concentration of Lipschitz functions implies that $\bu_k^f \in E_u(\Delta/2)$ with the desired high-probability.

This completes the proof of Lemma <ref>.

§ APPLICATIONS OF THE MARGINAL CHARACTERIZATION

In this section, we prove several consequences of the marginal characterization (Lemma <ref>).
In Section <ref>, we prove that $\phi_{\theta|1}(\mathsf{Cond}_1)$ and $\phi_{e|1}(\mathsf{Cond}_1)$ concentrate (Lemma <ref>).
In Section <ref>,
we prove that the good conditional characterization event occurs with high probability (Lemma <ref>).
The proof of Lemmas <ref> and <ref> will require some corollaries of the marginal characterization which are stated and proved in Section <ref>.

§.§ Corollaries of the marginal characterization

In this section, we prove Lemma <ref> and state and prove two corollaries of the marginal characterization which will be useful in proving Lemmas <ref> and <ref> and Theorem <ref>.
These proofs rely on the concentration of certain random-design quantities, which are easily established using Lemma <ref>.
For convenient reference, we collect these concentration statements in the next lemma.

Assume  and .
Then ,

\begin{equation}\label{eq:param-T}
\begin{gathered}
\Big\|
\bT\Big(
\sqrt{\frac{n_k}{p}}\, \bv_k ,\,
\sqrt{\frac{n_k}{p}}\, \bSigma^{1/2}(\hat \btheta_k^\de - \btheta_k)
\Big)
\E\Big[
\bT\Big(
\sqrt{\frac{n_k}{p}}\, \bv_k^f ,\,
\sqrt{\frac{n_k}{p}}\,\bg_k^f
\Big)
\Big]
\Big\|_{\sF}
< C'\epsilon,
\end{gathered}
\end{equation}

\begin{equation}\label{eq:noise-T}
\begin{gathered}
\Big\|
\bT\Big(
\frac{\be_{1,\cI_k}}{\sqrt{n_k}},\,
\frac{\be_{2,\cI_k}}{\sqrt{n_k}},\,
\frac{\bu_{k,\cI_k}}{\sqrt{n_k}}
\Big)
\E\Big[
\bT\Big(
\frac{\be_{1,\cI_k}}{\sqrt{n_k}},\,
\frac{\be_{2,\cI_k}}{\sqrt{n_k}},\,
\frac{\bu_{k,\cI_k}^f}{\sqrt{n_k}}
\Big)
\Big]
\Big\|_{\sF}
< C' \sqrt{\frac{p}{n_k}}\, \epsilon,
\end{gathered}
\end{equation}

\begin{equation}
\begin{gathered}
\E\Big[
\bT\Big(
\sqrt{\frac{n_k}{p}}\, \bv_k^f ,\,
\sqrt{\frac{n_k}{p}}\,\bg_k^f
\Big)
\Big]
\begin{pmatrix}
(n_k/p)\tau_{h_k}^2 & (n_k/p) \tau_{g_k}^2 \df_k \\[4pt]
(n_k/p) \tau_{g_k}^2 \df_k & n_k \tau_{g_k}^2
\end{pmatrix},
\\
\E\Big[
\bT\Big(
\frac{\be_{1,\cI_k}}{\sqrt{n_k}},\,
\frac{\be_{2,\cI_k}}{\sqrt{n_k}},\,
\frac{\bu_{k,\cI_k}^f}{\sqrt{n_k}}
\Big)
\Big]
\begin{pmatrix}
\tau_{e_1}^2 & \tau_{e_1}\tau_{e_2}\rho_e & \zeta_k \tau_{e_1} \tau_{e_k} \rho_e^{\indic{k=2}} \\[4pt]
\tau_{e_1}\tau_{e_2}\rho_e & \tau_{e_2}^2 & \zeta_k \tau_{e_2} \tau_{e_k} \rho_e^{\indic{k=1}} \\[4pt]
\zeta_k \tau_{e_1} \tau_{e_k} \rho_e^{\indic{k=2}} & \zeta_k \tau_{e_2} \tau_{e_k} \rho_e^{\indic{k=1}} & n_k \zeta_k^2 \tau_{g_k}^2
\end{pmatrix},
\end{gathered}
\end{equation}

where we interpret $0^0 = 1$ when $\rho_e = 0$,
and the entries of both matrices are bounded by $C$.

The same holds if we replace $\df_k$ by $\hat \df_k$ in the definition of $\hat \btheta_k^\de$ in Eq. (<ref>),
where $\hat \df_k$ is defined by Eq. (<ref>).

Both Eq. (<ref>) and (<ref>) follows from Lemma <ref> using Lemma <ref>.
by Eqs. (<ref>), (<ref>), and because proximal operators are $1$-Lispchitz [36],
the vectors $\sqrt{n_k/p}\, \bv_k^f, \sqrt{n_k/p}\,\bg_k^f$ are $\sqrt{n_k/p}\,\tau_{g_k}$-Lipschitz functions of $\bg_k^f / \tau_{g_k}$,
and by Lemma <ref>, we have $\sqrt{n_k/p}\,\tau_{g_k} \leq C/\sqrt{p}$.
Thus, we may take $M = C/\sqrt{p}$ in Lemma <ref>.
Further, by Eq. (<ref>),
we have $\bv_k = \bSigma^{1/2}(\hat \btheta_k - \btheta_k)$,
whence the marginal characterization (Lemma <ref>) gives us Eq. (<ref>) for the pair of vectors $\sqrt{n_k/p}\,\bv_k,\sqrt{n_k/p}\,\bSigma^{1/2}(\hat \btheta_k^{\de} - \btheta_k)$ with $K = C$.
Finally, by Lemma <ref>, $(n_k/p)\tau_{h_k}^2 \leq C$ and $n_k \tau_{g_k}^2 \leq C$, whence Eq. (<ref>) is satisfied.
Thus, Lemma <ref> implies Eq. (<ref>).

By Eqs. (<ref>) and (<ref>) and because $\zeta_k \leq 1$,
the vectors $\be_{1,\cI_k}/\sqrt{n_k},\be_{2,\cI_k}/\sqrt{n_k}$, and $\bu_{k,\cI_k}^f/\sqrt{n_k}$ are $M=1/\sqrt{n_k}$-Lipschitz functions of Gaussian vectors with variance $\tau_{e_1}^2$, $\tau_{e_2}^2$, and $\tau_{e_k}^2 + \tau_{h_k}^2$, respectively.
By Lemma <ref> these variance are all bounded by $C$.
Further, by Eq. (<ref>),
we have $\bu_{k,\cI_k} = \hat \be_{k,\cI_k}$,
whence the marginal characterization (Lemma <ref>) gives us Eq. (<ref>) for the vectors $\be_{1,\cI_k}/\sqrt{n_k},\be_{2,\cI_k}/\sqrt{n_k},\bu_{k,\cI_k}^f/\sqrt{n_k}$ with $K = \sqrt{p/n_k}$.
Finally, by Lemma <ref>,
Eq. (<ref>) is satisfied,
whence we may apply Lemma <ref> to conclude Eq. (<ref>).

We now prove Lemma <ref>.

Recall that $\hat \bg_k,\hat \bh_k$ are defined in Eq. (<ref>) as $\hat \bg_k = \tau_{g_k}\hat \bxi_g$ and $\hat \bh_k = \tau_{h_k} \hat \bxi_h$, where $\hat \bxi_g, \hat \bxi_h$ solve the system of equations (<ref>) and (<ref>).
(For notational simplicity, in most of the paper—including in Eqs. equations (<ref>) and (<ref>)—we considered conditioning on the first regression, $k = 1$. This was arbitrary, and in the current proof we consider general $k$.)
Writing the solution to these equations explicitly gives

\begin{equation}\label{eq:def-ggam-hgam}
\begin{gathered}
\hat \bxi_g
\frac{n_k \zeta_k \tau_{h_k}}{\| \bu_k \|_2 \| \bv_k \|_2}
\bSigma^{1/2}\big(\hat \btheta_k - \btheta_k\big)
\frac{\bSigma^{-1/2}\bX_{ \cI_k }^\top(\by_{ k,\cI_k }-\bX_{ \cI_k }\hat \btheta_k)}{\| \bu_k \|_2},
\\[4pt]
\hat \bxi_h
\begin{pmatrix}
- \frac{\bX_{ \cI_k }(\hat \btheta_k - \btheta_k)}{\|\bv_k\|_2} + \frac{\< \hat \bxi_g , \bv_k \> }{ \| \bu_k \|_2 \| \bv_k \|_2} (\by_{ k,\cI_k } - \bX_{ \cI_k }\hat \btheta_k)
\\[3pt]
-\frac{\bX_{ \cI_k ^c}(\hat \btheta_k - \btheta_k)}{\| \bv_k \|_2}
\end{pmatrix}.
\end{gathered}
\end{equation}

The first line follows from the second equation in Eq. (<ref>) by plugging in $n_k \zeta_k \tau_{h_k}$ for $\< \hat \bxi_h,\bu_k\>$, recalling the definition of $\bs_1$ in Eq. (<ref>), and rearranging.
The second line follows by rearranging the first equation in Eq. (<ref>) and using the definition of $\bt_1$ in Eq. (<ref>).

First consider $\big\| \hat \bg_k - \bSigma^{1/2}(\hat \btheta_k^{\de} - \btheta_k)\big\|_2$.
Recalling the definition of $\hat \btheta_k^{\de}$ in Eq. (<ref>) and that $n_k - \df_k = n_k\zeta_k $ by Eq. (<ref>),
the first line of Eq. (<ref>) and the triangle inequality imply

\begin{equation}
\big\| \hat \bg_k - \bSigma^{1/2}(\hat \btheta_k^{\de} - \btheta_k)\big\|_2
\leq
\Big|
\frac{n_k \zeta_k \tau_{h_k}\tau_{g_k}}{\|\bu_k\|_2\|\bv_k\|_2}
\Big|
\cdot
\| \hat \btheta_k - \btheta_k \|_{\bSigma}
\Big|
\frac{n_k\tau_{g_k}}{\|\bu_k\|_2} - \frac{1}{\zeta_k}
\Big|
\frac{\| \bSigma^{-1/2}\bX_{ \cI_k }^\top(\by_{ k,\cI_k }-\bX_{ \cI_k }\hat \btheta_k) \|_2}{n_k}.
\end{equation}

Because $\bv_k = \bSigma^{1/2}(\hat \btheta_k-\btheta_k)$ (see Eq. (<ref>)),
by Lemma <ref>
with probability at least $1 - Ce^{-cp}$ we have $\| \hat \btheta_k - \btheta_k \|_{\bSigma} \leq C \sqrt{p/n_k}$.
Further, by Lemma <ref>,
with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$

\begin{equation}
\Big|
\frac{n_k \zeta_k \tau_{h_k}\tau_{g_k}}{\|\bu_k\|_2\|\bv_k\|_2}
\Big|
\Big|
\frac{ (\sqrt{n_k}\,\tau_{g_k}\zeta_k) (\sqrt{n_k/p}\,\tau_{h_k}) }{(\|\bu_k\|_2/\sqrt{n_k})(\sqrt{n_k/p}\,\|\bv_k\|_2)}
\Big|
\end{equation}

where we use that $\sqrt{n_k/p}\,\|\bv_k\|_2$ and $\|\bu_k\|_2 / \sqrt{n_k} = \| \bu_{k,\cI_k} \|_2 / \sqrt{n_k}$ concentrate on $\sqrt{n_k/p}\,\tau_{h_k}$ and $\sqrt{n_k}\,\tau_{g_k}\zeta_k$, which are bounded above and below by $C$ and $c$.
Recalling that $\bu_{k,\cI_k} = \by_{k,\cI_k} - \bX_{\cI_k} \hat \btheta_k$ and applying <cit.> and Lemma <ref>,
we have with probability at least $1 - Ce^{-cp}$

\begin{equation}
\frac{\| \bSigma^{-1/2}\bX_{ \cI_k }^\top(\by_{ k,\cI_k }-\bX_{ \cI_k }\hat \btheta_k) \|_2}{n_k}
\leq
\frac{\| \bSigma^{-1/2} \bX_{\cI_k}^\top\|_{\mathrm{op}}}{\sqrt{n_k}}
\frac{\| \by_{k,\cI_k}-\bX_{\cI_k}\hat \btheta_k\|_2}{\sqrt{n_k}}
\leq
\end{equation}

and with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$,

\begin{equation}
\Big|
\frac{n_k\tau_{g_k}}{\|\bu_k\|_2} - \frac{1}{\zeta_k}
\Big|
\Big|
\frac{\sqrt{n_k}\,\tau_{g_k}}{\|\bu_k\|_2/\sqrt{n_k}} - \frac{1}{\zeta_k}
\Big|
< C'\sqrt{\frac{p}{n_k}}\,\epsilon,
\end{equation}

where we use that $\| \bu_k \|_2 / \sqrt{n_k}$ concentrates on $\sqrt{n_k}\,\tau_{g_k}$ which is bounded above and bewlo by $C$ and $c$.
Combining the above bounds gives the high-probability bound on $\big\| \hat \bg_k - \bSigma^{1/2}(\hat \btheta_k^{\de} - \btheta_k)\big\|_2$ in Lemma <ref>.

Next, consider $\big\| \hat \bh_{k,\cI_k} - (\hat \be_{k,\cI_k}^\de - \be_{k,\cI_k}) \big\|_2 / \sqrt{n_k}$.
Recalling the definition of $\hat \be_k^{\de}$ in Eq. (<ref>) and that $\df_k/(n_k - \df_k) = (1-\zeta_k)/\zeta_k $ by Eq. (<ref>),
the second line of Eq. (<ref>) and the triangle inequality imply

\begin{equation}
\frac{\big\| \hat \bh_{k,\cI_k} - (\hat \be_{k,\cI_k}^\de - \be_{k,\cI_k}) \big\|_2}{\sqrt{n_k}}
\leq
\Big|
\frac{\tau_{h_k}}{\| \bv_k \|_2} - 1
\Big|
\cdot
\frac{\| \bX_{\cI_k}(\hat \btheta_k - \btheta_k) \|_2}{\sqrt{n_k}}
\Big|
\frac{\tau_{h_k}\< \hat \bxi_g, \bv_k \>}{\| \bu_k \|_2 \| \bv_k \|_2}
\frac{1-\zeta_k}{\zeta_k}
\Big|
\cdot
\frac{\| \by_{k,\cI_k} - \bX_{\cI_k} \hat \btheta_k \|_2}{\sqrt{n_k}}.
\end{equation}

By <cit.> and Lemma <ref>,
with probability at least $1 - Ce^{-cp}$ we have $\| \bX_{\cI_k}(\hat \btheta_k - \btheta_k) \|_2/\sqrt{n_k} \leq \| \bX_{\cI_k} \bSigma^{-1/2} \|_2 \| \hat \btheta_k - \btheta_k \|_{\bSigma} \leq C' \sqrt{p/n_k}$
and $\| \by_{k,\cI_k} - \bX_{\cI_k} \hat \btheta_k \|_2/\sqrt{n_k} = \| \bu_{k,\cI_k}\|_2 / \sqrt{n_k} \leq C'$.
By Lemma <ref>,
with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$ we have

\begin{equation}
\Big|
\frac{\tau_{h_k}}{\| \bv_k \|_2} - 1
\Big|
\Big|
\frac{\sqrt{n_k/p}\,\tau_{h_k}}{\sqrt{n_k/p}\,\| \bv_k \|_2} - 1
\Big|
< C' \epsilon,
\end{equation}

where we have used that $\sqrt{n_k/p}\,\|\bv_k\|_2$ concentrates on $\sqrt{n_k/p}\,\tau_{h_k}$,
which is bounded above and below by $C$ and $c$.
By Lemma <ref> and the bound on $\big\| \hat \bg_k - \bSigma^{1/2}(\hat \btheta_k^{\de} - \btheta_k)\big\|_2$ in Lemma <ref> (which we have already proved),
with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$

\begin{equation}
\begin{aligned}
\frac{\< \hat \bxi_g , \bv_k\> }{\sqrt{n_k}}
\sqrt{n_k}\,\tau_{g_k}(1-\zeta_k)
\Big|
\frac{p}{n_k} \frac{1}{\sqrt{n_k}\,\tau_{g_k}}
\Big|
\Big\< \sqrt{\frac{n_k}{p}}\, \hat \bg_k , \sqrt{\frac{n_k}{p}}\,\bv_k\Big\>
\frac{n_k\,\tau_{g_k}^2\df_k}{p}
\Big|
\\
C' \| \hat \bg_k - \bSigma^{1/2}(\hat \btheta_k^\de - \btheta_k)\|_2 \| \bv_k\|_2
\Big|
\Big\< \sqrt{\frac{n_k}{p}}\, \bSigma^{1/2}(\hat \btheta_k^\de - \btheta_k) , \sqrt{\frac{n_k}{p}}\,\bv_k\Big\>
\frac{n_k\,\tau_{g_k}^2\df_k}{p}
\Big|
\leq C' \frac{p}{n_k}\,\epsilon,
\end{aligned}
\end{equation}

where in the first equality we have used that $n_k(1 - \zeta_k) =  \df_k$ by Eq. (<ref>),
and in the first inequality we have used that $\sqrt{n_k}\tau_{g_k} \geq c$ by Lemma <ref> to replace it by a constant.
Combining the above bounds,
we have with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$

\begin{equation}
\Big|
\frac{\tau_{h_k}\< \hat \bxi_g, \bv_k \>}{\| \bu_k \|_2 \| \bv_k \|_2}
\frac{1-\zeta_k}{\zeta_k}
\Big|
\Big|
\frac{(\< \hat \bxi_g, \bv_k \>/\sqrt{n_k})(\sqrt{n_k/p}\,\tau_{h_k})}{(\| \bu_k \|_2/\sqrt{n_k}) (\sqrt{n_k/p}\,\| \bv_k \|_2)}
\frac{1-\zeta_k}{\zeta_k}
\Big|
< C' \sqrt{\frac{p}{n_k}}\,\epsilon,
\end{equation}

where we use that $\sqrt{n_k/p}\,\|\bv_k\|_2$ concentrates on $\sqrt{n_k/p}\,\tau_{h_k}$,
$\|\bu_k\|_2 / \sqrt{n_k}$ concentrates on $\sqrt{n_k}\,\tau_{g_k}\zeta_k$, which are bounded above and below by $C$ and $c$,
and that $\< \hat \bxi_g,\bv_k\> / \sqrt{n_k}$ concentrates on $n_k \tau_{g_k}^2 \df_k / p$, which is bounded above by $C$ by Lemma <ref>.
Combining the above bounds gives the high probability bound on $\| \hat \bh_{k,\cI_k} - (\hat \be_{k,\cI_k}^\de - \be_{k,\cI_k})\|_2 / \sqrt{n_k}$ in Lemma <ref>.

To prove Lemmas <ref> and <ref> and Theorem <ref>,
we must extend the marginal characterization.
In particular, we extend the characterization to the vectors $\hat \bg_1$ and $\hat \bh_1$,
and extend the characterization from the index set $\cI_1$ to the index set $\cI_2$.
The next corollaries
perform these extensions.

Assume  and .

There exist $\cPmodel$, $\cPregr$ and regression method-dependent $c' > 0$ and $\sC,\sc: \reals_{>0} \rightarrow \reals_{>0}$ such that
for $\epsilon < c'$ and 1-Lipschitz $\phi_\theta:(\reals^p)^2 \rightarrow \reals$, $\phi_e: (\reals^{n_k})^4 \rightarrow \reals$, with probability at least $1 - \sC(\epsilon) e^{-\sc(\epsilon) p}$

\begin{equation}
\begin{gathered}
\big| \phi_\theta(\hat \btheta_k, \hat \bg_k) - \E[\phi_\theta(\hat \btheta_k^f,\bg_k^f)]\big| < \sqrt{\frac{p}{n_k}}\,\epsilon,
\\
\Big|
\phi_e\Big(
\frac{ \be_{1,\cI_k}}{\sqrt{n_k}},
\frac{ \be_{2,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}}{\sqrt{n_k}},
\frac{\hat \bh_{k,\cI_k}}{\sqrt{n_k}}
\Big)
\E\Big[
\phi_e\Big(
\frac{ \be_{1,\cI_k}}{\sqrt{n_k}},
\frac{ \be_{2,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}^f}{\sqrt{n_k}},
\frac{\bh_k^f}{\sqrt{n_k}}
\Big)\Big] \Big| < \sqrt{\frac{p}{ n_k }} \, \epsilon.
\end{gathered}
\end{equation}

By Lemma <ref>,

\begin{equation}
\Big|
\phi_e\Big(
\frac{ \be_{1,\cI_k}}{\sqrt{n_k}},
\frac{ \be_{2,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}}{\sqrt{n_k}},
\frac{\hat \bh_{k,\cI_k} }{\sqrt{n_k}}
\Big)
\phi_e\Big(
\frac{ \be_{1,\cI_k}}{\sqrt{n_k}},
\frac{ \be_{2,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}^{\de} - \be_{k,\cI_k}}{\sqrt{n_k}}
\Big)
\Big|
< \sqrt{\frac{p}{n_k}}\,\epsilon.
\end{equation}

Using the marginal characterization (Lemma <ref>) and recalling that $\hat \be_{k,\cI_k}^{f,\de} - \be_{k,\cI_k} = \bh_{k,\cI_k}^f$ (see Eqs. (<ref>) and
we have
\begin{equation}
\Big|
\phi_e\Big(
\frac{ \be_{1,\cI_k}}{\sqrt{n_k}},
\frac{ \be_{2,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}^{\de} - \be_{k,\cI_k}}{\sqrt{n_k}}
\Big)
\E\Big[
\phi_e\Big(
\frac{ \be_{1,\cI_k}}{\sqrt{n_k}},
\frac{ \be_{2,\cI_k}}{\sqrt{n_k}},
\frac{\hat \be_{k,\cI_k}^f}{\sqrt{n_k}},
\frac{\bh_k^f}{\sqrt{n_k}}
\Big)
\Big]
\Big|
< \sqrt{\frac{p}{n_k}}\,\epsilon.
\end{equation}

The corollary follows by combining the above two bounds using the triangle inequality.
The first line of the corollary follows similarly, using the bound on $\| \hat \bg_k - \bSigma^{1/2}(\hat \btheta_k^\de - \btheta_k)\|_2$ from Lemma <ref> and recalling that in the fixed design model $\bSigma^{1/2}(\hat \btheta_k^{f,\de} - \hat \btheta_k) = \bg_k^f$.

Assume  and .

Let $\phi_e: (\reals^{n_2})^4 \rightarrow \reals$ by $M_e$-Lipschitz in its first three arguments and $M_h$ Lipschitz in its final argument.
There exist $\cPmodel$, $\cPregr$ and regression method-dependent $c' > 0$ and $\sC,\sc: \reals_{>0} \rightarrow \reals_{>0}$ such that

\begin{equation}\label{eq:I2-marg}
\begin{gathered}
\Big|
\phi_e\Big(
\frac{ \be_{1,\cI_2}}{\sqrt{n_2}},
\frac{ \be_{2,\cI_2}}{\sqrt{n_2}},
\frac{\hat \be_{1,\cI_2}}{\sqrt{n_2}},
\frac{\hat \bh_{1,\cI_2}}{\sqrt{n_2}}
\Big)
\E\Big[
\phi_e\Big(
\frac{ \be_{1,\cI_2}}{\sqrt{n_2}},
\frac{ \be_{2,\cI_2}}{\sqrt{n_2}},
\frac{\hat \be_{1,\cI_2}^f}{\sqrt{n_2}},
\frac{\bh_{1,\cI_2}^f}{\sqrt{n_2}}
\Big)\Big] \Big| < \Big(
M_e \sqrt{\frac{p}{n_2}} + M_h \sqrt{\frac{p(p+n_{12})}{n_1n_2}}\;
\Big) \, \epsilon.
\end{gathered}
\end{equation}

Further, we have
\begin{equation}
\frac{\| \hat \bh_{1,\cI_2} - (\hat \be_{1,\cI_2}^\de - \be_{1,\cI_2})\|_2}{\sqrt{n_2}}
\leq \sqrt{\frac{p}{n_1}}\,\epsilon.
\end{equation}

Thus, if we replace $\hat \bh_{1,\cI_2}$ by $\hat \be_{1,\cI_2}^{\de}-\be_{1,\cI_2}$ in Eq. (<ref>),
the same bound holds except that we must replace the term $ M_h \sqrt{p(p+n_{12})/(n_1n_2)} $ on the right-hand side by $M_h \sqrt{p/n_1}$.

Part (ii) still holds if we replace $\df_k$ by $\hat \df_k$ in the definition of $\hat \be_1^\de$ in Eq. (<ref>),
where $\hat \df_k$ is defined by Eq. (<ref>).

Observe that the data $\be_{1,\cI_1},\be_{2,\cI_1}\bX_{\cI_1}$ is rotationally invariant in the sense that for any orthogonal matrix $\bQ \in \reals^{n_1\times n_1}$ (i.e., $\bQ^\top\bQ = \id_{n_k}$),
we have
$(\bQ \be_{1,\cI_1},\bQ \be_{2,\cI_1},\bQ \bX_{\cI_k}) \stackrel{\mathrm{d}}= (\be_{1,\cI_1},\be_{2,\cI_1},\bX_{\cI_1})$.
By the rotational invariance of the least-squares loss,
under this replacement $(\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_2})$
is replaced with $(\bQ\be_{1,\cI_1},\bQ\be_{2,\cI_1},\bQ\hat \be_{1,\cI_1},\bQ\hat \bh_{1,\cI_2})$ and $\bv_k$ is unaffected.
We conclude that $(\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_2})$ has a rotationally invariant distribution.
Further, from Eq. (<ref>) and recalling that $\bv_k = \bSigma^{1/2}(\hat \btheta_k - \btheta_k)$ (see Eq. (<ref>)) and $\bX_{\cI_1^c}$ is independent of $\bX_{\cI_1}$,
we have $\hat \bxi_{h,\cI_1^c} = - \bX_{\cI_1^c}(\hat \btheta_1-\btheta_1)/\| \hat \btheta_1 - \btheta_1 \|_{\bSigma} \sim \normal(0,\id_{N-n_1})$ independent of $\be_{1,\cI_1},\be_{2,\cI_1}\bX_{\cI_1}$.
Therefore, $\hat \bh_{1,\cI_1^c} = \tau_{h_1}\hat \bxi_{h,\cI_1^c} \sim \normal(0,\tau_{h_1}^2 \id_{N-n_1})$ independent of $\be_{1,\cI_1},\be_{2,\cI_1}\bX_{\cI_1}$.
Thus, we may represent

\begin{equation}
\begin{aligned}
\frac{\be_{1,\cI_2}}{\sqrt{n_2}},
\frac{\be_{2,\cI_2}}{\sqrt{n_2}},
\frac{\hat \be_{1,\cI_2}}{\sqrt{n_2}},
\frac{\hat \bh_{1,\cI_2}}{\sqrt{n_2}}
\Big)
\\
\frac1{\sqrt{n_2}}
\left(
\begin{pmatrix}
(\bQ \be_{1,\cI_1})_{\cI_1 \cap \cI_2} \\[3pt]
\tau_{e_1} \bxi^{(1)}
\end{pmatrix},
\begin{pmatrix}
(\bQ \be_{2,\cI_1})_{\cI_1 \cap \cI_2} \\[3pt]
\tau_{e_2} \bxi^{(2)}
\end{pmatrix},
\begin{pmatrix}
(\bQ \hat \be_{1,\cI_1})_{\cI_1 \cap \cI_2} \\[3pt]
\bzero
\end{pmatrix},
\begin{pmatrix}
(\bQ \hat \bh_{1,\cI_1})_{\cI_1 \cap \cI_2} \\[3pt]
\tau_{h_1} \bxi^{(3)}
\end{pmatrix}
\right)
=: \bD,
\end{aligned}
\end{equation}

where $\bxi^{(1)},\bxi^{(2)},\bxi^{(3)}\stackrel{\mathrm{iid}}\sim\normal(0,\id_{n_2 - n_{12}})$ and $\bQ\in \reals^{n_1 \times n_1}$ is drawn uniformly from the special orthogonal group $SO(n_1)$ and independently of everything else.
We introduce the notation $\bD \in \reals^{n_2 \times 4}$ for notational compactness,
and denote $\phi_e$ applied to its columns by $\phi_e(\bD)$.
By the distributional equivalence above,
it is enough to establish the concentration result for $\phi_e(\bD)$ in place of
\frac{\be_{1,\cI_2}}{\sqrt{n_2}},
\frac{\be_{2,\cI_2}}{\sqrt{n_2}},
\frac{\hat \be_{1,\cI_2}}{\sqrt{n_2}},
\frac{\hat \bh_{1,\cI_2}}{\sqrt{n_2}}
\Big)$.

Conditioning on $\bQ ,\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}$, the only randomness that remains is in $\bxi^{(1)},\bxi^{(2)},\bxi^{(3)}$.
By Lemma <ref> and assumption , we have $\tau_{e_1},\tau_{e_2} \leq C$ and $\tau_{h_1} \leq C\sqrt{p/n_1}$.
Thus, by Gaussian concentration,
with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$

\begin{equation}\label{eq:bound1}
\big| \phi_e(\bD) - \E[\phi_e(\bD) \mid \bQ ,\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]\big| < \Big(M_e\sqrt{\frac{p}{n_2}} + M_h \frac{p}{\sqrt{n_1n_2}}\;\Big)\epsilon.
\end{equation}

Now note that $\E[\phi_e(\bD) \mid \bQ ,\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]$ viewed as a function of $\bQ$ with $\be_1,\be_2,\hat \be_1,\hat \bh_1,\bxi^{(1)},\bxi^{(2)},\bxi^{(3)}$ fixed is $M_e(\| \be_{1,\cI_1}\|_2 + \| \be_{2,\cI_1} \|_2 + \| \hat \be_{1,\cI_1}\|_2)/\sqrt{n_2} + M_h \| \hat \bh_{1,\cI_1} \|_2 / \sqrt{n_2}$-Lipschitz in $\bQ$ in Frobenius norm.
Indeed, $\| \bQ \be_{1,\cI_1} - \bQ \be_{1,\cI_1}' \|_2/\sqrt{n_2} \leq \| \bQ - \bQ' \|_{\sF} \| \be_{1,\cI_1} \|_2/\sqrt{n_2}$, and likewise for the other arguments.
By concentration on the special orthogonal group (see, e.g., <cit.>),
with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$

\begin{equation}
\begin{aligned}
\E[\phi_e(\bD)\mid \bQ ,\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]
\E[\phi_e(\bD)\mid \be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]
\big|
\\
\leq \sqrt{\frac{p}{n_1}} \frac{M_e(\| \be_{1,\cI_1}\|_2 + \| \be_{2,\cI_1} \|_2 + \| \hat \be_{1,\cI_1}\|_2) + M_h \| \hat \bh_{1,\cI_1} \|_2}{ \sqrt{n_2} }\,\epsilon.
\end{aligned}
\end{equation}

Using Lemma <ref> and recalling that $\hat \be_{1,\cI_1} = \bu_{1,\cI_1}$ (see Eq. (<ref>)),
with probability at least $1 - Ce^{-cp}$
we have $\big(M_e(\| \be_{1,\cI_1}\|_2 + \| \be_{2,\cI_1} \|_2 + \| \hat \be_{1,\cI_1}\|_2) + M_h \| \hat \bh_{1,\cI_1} \|_2\big)/\sqrt{n_2} \leq C(M_e \sqrt{n_1/n_2} + M_h \sqrt{p/n_2})$.
for $\epsilon < c'$ with probability at $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$

\begin{equation}\label{eq:bound2}
\big|
\E[\phi_e(\bD)|\bQ ,\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]
\E[\phi_e(\bD)|\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]
\big| < \Big( M_e \sqrt{\frac{p}{n_2}} + M_h \frac{p}{\sqrt{n_1n_2}}\;\Big)\,\epsilon.
\end{equation}

Now denote $\tilde \phi_e(\be_{1,\cI_1}/\sqrt{n_1},\be_{2,\cI_1}/\sqrt{n_1},\hat \be_{1,\cI_1}/\sqrt{n_1},\hat \bh_{1,\cI_1}/\sqrt{n_1}) := \E[\phi_e(\bD)|\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]$.
Using the Lipschitz properties of $\phi_e$,
we have

\begin{equation}
\begin{aligned}
\tilde \phi_e
\Big(
\frac{\be_{1,\cI_1}}{\sqrt{n_1}},
\frac{\be_{2,\cI_1}}{\sqrt{n_1}},
\frac{\hat \be_{1,\cI_1}}{\sqrt{n_1}},
\frac{\hat \bh_{1,\cI_1}}{\sqrt{n_1}}
\Big)
\tilde \phi_e
\Big(
\frac{\be_{1,\cI_1}'}{\sqrt{n_1}},
\frac{\be_{2,\cI_1}}{\sqrt{n_1}},
\frac{\hat \be_{1,\cI_1}}{\sqrt{n_1}},
\frac{\hat \bh_{1,\cI_1}}{\sqrt{n_1}}
\Big)
\Big|
\\
&\quad \leq
M_e \sqrt{\frac{n_1}{n_2}}
\frac{
\E_{\bQ}\big[\|(\bQ (\be_{1,\cI_1}-\be_{1,\cI_1}'))_{\cI_1 \cap \cI_2}\|_2\big]
\leq
M_e \sqrt{\frac{n_1}{n_2}}
\frac{
\E_{\bQ}\big[\|(\bQ (\be_{1,\cI_1}-\be_{1,\cI_1}'))_{\cI_1 \cap \cI_2}\|_2^2\big]^{1/2}
\\
&\quad= M_e \sqrt{\frac{n_{12}}{n_2}} \frac{\|\be_{1,\cI_1}-\be_{1,\cI_1}'\|_2}{\sqrt{n_1}},
\end{aligned}
\end{equation}

where we have used that $\E_{\bQ}\big[\|(\bQ (\be_{1,\cI_1}-\be_{1,\cI_1}'))_{\cI_1 \cap \cI_2}\|_2^2\big] = n_{12} \| \be_{1,\cI_1} - \be_{1,\cI_1}'\|_2^2 / n_1$ by symmetry and the rotationally invariant distribution of $\bQ$.
Thus, $\tilde \phi_e$ is $M_e\sqrt{n_{12}/n_2}$-Lipschitz in $\be_{1,\cI_1}/\sqrt{n_1}$.
Similarly, it is $M_e\sqrt{n_{12}/n_2}$-Lipschitz in $\be_{2,\cI_1}/\sqrt{n_1}$ and $\hat \be_{1,\cI_1}/\sqrt{n_1}$
and $M_h\sqrt{n_{12}/n_2}$-Lipschitz in $\hat \bh_{1,\cI_1}/\sqrt{n_1}$.
In particular, it is $(M_e+M_h)\sqrt{n_{12}/n_2}$-Lipschitz in all its arguments.
Recalling the definition of $\tilde \phi_e$,
Corollary <ref> implies that
\begin{equation}\label{eq:bound3}
\big|
\E[\phi_e(\bD)|\be_{1,\cI_1},\be_{2,\cI_1},\hat \be_{1,\cI_1},\hat \bh_{1,\cI_1}]
\E[\phi_e(\bD)]
\big| < ( M_e + M_h) \sqrt{\frac{n_{12}p}{n_1n_2}}\,\epsilon.
\end{equation}

Combining Eqs. (<ref>), (<ref>), and (<ref>) and using $n_{12} \leq n_1$ gives Eq. (<ref>),
and the proof of Corollary <ref>(i) is complete.

Now we consider replacing $\hat \bh_{1,\cI_2}$ by $\hat \be_{1,\cI_2}^\de - \be_{1,\cI_2}$.
By Eq. (<ref>), with probability at least $1 - Ce^{-pc}$ we have $\| \hat \bh_{1,\cI_2}\|_2/\sqrt{n_2} \leq \E[\| \bh_{1,\cI_2}^f \|_2] / \sqrt{n_2} + C\sqrt{p/n_1} \leq \E[\| \bh_{1,\cI_2}^f \|_2^2]^{1/2} / \sqrt{n_2} + \sqrt{p(p+n_{12})/(n_1n_2)} \leq C\sqrt{p/n_1}$,
where in the second inequality we have used Jensen and that $p/n_2 \leq C$ and $n_{12} /n_2 \leq 1$, and in the third inequality have used that $\E[\| \bh_{1,\cI_2}^f \|_2^2]^{1/2} = \tau_{h_1} \leq C \sqrt{p/n_1}$ by Eq. (<ref>) and Lemma <ref>.
By Eq. (<ref>)
we have $\hat \be_{1,\cI_2\setminus \cI_1}^\de - \be_{1,\cI_2\setminus \cI_1} = - \bX_{\cI_2 \setminus \cI_1}(\hat \btheta_1 - \btheta_1)$ and by
Eq. (<ref>)
we have $\hat \bh_{1,\cI_2 \setminus \cI_1} = - (\tau_{h_1}/\|\bv_1\|_2)\bX_{\cI_2 \setminus \cI_1}(\hat \btheta_1 - \btheta_1)$, whence
\begin{equation}
\frac{\|\hat \bh_{1,\cI_2 \setminus \cI_1} - (\hat \be_{1,\cI_2\setminus \cI_1}^\de - \be_{1,\cI_2\setminus \cI_1})\|_2}{\sqrt{n_2}}
\Big|
\frac{\sqrt{n_1/p}\,\|\bv_1\|_2}{\sqrt{n_1/p}\,\tau_{h_1}}
\Big|
\cdot
\frac{\| \hat \bh_{1,\cI_2 \setminus \cI_1} \|_2}{\sqrt{n_2}}
\leq
C' \sqrt{\frac{p}{n_1}}\,\epsilon,
\end{equation}

where the inequality uses Lemma <ref> to bound $\big|1 - (\sqrt{n_1/p}\,\|\bv_1\|_2)/(\sqrt{n_1/p}\,\tau_{h_1})\big| \leq C'\epsilon$.
Further, by Eq. (<ref>)
we have $\hat \be_{1,\cI_2 \cap \cI_1}^\de - \be_{1,\cI_2 \cap \cI_1} = \hat \be_{1,\cI_2 \cap \cI_1}/\zeta_1 - \be_{1,\cI_2 \cap \cI_1} $,
whence by Eq. (<ref>) we have
\begin{equation}
\frac{\|\hat \bh_{1,\cI_2 \cap \cI_1} - (\hat \be_{1,\cI_2\cap \cI_1}^\de - \be_{1,\cI_2\cap \cI_1})\|_2}{\sqrt{n_2}}
\frac{\|\hat \bh_{1,\cI_2 \cap \cI_1} - (\hat \be_{1,\cI_2\cap \cI_1}/\zeta_1 - \be_{1,\cI_2\cap \cI_1})\|_2}{\sqrt{n_2}}
\leq
C' \sqrt{\frac{p(p+n_{12})}{n_1n_2}}\,\epsilon,
\end{equation}

where we have used that in the fixed design model $\bh_{1,\cI_2 \cap \cI_1}^f = \hat \be_{1,\cI_2\cap \cI_1}^f/\zeta_1 - \be_{1,\cI_2\cap \cI_1}$ (see Eq. (<ref>)).
Combining the previous two displays and using $\sqrt{p(p + n_{12})/(n_1n_2)} \leq C \sqrt{p/n_1}$ because $n_{12} \leq n_2$ and $p \leq Cn_2$ by , Corollary <ref>(ii) follows.

Corollary <ref>(iii) holds by applying Lemma <ref>.

§.§ Concentration of $\phi_{\theta|1}$ and $\phi_{e|1}$: proof of Lemma <ref>

Note $\phi_{\theta|1}$ is a function of $\mathsf{Cond}_1$ only via $\hat \bg_1$.
With some abuse of notation,
we will write $\phi_{\theta|1}(\hat \bg_1)$ for this function.
Similary, $\phi_{e|1}$ is a function of $\mathsf{Cond}_1$ only via $\hat \bh_1,\be_1,\be_2$.
In fact, because the coordinates of the noise are independent in the fixed-design model,
$\phi_{e|1}$ is a function of $\mathsf{Cond}_1$ only via $\hat \bh_{1,\cI_2}, \be_{1,\cI_2},\be_{2,\cI_2}$.
With some abuse of notation,
we will write $\phi_{e|1}(\be_{1,\cI_2}/\sqrt{n_2},\be_{2,\cI_2}/\sqrt{n_2},\hat \bh_{1,\cI_2}/\sqrt{n_2})$ for this function.

We claim that $\phi_{\theta|1}$ is a $M_1 + M_2(\tau_{g_2}\rho_g/\tau_{g_1}) \leq M_1 + C M_2 \sqrt{n_1/n_2}$-Lipschitz function of $\hat \bg_1$.
Indeed, by Eq. (<ref>) and because proximal operators are 1-Lipschitz [36],
we see that $\bv_k^f$ is a $1$-Lipschitz function of $\bg_k^f$.
Becuase $\hat \btheta_k^f = \btheta_k + \bSigma^{-1/2} \bv_k^f$ (see Eq. (<ref>)) and the singular values of $\bSigma$ are bounded below,
we have that $\hat \btheta_k^f$ is $C$-Lipschitz functions of $\bg_k^f$.
Because $\hat \btheta_k^{f,\de} = \btheta_k + \bg_k^f$, we have $\hat \btheta_2^{f,\de}$ is a 1-Lipschitz function of $\bg_k^f$ as well.
In the fixed design model, we may represent $\bg_2^f = (\tau_{g_2}\rho_g/\tau_{g_1})\bg_1^f + \tau_{g_2}\rho_g^\perp \bxi_g$, where $\bxi_g \sim \normal(0,\id_p)$ independent of $\bg_1^f$.
Because the expectation conditional on $\bg_1^f = \hat \bg_1$ in Eq. (<ref>) can be represented as an expectation over $\bxi_g$,
the above observations imply that
$\phi_{\theta|1}(\hat \bg_1)$ is $M_1 + M_2(\tau_{g_2}\rho_g/\tau_{g_1}) \leq M_1 + C M_2 \sqrt{n_1/n_2}$-Lipschitz in $\hat \bg_1$, where we have used Lemma <ref> to bound $\tau_{g_2}\rho_g/\tau_{g_1} \leq C\sqrt{n_1/n_2}$.
By the definition of $\phi_{\theta|1}$ (see Eq. (<ref>)),
we have $\E[\phi_{\theta|1}(\bg_1^f)] = \E\big[\phi_\theta\big( \hat \btheta_k^f , \hat \btheta_k^{f,\de}  \big)\big]$.
Thus, by Corollary <ref>,

\begin{equation}
\begin{aligned}
\Big|
\phi_{\theta|1}\big(\hat \bg_1\big)
\E\big[\phi_\theta\big( \hat \btheta_k^f , \hat \btheta_k^{f,\de}  \big)\big]
\Big|
\leq
\Big(M_1 + M_2\sqrt{\frac{n_1}{n_2}}\Big)\sqrt{\frac{p}{n_1}}\,\epsilon
\Big(M_1\sqrt{\frac{p}{n_1}}+ M_2 \sqrt{\frac{p}{n_2}}\Big)\,\epsilon,
\end{aligned}
\end{equation}

so that the first bound in Lemma <ref> is established.

Similarly, we claim that $\phi_{e|1}$ is a $C\sqrt{n_1/n_2}$-Lipschitz function of $\hat \bh_{1,\cI_2}/\sqrt{n_2}, \be_{1,\cI_2}/\sqrt{n_2}, \be_{2,\cI_2}/\sqrt{n_2}$.
by Eq. (<ref>),
$\hat \be_k^f = (\be_k + \bh_k^f)/\zeta_k$ and $\be_k^{f,\de} = \be_k + \bh_k^f$ are $C$-Lipschitz functions of $\bh_k^f$ because $1 / \zeta_k \leq C$ by Lemma <ref>.
In the fixed design model, we may represent $\bh_2^f = (\tau_{h_2}\rho_h/\tau_{h_1})\bh_1^f + \tau_{h_2}\rho_h^\perp \bxi_h$, where $\bxi_h \sim \normal(0,\id_N)$ independent of $\bh_1^f$.
Because the expectation conditional on $\bh_1^f = \hat \bh_1,\be_1,\be_2$ in Eq. (<ref>) can be represented as an expectation over $\bxi_h$,
we see that $\phi_{e|1}(\be_{1,\cI_2}/\sqrt{n_2},\be_{2,\cI_2}/\sqrt{n_2},\hat \bh_{1,\cI_2}/\sqrt{n_2})$
$M_1$-Lipschitz in $\be_{1,\cI_2}/\sqrt{n_2}$,
$M_2$-Lipschitz in $\be_{2,\cI_2}/\sqrt{n_2}$,
$M_1 + M_2(\tau_{h_2}\rho_h/\tau_{h_1}) \leq M_1 +  C M_2 \sqrt{n_1/n_2}$-Lipschitz in $\hat \bh_{1,\cI_2}/\sqrt{n_2}$, where we have used Lemma <ref> to bound $\tau_{h_2}/\tau_{h_1} \leq C \sqrt{n_1/n_2}$.
By the definition of $\phi_{e|1}$ (see Eq. (<ref>)),
we have $\E[\phi_{e|1}(\bh_{1,\cI_2}^f/\sqrt{n_2})] =                 \E\Big[
\phi_e\Big(\Big\{\frac{\hat \be_{k,\cI_2}^f}{\sqrt{n_2}}\Big\},\Big\{\frac{\hat \be_{k,\cI_2}^{\de,f}}{\sqrt{n_2}}\Big\}\Big)
\Big]$.
By Corollary <ref>,

\begin{equation}
\begin{aligned}
\Big|
\phi_{e|1}\Big(
\frac{\hat \bh_{1,\cI_2}}{\sqrt{n_2}}
\Big)
\E\Big[
\phi_e\Big(\Big\{\frac{\hat \be_{k,\cI_2}^f}{\sqrt{n_2}}\Big\},\Big\{\frac{\hat \be_{k,\cI_2}^{\de,f}}{\sqrt{n_2}}\Big\}\Big)
\Big]
\Big|
\Bigg[
(M_1 + M_2) \sqrt{\frac{p}{n_2}} + \Big(M_1 + M_2\sqrt{\frac{n_1}{n_2}}\Big)\sqrt{\frac{p(p+n_{12})}{n_1n_2}}\,
\Bigg]\epsilon
\\
&\leq C\big(M_1+M_2\big) \sqrt{\frac{p}{n_2}}\,\epsilon,
\end{aligned}
\end{equation}

where we have used that $M_1 \sqrt{p(p+n_{12})/(n_1n_2)} \leq CM_1 \sqrt{p/n_2}$ because $n_{12} \leq n_2$ and $p \leq Cn_1$,
and $M_2 \sqrt{n_1/n_2}\,\sqrt{p(p+n_{12})/(n_1n_2)} = M_2 \sqrt{p(p+n_{12})/n_2^2} \leq CM_2 \sqrt{p/n_2}$ because $n_{12} \leq n_2$ and $p \leq Cn_2$.
This complete the proof.

§.§ The good conditional characterization event: proof of Lemma <ref>

For many of the objects controlled on the event $\cG_{2|1}(\epsilon,\Delta)$, our proof in fact shows a bettr rate (i.e., have a prefactor on $\epsilon$ which is smaller in $p/n_k$).
We state the Lemma only with the rate that we need so as not to clutter notation.

First, we consider the concentration of
\frac{\be_{2,\cI_2}}{\sqrt{n_2}},
\frac{\bu_{1,\cI_2}}{\sqrt{n_1}},
\frac{\hat \bxi_{h,\cI_2}}{\sqrt{n_2}},
\frac{\bxi_{h,\cI_2}}{\sqrt{n_2}}
\Big)$.
The required concentration follows from applying Lemma <ref> to its arguments pairwise, as we now show.
Recall the definitions of $\be_2,\bu_1^f,\bh_1^f,\bh_2^f$ in Eqs. (<ref>) and (<ref>) and of $\tau_{h_1},\tau_{h_2},\rho_h$ in Section <ref>.
Using the bound $\zeta_1^2(\tau_{e_1}^2 + \tau_{h_1}^2) \leq C$ by Lemma <ref>,
we see that $\bu_1^f$ is a $1$-Lipschitz function of a Gaussian vector with variance bounded by $C$, and by the definition of $\tau_{h_1},\tau_{h_2},\rho_h$, we see that $\bh_{1,\cI_2}^f/\tau_{h_1}$ and $\bh_{2,\cI_2}^f - (\tau_{h_2}\rho_h/\tau_{h_1})/(\tau_{h_2}\rho_h^\perp)$ are standard Gaussian.
Thus, the arguments to $\bT$ inside the expectation in the first line of Eq. (<ref>) are $M = 1 / (\sqrt{n_1 \vee n_2}) \leq C/\sqrt{p}$-Lipschitz functions of a Gaussian vector with variance bounded by $C$.
Recall that $\bu_{1,\cI_2} = \hat \be_{1,\cI_2}$ (see Eq. (<ref>)) and $\hat \bxi_{h,\cI_2} = \hat \bh_{1,\cI_2} / \tau_{h_1}$ (see Eq. (<ref>)), and $\bxi_h$ is standard Gaussian independent of everything else.
Using that $\tau_{h_1} \geq c \sqrt{p/n_1}$ by Lemma <ref>,
we conclude that any function which is $1$-Lipschitz in $\be_{2,\cI_2}/\sqrt{n_2},\bu_{1,\cI_2}/\sqrt{n_1},\hat \bxi_{h,\cI_2}/\sqrt{n_2},\bxi_{h,\cI_2}/\sqrt{n_2}$ is in fact $M_e = (1 \vee \sqrt{n_2/n_1})$-Lipschitz in $\be_{1,\cI_2}/\sqrt{n_2},\be_{2,\cI_2}/\sqrt{n_2},\hat \be_{1,\cI_2}/\sqrt{n_2},\bxi_{h,\cI_2}/\sqrt{n_2}$ and $M_h = C \sqrt{n_1/p}$-Lipschitz in $\hat \bh_{1,\cI_2}/\sqrt{n_2}$.
Then, Corollary <ref> and Gaussian concentration of Lipschitz functions gives us Eq. (<ref>) with $K = C(1 \vee \sqrt{n_1/n_2})\sqrt{p/n_2} + C \sqrt{n_1/p}\,\sqrt{p(p+n_{12})/(n_1n_2)} \leq C$, where we use that $p/n_1< C$, $p/n_2 < C$ and $n_{12} \leq n_2$.
By Eq. (<ref>), Eq. (<ref>) is satisfied,
whence we may apply Lemma <ref> with $M = C/\sqrt{p}$ and $K = C$ to get the desired concentration.

Second, we consider the concentration of
\bT\Big(
\sqrt{\frac{n_1}{p}}\,\bv_1,
\sqrt{\frac{n_2}{p}}\,\bv_2^{cf},
\frac{\hat \bxi_g}{\sqrt{p}},
\frac{\bxi_g}{\sqrt{p}}
\Big).
As above, the required concentration follows from applying Lemma <ref> to its arguments pairwise, as we now show.
Recall the definitions of $\bv_1^f,\bv_2^f,\bg_1^f,\bv_2^f$ in Eqs. (<ref>), (<ref>)
and of $\tau_{g_1},\tau_{g_2},\rho_g$ in Section <ref>.
Using the bound $\tau_{g_k} \leq C/\sqrt{n_k}$ by Lemma <ref>,
we see that $\sqrt{n_1/p}\,\bv_1^f$, $\sqrt{n_2/p}\,\bv_2^f$ is a $1$-Lipschitz function of a Gaussian vector with variance bounded by $C$, and by the definition of $\tau_{g_1},\tau_{g_2},\rho_g$, we see that $\bg_{1,\cI_2}^f/\tau_{g_1}$ and $\bg_{2,\cI_2}^f - (\tau_{g_2}\rho_g/\tau_{g_1})/(\tau_{g_2}\rho_g^\perp)$ are standard Gaussian.
Thus, the arguments to $\bT$ inside the expectation in the second line of Eq. (<ref>) are $M = C/\sqrt{p}$-Lipschitz functions of a Gaussian vector with variance bounded by $C$.
By Eqs. (<ref>), (<ref>), (<ref>), the bounds $\tau_{g_2}  \leq C / \sqrt{n_2}$ and $\tau_{g_1} \geq c / \sqrt{n_1}$,
and because proximal operators are 1-Lipschitz [36],
the vectors $\sqrt{n_2/p}\,\bv_2^{cf},\hat \bxi_g/\sqrt{p},\bxi_g/\sqrt{p}$ are $C\sqrt{n_1/p}$-Lipschitz functions of $\hat \bg_1,\bxi_g/\sqrt{n_1}$;
and by Eq. (<ref>) and because the singular values of $\bSigma$ are bounded below by $c$,
we have $\sqrt{n_1/p}\,\bv_1$ is a $C\sqrt{n_1/p}$-Lipschitz function of $\hat \btheta_1$.
Then, Corollary <ref> and Gaussian concentration of Lipschitz functions give us Eq. (<ref>) with $K = C$.
By Eq. (<ref>), Eq. (<ref>) is satisfied,
whence we may apply Lemma <ref> with $M = C/\sqrt{p}$ and $K = C$ to get the desired concentration.

Third, we consider the concentration of $\bar \Omega_2(\bv_2^{cf})$.
In the case of least-squares, $\bar \Omega_2(\bv_2^{cf}) = 0$ always,
and there is nothing to show.
In the case of ridge-regression,
we apply Lemma <ref>.
Recall that $\bar \Omega_2(\bv_2^f) = \sqrt{p/n_2}\,(\lambda/2)\|\hat \btheta_2^f\|_2^2$.
By Eqs. (<ref>) and (<ref>), because proximal operators are 1-Lipschitz [36] and $\bSigma$ has singular values bounded below by $c$ and $\tau_{g_2} \leq C / \sqrt{n_2}$ by Lemma <ref>,
we have that $\hat \btheta_2^f$ is $M= 1 / \sqrt{n_2}$-Lipschitz in $\bg_2^f / \tau_{g_2}$, which is standard Gaussian.
Moreover, by the previous paragraph and because the singular values of $\bSigma$ are bounded below by $c$, we have that $\hat \btheta_2^{cf} = \btheta_2 + \bSigma^{-1/2}\bv_2^{cf}$
is a $C\sqrt{n_1/n_2}$-Lipschitz function of $\hat \bg_1,\bxi_g/\sqrt{n_1}$,
whence by Corollary <ref> and Gaussian concentration of Lipschitz functions,
Eq. (<ref>) holds for $K = \sqrt{p/n_2}$.
By  and Lemma <ref>, we have $\E[\|\btheta_2 + \bSigma^{-1/2}\bv_2^f\|_2^2] \leq 2\|\btheta_2\|_2^2 + 2 C\E[\|\bv_2^f\|_2^2] \leq C(2 + p/n_2) \leq C$.
Thus, using that $\omega_2 = \E[\bar \Omega_2(\bv_2^f)]$ (see Eq. (<ref>)),
we my apply Lemma <ref> to get the required concentration of $\bar \Omega_2(\bv_2^{cf})$, where we use additionally that $\lambda < C$ by .
In the case of the $\alpha$-smoothed Lasso,
we have that $\bar \Omega_2(\bv_2^{cf}) = \lambda \| \btheta_2 + \bSigma^{-1/2} \bv_2^{cf} \|_1/\sqrt{n_2}$ is
$C \lambda \sqrt{pn_1}/n_2 $-Lipschitz in $\hat \bg_1,\bxi_g/\sqrt{n_1}$.
The required concentration of $\bar \Omega_2(\bv_2^{cf})$ follows by Corollary <ref> and Gaussian concentration of Lipschitz functions.

To get the required concentration of $\bu_1$,
recall that $\bu_1 = \hat \be_1$ (see Eq. (<ref>)).
Then we apply Lemma <ref> to conclude that     we have $\big| \| \bu_{1,\cI_2} \|_2 / \sqrt{n_1} - \E[\| \bu_{1,\cI_2}^f \|_2 ]/\sqrt{n_1} \big| < \sqrt{p/n_1}\,\epsilon$ and $\big| \| \bu_{1,\cI_2^c} \|_2 / \sqrt{n_1} - \E[\| \bu_{1,\cI_2^c}^f \|_2 ]/\sqrt{n_1} \big| < \sqrt{p/n_1}\,\epsilon$.
Then, because $\bu_{1,\cI_1}$ is a Gaussian with variance bounded by $C$ (see Eq. (<ref>) and Lemma <ref>) and $\bu_{1,\cI_1^c} = 0$,
we can use Gaussian concentration of Lipschitz functions to conclude that $\big| \E[\| \bu_{1,\cI_2}^f \|_2]/\sqrt{n_1} - \E[\| \bu_{1,\cI_2}^f \|_2^2]^{1/2}/\sqrt{n_1} \big| < C/\sqrt{n_1}$ and $\big| \E[\| \bu_{1,\cI_2^c}^f \|_2 ]/\sqrt{n_1} - \E[\| \bu_{1,\cI_2^c}^f \|_2^2 ]^{1/2}/\sqrt{n_1}\big| < C $.
Thus, by adjusting $\sC(\epsilon)$ and $\sc(\epsilon)$ so that the bound becomes trivial with $\sqrt{p/n_1}\epsilon < C / \sqrt{n_1}$,
we conclude the required concentration of $\| \bu_{1,\cI_2} \|_2 / \sqrt{n_1}$ and $\| \bu_{1,\cI_2^c} \|_2 / \sqrt{n_1}$.

The required concentration of $\|\be_{2,\cI_2}\|_2/\sqrt{n_2}$ follows by Lemma <ref>.
The required concentration of $\| \proj_{\bv_1}\bxi_g\|_2 / \sqrt{n_2}$ and $\| \proj_{\bu_2}\bxi_h\|_2/\sqrt{n_2}$ holds because $\| \proj_{\bv_1}\bxi_g\|_2$ and $\| \proj_{\bu_2}\bxi_h\|_2$ are nothing but the norm of univariate Gaussians.

Finally, we show $\bv_2^{cf} \in E_{v|1}(\Delta/2)$ and $\bu_2^{cf} \in E_{u|1}(\Delta/2)$
By the definition of the conditional fixed design model (see Eq. (<ref>)),
we have $\E\big[ \phi_v(\bv_2^f) \bigm| \bg_1^f = \hat \bg_1 \big] = \E\big[ \phi_v(\bv_2^{cf}) \bigm| \hat \bg_1 \big]$.
Moreover, the above discussion implies that conditional on $\hat \bg_1$, $\bv_2^{cf}$ is a $C/\sqrt{n_2}$-Lipschitz function of $\bxi_g$. Thus, the required high-probability bound on $\bv_2^{cf} \in E_{v|1}(\Delta/2)$ holds by Gaussian concentration of Lipschitz functions and because $p/n_2 \leq C$ by .
we have $\E\big[ \phi_u(\bu_{2,\cI_2}^f/\sqrt{n_2}) \bigm| \bh_1^f = \hat \bh_1,\, \be_1,\be_2 \big] = \E\big[ \phi_u(\bu_{2,\cI_2}^{cf}/\sqrt{n_2}) \bigm| \hat \bh_1,\, \be_1,\be_2 \big]$.
Further, conditional on $\hat \bh_1,\be_1,\be_2$, we have $\bu_{2,\cI_2}^{cf}/\sqrt{n_2}$ is a $\tau_{h_2}/\sqrt{n_2} \leq C \sqrt{p}\,/n_2$-Lipschitz function of $\bxi_h$, where the inequality follows from Lemma <ref>.
Thus, we also conclude the required high-probability bound on $\bu_2^{cf} \in E_{u|1}(\Delta/2)$ by Gaussian concentration of Lipschitz functions.

The proof of Lemma <ref> is complete.

§ PROOF OF THEOREM <REF>: ESTIMATING NOISE COVARIANCE

We treat the on-diagonal and off-diagonal entries of $\bS_e$ separately.

On-diagonal estimation: noise variance.
In Theorem <ref>, $k$ indexes the set $\cI_k$ of samples used to perform the estimation,
but the theorem provides both an estimate of the noise for the first and second linear model.
We use $l$ to index the linear model whose noise variance we are estimating.

Note that $\| \hat \btheta_l^\de - \hat \btheta_l \|_{\bSigma}^2 = \| \bSigma^{1/2}(\hat \btheta_l^\de - \btheta_l) - \bv_l \|_2^2$,
were recall $\bv_l = \bSigma^{1/2}(\hat \btheta_l - \btheta_l)$ by Eq. (<ref>).
Applying Lemma <ref>,[Recall that Lemmas <ref> and <ref>, Corollary <ref>, and Theorem <ref> apply also if $\hat \btheta_k^\de$ is computed using $\hat \df_k$ in place of $\df_k$.]
, $\big| \| \hat \btheta_l^\de - \hat \btheta_l \|_{\bSigma}^2 - (p\tau_{g_l}^2 - 2\tau_{g_l}^2\,\df_l + \tau_{h_l}^2)\big| < C (p/n_l)\epsilon$.
We study the concentration of $\| \hat \be_{l,\cI_k}^\de \|_2^2 / n_k$ using Lemma <ref> and either Lemma <ref> (in the case the $l = k$) or Corollary <ref> (in the case that $l \neq k$).[See footnote <ref>.]
By Eq. (<ref>) and because $\tau_{\hat e_l^\de}^2 < C$ by Lemma <ref>,
we have that $\hat \be_{l,\cI_k}^{f,\de}$ is Gaussian with variance bounded by $C$.
Thus, in Lemma <ref>,
we may take $M = 1 / \sqrt{n_k}$.
Further, by Lemma <ref> (in the case that $\ell = k$) or Corollary <ref> (in the case the $l \neq k$),
we have Eq. (<ref>) for $\hat \be_{l,\cI_k}^\de/\sqrt{n_k}$
with $K = \sqrt{p/(n_1 \wedge n_2)}$.[In fact, for $l = k$, we get the better bound $K = \sqrt{p/n_l}$.]
Because $\tau_{\hat e_l^\de}^2 < C$,
Eq. (<ref>) is satisfied.
Thus, we may apply Lemma <ref> to conclude
$\big|\hat \tau_{\hat e_l^\de}^2 - \tau_{\hat e_l^{\de}}^2 \big| = \big|\| \hat \be_{l,\cI_k}^{\de} \|_2^2/n_k - \tau_{\hat e_l^{\de}}^2 \big| \leq   C \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.
Recall $\hat \tau_{g_l}^2 = \hat \tau_{\hat e_l^{\de}}^2 / n_l = \| \hat \be_{l,\cI_k}^{\de} \|_2^2 / (n_kn_l)$ (see Eq. (<ref>)),
so that $(p-2\,\hat \df_l)\hat \tau_{g_l}^2 = (p/n_l - 2\,\hat \df_l /n_l) \hat \tau_{\hat e_l^{\de}}^2 $.
Because $p/n_l \leq C$ by  and $\df_l \leq n_l = 1 - \zeta_l \leq 1$,
the concentration of $\hat \tau_{\hat e_l^\de}^2$ and Lemma <ref>
imply that  we have $\big|(p-2\,\hat \df_l)\hat \tau_{g_l}^2 - (p-2\,\df_l)\tau_{g_l}^2 \big| \leq C \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.
Combining these results,
and using that $\tau_{\hat e_l^\de}^2 = \tau_{e_l}^2 + \tau_{h_l}^2$ (see Eq. (<ref>)),
we have
\begin{equation}
\Big|\hat \tau_{\hat e_l^{\de}}^2 + (p-2\,\hat \df_l) \hat \tau_{g_l}^2 - \| \hat \btheta_l^{\de} - \hat \btheta_l \|_{\bSigma}^2 - \tau_{e_l}^2\Big| \leq \sqrt{\frac{p}{n_1 \wedge n_2}}\,\epsilon.
\end{equation}

Off-diagonal estimation: noise covariance.
We establish the concentration of $\< \hat \btheta_1^\de - \hat \btheta_1 , \hat \btheta_2^\de - \hat \btheta_2 \>_{\bSigma} = \< \bSigma^{1/2}(\hat \btheta_1^\de - \hat \btheta_1) , \bSigma^{1/2}(\hat \btheta_2^\de - \hat \btheta_2) \>$ using the joint characterization (Theorem <ref>) and Lemma <ref>.
Recall by Eq. (<ref>) that $\bSigma^{1/2} \hat \btheta^{f,\de} = \btheta_k + \bg_k^f$.
Further, because proximal operators are 1-Lipschitz [36] and $\tau_{g_k} < C /\sqrt{n_k}$ by Lemma <ref>,
we have that $\sqrt{n_1/p}\,\bSigma^{1/2}(\hat \btheta_1^{f,\de} - \hat \btheta_1^f)$ and $\sqrt{n_2/p}\,\bSigma^{1/2}(\hat \btheta_2^{f,\de} - \hat \btheta_2^f)$ are $C/\sqrt{p}$-Lipschitz in $\bg_1^f / \tau_{g_1}$ and $\bg_2^f / \tau_{g_2}^f$, which are Gaussian vectors with variance bounded by $C$.
Thus we will apply Lemma <ref> with $M = C/\sqrt{p}$.
Lemma <ref> gives us the first line in Eq. (<ref>) for vectors $\ba^{(k)} = \sqrt{n_k/p}\,\bSigma^{1/2}(\hat \btheta_k^\de - \hat \btheta_k)$.
Further, taking $M_1 = \sqrt{n_1/p}$ and $M_2 = \sqrt{n_2/p}$ in Theorem <ref> and using that the singular values of $\bSigma$ are bounded above by $C$ gives us the second line of Eq. (<ref>) for the these vectors with $K = \min\big\{ \sqrt{n_1/p} \sqrt{p/n_1} + \sqrt{n_2/p} , \sqrt{n_1/p} + \sqrt{n_2/p}\sqrt{p/n_2} \big\} \leq C \sqrt{(n_1 \wedge n_2)/p} $.[See footnote <ref>]
Further, as we computed above, $(n_k/p)\E[\| \hat \btheta_k^{f,\de} - \hat \btheta_k \|_{\bSigma}^2] = n_k\tau_{g_k}^2(1-2\,\df_k/p) + (n_k/p)\tau_{h_k}^2 < C$ by Lemma <ref>,
so that Eq. (<ref>) is satisfied.
Thus, we may apply Lemma <ref>.
First, using that $\bSigma^{1/2} (\hat \btheta^{f,\de} - \btheta_k) = \bg_k^f$,
we compute the expectation

\begin{equation}
\begin{aligned}
&\E[ \< \bSigma^{1/2}(\hat \btheta_1^{f,\de} - \hat \btheta_1^f), \bSigma^{1/2}(\hat \btheta_1^{f,\de} - \hat \btheta_1^f) \> ]
\\
&= \E[ \< \bg_1^f , \bg_2^f \>  ]
- \E[ \< \bg_2^f , \bSigma^{1/2}(\hat \btheta_1^f - \btheta_1) \> ]
- \E[ \< \bg_1^f , \bSigma^{1/2}(\hat \btheta_2^f - \btheta_2) \> ]
+ \E[ \< \hat \btheta_1^f - \btheta_1 , \hat \btheta_2^f - \btheta_2 \>_{\bSigma}  ]
\\
&= \tau_{g_1}\tau_{g_2}\rho_g(p - \df_1 - \df_2) + \tau_{h_1}\tau_{h_2}\rho_h = S_{g,12}(p - \df_1 - \df_2) + S_{h,12},
\end{aligned}
\end{equation}

where in the final equality we have used (<ref>) and Gaussian integration by parts with the definition of $\df_k$ (see Eq. (<ref>)).
Thus, Lemma <ref> implies that ,
$\big| \< \hat \btheta_1^\de - \hat \btheta_1 , \hat \btheta_2^\de - \hat \btheta_2 \>_{\bSigma} - S_{g,12}(p - \df_1 - \df_2) - S_{h,12} \big| < \sqrt{(n_1 \wedge n_2)/p} \sqrt{p^2/(n_1n_2)}\,\epsilon = \sqrt{p/(n_1 \vee n_2)}\,\epsilon$.

We now study the concentration of $\widehat S_{\hat e^d,12}^{(k)}$.
We decompose

\begin{equation}
\widehat S_{\hat e^{\de},12}^{(k)}
\frac{\< \be_{1,\cI_k},\be_{2,\cI_k}\>}{n_k}
\frac{\< \hat \be_{1,\cI_k}^\de - \be_{1,\cI_k},\be_{2,\cI_k}\>}{n_k}
\frac{\< \be_{1,\cI_k}, \hat \be_{2,\cI_k}^{\de} -  \be_{2,\cI_k}\>}{n_k}
\frac{\< \hat \be_{1,\cI_k}^\de - \be_{1,\cI_k}, \hat \be_{2,\cI_k}^{\de} -  \be_{2,\cI_k}\>}{n_k}.
\end{equation}

Because $\be_1,\be_2$ are jointly Gaussian,
for $\epsilon < c'$, with probability at least $1 - \sC(\epsilon)e^{-\sc(\epsilon)p}$, $\big| \< \be_{1,\cI_k},\be_{2,\cI_k}\>/n_k - S_{e,12}\big| \leq C\sqrt{p/n_k}\,\epsilon \leq C \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.
To establish the concentration of $\< \hat \be_{1,\cI_k}^\de - \be_{1,\cI_k},\be_{2,\cI_k}\>/n_k$,
we use Lemma <ref> (in the case $k = 1$) or Corollary <ref> (in the case $k = 2$) together with Lemma <ref>.
Recall by Eq. (<ref>) that $\hat \be_1^{f,\de} - \be_1 = \bh_1^f$.
Because $\tau_{h_1} < C \sqrt{p/n_1}$ by Lemma <ref> and $\tau_{e_1}^2 \leq C$ by ,
we have that $\sqrt{n_1/p}(\hat \be_{1,\cI_k}^{f,\de} - \be_1)/\sqrt{n_k}$ and $\be_{2,\cI_k}/\sqrt{n_k}$
are $C / \sqrt{n_k}$-Lipschitz functions of Gaussian vectors with variance bounded by $C$.
Thus we will apply Lemma <ref> with $M = C/\sqrt{n_k}$.
In the case $k=1$,
Lemma <ref> gives us Eq. (<ref>) for vectors $\sqrt{n_1/p}\,(\hat \be_{1,\cI_k}^\de - \be_{1,\cI_k})/\sqrt{n_k}$ and $\be_{1,\cI_k}/\sqrt{n_k}$ with $K = C$.
In the case $k = 2$,
Corollary <ref> with $M_e = 1$ and $M_h = \sqrt{n_1/p}$ gives us Eq. (<ref>) for these vectors with $K = \sqrt{p/n_2} + \sqrt{n_1/p}\sqrt{p/n_1} \leq  C$.
Further, we have $\E[\| \be_{2,\cI_k} \|_2^2]/n_k = \tau_{e_2}^2 \leq C$ and $(n_1/p)\E[\| \bh_{1,\cI_k}^f \|_2^2]/n_k = (n_1/p)\tau_{h_1}^2 \leq C$ by Eqs. (<ref>) and (<ref>) and Lemma <ref>,
whence Eq. (<ref>) is satisfied.
Thus, we may apply Lemma <ref>.
Because $\E[\< \bh_{1,\cI_k}^f , \be_{2,\cI_k} \>]/n_k = 0$, we conclude that ,
$|\< \hat \be_{1,\cI_k}^\de - \be_{1,\cI_k},\be_{2,\cI_k}\>/n_k| \leq \sqrt{p/n_1}\,\epsilon \leq \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.
An equivalent argument, switching the roles of the indices $1$ and $2$, gives that ,
$|\<\be_{1,\cI_k},\hat \be_{2,\cI_k}^\de - \be_{2,\cI_k}\>/n_k| \leq \sqrt{p/n_2}\,\epsilon \leq \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.

Finally, we deal with the last term
using Theorem <ref> and Lemma <ref>.
As justified in the previous paragraph,
we have that $\sqrt{n_1/p}\,(\hat \be_{1,\cI_k}^{f,\de} - \be_1)/\sqrt{n_k}$ and $\sqrt{n_2/p}\,(\hat \be_{2,\cI_k}^{f,\de} - \be_2)/\sqrt{n_k}$
are $C / \sqrt{n_k}$-Lipschitz functions of Gaussian vectors with variance bounded by $C$.
Thus, we will apply Lemma <ref> with $M = C / \sqrt{n_k}$.
In the case $k = 1$,
Lemma <ref> (in the case $k = 1$) and Corollary <ref> (in the case $k = 2$) give the first line of Eq. (<ref>) for the vector $\sqrt{n_1/p}\,(\hat \be_{1,\cI_k}^\de - \be_1)/\sqrt{n_k}$.
Likewise, Eq. (<ref>) is satisfied for the vector $\sqrt{n_2/p}\,(\hat \be_{2,\cI_k}^\de - \be_2)/\sqrt{n_k}$.
Theorem <ref> with $M_1 = \sqrt{n_1/p}$ and $M_2 = \sqrt{n_2/p}$
gives the second line of Eq. (<ref>) with $K = \sqrt{n_1/p}(\sqrt{p/n_2} + p/n_1) + \sqrt{n_2/p} \leq C \sqrt{(n_1 \vee n_2)/p}$ in the case $k = 2$ and $K = \sqrt{n_1/p} + \sqrt{n_2/p}(\sqrt{p/n_1} + p/n_2) \leq C\sqrt{(n_1 \vee n_2)/p}$ in the case $k = 1$.[Note that we have used here the rate given by Theorem <ref> because we want to accommodate the case that $\hat \be_k^\de$ is compute with $\hat \df_k$ in place of $\df_k$.]
Finally, as we checked above $(n_1/p)\E[\|\bh_{1,\cI_k}^f\|_2^2]/n_k \leq C$ and $(n_2/p)\E[\|\bh_{2,\cI_k}^f\|_2^2]/n_k \leq C$.
Thus, we may apply Lemma <ref>.
Because $\E[\< \bh_{1,\cI_k}^f, \bh_{2,\cI_k}^f\>]/n_k = S_{h,12} $ (see Eq. (<ref>)),
we conclude that ,
$\big|\< \hat \be_{1,\cI_k}^{\de} - \be_{1,\cI_k}, \hat \be_{2,\cI_k}^{\de} -  \be_{2,\cI_k}\>/n_k - S_{h,12}\big| \leq \sqrt{(n_1 \vee n_2)/p}\sqrt{p^2/(n_1n_2)}\,\epsilon = \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.

Combining the concentration bounds on each term in the decomposition,
we have , $\big| \widehat S_{\hat e^{\de},12}^{(k)} - S_{e,12} - S_{h,12}\big| = \big| \widehat S_{\hat e^{\de},12}^{(k)} - S_{\hat e^{\de},12}\big| < \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$,
where we have used that $ S_{\hat e^{\de},12} = S_{e,12} + S_{h,12}$ by Eq. (<ref>).
Recalling that $\widehat S_{g,12}^{(k)} = (n_{12}/(n_1n_2)) \widehat S_{\hat e^{\de},12}^{(k)} $,
we conclude that , $(p - \df_1 - \df_2)\widehat S_{g,12}^{(k)} = (n_{12}/(n_1n_2))(p-\df_1-\df_2)\widehat S_{\hat e^{\de},12}^{(k)} $.
Using Lemma <ref>,
we have that $(n_{12}/(n_1n_2))(p-\df_1-\df_2) \leq C$.
recalling that $(n_{12}/(n_1n_2))S_{\hat e^{\de},12} = S_{g,12}$ by Eq. (<ref>),
the concentration bound for $\widehat S_{\hat e^{\de},12}^{(k)}$ implies that
, $\big| (p - \df_1-\df_2)\widehat S_{g,12}^{(k)} - (p - \df_1-\df_2) S_{g,12}\big| < \sqrt{p/(n_1 \wedge n_2)}\,\epsilon$.
Combining the above results and using that $S_{\hat e^\de,12} = S_{e,12} + S_{h,12}$, we conclude that

\begin{equation}
\Big| \widehat S_{\hat e^{\de},12}^{(k)} + (p - \df_1 - \df_2) \widehat S_{g,12}^{(k)} - \<\hat \btheta_1^{\de} - \hat \btheta_1,\hat \btheta_2^{\de} - \hat \btheta_2\>_{\bSigma} - S_{e,12} \Big|
\leq
\sqrt{\frac{p}{n_1 \wedge n_2}}\,\epsilon.
\end{equation}

The proof of Theorem <ref> is complete. $\square$

§ PROOF OF LEMMA <REF>: THE CONDITIONAL GORDON INEQUALITY

The conditional Gordon inequality for regression (Lemma <ref>) is derived by applying the marginal Gordon inequality (Lemma <ref>) conditionally on $\mathsf{Cond}_1$, and then marginalizing over $\mathsf{Cond}_1$.
The key observation will be that conditioning on
is equivalent to conditioning on a set of linear constraints on $\bA$ (recall we define $\bA = - \bX \bSigma^{-1/2}$).
First, we develop a comparison inequality which holds when conditioning on a certain type of linear constraints on $\bA$.

For any vectors $\bu_1',\bt_1',\be_1',\be_2' \in \reals^N$, $\bv_1',\bs_1' \in \reals^p$
consider the event (over the randomness in $\bA$)

\begin{equation}\label{eq:A-lin-constraint}
\cE_{2|1} := \cE_{2|1}(\bu_1' , \bv_1', \bs_1', \bt_1')
\Big\{
\bA^\top \bu_1' +  \bs_1' = 0,
\;
\bA  \bv_1' -  \bt_1' = 0
\Big\}.
\end{equation}

Consider the system of equations, repeated from Eq. (<ref>) except now written for arbitrary $\bu_1',\bv_1',\bs_1',\bt_1'$ (not necessarily those defined in Eq. (<ref>)).

\begin{equation}\label{eq:hat-xi-rep}
\begin{gathered}
-\frac{ \bu_1'}{\| \bu_1'\|_2} \< \hat \bxi_g,  \bv_1' \> + \|  \bv_1'\|_2 \hat \bxi_h -  \bt_1' = 0,
\qquad
-\|  \bu_1' \|_2 \hat \bxi_g + \frac{ \bv_1'}{\|  \bv_1'\|_2} \< \hat \bxi_h,  \bu_1' \> +  \bs_1' = 0.
\end{gathered}
\end{equation}

These equations have a solution $\hat \bxi_g,\hat \bxi_h$
if and only if
the equations (<ref>) have a solution $\bA$:

The following are equivalent properties of the quadruplet $(\bu_1',\bv_1',\bs_1',\bt_1')$.

$\<\bv_1',\bs_1'\> + \< \bu_1',\bt_1'\> = 0$.

There exists $\bA$ such that the constraints (<ref>) are satisfied.

* There exist $\hat \bxi_g,\hat \bxi_h$ such that the constraints (<ref>) are satisfied.

For any $\alpha \in \reals$, under the additional constraint

\begin{equation}
\frac{\< \hat \bxi_h ,\bu_1 \>}{n_1} = \alpha,
\end{equation}

the solution $\hat \bxi_g,\hat \bxi_h$ to Eq. (<ref>) is unique.

Lemma <ref> is proved at the end of this section.
We are ready to state the conditional Gordon inequality, of which Lemma <ref> is corollary.

Fix $ \bu_1',\bt_1' \in \reals^N$ and $\bv_1', \bs_1' \in \reals^p$
such that $\<\bv_1',\bs_1'\> + \< \bu_1',\bt_1'\> = 0$.
Fix any $\hat \bxi_g,\hat \bxi_h$ satisfying Eq. (<ref>) (which exist by Lemma <ref>).

Let $\bA \in \reals^{N\times p}$ have entries $A_{ij} \stackrel{\mathrm{iid}}\sim \normal(0,1)$,
$\bxi_g \stackrel{\mathrm{iid}}\sim \normal(0,\id_p)$, and $\bxi_h \sim \normal(0,\id_N)$, all independent,
and define

\begin{equation}
\bg_{\mathrm{cg}}(\bu)
\frac{\<  \bu_1',\bu \>}{\| \bu_1'\|_2}\hat \bxi_g +  \| \sfP_{ \bu_1'}^\perp \bu\|_2  \sfP_{ \bv_1'}^\perp  \bxi_g,
\qquad
\bh_{\mathrm{cg}}(\bv)
\frac{\<  \bv_1' , \bv \> }{\|  \bv_1' \|_2} \hat \bxi_h + \| \sfP_{ \bv_1'}^\perp \bv\|_2  \sfP_{ \bu_1'}^\perp \bxi_h,
\end{equation}

(where the subscript stands for “conditional Gordon”).
Define $\cE_{2|1} = \cE_{2|1}(\bu_1',\bv_1',\bs_1',\bt_1')$ as in Eq. (<ref>).

If $E_u \in \reals^N$, $E_v \in \reals^p$ are compact sets,
then for any $t\in \reals$

\begin{equation}
\begin{aligned}
\PP\left( \min_{\bv \in E_v} \max_{\bu \in E_u} \bu^\top \bA \bv + \psi(\bu,\bv) \leq t \bigm| \cE_{2|1} \right)
\leq
\min_{\bv \in E_v} \max_{\bu \in E_u}
\<\bh_{\mathrm{cg}}(\bv),\bu\>
\psi(\bu,\bv) \leq t\right).
\end{aligned}
\end{equation}

If $E_u \in \reals^N$, $E_v \in \reals^p$ are compact, convex sets,
then for any $t\in \reals$

\begin{equation}
\begin{aligned}
\PP\left( \min_{\bv \in E_v} \max_{\bu \in E_u} \bu^\top \bA \bv + \psi(\bu,\bv) \geq t \bigm| \cE_{2|1} \right)
\leq
\min_{\bv \in E_v} \max_{\bu \in E_u}
\<\bh_{\mathrm{cg}}(\bv),\bu\>
\psi(\bu,\bv) \geq t\right).
\end{aligned}
\end{equation}

We prove Lemma <ref> at the end of this section.
First, we prove Lemma <ref> as a straightforward consequence.

Recall from Eq. (<ref>) that
$\bt_1 = \bA \bv_1$
and $\bs_1 = -\bA^\top \bu_1$.
The KKT conditions for the min-max problem (<ref>) state that $(\bv_1,\bu_1)$ is a saddle point for this min-max problem if and only if

\begin{equation}
\frac1{n_1} \bt_{1,\cI_1} + \frac1{n_1} \be_{1,\cI_1} - \frac1{n_1} \bu_{1,\cI_1} = 0,
\quad
-\frac1{n_1}\bs_1 \in \partial \bar \Omega_1(\bv_1),
\quad
\bu_{1,\cI_1^c} = 0.
\end{equation}

Now consider any vectors $\bu_1',\bt_1',\be_1',\be_2' \in \reals^N$, $\bv_1',\bs_1' \in \reals^p$ satisfying $\bu_{1,\cI_1^c}' = \bzero$,
$\bt_{1,\cI_1}' = \bu_{1,\cI_1}' - \be_{1,\cI_1}'$,
and $\bs_1' / n_1 \in \partial \bar \Omega_1(\bv_1)$.
(The prime indicates that these will be treated as dummy variables, not as those variables defined via Eq. (<ref>)).
The KKT conditions above imply that the event (over the randomness is $\bA,\be_1,\be_2$)

\begin{equation}
\cE_{2|1}(\bu_1' , \bv_1', \bs_1', \bt_1') \cap
\Big\{
\be_1 = \be_1',
\;
\be_2 = \be_2'
\Big\},
\end{equation}

is equivalent to the event

\begin{equation}
\Big\{
\hat \be_1 = \bu_1',
\;
\bSigma^{1/2}(\hat \btheta_1 - \btheta_1) = \bv_1',
\;
\bX(\hat \btheta_1 - \btheta_1) = -\bt_1',
\;
\bSigma^{-1/2}\bX^\top \hat \be_1 = \bs_1',
\;
\be_1 = \be_1',
\;
\be_2 = \be_2'
\Big\}.
\end{equation}

the event that $\mathsf{Cond}_1 = (\hat \btheta_1, \hat \be_1, \bX (\hat \btheta_1-\btheta_1), \bX^\top \hat \be_1,\be_1,\be_2)$ obtains a certain value is equivalent to the event $\cE_{2|1}(\bu_1' , \bv_1', \bs_1', \bt_1') \cap
\Big\{
\be_1 = \be_1',
\;
\be_2 = \be_2'
\Big\}$ for $\bu_1' = \bu_1 , \bv_1' = \bv_1 , \bs_1' = \bs_1, \bt_1' = \bt_1$, and vice versa.
We can thus apply Lemma <ref> conditionally on $\mathsf{Cond}_1$.

Take $\bg_{\mathrm{cg}}(\bu)$ and $\bh_{\mathrm{cg}}(\bv)$ as defined in Lemma <ref>.
Observe that $E_u = E_u(\mathsf{Cond}_1)$ is a deterministic set conditional on $\mathsf{Cond}_1$.
Thus, applying Lemma <ref>(i) conditionally,
we have for any $t \in \reals$ that with probability 1,

\begin{equation}
\PP\left( \min_{\bv \in E_v} \max_{\bu \in E_u} L_2(\bu,\bv) \leq t \Bigm| \cE_{2|1}(\mathsf{Cond}_1)\right)
\leq
\min_{\bv \in E_v} \max_{\bu \in E_u}
\ell_{2|1}(\bu,\bv) \leq t \Bigm|
\bu_1,\bv_1,\hat \bxi_g,\hat \bxi_h
\right).
\end{equation}

Taking the expectation of both sides of this inequality gives Lemma <ref>(i).
Lemma <ref>(ii) follows by the same argument using instead Lemma <ref>(ii).

We now prove Lemma <ref>.

Conditional on $\cE_{2|1}(\bu_1',\bv_1',\bs_1',\bt_1')$,

\begin{align*}
\bu^\top \bA \bv
\bu^\top \sfP_{ \bu_1'}^\perp \bA \sfP_{ \bv_1'}^\perp \bv
\bu^\top \bA \sfP_{ \bv_1'}\bv
\bu^\top \sfP_{ \bu_1'} \bA \bv
\bu^\top \sfP_{ \bu_1'} \bA \sfP_{ \bv_1'} \bv
\\
\bu^\top \sfP_{ \bu_1'}^\perp \bA \sfP_{ \bv_1'}^\perp \bv
\frac{\< \bu ,  \bt_1' \> \<  \bv_1' , \bv \> }{ \|  \bv_1' \|_2^2 }
\frac{ \< \bu ,  \bu_1' \> \<  \bs_1' , \bv \>  }{\|  \bu_1'\|_2^2 }
\frac{ \< \bu ,  \bu_1' \> \<  \bs_1' ,  \bv_1'\> \<  \bv_1' , \bv \>  }{\|  \bv_1' \|_2^2 \|  \bu_1'\|_2^2 }
\\
\bu^\top \sfP_{ \bu_1'}^\perp \bA \sfP_{ \bv_1'}^\perp \bv + \phi(\bu,\bv)
\stackrel{\mathrm{d}}=
\bu^\top \sfP_{ \bu_1'}^\perp \tilde \bA \sfP_{\bv_1'}^\perp \bv
\phi(\bu,\bv),
\end{align*}

where $\tilde \bA$ is independent of and identically distributed to $\bA$.
We emphasize that the distributional equivalence holds conditional on the event (<ref>).

\begin{equation}
\begin{aligned}
&\PP\left( \min_{\bv \in E_v} \max_{\bu \in E_u} \bu^\top \bA \bv + \psi(\bu,\bv) \leq t \bigm| \cE_{2|1} \right)
\PP\left(
\min_{\bv \in E_v} \max_{\bu \in E_u}
\bu^\top \sfP_{\bu_1'}^\perp \tilde \bA \sfP_{\bv_1'}^\perp \bv
\phi(\bu,\bv)
\psi(\bu,\bv) \leq t\right).
\end{aligned}
\end{equation}

Let $\bxi_g \sim \normal(0,\id_p)$, $\bxi_h \sim \normal(0,\id_N)$.
By the marginal Gordon inequality (Lemma <ref>),

\begin{equation}\label{eq:cond-gordon-unsimplified}
\begin{aligned}
\min_{\bv \in E_v}
\max_{\bu \in E_u}
\bu^\top \sfP_{\bu_1'}^\perp \tilde \bA \sfP_{\bv_1'}^\perp \bv
\phi(\bu,\bv)
\psi(\bu,\bv) \leq t\right)
\\
\min_{\bv \in E_v} \max_{\bu \in E_u}
-\| \sfP_{\bu_1'}^\perp \bu\|_2 \bxi_g^\top \sfP_{\bv_1'}^\perp \bv
\| \sfP_{\bv_1'}^\perp \bv\|_2 \bxi_h^\top \sfP_{\bu_1'}^\perp \bu
\phi(\bu,\bv)
\psi(\bu,\bv) \leq t\right).
\end{aligned}
\end{equation}

We now simplify the above expression.
Using Eq. (<ref>),

\begin{equation}
\begin{aligned}
\phi(\bu,\bv)
\frac{\big\< \bu , - \bu_1' \< \hat \bxi_g,\bv_1'\> /\| \bu_1' \|_2 + \| \bv_1' \|_2 \hat \bxi_h \big\> \< \bv_1' , \bv \> }{ \| \bv_1' \|_2^2 }
\frac{ \< \bu , \bu_1' \> \big\< - \| \bu_1'\|_2 \hat \bxi_g + \bv_1' \< \hat \bxi_h,\bu_1'\> /\|\bv_1'\|_2 , \bv \big\>  }{\| \bu_1'\|_2^2 }
\\
&\qquad\qquad- \frac{ \< \bu , \bu_1' \> \< -\| \bu_1'\|_2 \hat \bxi_g + \bv_1' \< \hat \bxi_h,\bu_1'\> /\|\bv_1'\|_2  ,\bv_1' \> \< \bv_1', \bv \> }{\| \bv_1'\|_2^2 \| \bu_1' \|_2^2}
\\
- \frac{\< \bu_1' , \bu \> }{\| \bu_1' \|_2} \< \hat \bxi_g , \bv \> + \frac{\< \bv_1' , \bv \> }{\| \bv_1' \|_2} \< \hat \bxi_h , \bu \> .
\end{aligned}
\end{equation}

Substituting into Eq. (<ref>),
we conclude

\begin{equation}
\begin{aligned}
&\PP\left( \min_{\bv \in E_v} \max_{\bu \in E_u}  \bu^\top \bA \bv + \psi(\bu,\bv) \leq t \bigm| \cE_{2|1} \right)
\\
\min_{\bv \in E_v} \max_{\bu \in E_u}
-\big(\| \sfP_{\bu_1'}^\perp \bu\|_2  \sfP_{\bv_1'}^\perp \bxi_g + \frac{\< \bu_1' , \bu \> }{\| \bu_1' \|_2} \hat \bxi_g \big)^\top \bv
\big(\| \sfP_{\bv_1'}^\perp \bv\|_2  \sfP_{\bu_1'}^\perp \bxi_h + \frac{\< \bv_1' , \bv \> }{\| \bv_1' \|_2} \hat \bxi_h\big)^\top \bu
\psi(\bu,\bv) \leq t\right).
\end{aligned}
\end{equation}

The argument when $E_v,E_u$ are also convex and $\psi$ is convex-concave is equivalent but applies the convex version of the marginal Gordon inequality (Lemma <ref>(ii)).
The proof is complete.

We first prove Lemma <ref>.
We write the system of equations (<ref>) in matrix-form

\begin{equation}
\begin{pmatrix}
\bs_1 \\  \bt_1
\end{pmatrix}
\begin{pmatrix}
\|  \bu_1' \|_2 \id_p  &  -  \bv_1'  \bu_1'^\top / \|  \bv_1' \|_2 \\
-  \bu_1'  \bv_1'^\top / \|  \bu_1' \|_2 & \|  \bv_1' \|_2 \id_N
\end{pmatrix}
\begin{pmatrix}
\bg \\ \bh
\end{pmatrix}
\bM \begin{pmatrix}
\bg \\ \bh
\end{pmatrix}.
\end{equation}

The matrix $\bM$ in the preceding display is rank $p+N-1$,
so that its column space has codimension 1.
Checking that $\begin{pmatrix}  \bv_1'^\top &  \bu_1'^\top \end{pmatrix} \bM = 0$,
we conclude Lemma <ref>.

Lemma <ref> follows from Lemma <ref> taking $\alpha = \zeta_1 \tau_{h_1}$ and observing that by the KKT conditions, Eq. (<ref>) is satisfied for $\bu_1' = \bu_1$, $\bs_1' = \bs_1$, $\bv_1' = \bv_1$, $\bt_1' = \bt_1$, and $\bA = - \bX\bSigma^{-1/2}$.

The conditional Gordon inequality makes an interesting—though to us still mysterious—connection between the primal and auxilliary KKT conditions.
In particular,
Eqs. (<ref>) are the KKT conditions for the regression optimization (<ref>), and Eqs. (<ref>) are the KKT conditions for the marginal auxilliary objective (<ref>).
Indeed, the gradient of $- \<\bg_{\mathrm{mg}}(\bu), \bv\> + \<\bh_{\mathrm{mg}}(\bv), \bu\>$ with respect to $\bu$ is $- (\bu / \| \bu \|_2)\< \bxi_g,\bv\> + \| \bv \|_2 \bxi_h$,
and its gradient with respect to $\bv$ is $- \| \bu \|_2 \bxi_g + (\bv/\|\bv\|_2)\< \bxi_h,\bu\>$.
Thus, for a quadruplet $(\bu_1',\bv_1',\bs_1',\bt_1')$
for which $\bu_{1,\cI_1^c}' = \bzero$,
$\bt_{1,\cI_1}' = \bu_{1,\cI_1}' - \be_{1,\cI_1}'$,
and $\bs_1' / n_1 \in \partial \bar \Omega_1(\bv_1)$,
Eq. (<ref>) identifies those realizations of the random matrix $\bA$ for which the primal KKT conditions are satisfied at $(\bu_1',\bv_1',\bs_1',\bt_1')$,
and Eq. (<ref>) identifies those realizations of the random noise in Gordon's objective for which the auxilliary KKT conditions are satisfied at $(\bu_1',\bv_1',\bs_1',\bt_1')$.

§ TECHNICAL LEMMAS

§.§ Properties of fixed-point solutions: proofs of Lemmas <ref>, <ref>, <ref>, and <ref>

We first show that, for fixed $k$, the equations (<ref>)
have a unique solution $\tau_{g_k},\zeta_k$ (note that these equations do not depend on $\rho_g$ or $\tau_{g_l},\zeta_l$ for $l \neq k$).
We consider least squares, ridge regression, and the Lasso separately.

In the case of least squares, we may solve these two equations explicitly.
Indeed, $\sR_k(\tau^2,\zeta) = p \tau^2$ and $\df_k(\tau^2,\zeta) = p$,
whence by Eq. (<ref>)

\begin{equation}\label{eq:ols-fixed-pt}
\text{for least squares,}
\;\;
\tau_{g_k}^2 = \frac{\tau_{e_k}^2/n_k}{1 - p / n_k}
\;\;
\text{and}
\;\;
\zeta_k = 1 - p/n_k
\;\;
\text{uniquely}.
\end{equation}

Because $\tau_{e_k}^2 > 0$ and $p < n_k$,
we have $\tau_{g_k}^2 > 0$

In the case of ridge regression with $\lambda  > 0$,
we have the explicit form

\begin{equation}\label{eq:ridge-R-df}
\begin{gathered}
\sR_k(\tau^2,\zeta)
\frac{p\lambda^2}{n_k \zeta^2} \btheta_k \bSigma^{1/2}\Big(\bSigma + \sqrt{\frac{p}{n_k}} \, \frac{\lambda}{\zeta} \id_p\Big)^{-2}\bSigma^{1/2} \btheta_k
\tau^2 \Tr\Big(\bSigma^2\Big(\bSigma + \sqrt{\frac{p}{n_k}}\,\frac{\lambda}{\zeta}\id_p\Big)^{-2}\Big),
\\
\df_k(\tau^2,\zeta)
\Tr\Big(\bSigma\Big(\bSigma + \sqrt{\frac{p}{n_k}}\,\frac{\lambda}{\zeta}\id_p\Big)^{-1}\Big).
\end{gathered}
\end{equation}

The degrees-of-freedom $\df_k(\tau^2,\zeta)$ does not depend on $\tau^2$, is continuous and strictly increasing in $\zeta$,
and is 0 when $\zeta = 0$.
Thus, the second fixed-point equation (i.e., $\zeta_k =  1 - \df_k(\tau_{g_k}^2,\zeta_k)/n_k$) has a unique solution $\zeta_k > 0$.
Because $\bSigma( \bSigma + \sqrt{p/n_k}\,(\lambda/\zeta_k) \id_p)^{-1} \preceq \id_p$ and is diagonalizable,
$\Tr\big(\bSigma^2( \bSigma + \sqrt{p/n_k}\,(\lambda/\zeta_k) \id_p)^{-2} \big) \leq \df_k(\tau_{g_2}^2,\zeta_k) < n_k$.
Using the previous display and Eq. (<ref>),
we may solve

\begin{equation}\label{eq:ridge-tauO*}
\tau_{g_k}^2
\frac{\tau_{e_k}^2 + p\lambda^2/(n\zeta_k^2) \btheta_k^\top \bSigma^{1/2} (\bSigma + \sqrt{p/n_k}\, (\lambda / \zeta_k) \id_p)^{-2}\bSigma^{1/2} \btheta_k }{n_k - \Tr\big(\bSigma^2( \bSigma + \sqrt{p/n_k}\,(\lambda/\zeta_k) \id_p)^{-2} \big) } > 0.
\end{equation}

In the case of the Lasso and the $\alpha$-smoothed Lasso,
we apply results from the paper [15], which considers the same setting with a different covariate normalization.
In Section <ref>,
we provide the change of variables to translate our normalization into theirs.
Under this change of variables,
the fixed point equations (<ref>) are exactly equations (8a) and (8b) of [15].
Theorem 1 and Lemma A.2 of [15] guarantee these equations have a unique solution when $\bSigma$
is invertible and $\tau_{e_k}^2 > 0$,
and the solution $\tau_{g_k}^2 > 0$.

Finally, we show that given a solution $\tau_{g_1},\zeta_1,\tau_{g_2},\zeta_2$ to Eqs. (<ref>),
there is a solution $\rho_g$ to the equation Eq. (<ref>).
By Cauchy-Schwartz, $\sR_{12}(\bS_g,\{\zeta_k\}) \leq \sqrt{\sR_1(\tau_{g_1}^2,\zeta_1)\sR_2(\tau_{g_2}^2,\zeta_2)}$. Because also $n_{12}/\sqrt{n_1 n_2} \leq 1$,
the right-hand side of the fixed point equation (<ref>) is between

\begin{equation}\label{eq:Q-bound}
\pm
\frac{1}{\sqrt{n_1}}\sqrt{\tau_{e_1}^2\rho_e + \sR_1(\tau_{g_1}^2,\zeta_1)}
\frac{1}{ \sqrt{n_2} } \sqrt{\tau_{e_2}^2\rho_e + \sR_2(\tau_{g_2}^2,\zeta_2)}
\pm\tau_{g_1}\tau_{g_2}\sqrt{\big(1 - \tau_{e_1}^2\rho_e^{\perp2}/(n_1\tau_{g_1}^2)\big)\big(1 - \tau_{e_2}^2\rho_e^{\perp2}/(n_2\tau_{g_2}^2)\big)},
\end{equation}

where we have used the fixed point equations (<ref>) to get the final equality.
Because $\tau_{e_1},\tau_{e_2},\rho_e^\perp > 0$ (see Lemma <ref>),
we conclude the right-hand side of the equation (<ref>) is always in the open interval $(-\tau_{g_1}\tau_{g_2},\tau_{g_1}\tau_{g_2})$.
Moroever, it is continuous in $\rho_g$.
in the definition of $\sR_{12}$,
we may represent $\bg_2^f = (\tau_{g_2}/\tau_{g_1})\rho_g \bg_1^f + \tau_{g_2}\sqrt{1-\rho_g^2}\bxi_g$ where $\bxi_g \stackrel{\mathrm{iid}}\sim \normal(0,\id_p)$ independent of everything else.
Then continuity in $\rho_g$ follows by dominated convergence using that
$\hat \btheta_2^{f,\de}$ and $\hat \btheta_2$ are $\tau_{g_2}$-Lipshitz in $\bg_2^f$ (because proximal operators are $1$-Lipschitz [36]).
At $\rho_g = 1$, the left-hand side of the fixed point equation (<ref>) is $\tau_{g_1}\tau_{g_2}$, and at $\rho_g = -1$, the left-hand side is $-\tau_{g_1}\tau_{g_2}$.
The existence of a solution $\rho_g \in (-1,1)$ follows by the intermediate value theorem.

We now show uniqueness of the solution $\rho_g$.
We denote by $\partial_{\rho_g} \sR_{12}(\bS_g,\{\zeta_k\}) $ the partial derivative taken with $\tau_{g_1},\tau_{g_2},\zeta_1,\zeta_2$ held fixed;
by $\nabla_{\bg_k^f}$ the Jacobian with respect to $\bg_k^f$;
and for matrices $\bA,\bB \in \reals^{p\times p} $ by $\< \bA ,\bB \> = \Tr(\bA^\top \bB)$.
Then using the Dirichlet form for the Ornstein-Uhlenbeck process (see, for example, Sections 2.6 and 2.7 of [4]),
we compute

\begin{equation}\label{eq:cor-deriv}
\begin{aligned}
\big|\partial_{\rho_g} &\sR_{12}(\bS_g,\{\zeta_k\})\big|
\tau_1\tau_2 \big|
\E\big[ \big\< \nabla_{\bg_1^f} \bSigma^{1/2} \eta_1(\bSigma^{1/2} \btheta_1 + \bg_1^f;\zeta_1) , \nabla_{\bg_2^f} \bSigma^{1/2} \eta_2(\bSigma^{1/2} \btheta_2 + \bg_2^f;\zeta_2) \big\>  \big]\big|
\\
\tau_{g_1}\tau_{g_2}
\E\big[\Tr\big((\nabla_{\bg_1^f} \bSigma^{1/2} \eta_1(\bSigma^{1/2} \btheta_1 + \bg_1^f;\zeta_1))^2\big)\big]^{1/2}
\E\big[\Tr\big((\nabla_{\bg_2^f} \bSigma^{1/2} \eta_2(\bSigma^{1/2} \btheta_2 + \bg_2^f;\zeta_2))^2\big)\big]^{1/2}
\\
\tau_{g_1}\tau_{g_2}
\sqrt{\df_1(\tau_{g_1}^2,\zeta_1)\df_2(\tau_{g_2}^2,\zeta_2)}
< \tau_{g_1} \tau_{g_2} \sqrt{n_1n_2},
\end{aligned}
\end{equation}

where in the second-to-last inequality we have used that
$\nabla_{\bg_1^f} \bSigma^{1/2} \eta_1(\bSigma^{1/2} \btheta_1 + \bg_1^f;\zeta_1)$ is symmetric (because $\bSigma^{1/2} \eta_1(\bSigma^{1/2} \btheta_1 + \bg_1^f;\zeta_1)$ is the solution to an optimization problem), and because $\bSigma \eta_k$ is $1$-Lipschitz, its eigenvalues are bounded by 1.
Thus, the magnitude of the partial derivative with respect to $\rho_g$ of the right-hand side of Eq. (<ref>) is strictly less than $n_{12}/(n_1n_2)\tau_1\tau_2 \sqrt{n_1n_2} \leq \tau_1\tau_2$, where we have used that $n_{12} \leq n_1 \wedge n_2 \leq  \sqrt{n_1n_2}$.
Because the partial derivative of the left-hand side of Eq. (<ref>) with respect to $\rho_g$ is $\tau_1\tau_2$,
we conclude the solution $\rho_g$ is unique.

To provide bounds on the fixed-point parameters in the proof of Lemma <ref>,
we will need the following lemma.

Taking any $\tau_{g_k},\zeta_k$ (not necessarily solutions to the fixed-point equations (<ref>)),
there exists constant $C$ depending only on $\cPmodel,\cPregr$ such that when the $\alpha$-smoothed lasso is used in the fixed design model

\begin{equation}
\| \hat \btheta_k^f - \btheta_k \|_{\bSigma} \leq
\| \bg_k^f\|_2
\frac{C}{\zeta_k} \sqrt{\frac{p}{n_k}}.
\end{equation}

\begin{equation}
\E[\| \hat \btheta_k^f - \btheta_k \|_{\bSigma}^2]
\geq
\frac{c}{\zeta_k^2} \frac{p}{n_k} \Phi\Big(- \frac{C}{\sqrt{n_k}\,\tau_k \zeta_k}\Big),
\end{equation}

where $\Phi(z) = \P(Z \leq z)$ for $Z \sim \normal(0,1)$ is the Gaussian cdf.

The KKT conditions for the optimization (<ref>) are

\begin{equation}
\bSigma^{1/2}(\hat \btheta_k^f - \btheta_k )
\in
\bg_k^f - \frac1{\zeta_k} \bSigma^{-1/2} \partial \Omega_k(\hat \btheta_k),
\end{equation}

where for the Lasso and the $\alpha$-smoothed Lasso, we have $\| \bd \|_2 \leq C\sqrt{p/n_k}$ for any $\bd \in \partial \Omega_k(\hat \btheta_k)$ and any $\hat \btheta_k$.
The first result follows.

To derive the second result,
we write

\begin{equation}\label{eq:lasso-coord-bound}
\bSigma(\hat \btheta_k^f - \btheta_k )
\in
\bSigma^{1/2}\bg_k^f - \frac1{\zeta_k} \partial \Omega_k(\hat \btheta_k^f).
\end{equation}

Note $[\partial \Omega_k(\hat \btheta_k^f)]_j \in [-\lambda/\sqrt{n_k},\lambda /\sqrt{n_k}]$,
whence $\big| [\bSigma(\hat \btheta_k^f - \btheta_k )]_j \big| \geq \big| [\bSigma^{1/2}\bg_k^f]_j \big| - \lambda / (\sqrt{n_k}\,\zeta_k)$.
$\E\big[[\bSigma(\hat \btheta_k^f - \btheta_k )]_j^2\big] \geq \P\big( \big| [\bSigma^{1/2}\bg_k^f]_j \big| > 2 \lambda / (\sqrt{n_k}\zeta_k) \big) \lambda^2/(n_k\zeta_k^2) \geq c\Phi\big( - C/(\sqrt{n_k}\,\tau_{g_k} \zeta_k) \big) / (n_k \zeta_k^2)$.
Because $\| \hat \btheta_k^f - \btheta_k \|_{\bSigma}^2 \geq c \| \bSigma(\hat \btheta_k^f - \btheta_k ) \|_2^2 = c \sum_{j=1}^p [\bSigma(\hat \btheta_k^f - \btheta_k )]_j^2$,
the result follows.

By the first equation of (<ref>),
we have $\tau_{g_k}^2 \geq \tau_{e_k}^2 / n_k \geq c/n_k$.
We next determine an upper bound on $\tau_{g_k}$ and lower bound on $\zeta_k$.
We do so for least-squares, ridge regression, and the $\alpha$-smoothed Lasso separately (where $\alpha= 0$ is the Lasso).

For least squares,
by Eq. (<ref>) we have $\zeta_k = 1 - p/n_k > c > 0$.
Also by Eq. (<ref>),
we have $\tau_{g_k}^2 \leq (\tau_{e_k}^2 /n_k)/\zeta_k \leq C/n_k$.

For ridge regression, there are two cases.
In the first case, $n_k/p \geq c > 1$.
Then we use $\df_k(\tau^2,\zeta) \leq p$ (see Eq. (<ref>))
to get $\zeta_k = 1 - \df_k(\tau_{g_k}^2,\zeta_k)/n_k \geq 1 - 1/c > 0$.
Note $n_k - \Tr(\bSigma^2(\bSigma + \sqrt{p/n_k}(\lambda / \zeta_k)\id_p)^{-2}) \geq n_k - \Tr(\bSigma(\bSigma + \sqrt{p/n_k}(\lambda / \zeta_k)\id_p)^-1) = n_k \zeta_k > n_k c$.
Thus, by Eq. (<ref>),

\begin{equation*}
\leq
\frac{\tau_{e_k}^2 + (p/n_k)(\lambda / \zeta_k)^2 \| \btheta_k \|_2^2 \| \bSigma(\bSigma + \sqrt{p/n_k}(\lambda / \zeta_k)\id_p)^{-2}\|_{\mathrm{op}}}{\zeta_k} \leq C.
\end{equation*}

In the second case, $\lambda \geq c > 0$.
Then we use the second line of Eq. (<ref>) to bound
$\df_k(\tau^2,\zeta_k) \leq Cp\zeta_k$.
The second line of Eq. (<ref>) implies $\zeta_k \geq 1 - C(p/n_k)\zeta_k \geq 1 - C\zeta_k$,
whence $\zeta_k \geq 1/(1+C)$.
Using, as above that $n_k - \Tr(\bSigma^2(\bSigma + \sqrt{p/n_k}(\lambda / \zeta_k)\id_p)^{-2}) \geq n_k c$, we conclude that $n_k \tau_{g_k}^2 \leq C$.

For the Lasso and the $\alpha$-smoothed Lasso,
an upper bound on $\tau_{g_k}$ and lower bound on $\zeta_k$ holds by Theorem 2 and Lemma A.3 of [15]. See Section <ref> for the change of normalization and notation connecting the current paper to [15].

We now derive bounds on $\tau_{h_k}$.
For least squares, $\tau_{h_k}^2 = \sR_k(\tau_{g_k}^2,\zeta_k) = p \tau_{g_k}^2$, whence $cp/n_k \leq \tau_{h_k}^2 \leq Cp/n_k$ by the bounds on $\tau_{g_k}$.
For ridge regression,
we upper bound

\begin{align*}
\tau_{h_k}^2
\frac{p\lambda^2}{n_k \zeta_k^2} \btheta_k \bSigma^{1/2}\Big(\bSigma + \sqrt{\frac{p}{n_k}} \, \frac{\lambda}{\zeta_k} \id_p\Big)^{-2}\bSigma^{1/2} \btheta_k
\tau_{g_k}^2 \Tr\Big(\bSigma^2\Big(\bSigma + \sqrt{\frac{p}{n_k}}\,\frac{\lambda}{\zeta_k}\id_p\Big)^{-2}\Big)
\\
C\frac{p}{n_k} \frac{\| \btheta_k\|_2^2}{\sigma_{\min}(\bSigma + \sqrt{p/n_k}(\lambda / \zeta_k) \id_p)} + \tau_{g_k}^2 p \leq C p /n_k.
\end{align*}

For the lower bound, we use that $\Tr(\bSigma^2(\bSigma + \sqrt{p/n_k} (\lambda / \zeta_k)\id_p)^{-2}) \geq p\sigma_{\min}(\bSigma)^2(\sigma_{\min}(\bSigma)^2 + \sqrt{p/n_k}(\lambda /\zeta_k)) \geq cp$,
whence, using that $\tau_{g_k}^2 \geq c/n_k$, we have $\tau_{h_k}^2 \geq cp/n_k$.
Eq. (<ref>) gives
$\tau_{h_k}^2 \leq (p/n_k)(\lambda/\zeta_k)^2 \| \btheta_k \|_2^2 / \sigma_{\min}(\bSigma + \sqrt{p/n_k}(\lambda / \zeta_k)\id_p) + \tau_{g_k}^2 p \leq Cp/n_k$.
For the $\alpha$-smoothed Lasso,
we use that by Lemma <ref>,
we have $\tau_{h_k}^2 = \E[\| \hat \btheta_k^f - \btheta_k \|_{\bSigma}^2 ] \leq 2\E[\| \bg_k^f \|_2^2] + (C/\zeta_k)^2 p/n_k \leq Cp/n_k$ by the upper bound on $\tau_{g_k}$.
For the lower bound, we apply the second display in Lemma <ref>, and used that $\zeta_k \leq  1$ and $\sqrt{n_k}\,\tau_{g_k} \zeta_k \geq c$, which gives $\tau_{h_k}^2 = \E[\| \hat \btheta_k^f - \btheta_k\|_{\bSigma}^2] \geq cp/n_k$.

Recalling that $\tau_{\hat e_k^\de}^2 = \tau_{e_k}^2 + \tau_{h_k}^2$ (see Eq. (<ref>)),
the above bounds imply $c < \tau_{\hat e_k^\de}^2 < C$.

Next we upper bound $|\rho_e|$. Using the change of variables given in Section <ref>,
we have $\rho_e^2 = \kappa^2\beta^2/(\kappa^2 \beta^2 + \sigma^2) = 1 - \sigma^2/(\kappa^2 \beta^2 + \sigma^2)  \leq 1 - c$ using that $C > \sigma^2 > c$ and $\kappa^2< C$, $\beta^2 < C$.
We now upper bound $|\rho_g|$ and $|\rho_{\hat e^\de}|$.
By Eq. (<ref>) and recalling the bounds on its right-hand side given by Eq. (<ref>),
we have $|\rho_g| \leq 1 - c$.
Further, by Eq. (<ref>) and using $|\rho_h| \leq 1 $, we have $|\rho_{\hat e^{\de}}| \leq (\tau_{e_1}\tau_{e_2} + \tau_{h_1}\tau_{h_2})/(\tau_{\hat e_1^\de}\tau_{\hat e_2^\de}) - \tau_{e_1}\tau_{e_2}(1-\rho_e)/(\tau_{\hat e_1^\de}\tau_{\hat e_2^\de})$.
By Cauchy-Schwartz we have $(\tau_{e_1}\tau_{e_2} + \tau_{h_1}\tau_{h_2})/(\tau_{\hat e_1^\de}\tau_{\hat e_2^\de}) \leq 1$.
Further, because $\tau_{h_k}^2 \leq Cp/n_k \leq C$, we have by Eq. (<ref>) that $\tau_{\hat e_k^{\de}}^2 \leq C$, whence $\tau_{e_1}\tau_{e_2}(1-\rho_e)/(\tau_{\hat e_1^\de}\tau_{\hat e_2^\de}) \geq c$, where we have used Lemma <ref>.
We conclude that $|\rho_{\hat e^\de}| \leq 1 - c$.

We now upper bound $|\rho_h|$.
We may write
$\bg_2^f = (\tau_{g_2} \rho_g / \tau_{g_1}) \bg_1^f + \tau_{g_2}\rho_g^\perp \bxi_g$ for $\bxi_g \sim \normal(0,\id_p)$ independent of everything else.
Define $\tilde \bg_2^f = (\tau_{g_2} \rho_g / \tau_{g_1}) \bg_1^f + \tau_{g_2}\rho_g^\perp \tilde \bxi_g$, where $\tilde \bxi_g \sim \normal(0,\id_p)$ independent of everything else.
Define $\tilde {\hat \btheta}_2^f$ as we defined $\hat \btheta_2^f$ except with $\tilde \bg_2^f$.
We first claim that

\begin{equation}\label{eq:corr-diff}
\E[\| \hat \btheta_2^f - \tilde {\hat \btheta}_2^f\|_{\bSigma}^2]
\geq
c p / n_k.
\end{equation}

For ridge-regression (possibly with $\lambda = 0$ for least-squares),
observe that $\|\hat \btheta_2^f - \tilde {\hat \btheta}_2^f\|_{\bSigma} = \|\bSigma(\bSigma + \sqrt{p/n_k}\,(\lambda / \zeta_k)\id_p)^{-1}(\bg_k^f - \tilde \bg_k^f)\|_2 \geq c \| \bg_k^f - \tilde \bg_k^f\|_2$,
where we have used that the singular values of $\bSigma$ are bounded below and $\sqrt{p/n_k}(\lambda / \zeta_k) \leq C$.
Then $\E[\| \hat \btheta_2^f - \tilde {\hat \btheta}_2^f\|_{\bSigma}^2] \geq c \E[\|\bg_2^f - \tilde \bg_2^f\|_2^2] = \E[\|\tau_{g_2}\rho_g^\perp(\bxi_g - \tilde \bxi_g)\|_2^2] = c \tau_{g_2}^2(1-\rho_g^2) \E[\| \bxi_g - \tilde \bxi_g\|_2^2] \geq cp/n_k$, where we have used that $|\rho_g| \leq 1-c$.

For the $\alpha$-smoothed Lasso,
observe that for any $j \in [p]$,

\begin{equation}
\P\big([\bSigma^{1/2}\bg_2^f]_j > 2 \lambda / (\sqrt{n_2}\,\zeta_2) \; \text{and} \; [\bSigma^{1/2}\tilde \bg_2^f]_j < - 2 \lambda / (\sqrt{n_2}\,\zeta_2)\big) > c.
\end{equation}

Indeed, this event is implied by the simultaneous occurrence of $[\bSigma^{1/2}\bg_2^f]_j \in [2 \lambda / (\sqrt{n_2}\,\zeta_2),3 \lambda / (\sqrt{n_2}\,\zeta_2)]$ and $\tau_{g_2}\rho_g^\perp [\bSigma^{1/2}\tilde \bxi_g]_j < - 5 \lambda / (\sqrt{n_2}\,\zeta_2)$.
Because $\bSigma$ has bounded singular values and $\rho_g^\perp > c$,
the variance of both $[\bSigma^{1/2}\bg_2^f]_j$ and $\tau_{g_2}\rho_g^\perp [\bSigma^{1/2}\tilde \bxi_g]_j$ are bounded below and above by $c/n_k$ and $C/n_k$.
Because further $\lambda$ and $\zeta_2$ are bounded above and below by a constant,
we conclude that each of these events has probability lower bounded by $c > 0$.
Because $[\bSigma^{1/2}\bg_2^f]_j $ and $\tau_{g_2}\rho_g^\perp [\bSigma^{1/2}\tilde \bxi_g]_j$ are independent, we conclude the previous display.
Using Eq. (<ref>) and the fact that $[\partial \Omega_k(\hat \btheta_k^f)]_j \in [-\lambda/\sqrt{n_k},\lambda /\sqrt{n_k}]$ for the $\alpha$-smoothed Lasso,
we conclude that on the event in the previous display, $\big| [\bSigma(\hat \btheta_2^f - \btheta_2)]_j - [\bSigma(\tilde{\hat \btheta}_2^f - \btheta_2)]_j\big| > 2\lambda/(\sqrt{n_2}\zeta_2)$.
Summing over $j$ and using the the singular values of $\bSigma$ are bounded below,
we conclude that

\begin{equation}
\E[\| \hat \btheta_2^f - \tilde {\hat \btheta}_2^f\|_{\bSigma}^2] \geq c\E[\| \bSigma(\hat \btheta_2 - \tilde{\hat \btheta}_2^f) \|_2^2] \geq cp\frac{4\lambda^2}{n_2\zeta_2^2} \geq cp/n_2.
\end{equation}

Thus, we have established Eq. (<ref>) in all cases (i.e., least-squares, ridge regression, and the $\alpha$-smoothed Lasso).

Now we use the triangle inequality to get $\| \hat \btheta_2^f - \tilde {\hat \btheta}_2^f\|_{\bSigma}^2 \leq 2\| (\hat \btheta_2^f - \btheta_2) - (\tau_{h_2}\rho_h/\tau_{h_1}) (\hat \btheta_1^f - \btheta_1) \|_{\bSigma}^2 + 2\| (\tilde{\hat \btheta}_2^f - \btheta_2) - (\tau_{h_2}\rho_h/\tau_{h_1}) (\hat \btheta_1^f - \btheta_1) \|_{\bSigma}^2$, and that $\E[\| (\hat \btheta_2^f - \btheta_2) - (\tau_{h_2}\rho_h/\tau_{h_1}) (\hat \btheta_1^f - \btheta_1) \|_{\bSigma}^2] = \E[\| (\tilde{\hat \btheta}_2^f - \btheta_2) - (\tau_{h_2}\rho_h/\tau_{h_1}) (\hat \btheta_1^f - \btheta_1) \|_{\bSigma}^2]$, to get

\begin{equation}
\begin{aligned}
\frac{cp}{n_2} \leq \E[\| \hat \btheta_2^f - \tilde {\hat \btheta}_2^f\|_{\bSigma}^2]
(\hat \btheta_2^f - \btheta_2) - (\tau_{h_2}\rho_h/\tau_{h_1}) (\hat \btheta_1^f - \btheta_1)
\big\|_{\bSigma}^2\big]
= \tau_{h_2}^2(1 - \rho_h^2) \leq \frac{Cp}{n_2}(1 - \rho_h^2).
\end{aligned}
\end{equation}

We conclude $|\rho_h| \leq 1-c$, as claimed.

Finally, by Eq. (<ref>) and because proximal operators are 1-Lipschitz,
$\bSigma^{1/2} \eta_k(\by_k^f;\zeta_k)$ is $1$-Lipschitz in $\by_k^f$.
Thus, $\div \bSigma^{1/2} \eta_k(\by_k^f;\zeta_k) \leq p$ almost surely, so by Eq. (<ref>), $\df_k \leq p$.

We prove Lemmas <ref> and <ref> simultaneously, because both proofs involve coupling the fixed design models corresponding to two different regression estimators.

We compare two settings: one in which the second regression uses the Lasso with regularization parameter $\lambda$, and one in which the second regression uses the $\alpha$-smoothed Lasso with regularization parameter $\lambda$.
Across the the two settings, all other model parameters and the estimators in the first regression are the same.
Each setting gives rise to a set of solutions to the fixed-point equations (<ref>) and corresponding fixed design model (<ref>).
We will couple the fixed design models from the two settings so that the estimates from the second regression are close.

We introduce some notation.
The function $\sR$ appearing in the fixed point equations (<ref>) will be denoted $\sR^{(0)}$ and $\sR^{(\alpha)}$ in the model in which the Lasso and $\alpha$-smoothed Lasso is used in the second regression, respectively.
Likewise we define $\df_k^{(0)}$ and $\df_k^{(\alpha)}$.
The solutions to the fixed point equations, in the notation of Section <ref>,
will be denoted $\tau_{g_1,0},\tau_{g_2,0},\rho_{g,0},\zeta_{1,0},\zeta_{2,0}$ for the Lasso and  $\tau_{g_1,\alpha},\tau_{g_2,\alpha},\rho_{g,\alpha},\zeta_{1,\alpha},\zeta_{2,\alpha}$ for the $\alpha$-smoothed Lasso.
We will denote by $\tau_{g_1},\tau_{g_2},\rho_{g},\zeta_{1},\zeta_{2}$ (i.e., with the second subscript omitted) generic arguments to the functions $\sR^{(0)},\df_k^{(0)},\sR^{(\alpha)},\df_k^{(\alpha)}$ which do not necessarily satisfy the fixed point equations (<ref>) (or equivalently, (<ref>) and (<ref>)).
Note that because $\tau_{g_1,0},\zeta_{1,0}$ and $\tau_{g_1,\alpha},\zeta_{1,\alpha}$ are the solutions to the same equations (<ref>) for $k = 1$, we have $\tau_{g_1,0} = \tau_{g_1,\alpha}$ and $\zeta_{1,0} = \zeta_{1,\alpha}$.
Denote the noise and estimates from the fixed design model in which the Lasso is used in the second regression by $\bg_{1,0}^f,\bg_{2,0}^f,\hat \btheta_{1,0}^f,\hat \btheta_{2,0}^f$,
and denote by $\bg_{1,\alpha}^f,\bg_{2,\alpha}^f,\hat \btheta_{1,\alpha}^f,\hat \btheta_{2,\alpha}^f$ the corresponding objects for the fixed design model in which the $\alpha$-smoothed Lasso is used.

We now construct a coupling between the two fixed design models.
First, define $\bg_{2,*}^f = (\tau_{g_2,\alpha}/\tau_{g_2,0})\bg_{2,0}^f$ and

\begin{equation}
\hat \btheta_{2,*}^f
\argmin_{\bpi \in \reals^p}
\Big\{
\frac12\| \bg_{2,*}^f - \bSigma^{1/2} (\bpi - \btheta_2)\|_2^2 + \frac1{\zeta_{2,\alpha}} \frac{\lambda}{\sqrt{n_2}} \sM_{\alpha/\sqrt{n_2}}(\bpi)
\Big\}.
\end{equation}

Note that $(\bg_{2,*}^f, \hat \btheta_{2,*}^f) \stackrel{\mathrm{d}}= (\bg_{2,\alpha}^f,\hat \btheta_{2,\alpha}^f)$.
The quantity $\E[\| \hat \btheta_{2,*}^f - \hat \btheta_{2,0}^f \|_{\bSigma}^2]$ corresponds to the quantity $\| \bv_\alpha^* - \bv_0^* \|_{L_2}^2/n$ in the proof Lemma A.5 of [15].
There it is shown that this quantity is bounded by $C \alpha$ for $\alpha < c'$.
Moreover, Lemma A.5 of [15] states that $|\sqrt{n_2}\,\tau_{g_2,\alpha} - \sqrt{n_2}\,\tau_{g_2,0}| < C \sqrt{\alpha}$ for $\alpha < c'$,
whence, using Lemma <ref>, we conclude that $|\tau_{g_2,\alpha}/\tau_{g_2,0} - 1| < C \sqrt{\alpha}$.
We conclude that $\E[\| \bg_{2,*}^f - \bg_{2,0}^f \|_2^2] < C \alpha p / n_2 < C \alpha$.
In particular,
for any $1$-Lipschitz function $\phi_\theta:(\reals^p)^2 \rightarrow \reals$,
we have

\begin{equation}
\big| \E[\phi_\theta(\hat \btheta_{2,\alpha}^f,\hat \btheta_{2,\alpha}^{f,\de})] - \E[\phi_\theta(\hat \btheta_{2,0}^f,\hat \btheta_{2,0}^{f,\de})] \big| < C\sqrt{\alpha}.
\end{equation}

Lemma <ref> follows by applying this bound to $\E[\phi_\theta(\hat \btheta_{2,\alpha'}^f,\hat \btheta_{2,\alpha'}^{f,\de})]$ for $\alpha' \leq \alpha$ and using the triangle inequality and $C \sqrt{\alpha'} + C \sqrt{\alpha} \leq C \sqrt{\alpha}$.

Next observe that

\begin{equation}
\begin{aligned}
\sR_{12}^{(\alpha)}( \tau_{g_1,0},\tau_{g_2,\alpha},\rho_{g,0},\zeta_{1,0},\zeta_{2,\alpha} )
\sR_{12}^{(0)}( \tau_{g_1,0},\tau_{g_2,0},\rho_{g,0},\zeta_{1,0},\zeta_{2,0} )
\big|
\\
\Big|
\E[\< \hat \btheta_{1,0}^f - \btheta_1 , \hat \btheta_{2,*}^f - \btheta_2 \>_{\bSigma} ]
\E[\< \hat \btheta_{1,0}^f - \btheta_1 , \hat \btheta_{2,0}^f - \btheta_2 \>_{\bSigma} ]
\Big|
\leq C\tau_{h_1}\sqrt{\alpha} \leq C \sqrt{\frac{\alpha p}{n_1}}.
\end{aligned}
\end{equation}

Recall that $\tau_{g_1,0} = \tau_{g_1,\alpha}$ and $\zeta_{2,0} = \zeta_{2,\alpha}$.
Using the fixed point equation (<ref>),
we conclude that

\begin{equation}
\Big|
\tau_{g_1,\alpha}\tau_{g_2,0}\rho_{g,0}
\frac{n_{12}}{n_1n_2}\big( \tau_{e_1}\tau_{e_2} \rho_e + \sR_{12}^{(\alpha)}(\tau_{g_1,\alpha},\tau_{g_2,\alpha},\rho_{g,0},\zeta_{1,\alpha},\zeta_{2,\alpha})\big)
\Big|
\leq C\frac{n_{12}}{n_1n_2} \sqrt{\frac{ p}{n_1}}\,\sqrt{\alpha}
\leq C \sqrt{\frac{\alpha}{n_1n_2}},
\end{equation}

where in the last inequality we use that $p/n_1 \leq C$ and $n_{12} \leq \sqrt{n_1n_2}$.
Because $\big|\sqrt{n_2}\,\tau_{g_2,\alpha} - \sqrt{n_2}\,\tau_{g_2,0}\big|<C\sqrt{\alpha}$ and $\tau_{g_1,\alpha} \leq C / \sqrt{n_1}$,
we conclude that

\begin{equation}\label{eq:approx-fix-pt}
\Big|
\tau_{g_1,\alpha}\tau_{g_2,\alpha}\rho_{g,0}
\frac{n_{12}}{n_1n_2}\big( \tau_{e_1}\tau_{e_2} \rho_e + \sR_{12}^{(\alpha)}(\tau_{g_1,\alpha},\tau_{g_2,\alpha},\rho_{g,0},\zeta_{1,\alpha},\zeta_{2,\alpha})\big)
\Big|
\leq C \sqrt{\frac{\alpha}{n_1n_2}}.
\end{equation}

Finally, as argued in the proof of Lemma <ref> (see Eq. (<ref>)),
we have

\begin{equation}
\begin{aligned}
\frac{n_{12}}{n_1n_2}
\big|
\partial_{\rho_g}\sR_{12}^{(\alpha)}(\tau_{g_1,\alpha},\tau_{g_2,\alpha},\rho_g,\zeta_{1,\alpha},\zeta_{2,\alpha})\big)
\big|
&\leq \tau_{g_1,\alpha}\tau_{g_2,\alpha} \frac{n_{12}}{n_1n_2} \sqrt{\df_1^{(\alpha)}(\tau_{g_1,\alpha}^2,\zeta_{1,\alpha})\df_2^{(\alpha)}(\tau_{g_2,\alpha}^2,\zeta_{2,\alpha})}
\\
&\leq \tau_{g_1,\alpha}\tau_{g_2,\alpha} (1-c),
\end{aligned}
\end{equation}

where we use that
$\df_k^{(\alpha)}(\tau_{g_k,\alpha}^2,\zeta_{k,\alpha})/n_k \leq 1-c$ by Lemma <ref>.
We see that the derivative of the term in the absolute value in Eq. (<ref>) has partial derivative with respect to $\rho_{g,0}$ no smaller than $c\tau_{g_1,\alpha}\tau_{g_2,\alpha}$.
Because $\tau_{g_1,\alpha} \geq c/\sqrt{n_1}$ and $\tau_{g_2,\alpha} \geq c/\sqrt{n_2}$ by Lemma <ref>,
Eq. (<ref>) and the derivative bound imply $\big|\rho_{g,\alpha} - \rho_{g,0}\big| < C \sqrt{\alpha}$.
Because $\rho_{g,\alpha}^\perp = \sqrt{1-\rho_{g,\alpha}^2}$ and $\rho_{g,0}^\perp = \sqrt{1 - \rho_{g,0}^2}$,
and $|\rho_{g,\alpha}|,|\rho_{g,0}| < 1 -c$ by Lemma <ref>,
we conclude also that $|\rho_{g,\alpha}^\perp - \rho_{g,0}^\perp| < C\sqrt{\alpha}$.

We now construct the coupling between $(\bg_{1,0}^f,\bg_{2,0}^f)$ and $(\bg_{1,\alpha}^f,\bg_{2,\alpha}^f)$.
Note that we may represent $\bg_{2,0}^f = (\tau_{g_2,0}\rho_{g,0}/\tau_{g_1,0}) \bg_{1,0}^f + \tau_{g_2,0}\rho_{g,0}^\perp \bxi_g$, where $\bxi_g\sim\normal(0,\id_p)$ independent of everything else.
Using this representation, we construct the coupling $\bg_{1,\alpha}^f = \bg_{1,0}^f$ and $\bg_{2,\alpha}^f = (\tau_{g_2,\alpha}\rho_{g,\alpha}/\tau_{g_1,\alpha}) \bg_{1,0}^f + \tau_{g_2,\alpha}\rho_{g,\alpha}^\perp \bxi_g$.
Note that

\begin{equation}
\| \bg_{2,\alpha}^f - \bg_{2,*}^f \|_2
\leq
\| (\tau_{g_2,\alpha}/\tau_{g_1,\alpha}) \bg_{1,0}^f \|_2 |\rho_{g,\alpha} - \rho_{g,0}|
\| \tau_{g_2,\alpha} \bxi_g \|_2 |\rho_{g,\alpha}^\perp - \rho_{g,0}^\perp |.
\end{equation}

Using the bounds on $\big|\rho_{g,\alpha} - \rho_{g,0}\big|$ and $|\rho_{g,\alpha}^\perp - \rho_{g,0}^\perp|$ above and the bounds on $\tau_{g_2,\alpha}$ and $\tau_{g_2,0}$ from Lemma <ref>,
we conclude that $\E[\| \bg_{2,\alpha}^f - \bg_{2,*}^f \|_2^2] \leq C \alpha$ for $\alpha < c'$.
Because proximal operators are $1$-Lipschitz,
we then also conclude that $\| \hat \btheta_{2,\alpha}^f - \hat \btheta_{2,*}^f \|_{\bSigma}^2] \leq C\alpha$.
We have already shown that $\E[\| \hat \btheta_{2,*}^f - \hat \btheta_{2,0}^f \|_{\bSigma}^2] < C\alpha$ and $\E[\| \bg_{2,*}^f - \bg_{2,0}^f \|_2^2] < C \alpha$.
we have that $\E[\| \hat \btheta_{2,\alpha}^f - \hat \btheta_{2,0}^f \|_{\bSigma}^2] < C\alpha$ and $\E[\| \bg_{2,\alpha}^f - \bg_{2,0}^f \|_2^2] < C \alpha$ under this coupling.

Using this coupling, we can derive the conclusion of the lemma.
As argued in the proof of Lemma <ref> (see Section <ref>),
$\phi_{\theta|1}^{(\alpha)}$ and $\phi_{\theta|1}^{(0)}$ can be viewed as a function of $\hat \bg_1$.
Writing it as a function of this argument and computing expectations under this coupling, we get

\begin{equation}
\begin{aligned}
\E\big[
\big|
\phi_{\theta|1}^{(\alpha)}(\bg_{1,0}^f)
\phi_{\theta|1}^{(0)}(\bg_{1,0}^f)
\big|
\big]
\E\big[
\big|
\E\big[\phi_\theta(\{ \hat \btheta_{k,\alpha}^f\},\{\hat \btheta_{k,\alpha}^{f,\de}\})\bigm| \bg_{1,0}^f\big]
\E\big[\phi_\theta(\{ \hat \btheta_{k,0}^f\},\{\hat \btheta_{k,0}^{f,\de}\})\bigm| \bg_{1,0}^f\big]
\big|
\big]
\\
\E\big[
\big|
\phi_\theta(\{ \hat \btheta_{k,\alpha}^f\},\{\hat \btheta_{k,\alpha}^{f,\de}\})
\phi_\theta(\{ \hat \btheta_{k,0}^f\},\{\hat \btheta_{k,0}^{f,\de}\})
\big|
\big]
\leq C M_2\sqrt{\alpha}.
\end{aligned}
\end{equation}

As argued in the proof of Lemma <ref> (see Section <ref>),
$\phi_{\theta|1}^{(\alpha)}$ and $\phi_{\theta|1}^{(0)}$ are $C(M_1 + M_2 \sqrt{n_1/n_2})$-Lipschitz in $\hat \bg_1$,
whence $\big|
\phi_{\theta|1}^{(\alpha)}(\cdot)
\phi_{\theta|1}^{(0)}(\cdot)
\big|$ is $C(M_1 + M_2 \sqrt{n_1/n_2})$-Lipschitz.
By Corollary <ref>,
we have for $\epsilon < c'$ with probability at least $1 - \sC(\epsilon) e^{-\sc(\epsilon)p}$

\begin{equation}
\Big|
\big|
\phi_{\theta|1}^{(\alpha)}(\hat \bg_1)
\phi_{\theta|1}^{(0)}(\hat \bg_1)
\big|
\E\big[
\big|
\phi_{\theta|1}^{(\alpha)}(\hat \bg_1)
\phi_{\theta|1}^{(0)}(\hat \bg_1)
\big|
\big]
\Big|
< C (M_1 \sqrt{p/n_1} + M_2 \sqrt{p/n_2})\,\epsilon \leq C(M_1 \sqrt{p/n_1} + M_2)\epsilon.
\end{equation}

Taking $\epsilon = \sqrt{\alpha}$ gives Lemma <ref> for $\alpha' = 0$.
Lemma <ref> for $\alpha' \leq \alpha$ follows by applying the previous display with $\alpha'$ in place of $\alpha$ and using the triangle inequality and $\sqrt{\alpha'} +\sqrt{\alpha} \leq 2 \sqrt{\alpha}$.

§.§ Gram-Schmidt approximation bound

Consider $\bA \in \reals^{n \times k}$ such that $\big\|\bA^\top \bA - \sT\big\|_{\sF} < \epsilon$.
Denoting the columns of $\bA$ by $\ba_1,\ldots,\ba_k$, assume also that  $\sT$ has singular values bounded below and above by $c$ and $C$ respectively.
Consider applying the Gram-Schmidt procedure the $\ba_1,\ldots,\ba_k$ in that order, to produce $\ba_1^{\mathrm{gs}},\ldots,\ba_k^{\mathrm{gs}}$ which satisfy $\bA^{\mathrm{gs}\top}\bA^{\mathrm{gs}} = \sT$.
Then there exists constants $C',c' >0 $ depending only on $c,C,k$ such that for $\epsilon < c'$

\begin{equation}
\max_{j \in [k]}\Big\{  \|\ba_j^{\mathrm{gs}} - \ba_j\|_2 \Big\} < C'\epsilon.
\end{equation}

We denote the matrix in $\reals^{n \times j}$ formed by the first $j$ columns of $\bA$ by $\bA(j)$, and the submatrix of $\sT$ formed by the first $j$ rows and columns by $\sT(j)$.
We prove the result inductively.
Throughout the proof, $C',c'$ may change at each appearance, but only depend on $c,C,k$.

For the base case, observe $\ba_1^{\mathrm{gs}} = \sqrt{\sT_{11}} \ba_1 / \| \ba_1 \|_2$. Because $\sT_{11} > c$, for $\epsilon < c'$ we have $\big| \sqrt{\sT_{11}}/\|\ba_1\|_2 - 1\big| < C'\epsilon$ and $\| \ba_1 \|_2 \leq C'$.
Thus, $\| \ba_1^{\mathrm{gs}} - \ba_1 \|_2 < C'\epsilon$.

Now assume the inductive hypothesis, that $\max_{j \leq i} \| \ba_j^{\mathrm{gs}} - \ba_j\|_2 < C'\epsilon$.
We have

\begin{equation}
\ba^{\mathrm{gs}}_{i+1}
\bA^{\mathrm{gs}}(i) \sT(i)^{-1} \sT_{1:i,i+1}
\sqrt{T_{i+1,i+1}}
\frac{\proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1}}{\|\proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1}\|_2},
\end{equation}

\begin{equation}
\ba_{i+1}
\proj_{\bA^{\mathrm{gs}}(i)} \ba_{i+1}
\proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1}
\bA^{\mathrm{gs}}(i) \sT(i)^{-1} \bA^{\mathrm{gs}}(i)^\top \ba_{i+1}
\proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1}.
\end{equation}

\begin{equation}
\big\| \bA^{\mathrm{gs}}(i)^\top \ba_{i+1} - \sT_{1:i,i+1}\big\|_2
\leq
\big\| \bA^{\mathrm{gs}}(i)^\top \ba_{i+1} - \bA(i)^\top \ba_{i+1}\big\|_2
\big\| \bA(i)^\top \ba_{i+1} - \sT_{1:i,i+1}\big\|_2
< C'\epsilon,
\end{equation}

whence because $\|\bA^{\mathrm{gs}}(i)\sT(i)^{-1}\|_{\mathsf{op}} < C'$,
we conclude

\begin{equation}
\big\| \bA^{\mathrm{gs}}(i) \sT(i)^{-1} \sT_{1:i,i+1} - \proj_{\bA^{\mathrm{gs}}(i)} \ba_{i+1}\big\|_2
< C'\epsilon.
\end{equation}

This implies that

\begin{equation}\label{eq:gs-par-bound}
\Big|
\| \proj_{\bA^{\mathrm{gs}}(i)} \ba_{i+1}^{\mathrm{gs}} \big\|_2 - \| \proj_{\bA^{\mathrm{gs}}(i)} \ba_{i+1}\big\|_2
\Big|
\Big|
\sT_{i+1,1:i} \sT(i)^{-1} \sT_{1:i,i+1} - \| \proj_{\bA^{\mathrm{gs}}(i)} \ba_{i+1}\big\|_2
\Big| < C' \epsilon.
\end{equation}

Using the previous display and that $\big| \|\ba_{i+1}\|_2^2 - T_{i+1,i+1} \big| < \epsilon$,
we have
$\|\proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1}\|_2
= \sqrt{\|\ba_{i+1}\|_2^2 - \| \proj_{\bA^{\mathrm{gs}}(i)} \ba_{i+1}^2 \| }$
differs from $\sqrt{\sT_{i+1,i+1} - \sT_{i+1,1:i} \sT(i)^{-1} \sT_{1:i,i+1}}$ by at most $C'\epsilon$,
where we use that $\sT_{i+1,i+1} - \sT_{i+1,1:i} \sT(i)^{-1} \sT_{1:i,i+1} > c$ because $\sT$ has singular values bounded below by $c$.

\begin{equation}
\Big|
\| \proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1}^\mathrm{gs} \|_2
\| \proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1} \|_2
\Big|
\Big|
\sqrt{\sT_{i+1,i+1} - \sT_{i+1,1:i} \sT(i)^{-1} \sT_{1:i,i+1}}
\| \proj_{\bA^{\mathrm{gs}}(i)}^\perp \ba_{i+1} \|_2
\Big| < C'\epsilon.
\end{equation}

Combined with Eq. (<ref>), we conclude $\| \ba_{i+1}^{\mathrm{gs}} - \ba_{i+1} \|_2 < C'\epsilon$, and the inductive step is complete.

§.§ Comparison of cosines

We prove the following lemma.

For a positive definite matrix $\bK$, define $\cos_{\bK}(\ba,\bb) = \ba^\top \bK \bb / (\| \ba \|_{\bK} \|\bb\|_{\bK})$,
and define $\sin_{\bK}^2(\ba,\bb) = 1 - \cos_{\bK}^2(\ba,\bb)$.
Let $\sigma_{\max}(\bK),\sigma_{\min}(\bK)$, and $\kappa_{\mathrm{cond}}(\bK)$ be the maximal and minimal singular values and the condition number of $\bK$ respectively.
Then for any $\ba,\bb$

\begin{equation}
1 - \cos_{\bK}^2(\ba,\bb)
\geq
\frac{1 - \cos_{\id}^2(\ba,\bb)}{4\kappa_{\mathrm{cond}}(\bK)}.
\end{equation}

Because the cosine is invariant to scaling $\ba$ and $\bb$, we may without loss of generality assume $\| \ba \|_2 = \| \bb \|_2 = 1$.

\begin{equation}
\begin{aligned}
\cos_{\bK}^2(\ba,\bb)
\cos_{\id}^2\Big(
\frac{\bK^{1/2}\ba}{\|\bK^{1/2}\ba\|_2},
\frac{\bK^{1/2}\bb}{\|\bK^{1/2}\bb\|_2}
\Big)
= 1 - \frac14\Big\| \frac{\bK^{1/2}\ba}{\|\bK^{1/2}\ba\|_2} - \frac{\bK^{1/2}\bb}{\|\bK^{1/2}\bb\|_2}\Big\|_2^2
\\
1 - \frac{\sigma_{\min}(\bK)}4\Big\| \frac{\ba}{\|\bK^{1/2}\ba\|_2} - \frac{\bb}{\|\bK^{1/2}\bb\|_2}\Big\|_2^2
\leq
1 - \frac{\sigma_{\min}(\bK)}4\Big\| \frac{\ba}{\|\bK^{1/2}\ba\|_2} - \frac{\proj_{\bb}\ba }{\|\bK^{1/2}\bb\|_2}\Big\|_2^2
\\
&= 1 - \frac{\sigma_{\min}(\bK) \sin_{\id}(\ba,\bb)^2 }{4\| \bK^{1/2}\ba\|_2^2}
\leq 1 - \frac{\sigma_{\min}(\bK) \sin_{\id}(\ba,\bb)^2 }{4\sigma_{\max}(\bK)}
= 1 - \frac{1 - \cos_{\id}^2(\ba,\bb)}{4\kappa_{\mathrm{cond}}(\bK)}.
\end{aligned}
\end{equation}

The proof is complete.

§.§ Change of normalization from [15]

Our results rely on the lasso characterization provided by [15], which uses a different normalization: in our case the covariates are $\bx_i \stackrel{\mathrm{iid}}\sim \normal(0,\bSigma)$, whereas in [15] the covariates are $\bx_i \stackrel{\mathrm{iid}}\sim \normal(0,\bSigma/n_{\cdot})$.
We here provide the change of variables to tranfer results between the two papers.

We will denote quantities which occur in their paper with a prime.
They consider a design matrix $\bX'$, observation $\by'$, true parameter $\btheta'$, noise standard deviation $\sigma'$, Lasso regularization $\lambda'$, and fixed-point parameter $\tau',\zeta'$.
These are related to the quantites in the present paper by

\begin{equation}\label{eq:COV-lasso}
\begin{aligned}
\bX' &= \frac1{\sqrt{n_{\sx}}} \bX_{\cI_k},\qquad& \by' &= \by_k,\qquad& \btheta' &= \sqrt{n_k}\, \btheta_k,\qquad& \bSigma' &= \bSigma, \qquad & \alpha' &= \alpha_k,\\
\sigma'^2 &= \tau_{e_k}^2,\qquad& \lambda'&= \lambda_k,\qquad &\tau' &= \sqrt{n_k}\,\tau_{g_k},\qquad & \zeta'& = \zeta_k.
\end{aligned}
\end{equation}

With some algebra, one can check that the fixed point system equations (8) from [15] is equivalent to the outcome regression equations and precision regression equations in Eq. (<ref>) in the present paper.

§.§ Reference on second moment identities

We write explicitly the expectations appearing in Eq. (<ref>). They are

\begin{equation}\label{eq:T-marg-ref}
\begin{gathered}
\sT_{\sN,k}
\E\Big[
\bT\Big(\frac{\be_{k,\cI_k}}{\sqrt{n_k}},\frac{\bxi_{h,\cI_k}}{\sqrt{n_k}}\Big)
\Bigm|
\be_k
\Big]
\begin{pmatrix}
\tau_{e_k}^{*2} & 0 \\[2pt]
0 & 1
\end{pmatrix},
\\
\sT_{\sP,k}
\E\Big[\bT\Big(\sqrt{\frac{n_k}{p}}\,\bv_k^f,\frac1{\sqrt{p}}\bxi_g\Big)\Big]
\begin{pmatrix}
(n_k/p)\tau_{h_k}^2 & \sqrt{n_k}\,\tau_{g_k}\df_k/p \\[2pt]
\sqrt{n_k}\,\tau_{g_k}\df_k/p & 1
\end{pmatrix},
\\
\text{and $\sT_{\sP,k}$ has entries bounded above by $C$,}
\end{gathered}
\end{equation}

which we now check.
Because $\bxi_{h,\cI_k} \sim \normal(0,\id_{n_k})$ is independent of $\be_{k,\cI_k}$ and $\tau_{e_k}^* = \| \be_{k,\cI_k} \|_2/\sqrt{n_k}$ by definition,
the expression of $\sT_{\sN,k}$ is easily checked.
By the definition of $\bv_k^f$ (see Eq. (<ref>)) and the identity Eq. (<ref>), we have $\E[\|\bv_k^f\|_2^2] = \tau_{h_k}^2$. Because $\bxi_g \sim \normal(0,\id_p)$, we have $\E[\| \bxi_g \|_2^2] / p = 1$.
Thus, the diagonal entries of $\sT_{\sP,k}$ are as stated.
By the definition of $\bv_k^f$ (see Eqs. (<ref>) and (<ref>)) and
because proximal operators are 1-Lipschitz,
we see that $\bv_k^f$ is $1$-Lipschitz in $\bg_k^f$.
Thus, we may apply Gaussian integration by parts,
which gives $\E[\bg_k^{f\top} \bv_k^f] = \tau_{g_k}^2 \df_k $ by the definition of $\df_k$ (see Eq. (<ref>)) and because $\bg_k^f \sim \normal(0,\tau_{g_k}^2 \id_p)$.
Thus, the off-diagonal entries of $\sT_{\sP,k}$ are as stated.
By Lemma <ref>, $(n_k/p)\tau_{h_k}^2 \leq C$, so Eq. (<ref>) follows.

We also rewrite explicitly the expectations appearing in Eq. (<ref>).
They are

\begin{equation}\label{eq:T-cond-ref}
\begin{gathered}
\begin{aligned}
\sT_{\sN,2|1}
\E\Big[
\bT\Big(
\frac{\be_{2,\cI_2}}{\sqrt{n_2}},
\frac{\bu_{1,\cI_2}^f}{\sqrt{n_1}},
\frac{\bh_{1,\cI_2}^f}{\sqrt{n_2}\,\tau_{h_1}},
\frac{\bh_{2,\cI_2}^f - (\tau_{h_2}\rho_h/\tau_{h_1})\bh_{1,\cI_2}^f}{\sqrt{n_2}\,\tau_{h_2}\rho_h^\perp}
\Big)
\Big]
\\
\begin{pmatrix}
\tau_{e_2}^2
& \frac{n_{12}}{\sqrt{n_1n_2}}\tau_{e_1}\tau_{e_2} \rho_e \zeta_1
& 0
& 0  \\[4pt]
\frac{n_{12}}{\sqrt{n_1n_2}}\tau_{e_1}\tau_{e_2} \rho_e \zeta_1
& n_{12} \tau_{g_1}^2 \zeta_1^2
& \frac{n_{12}}{\sqrt{n_1n_2}}\, \tau_{h_1} \zeta_1
& 0 \\[4pt]
& \frac{n_{12}}{\sqrt{n_1n_2}}\, \tau_{h_1} \zeta_1
& 1
& 0 \\[4pt]
& 0
& 0
& 1
\end{pmatrix},
\\[4pt]
\sT_{\sP,2|1}
\E\Big[
\bT\Big(
\sqrt{\frac{n_1}{p}}\,\bv_1^f,
\sqrt{\frac{n_2}{p}}\,\bv_2^f,
\frac{\bg_1^f}{\sqrt{p}\,\tau_{g_1}},
\frac{\bg_2^f - (\tau_{g_2}\rho_g/\tau_{g_1})\bg_1^f}{\sqrt{p}\,\tau_{g_2}\rho_g^\perp}
\Big)
\Big]
\\
\begin{pmatrix}
\frac{n_1}{p} \tau_{h_1}^2
& \frac{\sqrt{n_1n_2}}{p} \tau_{h_1}\tau_{h_2} \rho_h
& \sqrt{n_1}\, \tau_{g_1}\frac{\df_1}{p}
& 0  \\[4pt]
\frac{\sqrt{n_1n_2}}{p} \tau_{h_1}\tau_{h_2} \rho_h
& \frac{n_2}{p} \tau_{h_2}^2
& \sqrt{n_2}\, \tau_{g_2}\rho_g \frac{\df_2}{p}
& \sqrt{n_2}\, \tau_{g_2}\rho_g^\perp \frac{\df_2}{p} \\[4pt]
\sqrt{n_1}\, \tau_{g_1}\frac{\df_1}{p}
& \sqrt{n_2}\, \tau_{g_2}\rho_g \frac{\df_2}{p}
& 1
& 0 \\[4pt]
& \sqrt{n_2}\, \tau_{g_2}\rho_g^\perp \frac{\df_2}{p}
& 0
& 1
\end{pmatrix},
\end{aligned}\\[4pt]
\text{and $\sT_{\sN,2|1}$ and $\sT_{\sP,2|1}$ have entries bounded above by $C$,}
\end{gathered}
\end{equation}

which we now check.
By Eqs. (<ref>) and (<ref>),
we have

\begin{equation}
\bu_{1,\cI_2}^f
\begin{pmatrix}
\zeta_1(\be_{1,\cI_1\cap\cI_2} + \bh_{1,\cI_1 \cap \cI_2}^f)\\[4pt]
\bzero_{n_2 - n_{12}}
\end{pmatrix},
\end{equation}

where with some abuse of notation, we have written those indices in $\cI_1 \cap \cI_2$ first, and those in $\cI_2 \setminus \cI_1$ second.
This gives us that $\E[\| \be_{2,\cI_2}\|_2^2]/n_2 = \tau_{e_2}^2$, $\E[\< \be_{2,\cI_2} , \bu_{1,\cI_2}^f \> ] / \sqrt{n_1n_2} = \zeta_1\E[\< \be_{2,\cI_1 \cap \cI_2} , \be_{1,\cI_1\cap\cI_2} + \bh_{1,\cI_1 \cap \cI_2}^f \> ] / \sqrt{n_1n_2} = (n_{12}/\sqrt{n_1n_2})\tau_{e_1}\tau_{e_2}\rho_e \zeta_1$, and $\E[\| \bu_{1,\cI_2}^f \|_2^2]/n_1 = \zeta_1^2\E[\| \be_{1,\cI_1\cap\cI_2} + \bh_{1,\cI_1 \cap \cI_2}^f \|_2^2]/n_1 = \zeta_1^2 n_{12} (\tau_{e_1}^2 + \tau_{h_1}^2)/n_1 = n_{12}\tau_{g_1}^2\zeta_1^2 $, where we have applied the independence of $\be_2$ and $\bh_1$ and Eq. (<ref>).
Note that $\tau_{e_2}^2 \leq C$ by 4 and $n_{12}\tau_{g_1}^2 \zeta_1^2 \leq n_1 \tau_{g_1}^2 \leq C$ by Lemma <ref>.
Thus, we have computed the top left $2\times2$ block of $\sT_{\sN,2|1}$.

By the definition of $\tau_{h_1},\tau_{h_2},\rho_h$ (see Section <ref>),
we have $\bh_{1,\cI_2}^f/\tau_{h_1}$ and $(\bh_{2,\cI_2}^f - (\tau_{h_2}\rho_h/\tau_{h_1})\bh_{1,\cI_2}^f)/(\tau_{h_2}\rho_h^\perp$ are standard Gaussian and independent, both are independent of $\be_{2,\cI_2}$, and the latter is independent of $\bu_{1,\cI_2}^f$.
This gives the computation for the bottom right $2\times2$ block of $\sT_{\sN,2|1}$.
For the top right $2\times2$ block of $\sT_{\sN,2|1}$, the only non-zero entry comes from $\E[\<\bu_{1,\cI_2}^f,\bh_{1,\cI_2}^f\>]/(\sqrt{n_1n_2}\,\tau_{h_1})$, because the remaining entries are 0 by independence.
We compute $\E[\<\bu_{1,\cI_2}^f,\bh_{1,\cI_2}^f\>]/(\sqrt{n_1n_2}\,\tau_{h_1}) = \zeta_1\E[\<\be_{1,\cI_1 \cap \cI_2} + \bh_{1,\cI_1 \cap \cI_2},\bh_{1,\cI_1 \cap \cI_2}^f\>]/(\sqrt{n_1n_2}\,\tau_{h_1}) = (n_{12}/\sqrt{n_1n_2})\tau_{h_1}\zeta_1$ as claimed.

The top left $2\times 2$ block of $\cT_{\sP,2|1}$ is computed using Eq. (<ref>) and recalling that $\bv_k = \bSigma^{1/2}(\hat \btheta_k - \btheta_k)$ (see Eq. (<ref>)).
By Lemma <ref>, $(n_k/p)\tau_{h_k}^2 \leq C$.
By the definition of $\tau_{g_1},\tau_{g_2},\rho_g$ (see Section <ref>),
we have $\bg_1^f/\tau_{g_1}$ and $(\bg_2^f - (\tau_{g_2}\rho_g/\tau_{g_1})\bg_1^f)/(\tau_{g_2}\rho_g^\perp)$ are standard Gaussian and independent, which gives the bottom right block.
The top right block is computed using Gaussian integration by parts, recalling that because $\bv_k = \bSigma^{1/2}(\eta_k(\btheta_k + \bg_k^f;\zeta_k) - \btheta_k)$ (see Eqs. (<ref>) and (<ref>)), the definition of $\df_k$ in Eq. (<ref>) gives $\E[\< \bg_l^f , \bv_k^f \>] = \Cov(\bg_k^f,\bg_l^f) \df_k$.