we can choose any convenient basis to work with.

\[rem:weight-dist\] Corollary \[cor:weight-dist\] can be used to get a similar result to Theorem 3 by @liu2019understanding. In this remark, we consider mostly their notations. Their theorem states that under some conditions, for 2-layer neural networks with a RELU activation function, with probability no less than $1 - \delta$ over random initialisation, $$\begin{aligned}
        \norm{{\mbox{${\mbox{$\mathbf{W}$}}$}}(P) - {\mbox{${\mbox{$\mathbf{W}$}}$}}(Q)}_\mathrm{F} &\leq \sqrt{ {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_{P \rightarrow Q}^T H_P^{\infty \space -1} {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_{P \rightarrow Q}}
+ \epsilon, &\\
        \intertext{where, in their work:}
        {\mbox{${\mbox{$\mathbf{y}$}}$}}_{P \rightarrow Q} &= H_{PQ}^{\infty, T} H_P^{\infty \space -1} {\mbox{${\mbox{$\mathbf{y}$}}$}}_P,\\
        {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_{P \rightarrow Q} &= {\mbox{${\mbox{$\mathbf{y}$}}$}}_Q - {\mbox{${\mbox{$\mathbf{y}$}}$}}_{P \rightarrow Q}.
    \end{aligned}$$

Note that $H_P^{\infty}$ is a Gram matrix, which also corresponds to the NTK of the neural network they consider. We see an analogy with our result, where we work directly with the NTK, with no assumptions on the neural network. One important observation is that, to our knowledge, since there are no guarantees for the invertibility of our Gram matrix, we add a ridge regularisation to work with a regularised matrix, which is then invertible. In our setting, by considering $\lambda \rightarrow 0$, and with the additional assumption of invertibility of ${\mbox{${\mbox{$\mathbf{H}$}}$}}_{\tau, 0}$,which is valid in the two-layer 