online SGD to minimize the population loss $\calL(w)$. Formally, we first choose a mini-batch size $b$ and initial weight matrices $\sth{W_{0,1}, W_{0,2}}$. In each iteration $t$, we draw $b$ independent and identically distributed samples $x_{t,i}, i\in[b]$ from $\calN(0, I_p)$ to form the mini-batch $\calB_t^b$ and update the weight matrices by $W_{t+1,1}^b = W_{t,1}^b - \alpha_t g_{t,1}^b$ and $W_{t+1, 2}^b = W_{t,2}^b - \alpha_t g_{t,2}^b$, where $$\begin{aligned}
        g_{t, 1}^b &= \frac{1}{b} \sum_{i=1}^b \nabla_{W_{t,1}^b} \pth{\frac{1}{2} \norm{W_{t,2}^b W_{t,1}^b x_{t,i} - W_2^* W_1^* x_{t,i}}^2} \notag \\
        &= \frac{1}{b}\sum_{i=1}^b {W_{t,2}^b}^\transp \pth{W_{t,2}^b W_{t,1}^b - W_{2}^* W_{1}^*} x_{t,i} x_{t,i}^\transp, \label{eq:gt1}\\
        g_{t, 2}^b &= \frac{1}{b} \sum_{i=1}^b \nabla_{W_{t,2}^b} \pth{\frac{1}{2} \norm{W_{t,2}^b W_{t,1}^b x_{t,i} - W_2^* W_1^* x_{t,i}}^2} \notag \\
        &= \frac{1}{b}\sum_{i=1}^b \pth{W_{t,2}^b W_{t,1}^b - W_{2}^* W_{1}^*} x_{t,i} x_{t,i}^\transp{W_{t,1}^b}^\transp. \label{eq:gt2}
    \end{aligned}$$

The derivation follows from the formulas in [@IMM2012-03274]. In the following, we use $\calW_t^b = W_{t,2}^b W_{t,1}^b - W_{2}^* W_{1}^* $ to denote the gap between the product of model weights and ground-truth weights.

For ease of developing our proofs, we first introduce the definition of a *multiplicative term* in Definition \[def:mul\]. Intuitively, a multiplicative term is a matrix which equals to the product of its parameter matrices and constant matrices (and their transpose). The degree of a matrix $A$ 