in a multiplicative term $M$ is the number of appearance of $A$ and $A^\transp$ in $M$. The degree of $M$ is exactly the number of appearances of all weight matrices in $M$.

\[def:mul\] For any set of matrices $\calS$, we denote $\widebar{\calS} = \calS \cup \{M^\transp: M \in \calS\}$. Given a set of parameter matrices $\calX = \{X_1, X_2, \cdots, X_{n_v}\}$ and constant matrices $\calC = \{C_1, C_2, \cdots, C_{n_c}\}$, we say that a matrix $M$ is *a multiplicative term of parameter matrices $\calX$ and constant matrices $\calC$* if it can be written in the form of $$M = M(\calX, \calC) = \prod_{i=1}^k A_i,$$ where $A_i \in \widebar{\calX} \cup \widebar{\calC}$. We write $\deg(X_j; M) = \sum_{i=1}^k \pth{\mathbbm{1}\sth{X_j = A_i} + \mathbbm{1}\sth{X_j = A_i^\transp}}, j \in [n_v]$ as the degree of parameter matrix $X_j$ in $M$, $\deg(C_j; M) = \sum_{i=1}^k \pth{\mathbbm{1}\sth{C_j = A_i} + \mathbbm{1}\sth{C_j = A_i^\transp}}, j \in [n_c]$ as the degree of constant matrix $C_j$ in $M$, and $\deg(M) = \sum_{i=1}^k \mathbbm{1}\sth{A_i \in \widebar{\calX}} = \sum_{j=1}^{n_v} \deg(X_j; M)$ as the total degree of the parameter matrices of $M$.

As pointed out in the Section \[sec:1\], the difficulty of studying the dynamics of SGD is how to connect the quantities 