in iteration $t$ with fixed variables, like initial weights $W_{0,1}, W_{0,2}$ and mini-batch size $b$. We overcome this challenge by the following two lemmas. Lemma \[lemma:GtoW\] provides the relationship between $g_{t,i}^b, i=1,2$ and $W_{t,i}^b, i=1,2$ by taking expectation over the distribution of random samples in $\calB_t^b$. Lemma \[lemma:WtoG\] shows the relationship between $W_{t,i}^b, i=1,2$ and $g_{t-1,i}^b, i=1,2$ using and .

\[lemma:GtoW\] For multiplicative terms $M_i, i \in \fromto{0}{m} $ of parameter matrices $\left\{ g_{t,1}^b, g_{t,2}^b \right\}$ and constant matrices $\sth{W_{t,1}^b, W_{t,2}^b, W_1^*, W_2^*}$ with degree $d_i$, respectively, we denote $M = \prod_{i=1}^m \trace{M_i} M_0$ and $d = \sum_{i=0}^m d_i$. There exists a set of multiplicative terms $\sth{M_{ij}^k, i\in [m_k], j \in \fromto{0}{m_{ki}}, k \in \fromto{0}{q}}$ of parameter matrices $\sth{W_{t,1}^b, W_{t,2}^b}$ and constant matrices $\sth{W_1^*, W_2^*}$ such that $$\begin{aligned}
                \condexp{}{M}{\calF_t^b} = N_0 + N_1 \frac{1}{b} + \cdots + N_d \frac{1}{b^d},
                            \end{aligned}$$ where $N_k = \sum_{i=1}^{m_k} \prod_{j=1}^{m_{ki}} \trace{M_{ij}^k} M_{i0}^k, k\in \fromto{0}{d}  $. Here $m_k, m_{ki}$ are constants independent of $b$, and $\sum_{j=0}^{m_{ki}} \deg\pth{M_{ij}^k} \le 3d + \sum_{i=0}^m \pth{\deg\pth{W_{t,1}^b; M_i} + \deg(W_{t,2}^b; M_i)}$.

\[lemma:WtoG\] For multiplicative term $M_i, i \in \fromto{0}{m}$ of parameter matrices $\sth{W_{t,1}^b, W_{t,2}^b}$ and constant matrices $\sth{W_1^*, W_2^*}$ of degree $d_i$, let $d = 2^{d_0+\cdots +d_m}$. There exists a set 