dynamics of the stochastic differential equation and the initial point.

In this section, we address these questions under two settings: linear regression and a deep linear network. In Section \[subsec:linearRegression\] with a linear regression setting, we provide explicit formulas for calculating any norm of the linear combination of sample-wise gradients. We therefore show that the $\var \pth{\grad_w L_{\calB_t}(w_t)}$ is a decreasing function of the mini-batch size $b$. In Section \[subsec:linearNetwork\] with a deep linear network setting and samples drawn from a normal distribution, we show that any trace of the product of weight matrices and stochastic gradient estimators is a polynomial in $1/b$ with finite degree. We further prove that $\var \pth{\grad_w L_{\calB_t}(w_t)}$ is a decreasing function of the mini-batch size $b > b_0$ for some constant $b_0$.

For a random matrix $M$, we define $\var\pth{M} \triangleq \Expect \norm{\text{vec}(M)}^2 - \norm{\Expect\text{vec}(M)}^2$ where $\textrm{vec}(M)$ denotes the vectorization of matrix $M$. We denote $\fromto{m}{n} \triangleq \{m, m+1, \ldots,n\}$ if $m\le n$, and $\emptyset$ otherwise. We use $[n] \triangleq \fromto{1}{n}$ as an abbreviation. For clarity, we use the superscript $b$ to distinguish the variables with different choices of the mini-batch size $b$. In each iteration $t$, we use $\calB_t^b$ to denote the batch 