to derive other properties of the SGD dynamics in regard to the mini-batch size and initial weights. To the best of authors’ knowledge, the work is the first one to theoretically study the impact of the mini-batch size on the variance of the gradient, under mild assumptions on the network and the loss function. We support our theoretical results by experiments. We further experiment on other state-of-the-art deep learning models and datasets to empirically show the validity of the conjectures about the impact of mini-batch size on average loss, average accuracy and the generalization ability of the model.

The major contributions of this paper are as follows.

-   For linear regression, we show that the norm of any number of linear combinations of the coordinates of the gradient is a decreasing function of the mini-batch size (Theorem \[thm:2\]). As a special case, the variance of the stochastic gradient estimators is also a decreasing function of the mini-batch size, for all iterations and all choices of learning rates (Corollary \[col:1\]) that are independent of the mini-batch size.

-   For a two-layer linear network, we show that any non-negative trace of the product of weight matrices and stochastic gradient estimators is a 