intuitively imply convergence to lower training loss values and in turn better prediction and generalization ability (these relationships are yet to be confirmed analytically; but we provide empirical evidence to their validity).

Another line of research focuses on how to choose an optimal mini-batch size based on different criteria [@smith2017bayesian; @gower2019sgd]. However, these papers make strong assumptions on the loss function properties (strong or point or quasi convexity, or constant variance near stationary points) or about the formulation of the SGD algorithm (continuous time interpretation by means of differential equations). The statements are approximate in nature and thus not mathematical claims. They also focus on convergence and generalization while our goal is variance. The theoretical results regarding the relationship between the mini-batch size and the performance (variance, loss, generalization ability, etc.) of the SGD algorithm applied to general machine learning models are still missing. The work herein partially addresses this gap by showing the impact of the mini-batch size on the variance of gradients in SGD.

In the linear regression case, we show that in each iteration the norm of any linear combination of sample-wise gradients is a decreasing function of the mini-batch size $b$. As a special case, the 