worse generalization ability [@lecun2012efficient; @keskar2019large]. Therefore, many efforts have been made to develop specialized training procedures that achieve good generalization using large mini-batch sizes [@hoffer2017train; @goyal2017accurate]. Smaller batch sizes have the advantage of allegedly offering better generalization (at the expense of a higher training time).

We hypothesize that smaller sizes lead to lower training loss and, unfortunately, decrease stability of the algorithm. The latter follows from the fact that the smaller is the batch size, more stochasticity and volatility is introduced. After all, if the batch size equals to the number of samples, there is no stochasticity in the algorithm. To this end, we conjecture that the variance of the gradient in each iteration is a decreasing function of the mini-batch size. The conjecture is the focus of the work herein. We are able to prove it in the convex linear regression case and to show significant progress in a two layer neural network setting with samples based on a normal distribution. In this case we show that the variance is a polynomial in the reciprocal of the mini-batch size and that it is decreasing for large enough mini-batch sizes. The increased variance as the mini-batch size decreases should also 