including natural language processing, computer vision, and reinforcement learning [@goodfellow2016deep]. Despite their practical success, there are only limited studies of the theoretical properties of deep learning; see survey papers [@sun2019optimization; @fan2019selective] and references therein. The general problem underlying deep learning models is to optimize (minimize) a loss function, defined by the deviation of model predictions on data samples from the corresponding true labels. The prevailing method to train deep learning models is the mini-batch stochastic gradient descent (SGD) algorithm and its variants [@bottou1998online; @bottou2018optimization]. SGD updates model parameters by calculating a stochastic approximation of the full gradient of the loss function, based on a random selected subset of the training samples called a mini-batch.

It is well-accepted that selecting a large mini-batch size reduces the training time of deep learning models, as computation on large mini-batches can be better parallelized on processing units. For example, Goyal et. al. [@goyal2017accurate] scale ResNet-50 [@he2016deep] from a mini-batch size of 256 images and training time of 29 hours, to a larger mini-batch size of 8,192 images. Their training achieves the same level of accuracy while reducing the training time to one hour. However, noted by many researchers, larger mini-batch sizes suffer from a 