to support the theorems in Section \[sec:3\], to backup the hypotheses discussed in the introduction, and to provide further insights.

For all experiments, we perform mini-batch SGD multiple times starting from the same initial weights and following the same choice of the learning rates and other hyper-parameters, if applicable. This enables us to calculate the variance of the gradient estimators and other statistics in each iteration, where the randomness comes only from different samples of SGD. The learning rate $\alpha_t$ is selected to be inversely proportional to iteration $t$, or fixed, depending on the task at hand.

All models are implemented using PyTorch version 1.4 [@paszke2019pytorch] and trained on NVIDIA 2080Ti/1080 GPUs. We report the details about the hyperparameters and training procedures in Appendix \[appendix:exp\].

Datasets and Settings
---------------------

The Graduate Admission dataset[^1] [@acharya2019comparison] is to predict the chance of a graduate admission using linear regression. The dataset contains 500 samples with 6 features. This is a popular regression dataset with clean data. We build a linear regression model to predict the chance of acceptance (we include the intercept term in the model) and minimize the empirical $L_2$ loss using mini-batch SGD, as stated in Section \[subsec:linearRegression\]. The purpose of 