variance of the stochastic gradient estimator and the full gradient at the iterate in step $t$ are also decreasing functions of $b$ at any iteration step $t$. In addition, the proof provides a recursive relationship between the norm of gradients and the model parameters at each iteration. This recursive relationship can be used to calculate any quantity related to the stochastic gradient or full gradient at any iteration with respect to the initial weights. We give structural results and not explicit formulas which are impossible to obtain. For the two-layer linear neural network with $L_2$-loss and samples drawn from a normal distribution, we show that in each iteration step $t$ the trace of any product of the stochastic gradient estimators and weight matrices is a polynomial in $1/b$ with coefficients a sum of products of the initial weights. As a special case, the variance of the stochastic gradient estimator is a polynomial in $1/b$ without the constant term and therefore it is a decreasing function of $b$ when $b$ is large enough. The results can be easily extended to general deep linear networks. As a comparison, other papers that study theoretical properties of two-layer networks either fix one layer of 