most existing SGD settings, $\tilde{g}(\theta)$ is not an unbiased estimator for $g(\theta)$, as suggested by Theorem \[thm:control\_bias\_mse\]. Therefore, we refer to the optimization method based on such a $\tilde{g}(\theta)$ as the stochastic *approximate* gradient descent. The outline of SAGD is given in Algorithm \[alg:sagd\].

Despite the fact that $\tilde{g}(\theta)$ is a biased estimator for the true gradient, we show that by carefully choosing the hyperparameters, we can actually guarantee the overall convergence of SAGD. Interestingly, the convergence rate for a convex objective function, in terms of the number of gradient updates, is the same as the vanilla SGD method with an order of $\mathcal{O}(1/\sqrt{T})$, as is shown in Theorem \[thm:convergence\_convex\].

\[assu:f\_condition\]$f(\theta;\cdot)\in\mathscr{C}_{poly}^{r+5}$ for each $\theta\in\Theta$, and there exist a constant $C>0$ and an integer $m\ge0$ such that $\Vert\nabla^{i}f(\theta;\cdot)\Vert\le C(1+\Vert\cdot\Vert^{m})$ for all $\theta\in\Theta$ and $0\le i\le r+5$.

\[thm:convergence\_convex\]Suppose that $F(\theta)$ is convex and $L$-Lipschitz continuous in $\theta\in\Theta$, and $\Theta$ is a closed convex set with diameter $D<\infty$. Also assume that Assumption \[assu:f\_condition\] and the conditions in Theorem \[thm:finite\_moments\] hold. Then by choosing $\delta_{t}=C_{1}/\sqrt{t}$, $K_{t}=C_{2}t$, and $\alpha_{t}=\alpha_{0}/\sqrt{t}$, where $C_{1},C_{2},\alpha_{0}>0$ are constants, we have $\mathbb{E}[F(\hat{\theta})]-F^{*}\le\mathcal{O}(1/\sqrt{T})$.

The significance of Theorem \[thm:convergence\_convex\] is that it provides clear rates for the hyperparameters $\delta_{t}$ and $K_{t}$ in the 