# Generation of 14.0W of single frequency light at 770 nm by intracavity
frequency doubling

Minho Kwon  Department of Physics, University of Wisconsin-Madison, 1150
University Ave., Madison, WI 53706, USA  Corresponding author:
<EMAIL_ADDRESS>Peiyu Yang  Department of Physics, University of Wisconsin-
Madison, 1150 University Ave., Madison, WI 53706, USA  Quantum Institute for
Light and Atoms, School of Physics and Material Science, East China Normal
University, Shanghai 200062, China  Preston Huft  Department of Physics,
University of Wisconsin-Madison, 1150 University Ave., Madison, WI 53706, USA
Christopher Young  Department of Physics, University of Wisconsin-Madison,
1150 University Ave., Madison, WI 53706, USA  Matthew Ebert  Department of
Physics, University of Wisconsin-Madison, 1150 University Ave., Madison, WI
53706, USA  Mark Saffman  Department of Physics, University of Wisconsin-
Madison, 1150 University Ave., Madison, WI 53706, USA

###### Abstract

We present a continuous, narrow-linewidth, tunable laser system that outputs
up to 14.0 W at 770 nm. The light is generated by frequency doubling 18.8 W of
light from a 1540 nm fiber amplifier that is seeded by a single mode diode
laser achieving >74% conversion efficiency. We utilize a Lithium Triborate
Crystal in an enhancement ring cavity. The low intensity noise and narrow
linewidth of the 770 nm output are suitable for cold atom experiments.

###### doi:

††journal: ol

Near-infrared(NIR) lasers around 760 - 780 nm have many applications including
laser cooling and trapping of atoms, atomic state manipulation, and
spectroscopy of Oxygen(760.8, 763.8 nm)[1], Potassium(766.7, 770.1 nm)[2] and
Rubidium(780.2 nm)[3]. Iodine $\text{I}_{2}$ rovibrational lines[4] at 770.7
nm provide excellent frequency references. In the context of quantum
computing, magic wavelengths for optical trapping of cold alkali atoms[5, 6]
near this wavelength exist, and the light can be used to rapidly drive Raman
transitions between Potassium and Rubidium hyperfine ground states.

Common diode lasers at these wavelengths are typically power limited up to
$\sim 100$ mW. Semiconductor optical amplifiers can boost the power to $\sim
3$ W, although typically with a poor spatial profile that reduces the usable
power when diffraction-limited performance is required. Furthermore, amplified
spontaneous emission generates incoherent frequency noise extending over tens
of nm, which leads to significant reduction of atomic coherence when the light
interacts with atoms. For atom trapping applications at IR wavelengths, fiber
lasers are typically used for their high optical power and low frequency
noise[7].

An alternative route to high power tunable sources for atomic spectroscopy is
based on second harmonic generation (SHG) or sum frequency generation(SFG)[8]
of a high power fiber laser. This approach benefits from the availability of
tunable, single mode laser diode sources, and high power fiber amplifiers that
utilize mature industrial technology in telecom wavelength bands. A variety of
nonlinear crystals have been explored for this application. Early efforts
utilized the high nonlinearity of periodically-poled Lithium Niobate(PPLN) to
achieve high power output[12, 13, 10, 11, 9, 14, 15] without introducing an
enhancement cavity. Examples include 11 W at 780 nm for Rb cooling[9], 313 nm
for Be ion cooling and quantum gates[10], and 319.8 nm for metastable Helium
trapping[11]. Other efforts have utilized Magnesium Oxide doped PPLN
(PPMgO:LN) for Rb/Cs cooling[16] and K cooling[17]. Periodically-poled
Potassium Titanyl Phosphate(PPKTP) in a Fabry-Perot enhancement cavity was
used to achieve 95% conversion efficiency[18] and an output power of 1.05 W at
775 nm.

Periodically-poled crystals are engineered to exhibit high optical non-
linearity utilizing quasi-phase matching. Typically a single or double pass
interaction is sufficient to reach high conversion efficiency with high power
sources. Despite their high optical nonlinearity, these crystals are prone to
thermal effects[19, 20, 21, 22] leading to photorefractive beam distortion.
Furthermore, nonlinear induced absorption[23] can cause permanent photochromic
damage to the crystal, imposing another challenge for high power cw operation.

Instead, we use Lithium triborate(LBO), which is known for its robustness at
high power[24] and wide transparency window. One drawback is the much weaker
optical nonlinearity ($d_{32}=0.67\leavevmode\nobreak\ \text{pm/V}$, Type-I
NCPM), compared to PPLN ($d_{33}=25\leavevmode\nobreak\ \text{pm/V}$) and
PPKTP ($d_{33}=15-17\leavevmode\nobreak\ \text{pm/V}$)[25]. This necessitates
the use of a pulsed laser or an enhancement cavity. Cavity-assisted conversion
at 1064 nm was demonstrated[26, 27], leading to >100 W of continuous 532 nm
light. We report here on the first high power, high efficiency frequency
doubling of a 1540 nm laser using an enhancement cavity and LBO crystal[28].
The combination of tunability, low intensity and frequency noise, and
excellent spatial beam quality make this laser source suitable for experiments
with Rb and K atoms.

Figure 1: Setup for SHG of 1540 nm light. The beam is mode-matched to the ring
cavity with a plano-convex lens (f=250 mm). The cavity is comprised of two
plano-concave mirrors(ROC=50 mm) and two flat mirrors in a bow-tie
configuration, with angles of incidents(AOI) nominally $8^{\circ}$. Input
coupler($\text{M}_{1}$) is partially reflective at 1540 nm and the
transmittance is optimized for impedance-matching. The other cavity
mirrors($\text{M}_{2},\text{M}_{3}$ and $\text{M}_{4}$) are front-side HR
coated at 1540 nm and rear-side AR coated for 1540 and 770 nm. The cavity
round trip length is 29.5 cm and the distance between the curved mirrors is
6.6 cm.

A schematic of the setup is shown in Fig.1. A 20 W fiber amplifier (IPG
Photonics EAR-20K-C-LP-SF) is seeded by a 1540 nm single frequency fiber laser
(Redfern Integrated Optics RIO3135-3-46-1). A fiber phase modulator (iXblue
MPX-LN-0.1-P-P-FA-FA) is used for Pound-Drever-Hall[29] locking of the SHG
cavity. The fiber amplifier output is linearly polarized with $1/e^{2}$ beam
diameter of 1.1 mm. The fundamental resonator mode is
circular($w_{y,z}=40.6\leavevmode\nobreak\ \mu$m) at the crystal center, which
is also aligned to the midpoint between the concave mirrors. AR coated LBO
crystals cut at ($\theta=90^{\circ},\phi=0^{\circ}$), with dimensions of
$3\times 3\times 30$ mm were purchased from United Crystals and UVisIR.

Type-I Non-Critical Phase Matching of LBO converts two $z$-polarized photons
to a single $y$-polarized photon propagating along the $x$-axis. The crystal
is temperature-tuned to satisfy phase matching conditions, dictated by
Sellmeier equations[30]. The observed single-pass temperature bandwidth of
$\Delta T\cdot l=8.68\pm 0.15^{\circ}\text{C}\cdot\text{cm}$ is consistent
with reported values[30]. However, we find a phase-matching temperature of
$88^{\circ}$C, which is $22^{\circ}$C lower than the predicted
value($110^{\circ}$C) [30]. The measurement was done with low power input such
that the contribution from absorption induced heating was negligible. We do
not have an explanation for the discrepancy. Heating becomes significant when
operating the enhancement cavity with high power input, due to high
circulating power and finite absorption. It is therefore necessary to make
small adjustments to the crystal oven temperature to achieve optimum output,
dependent upon input power. For the maximum input power of 18.8 W, optimum
temperatures are about $1.2^{\circ}$C lower than that of the low-power input.
We employed two-stage active temperature control, where the base plate and the
crystal oven are independently controlled by TECs. This minimizes the impact
on resonator path length while the crystal temperature is altered.
Furthermore, smaller thermal mass of the crystal and the oven allows faster
thermal control. Mechanical parts are all made from metal to minimize
outgassing and degradation at high temperature operation.

The cavity finesse is $\mathcal{F}_{\text{LP}}\sim 102$ at low circulating
power and when frequency doubling is operating at the maximum input power the
finesse is $\mathcal{F}_{\text{HP}}\sim 59$, corresponding to a cavity
linewidth of $\kappa_{\text{HP}}/2\pi\sim 16$ MHz and build-up factor of $\sim
19$. The round-trip length of the cavity is adjusted with a ring piezo
(Piezomechanik HPSt 150/14-10/12 HAg) attached to the flat
mirror($\text{M}_{2}$), providing up to $12\leavevmode\nobreak\ \mu$m stroke
length. The actuator range is particularly important for handling thermal
effects from the crystal. With a given stroke it can compensate a $\sim
24\leavevmode\nobreak\ \mu\text{m}$ round-trip length change, which is enough
to compensate $>4^{\circ}$C of crystal temperature variation. The seed laser
can be locked to the enhancement cavity for applications where intensity
stability is more important than frequency excursion. Our laser had an optical
tuning range of $\sim$500 MHz with modulation bandwidth of tens of kHz.

We initially tried the Hänsch-Couillaud[31, 32](HC) technique to obtain an
error signal for cavity locking, as it does not require any modulation of the
laser. However the HC error signal could not withstand transient thermal
effects at high power, and we switched to the PDH technique. A detailed
discussion of thermal effects is provided below. Reflection from the input
coupler($\text{M}_{1}$) is sampled from an AR-coated window, and detected by a
fast photodetector (Thorlabs DET08C). The signal is amplified and demodulated
to obtain a dispersive PDH error signal. The chosen phase-modulation frequency
of 130 MHz provides large capture range covering $>25\%$ of the free spectral
range(FSR = 0.96 GHz) of the cavity. Modulation depth is kept small such that
the carrier has $>97\%$ of the power. This keeps the nonlinear process power-
efficient, while minimizing intensity noise in the harmonic light caused by
the sidebands, which are off-resonant and thus attenuated by >18 dB.

Figure 2: SHG output(Black points) and conversion efficiency(Red points)
versus fundamental power. The black line is calculated SHG output based on
measured optical parameters described in the text.

A maximum harmonic output of 14.0 W at 18.8 W input power is observed as shown
in Fig. 2. The conversion efficiency $\epsilon=\text{P}_{2}/\text{P}_{1}$
saturates due to depletion of the circulating fundamental power and cavity
loss. Intracavity SHG output can be calculated[33] by solving the equation,

$\sqrt{\epsilon}=\frac{4T_{1}\sqrt{E_{NL}P_{m,1}}}{\left[2-\sqrt{1-T_{1}}(2-L-\sqrt{\epsilon
E_{NL}P_{m,1}})\right]^{2}},$ (1)

where $\text{T}_{1}$ is the transmission of the input coupler, $L$ is the
round-trip linear loss excluding the input coupler, $E_{NL}$ is a single-pass
nonlinear conversion coefficient, $P_{m,1}=mP_{1}$ is the fundamental power
coupled to the cavity, and $m$ is the mode-matching coefficient to the
$\text{TEM}_{00}$ resonator mode. We found $\text{T}_{1}=5\%$ and
$E_{NL}=1.23\times 10^{-6}\leavevmode\nobreak\ \text{W}^{-1}$ from direct
measurements, $L\sim 1\%$ from the measured cavity finesse, and $m\sim 0.95$
from the reflection dip. Calculated SHG output is consistent with the
parameters, as shown in Fig. 2. The measured spatial profile of the SHG output
had $\text{M}^{2}<1.4$ and circular($1:0.97$), as shown in Fig. 1.

Continuous wave operation requires active stabilization of the cavity length.
A servo-loop using an error signal generated by the PDH technique provided a
robust lock.

Figure 3: Relative Intensity Noise spectrum of the seed, fiber amplifier(FA),
and SHG output from 128 Hz to 102.4 kHz. Inset shows the RIN at 0.1-300 MHz
range, covering the rf frequency driving the fiber EOM.

Relative Intensity Noise(RIN) of the SHG output in continuous operation is
shown in Fig. 3. We observe noisier RIN than the input light. This is expected
because mechanical noise also contributes to the intensity noise in the SHG
output. The observed RIN also depended on the servo configuration. Best RIN
performance is observed when mechanical and laser modulation are both engaged.
Mechanical feedback alone left residual noise in the audio spectrum. Residual
amplitude and phase modulation of the fiber EOM caused weak RIN peaks at 130
MHz and 260 MHz.

SHG output and crystal temperature variation showing the long-term stability
are presented in Fig. 4.

Figure 4: SHG output and LBO oven temperature from a cold-start. The laser is
locked to the fundamental resonator mode at $t=20$ s, and stayed locked during
the acquisition window. Scale breaker indicates the crossover from transient
to steady-state dynamics. SHG output is normalized by the observed maximum,
4.5 W, for this data.

When the lock engages, the oven temperature suddenly rises from absorption
induced heating in the crystal at high circulating power. This thermal
perturbation settles down after a few minutes of active temperature control.
Sub-optimal choice of the oven set-point temperature caused the steady-state
SHG output to be slightly lower than the true maximum.

The spectral quality of the harmonic light is characterized by a self-
heterodyne measurement and is shown in Fig. 5.

Figure 5: Self-heterodyne beatnote of the harmonic light using an 11 km long
fiber delay line. Note that the FWHM of the self-heterodyne beatnote is twice
the laser linewidth. We observe FWHM/2 of 25(49) kHz without(with) the fast
servo loop engaged. The Voigt linewidths[34, 35]are 25(48) kHz for Slow
only(Slow & Fast servo loops). Measurements are averaged over $100\times$10 ms
sweeps(RBW 10 kHz).

Extracted linewidths are 25 kHz for piezo stabilization only, and 49 kHz when
the laser is additionally allowed to be modulated, which effectively broadens
the linewidth. The seed laser is specified to have < 10 kHz linewidth,
although we could not verify it due to spectral incompatibility of our self-
heterodye setup. The observed linewidths are not significantly broader then
the seed.

Now we discuss the cavity stabilization in more detail. Overall, the error
signal generated by the PDH technique resulted in the best performance
compared to two other techniques we tested. The HC error signal could not
withstand thermal transients, making it very difficult to cold start. The HC
error signal relies on intracavity birefringence, for our case $n_{y}\neq
n_{z}$. The birefringence is also temperature-
dependent($\frac{dn_{y}}{dT}\neq\frac{dn_{z}}{dT}$), meaning that the indices
of refraction for the two polarizations change at different rates. As the
temperature changes, their spectra in the cavity shift relative to each other,
and so does the error signal. If the change is too large, they eventually
become degenerate and the correct error signal shape is lost. We directly
observed the temperature-dependent HC error signal as shown in Fig. 6. When
both polarizations become degenerate in the cavity, the slope of the error
signal gets flipped and the servo lock is lost. For our cavity, degenerate
birefringence occurs if the optical path difference(OPD) between $y-$ and
$z-$polarizations become an integer multiple of $\lambda=1540$ nm, where the
OPD can be written as

$\text{OPD}_{\lambda}(T)=\left[n_{y}(T)-n_{z}(T)\right]\text{L}_{c}\left(1+\text{a}_{x}T\right).$
(2)

Here, $\text{L}_{c}$ is the length of the crystal, $n_{y(z)}(T)$ are
refractive indices of LBO for $\lambda$ at the temperature T, and
$\text{a}_{x}$ is the coefficient of thermal expansion along the x-axis. The
exact condition for the degeneracy may vary depending upon the cavity
configuration and spatial modes. Based on the Sellmeier equations[30], at
$T_{0}=$$81.77^{\circ}$C, we calculate the rate is
$\frac{d\text{OPD}}{dT}|_{T=T_{0}}$=538 nm/∘C, suggesting that the HC lock
cannot tolerate more than $2.86^{\circ}$C change. This agrees to better than
2% with the observed rate of 533 nm/∘C (or 0.346 FSR$/^{\circ}$C for our
cavity). We were able to analytically reproduce the observed distortion based
upon a theoretical framework[36]. Temperature-dependent non-linear loss
changes the amplitude of the error signal slightly but the effect is minor.

Figure 6: Temperature-dependent HC error signal. Color represents different
crystal temperatures, from $80.57^{\circ}$C to $82.97^{\circ}$C, in steps of
$\Delta\text{T}=+0.3^{\circ}$C. Cavity detuning of zero corresponds to the
$y$-pol. cavity resonance. $y$\- and $z$-pol. become simultaneously resonant
at $T_{0}$, where the original error shape is lost.

In principle, one may consider slowly ramping the input power allowing the
crystal to thermalize, minimizing temperature variation. However, input-power
dependent zero-crossing of the HC error signal requires in-situ adjustments
and we found it to be impractical. We concluded that the HC technique alone
cannot be reliably used for this application. A dither lock provided better
stability than the HC lock, although its narrow capture range and dither
frequency at tens of kHz caused significant intensity modulation, which can
cause parametric heating[37] of optically trapped atoms.

We believe that the limiting factor for more efficient conversion is linear
loss associated with the crystal. The LBO crystal exhibits two orders of
magnitude larger photoacoustic absorption($y$-pol., $\sim
10^{-3}\leavevmode\nobreak\ \text{cm}^{-1}$) at $1.5\leavevmode\nobreak\
\mu$m[38] than for $\lambda\sim 0.5-1.1\leavevmode\nobreak\ \mu$m. Assuming
the z-pol absorption is similar, a 30 mm long crystal will exhibit 0.3% of
absorption, which cannot be neglected. Finite absorption combined with high
circulating power causes substantial crystal heating and thermal effects.
These can lead to thermal lensing which will degrade the spatial mode quality
and conversion efficiency at high power. Using a loosely focused beam may
mitigate the thermal lensing[39] which could make the HC technique a viable
approach for stabilization.

In conclusion, we generated 14.0 W of cw single frequency, single transverse
mode light at 770 nm, which is an unexplored wavelength for LBO at high power.
The achievable output power is mainly limited by the available fundamental
power and residual absorption in the LBO crystal.

Funding National Science Foundation awards PHY-1104531, PHY-1521374, and the
Air Force Office of Scientific Research Quantum Memories MURI. PY acknowledges
support from the China Scholarship Council.

## References

  * [1] G. D. Greenblatt, J. J. Orlando, J. B. Burkholder, and A. R. Ravishankara, Journal of Geophysical Research 95, 18577 (1990).
  * [2] T. G. Tiecke, “Properties of Potassium,” Tech. Rep. 20 (2010).
  * [3] D. A. Steck, “Rubidium 87 D Line Data,” (2001).
  * [4] S. Rakowsky, D. Zimmermann, and W. E. Ernst, Applied Physics B Photophysics and Laser Chemistry 48, 463 (1989).
  * [5] G. Li, P. Zhang, and T. Zhang, arxiv:1905.11094 (2019).
  * [6] J. Wang, Y. Cheng, S. Guo, B. Yang, and J. He,Proc. SPIE, Quantum Optics II 8440, 84400Q (2012)
  * [7] A. Mazurenko, S. Blatt, F. Huber, M. F. Parsons, C. S. Chiu, G. Ji, D. Greif, and M. Greiner, Review of Scientific Instruments 90, 033101 (2019).
  * [8] S. Guo, Y. Ge, J. He, and J. Wang, Journal of Modern Optics 62, 1583 (2015).
  * [9] S. S. Sané, S. Bennetts, J. E. Debs, C. C. N. Kuhn, G. D. McDonald, P. A. Altin, J. D. Close, and N. P. Robins, Optics Express 20, 8915 (2012).
  * [10] A. C. Wilson, C. Ospelkaus, A. P. VanDevender, J. A. Mlynek, K. R. Brown, D. Leibfried, and D. J. Wineland, Applied Physics B 105, 741 (2011).
  * [11] R. J. Rengelink, R. P. M. J. W. Notermans, and W. Vassen, Applied Physics B 122, 122 (2016).
  * [12] R. Thompson, M. Tu, D. Aveline, N. Lundblad, and L. Maleki, Optics Express 11, 1709 (2003).
  * [13] F. Lienhart, S. Boussen, O. Carraz, N. Zahzam, Y. Bidel, and A. Bresson, Applied Physics B 89, 177 (2007).
  * [14] M. Lichtman, “Coherent Operations, Entanglement, and Progress Toward Quantum Search in a Large 2D Array of Neutral Atom Qubits,” Ph.D. thesis, University of Wisconsin-Madison (2015).
  * [15] T. H. Runcorn, R. T. Murray, and J. R. Taylor, IEEE Photonics Technology Letters 29, 1576 (2017).
  * [16] K. Zhang, J. He, and J. Wang, arxiv:1811.03799 (2018).
  * [17] G. Stern, B. Allard, M. Robert-de Saint-Vincent, J.-P. Brantut, B. Battelier, T. Bourdel, and P. Bouyer., Applied Optics 49, 3092 (2010).
  * [18] S. Ast, R. M. Nia, A. Schönbeck, N. Lastzka, J. Steinlechner, T. Eberle, M. Mehmet, S. Steinlechner, and R. Schnabel, Optics Letters 36, 3467 (2011).
  * [19] Z. M. Liao, S. A. Payne, J. W. Dawson, A. D. Drobshoff, C. A. Ebbers, D. M. Pennington, I. Jovanovic, and L. R. Taylor, _Nonlinear Optics: Materials, Fundamentals and Applications,_ , vol. 21 (OSA, Washington, D.C., 2004), p. WD17.
  * [20] S. Spiekermann, F. Laurell, V. Pasiskevicius, H. Karlsson, and I. Freitag, Applied Physics B: Lasers and Optics 79, 211 (2004).
  * [21] O. A. Louchev, N. E. Yu, S. Kurimura, and K. Kitamura, Applied Physics Letters 87, 131101 (2005).
  * [22] I. Ricciardi, M. De Rosa, A. Rocco, P. Ferraro, and P. De Natale, Optics Express 18, 10985 (2010).
  * [23] Y. Furukawa, K. Kitamura, A. Alexandrovski, R. K. Route, M. M. Fejer, and G. Foulon, Applied Physics Letters 78, 1970 (2001).
  * [24] T. Meier, “High-Power CW Green Lasers for Optical Metrology and Their Joint Benefit in Particle Physics Experiments,” Ph.D. thesis, Gottfried Wilhelm Leibniz Universität Hannover (2011).
  * [25] I. Shoji, T. Kondo, A. Kitamoto, M. Shirane, and R. Ito, Journal of the Optical Society of America B 14, 2268 (1997).
  * [26] S. Cui, L. Zhang, H. Jiang, W. Pan, X. Yang, G. Qin, and Y. Feng, Laser Physics Letters 16, 035105 (2019).
  * [27] T. Meier, B. Willke, and K. Danzmann, Optics Letters 35, 3742 (2010).
  * [28] M. Kwon, “Rydberg-mediated Atomic Ensemble Entanglement and Hyperfine Qubit Detection,” Ph.D. thesis, University of Wisconsin-Madison (2019).
  * [29] R. W. P. Drever, J. L. Hall, F. V. Kowalski, J. Hough, G. M. Ford, A. J. Munley, and H. Ward, Applied Physics B Photophysics and Laser Chemistry 31, 97 (1983).
  * [30] K. Kato, IEEE Journal of Quantum Electronics 30, 2950 (1994).
  * [31] T. Hansch and B. Couillaud, Optics Communications 35, 441 (1980).
  * [32] J. Boon-Engering, W. van der Veer, E. Bente, and W. Hogervorst, Optics Communications 140, 285 (1997).
  * [33] E. S. Polzik and H. J. Kimble, Optics Letters 16, 1400 (1991).
  * [34] M. Chen, Z. Meng, J. Wang, and W. Chen, Optics Express 23, 6803 (2015).
  * [35] L. Mercer, Journal of Lightwave Technology 9, 485 (1991).
  * [36] M. Vainio, J. E. Bernard, and L. Marmet, Applied Physics B 104, 897 (2011).
  * [37] T. Savard, K. O’Hara, and J. Thomas, Physical Review A 56, R1095 (1997).
  * [38] N. Waasem, S. Fieberg, J. Hauser, G. Gomes, D. Haertle, F. Kühnemann, and K. Buse, Review of Scientific Instruments 84 (2013).
  * [39] R. Le Targat, J.-J. Zondy, and P. Lemonde, Optics Communications 247, 471 (2005).

## References

  * [1] G. D. Greenblatt, J. J. Orlando, J. B. Burkholder, and A. R. Ravishankara, “Absorption measurements of oxygen between 330 and 1140 nm,” Journal of Geophysical Research 95, 18577 (1990).
  * [2] T. G. Tiecke, “Properties of Potassium,” Tech. Rep. 20 (2010).
  * [3] D. A. Steck, “Rubidium 87 D Line Data,” (2001).
  * [4] S. Rakowsky, D. Zimmermann, and W. E. Ernst, “Accurate determination of wavenumbers for iodine molecular lines in the red spectral region,” Applied Physics B Photophysics and Laser Chemistry 48, 463–466 (1989).
  * [5] G. Li, P. Zhang, and T. Zhang, “Triply magic conditions for microwave transition of optically trapped alkali-metal atoms,” arxiv:1905.11094 (2019).
  * [6] J. Wang, Y. Cheng, S. Guo, B. Yang, and J. He, “Magic-wavelength optical dipole trap of cesium and rubidium atoms,” Proc. SPIE, Quantum Optics II 8440, 84400Q (2012).
  * [7] A. Mazurenko, S. Blatt, F. Huber, M. F. Parsons, C. S. Chiu, G. Ji, D. Greif, and M. Greiner, “Implementation of a stable, high-power optical lattice for quantum gas microscopy,” Review of Scientific Instruments 90, 033101 (2019).
  * [8] S. Guo, Y. Ge, J. He, and J. Wang, “Singly resonant sum-frequency generation of 520-nm laser via a variable input-coupling transmission cavity,” Journal of Modern Optics 62, 1583–1590 (2015).
  * [9] S. S. Sané, S. Bennetts, J. E. Debs, C. C. N. Kuhn, G. D. McDonald, P. A. Altin, J. D. Close, and N. P. Robins, “11 W narrow linewidth laser source at 780nm for laser cooling and manipulation of Rubidium,” Optics Express 20, 8915 (2012).
  * [10] A. C. Wilson, C. Ospelkaus, A. P. VanDevender, J. A. Mlynek, K. R. Brown, D. Leibfried, and D. J. Wineland, “A 750-mW, continuous-wave, solid-state laser source at 313 nm for cooling and manipulating trapped 9Be+ ions,” Applied Physics B 105, 741–748 (2011).
  * [11] R. J. Rengelink, R. P. M. J. W. Notermans, and W. Vassen, “A simple 2 W continuous-wave laser system for trapping ultracold metastable helium atoms at the 319.8 nm magic wavelength,” Applied Physics B 122, 122 (2016).
  * [12] R. Thompson, M. Tu, D. Aveline, N. Lundblad, and L. Maleki, “High power single frequency 780nm laser source generated from frequency doubling of a seeded fiber amplifier in a cascade of PPLN crystals,” Optics Express 11, 1709 (2003).
  * [13] F. Lienhart, S. Boussen, O. Carraz, N. Zahzam, Y. Bidel, and A. Bresson, “Compact and robust laser system for rubidium laser cooling based on the frequency doubling of a fiber bench at 1560 nm,” Applied Physics B 89, 177–180 (2007).
  * [14] M. Lichtman, “Coherent Operations, Entanglement, and Progress Toward Quantum Search in a Large 2D Array of Neutral Atom Qubits,” Ph.D. thesis, University of Wisconsin-Madison (2015).
  * [15] T. H. Runcorn, R. T. Murray, and J. R. Taylor, “High Average Power Second-Harmonic Generation of a CW Erbium Fiber MOPA,” IEEE Photonics Technology Letters 29, 1576–1579 (2017).
  * [16] K. Zhang, J. He, and J. Wang, “Compact single-pass laser frequency conversion to 780.2 nm and 852.3 nm based on PPMgO:LN bulk crystals and diode-laser-seeded fiber amplifiers,” arxiv:1811.03799 (2018).
  * [17] G. Stern, B. Allard, M. Robert-de Saint-Vincent, J.-P. Brantut, B. Battelier, T. Bourdel, and P. Bouyer., “Frequency doubled 1534 nm laser system for potassium laser cooling,” Applied Optics 49, 3092 (2010).
  * [18] S. Ast, R. M. Nia, A. Schönbeck, N. Lastzka, J. Steinlechner, T. Eberle, M. Mehmet, S. Steinlechner, and R. Schnabel, “High-efficiency frequency doubling of continuous-wave laser light,” Optics Letters 36, 3467 (2011).
  * [19] Z. M. Liao, S. A. Payne, J. W. Dawson, A. D. Drobshoff, C. A. Ebbers, D. M. Pennington, I. Jovanovic, and L. R. Taylor, “Thermally induced dephasing in periodically poled KTP frequency-doubling crystals,” in _Nonlinear Optics: Materials, Fundamentals and Applications_ , vol. 21 (OSA, Washington, D.C., 2004), p. WD17.
  * [20] S. Spiekermann, F. Laurell, V. Pasiskevicius, H. Karlsson, and I. Freitag, “Optimizing non-resonant frequency conversion in periodically poled media,” Applied Physics B: Lasers and Optics 79, 211–219 (2004).
  * [21] O. A. Louchev, N. E. Yu, S. Kurimura, and K. Kitamura, “Thermal inhibition of high-power second-harmonic generation in periodically poled LiNbO3 and LiTaO3 crystals,” Applied Physics Letters 87, 131101 (2005).
  * [22] I. Ricciardi, M. De Rosa, A. Rocco, P. Ferraro, and P. De Natale, “Cavity-enhanced generation of 6 W cw second-harmonic power at 532 nm in periodically-poled MgO:LiTaO3,” Optics Express 18, 10985 (2010).
  * [23] Y. Furukawa, K. Kitamura, A. Alexandrovski, R. K. Route, M. M. Fejer, and G. Foulon, “Green-induced infrared absorption in MgO doped LiNbO3,” Applied Physics Letters 78, 1970–1972 (2001).
  * [24] T. Meier, “High-Power CW Green Lasers for Optical Metrology and Their Joint Benefit in Particle Physics Experiments,” Ph.D. thesis, Gottfried Wilhelm Leibniz Universität Hannover (2011).
  * [25] I. Shoji, T. Kondo, A. Kitamoto, M. Shirane, and R. Ito, “Absolute scale of second-order nonlinear-optical coefficients,” Journal of the Optical Society of America B 14, 2268 (1997).
  * [26] S. Cui, L. Zhang, H. Jiang, W. Pan, X. Yang, G. Qin, and Y. Feng, “High efficiency frequency doubling with a passive enhancement cavity,” Laser Physics Letters 16, 035105 (2019).
  * [27] T. Meier, B. Willke, and K. Danzmann, “Continuous-wave single-frequency 532 nm laser source emitting 130 W into the fundamental transversal mode,” Optics Letters 35, 3742 (2010).
  * [28] M. Kwon, “Rydberg-mediated Atomic Ensemble Entanglement and Hyperfine Qubit Detection,” Ph.D. thesis, University of Wisconsin-Madison (2019).
  * [29] R. W. P. Drever, J. L. Hall, F. V. Kowalski, J. Hough, G. M. Ford, A. J. Munley, and H. Ward, “Laser phase and frequency stabilization using an optical resonator,” Applied Physics B Photophysics and Laser Chemistry 31, 97–105 (1983).
  * [30] K. Kato, “Temperature-tuned 90∘ phase-matching properties of LBO,” IEEE Journal of Quantum Electronics 30, 2950–2952 (1994).
  * [31] T. Hansch and B. Couillaud, “Laser frequency stabilization by polarization spectroscopy of a reflecting reference cavity,” Optics Communications 35, 441–444 (1980).
  * [32] J. Boon-Engering, W. van der Veer, E. Bente, and W. Hogervorst, “Stabilization of an optical cavity containing a birefringent element,” Optics Communications 140, 285–288 (1997).
  * [33] E. S. Polzik and H. J. Kimble, “Frequency doubling with KNb$\text{O}_{3}$ in an external cavity,” Optics Letters 16, 1400 (1991).
  * [34] M. Chen, Z. Meng, J. Wang, and W. Chen, “Ultra-narrow linewidth measurement based on Voigt profile fitting,” Optics Express 23, 6803 (2015).
  * [35] L. Mercer, “1/f frequency noise effects on self-heterodyne linewidth measurements,” Journal of Lightwave Technology 9, 485–493 (1991).
  * [36] M. Vainio, J. E. Bernard, and L. Marmet, “Cavity-enhanced optical frequency doubler based on transmission-mode Hänsch–Couillaud locking,” Applied Physics B 104, 897–908 (2011).
  * [37] T. Savard, K. O’Hara, and J. Thomas, “Laser-noise-induced heating in far-off resonance optical traps,” Physical Review A 56, R1095–R1098 (1997).
  * [38] N. Waasem, S. Fieberg, J. Hauser, G. Gomes, D. Haertle, F. Kühnemann, and K. Buse, “Photoacoustic absorption spectrometer for highly transparent dielectrics with parts-per-million sensitivity,” Review of Scientific Instruments 84 (2013).
  * [39] R. Le Targat, J.-J. Zondy, and P. Lemonde, “75%-Efficiency blue generation from an intracavity PPKTP frequency doubler,” Optics Communications 247, 471–481 (2005).


marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio

Anonymous Authors1

††footnotetext:  1Anonymous Institution, Anonymous City, Anonymous Region,
Anonymous Country.

###### Abstract

It is well known that we need to choose the hyper-parameters in Momentum,
AdaGrad, AdaDelta, and other alternative stochastic optimizers. While in many
cases, the hyper-parameters are tuned tediously based on experience becoming
more of an art than science. We present a novel per-dimension learning rate
method for gradient descent called AdaSmooth. The method is insensitive to
hyper-parameters thus it requires no manual tuning of the hyper-parameters
like Momentum, AdaGrad, and AdaDelta methods. We show promising results
compared to other methods on different convolutional neural networks, multi-
layer perceptron, and alternative machine learning tasks. Empirical results
demonstrate that AdaSmooth works well in practice and compares favorably to
other stochastic optimization methods in neural networks.

##  1 Introduction

Over the years, stochastic gradient-based optimization has become a core
method in many fields of science and engineering such as computer vision and
automatic speech recognition processing (Krizhevsky et al., 2012; Hinton et
al., 2012a; Graves et al., 2013). Stochastic gradient descent (SGD) and deep
neural network (DNN) play a core role in training stochastic objective
functions. When a new deep neural network is developed for a given task, some
hyper-parameters related to the training of the network must be chosen
heuristically. For each possible combination of structural hyper-parameters, a
new network is typically trained from scratch and evaluated over and over
again. While much progress has been made on hardware (e.g. Graphical
Processing Units) and software (e.g. cuDNN) to speed up the training time of a
single structure of a DNN, the exploration of a large set of possible
structures remains very slow making the need of a stochastic optimizer that is
insensitive to hyper-parameters.

###  1.1 Gradient Descent

Gradient descent (GD) is one of the most popular algorithms to perform
optimization and by far the most common way to optimize machine learning
tasks. And this is particularly true for optimizing neural networks. The
neural networks or machine learning in general find the set of parameters
$\bm{x}\in{}^{d}$ in order to optimize an objective function $L(\bm{x})$. The
gradient descent finds a sequence of parameters

$\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{T},$ (1)

such that when $T\rightarrow\infty$, the objective function $L(\bm{x}_{T})$
achieves the optimal minimum value. At each iteration $t$, a step
$\Delta\bm{x}_{t}$ is applied to change the parameters. Denoting the
parameters at the $t$-th iteration as $\bm{x}_{t}$. Then the update rule
becomes

$\bm{x}_{t+1}=\bm{x}_{t}+\Delta\bm{x}_{t}.$ (2)

The most naive method of stochastic gradient descent is the vanilla update:
the parameter moves in the opposite direction of the gradient which finds the
steepest descent direction since the gradients are orthogonal to level curves
(a.k.a., level surface, see Lemma 16.4 in Lu (2022b)):

$\Delta\bm{x}_{t}=-\eta\bm{g}_{t}=-\eta\frac{\partial
L(\bm{x}_{t})}{\partial\bm{x}_{t}}=-\eta\nabla L(\bm{x}_{t}),$ (3)

where the positive value $\eta$ is the learning rate and depends on specific
problems, and $\bm{g}_{t}=\frac{\partial
L(\bm{x}^{t})}{\partial\bm{x}^{t}}\in{}^{d}$ is the gradient of the
parameters. The learning rate $\eta$ controls how large of a step to take in
the direction of negative gradient so that we can reach a (local) minimum.
While if we follow the negative gradient of a single sample or a batch of
samples iteratively, the local estimate of the direction can be obtained and
is known as the stochastic gradient descent (SGD) (Robbins & Monro, 1951). In
the SGD framework, the objective function is stochastic that is composed of a
sum of subfunctions evaluated at different subsamples of the data.

For a small step-size, gradient descent makes a monotonic improvement at every
iteration. Thus, it always converges, albeit to a local minimum. However, the
speed of the vanilla GD method is usually slow, while it can take an
exponential rate when the curvature condition is poor. While choosing higher
than this rate may cause the procedure to diverge in terms of the objective
function. Determining a good learning rate (either global or per-dimension)
becomes more of an art than science for many problems. Previous work has been
done to alleviate the need for selecting a global learning rate (Zeiler,
2012), while it is still sensitive to other hyper-parameters.

The main contribution of this paper is to propose a novel stochastic
optimization method which is insensitive to different choices of hyper-
parmeters resulting in adaptive learning rates, and naturally performs a form
of step size annealing. We propose the AdaSmooth algorithm to both increase
optimization efficiency and out-of-sample accuracy. While previous works
propose somewhat algorithms that are insensitive to the global learning rate
(e.g., (Zeiler, 2012)), the methods are still sensitive to hyper-parameters
that influence per-dimension learning rates. The proposed AdaSmooth, a method
for efficient stochastic optimization that only requires first-order gradients
and (accumulated) past update steps with little memory requirement, allows
flexible and adaptive per-dimension learning rates. Meanwhile, the method is
memory efficient and easy to implement. Our method is designed to combine the
advantages of the following methods: AdaGrad (Duchi et al., 2011), which works
well with sparse gradients, RMSProp (Hinton et al., 2012b), which works well
in online and non-stationary settings, and AdaDelta (Zeiler, 2012), which is
less sensitive in global learning rate.

##  2 Related Work

There are several variants of gradient descent to use heuristics for
estimating a good learning rate at each iteration of the progress. These
methods either attempt to accelerate learning when suitable or to slow down
learning near a local minima (Zeiler, 2012; Kingma & Ba, 2014; Ruder, 2016).

(a) Optimization without Momentum. A higher learning rate may result in larger
parameter updates in dimension across the valley (direction of $x_{2}$) which
could lead to oscillations back as forth across the valley.

(b) Optimization with Momentum. Though the gradients along the valley
(direction of $x_{1}$) are much smaller than the gradients across the valley
(direction of $x_{2}$), they are typically in the same direction and thus the
momentum term accumulates to speed up movement, dampen oscillations and cause
us to barrel through narrow valleys, small humps and (local) minima.

Figure 1: A 2-dimensional convex function
$L(\bm{x})=2(x_{1}-3)^{2}+20(x_{2}-2)^{2}+5$ and $\frac{\partial
L(\bm{x})}{\partial\bm{x}}=(4x_{1}-12,8x_{2}-16)^{\top}$. Starting point to
descent is $(-2,5)^{\top}$.

###  2.1 Momentum

If the cost surface is not spherical, learning can be quite slow because the
learning rate must be kept small to prevent divergence along the steep
curvature directions (Rumelhart et al., 1986; Qian, 1999; Sutskever et al.,
2013). The SGD with Momentum (that can be applied to full batch or mini-batch
learning) attempts to use previous step to speed up learning when suitable
such that it enjoys better converge rates on deep networks. The main idea
behind the Momentum method is to speed up the learning along dimensions where
the gradient consistently point in the same direction; and to slow the pace
along dimensions in which the sign of the gradient continues to change. Figure
1(a) shows a set of updates for vanilla GD where we can find the update along
dimension $x_{1}$ is consistent; and the move along dimension $x_{2}$
continues to change in a zigzag pattern. The GD with Momentum keeps track of
past parameter updates with an exponential decay, and the update method has
the following step:

$\displaystyle\Delta\bm{x}_{t}$
$\displaystyle=\rho\Delta\bm{x}_{t-1}-\eta\frac{\partial
L(\bm{x}_{t})}{\partial\bm{x}_{t}},$ (4)

where the algorithm remembers the latest update and adds it to present update
by multiplying a parameter $\rho$ called momentum parameter. That is, the
amount we change the parameter is proportional to the negative gradient plus
the previous weight change; the added momentum term acts as both a smoother
and an accelerator. The momentum parameter $\rho$ works as a decay constant
where $\Delta\bm{x}_{1}$ may has effect on $\Delta\bm{x}_{100}$; however, its
effect is decayed by this decay constant. In practice, the momentum parameter
$\rho$ is usually set to be 0.9 by default. Momentum simulates the concept
inertia in physics. It means that in each iteration, the update mechanism is
not only related to the gradient descent, which refers to the dynamic term,
but also maintains a component which is related to the direction of last
update iteration, which refers to the momentum.

The momentum works extremely better in ravine-shaped loss curve. Ravine is an
area, where the surface curves are much more steeply in one dimension than in
another (see the loss curve in Figure 1, i.e., a long narrow valley). Ravines
are common near local minima in deep neural networks and vanilla SGD has
trouble navigating them. As shown by the toy example in Figure 1(a), SGD will
tend to oscillate across the narrow ravine since the negative gradient will
point down one of the steep sides rather than along the ravine towards the
optimum. Momentum helps accelerate gradients in the correct direction (Figure
1(b)).

###  2.2 AdaGrad

The learning rate annealing procedure modifies a single global learning rate
that applies to all dimensions of the parameters (Smith, 2017). Duchi et al.
(2011) proposed a method called AdaGrad where the learning rate is updated on
a per-dimension basis. The learning rate for each parameter depends on the
history of gradient updates of that parameter in a way such that parameters
with a scarce history of updates are updated faster using a larger learning
rate. In other words, parameters that have not been updated much in the past
are more likely to have higher learning rates now. Denoting the element-wise
vector multiplication between $\bm{a}$ and $\bm{b}$ by $\bm{a}\odot\bm{b}$,
formally, the AdaGrad has the following update step:

$\displaystyle\Delta\bm{x}_{t}$
$\displaystyle=-\frac{\eta}{\sqrt{\sum_{\tau=1}^{t}\bm{g}_{\tau}^{2}+\epsilon}}\odot\bm{g}_{t},$
(5)

where $\epsilon$ is a smoothing term to better condition the division, $\eta$
is a global learning rate shared by all dimensions, $\bm{g}_{\tau}^{2}$
indicates the element-wise square $\bm{g}_{\tau}\odot\bm{g}_{\tau}$, and the
denominator computes the $l$2 norm of sum of all previous squared gradients in
a per-dimension fashion. Though the $\eta$ is shared by all dimensions, each
dimension has its own dynamic learning rate controlled by the $l$2 norm of
accumulated gradient magnitudes. Since this dynamic learning rate grows with
the inverse of the accumulated gradient magnitudes, larger gradient magnitudes
have smaller learning rates and smaller absolute values of gradients have
larger learning rates. Therefore, the accumulated gradient in the denominator
has the same effects as the learning rate annealing.

One pro of the AdaGrad method is that it partly eliminates the need to tune
the learning rate controlled by the accumulated gradient magnitude. However,
AdaGrad’s main weakness is its accumulation of the squared gradients in the
denominator. Since every added term is positive, the accumulated sum keeps
growing or exploding during every training step. This in turn causes the per-
dimension learning rate to shrink and eventually decrease throughout training
and become infinitesimally small, eventually falling to zero and stopping
training any more. Moreover, since the magnitudes of gradients are factored
out in AdaGrad, this method can be sensitive to the initialization of the
parameters and the corresponding gradients. If the initial magnitudes of the
gradients are large or infinitesimally huge, the per-dimension learning rates
will be low for the remainder of training. This can be partly combated by
increasing the global learning rate, making the AdaGrad method sensitive to
the choice of learning rate. Further, since AdaGrad assumes the parameter with
fewer updates should favor a larger learning rate; and one with more movement
should employ a smaller learning rate. This makes it consider only the
information from squared gradients, or the absolute value of the gradients.
And thus AdaGrad does not include information from total move (i.e., the sum
of updates; not the sum of absolute updates).

To be more succinct, AdaGrad has the following main drawbacks: 1) the
continual decay of learning rates throughout training; 2) the need for a
manually selected global learning rate; 3) considering only the absolute value
of gradients.

###  2.3 AdaDelta

AdaDelta is an extension of AdaGrad that overcomes the main weakness of
AdaGrad (Zeiler, 2012). The original idea of AdaDelta is simple: it restricts
the window of accumulated past gradients to some fixed size $w$ rather than
$t$ (i.e., current time step). However, since storing $w$ previous squared
gradients is inefficient, the AdaDelta introduced in Zeiler (2012) implements
this accumulation as an exponentially decaying average of the squared
gradients. This is very similar to the idea of momentum term (or decay
constant).

####  2.3.1 AdaDelta: Form 1 (RMSProp)

We first discuss the exact form of the window AdaGrad (we here call it
AdaGradWin for short). Assume at time $t$ this running average is
$E[\bm{g}^{2}]_{t}$ then we compute:

$E[\bm{g}^{2}]_{t}=\rho E[\bm{g}^{2}]_{t-1}+(1-\rho)\bm{g}_{t}^{2},$ (6)

where $\rho$ is a decay constant similar to that used in the momentum method
and $\bm{g}_{t}^{2}$ indicates the element-wise square
$\bm{g}_{t}\odot\bm{g}_{t}$.

As Eq (6) is just the root mean squared (RMS) error criterion of the
gradients, we can replace it with the criterion short-hand. Let
$\text{RMS}[\bm{g}]_{t}=\sqrt{E[\bm{g}^{2}]_{t}+\epsilon}$, where again a
constant $\epsilon$ is added to better condition the denominator. Then the
resulting step size can be obtained as follows:

$\Delta\bm{x}_{t}=-\frac{\eta}{\text{RMS}[\bm{g}]_{t}}\odot\bm{g}_{t},$ (7)

where again $\odot$ is the element-wise vector multiplication.

As aforementioned, the form in Eq (6) is originally from the exponential
moving average (EMA). In the original form of EMA, $1-\rho$ is also known as
the smoothing constant (SC) where the SC can be written as $\frac{2}{N+1}$ and
the period $N$ can be thought of as the number of past values to do the moving
average calculation (Lu, 2022a):

$\text{SC}=1-\rho=\frac{2}{N+1}.$ (8)

The above Eq (8) links different variables: the decay constant $\rho$, the
smoothing constant (SC), and the period $N$. If $\rho=0.9$, then $N=19$. That
is, roughly speaking, $E[\bm{g}^{2}]_{t}$ at iteration $t$ is approximately
equal to the moving average of past 19 squared gradients and the current one
(i.e., moving average of 20 squared gradients totally). The relationship in Eq
(8) though is not discussed in Zeiler (2012), it is important to decide the
lower bound of the decay constant $\rho$. Typically, a time period of $N=3$ or
7 is thought to be a relatively small frame making the lower bound of decay
constant $\rho=0.5$ or 0.75; when $N\rightarrow\infty$, the decay constant
$\rho$ approaches $1$.

However, we can find that the AdaGradWin still only considers the absolute
value of gradients and a fixed number of past squared gradients is not
flexible which can cause a small learning rate near (local) minima as we will
discuss in the sequel.

##### RMSProp

The AdaGradWin is actually the same as the RMSProp method developed
independently by Geoff Hinton in Hinton et al. (2012b) both of which are
stemming from the need to resolve AdaGrad’s radically diminishing per-
dimension learning rates. Hinton et al. (2012b) suggests $\rho$ to be set to
0.9 and the global learning rate $\eta$ to be $0.001$ by default.

####  2.3.2 AdaDelta: Form 2

Zeiler (2012) shows the units of the step size shown above do not match (so as
the vanilla SGD, the momentum, and the AdaGrad). To overcome this weakness,
from the correctness of the second order method, the author considers to
rearrange Hessian to determine the quantities involved. It is well known that,
though the calculation of Hessian or approximation to the Hessian matrix is a
tedious and computationally expensive task, its curvature information is
useful for optimization, and the units in Newton’s method are well matched.
Given the Hessian matrix $\bm{H}$, the update step in Newton’s method can be
described as follows (Becker & Le Cun, 1988; Dauphin et al., 2014):

$\Delta\bm{x}_{t}\propto-\bm{H}^{-1}\bm{g}_{t}\propto\frac{\frac{\partial
L(\bm{x}_{t})}{\partial\bm{x}_{t}}}{\frac{\partial^{2}L(\bm{x}_{t})}{\partial\bm{x}^{2}}}.$
(9)

This implies

$\frac{1}{\frac{\partial^{2}L(\bm{x}_{t})}{\partial\bm{x}_{t}^{2}}}=\frac{\Delta\bm{x}_{t}}{\frac{\partial
L(\bm{x}_{t})}{\partial\bm{x}_{t}}},$ (10)

i.e., the units of the Hessian matrix can be approximated by the right-hand
side term of the above equation. Since the RMSProp update in Eq (7) already
involves $\text{RMS}[\bm{g}]_{t}$ in the denominator, i.e., the units of the
gradients. Putting another unit of the order of $\Delta\bm{x}_{t}$ in the
numerator can match the same order as Newton’s method. To do this, define
another exponentially decaying average of the update steps:

$\displaystyle\text{RMS}[\Delta\bm{x}]_{t}$
$\displaystyle=\sqrt{E[\Delta\bm{x}^{2}]_{t}}$ (11) $\displaystyle=\sqrt{\rho
E[\Delta\bm{x}^{2}]_{t-1}+(1-\rho)\Delta\bm{x}_{t}^{2}}.$

Since $\Delta\bm{x}_{t}$ for the current iteration is not known and the
curvature can be assumed to be locally smoothed making it suitable to
approximate $\text{RMS}[\Delta\bm{x}]_{t}$ by
$\text{RMS}[\Delta\bm{x}]_{t-1}$. So we can use an estimation of
$\frac{1}{\frac{\partial^{2}L(\bm{x}_{t})}{\partial\bm{x}_{t}^{2}}}$ to
replace the computationally expensive $\bm{H}^{-1}$:

$\frac{\Delta\bm{x}_{t}}{\frac{\partial
L(\bm{x}_{t})}{\partial\bm{x}_{t}}}\sim\frac{\text{RMS}[\Delta\bm{x}]_{t-1}}{\text{RMS}[\bm{g}]_{t}}.$
(12)

This is an approximation to the diagonal Hessian using only RMS measures of
$\bm{g}$ and $\Delta\bm{x}$, and results in the update step whose units are
matched:

$\Delta\bm{x}_{t}=-\frac{\text{RMS}[\Delta\bm{x}]_{t-1}}{\text{RMS}[\bm{g}]_{t}}\odot\bm{g}_{t}.$
(13)

The idea of AdaDelta from the second method overcomes the annoying choosing of
learning rate. Similarly, in Kingma & Ba (2014), an exponentially decaying
average is incorporated into the gradient information such that convergence in
online convex setting is improved. Meanwhile, the diminishing problem was also
attacked in second order methods (Schaul et al., 2013). The idea of averaging
gradient or its alternative is not unique, previously in Moulines & Bach
(2011), Polyak-Ruppert averaging has been shown to improve the convergence of
vanilla SGD (Ruppert, 1988; Polyak & Juditsky, 1992). Other stochastic
optimization methods, including the natural Newton method, AdaMax, Nadam, all
set the step-size per-dimension by estimating curvature from first-order
information (Le Roux & Fitzgibbon, 2010; Kingma & Ba, 2014; Dozat, 2016). The
LAMB adopts layerwise normalization due to layerwise adaptivity (You et al.,
2019) and we shall not go into the details.

##  3 AdaSmooth Method

In this section we will discuss the effective ratio based on previous updates
in the stochastic optimization process and how to apply it to accomplish
adaptive learning rates per-dimension via the flexible smoothing constant,
hence the name AdaSmooth. The idea presented in the paper is derived from the
first form of AdaDelta (Zeiler, 2012) in order to improve two main drawbacks
of the method: 1) consider only the absolute value of the gradients rather
than the total movement in each dimension; 2) the need for manually selected
hyper-parameters.

###  3.1 Effective Ratio (ER)

Kaufman (2013; 1995) suggested replacing the smoothing constant in the EMA
formula with a constant based on the efficiency ratio (ER). And the ER is
shown to provide promising results for financial forecasting via classic
quantitative strategies Lu (2022a) where the ER of the closing price is
calculated to decide the trend of the asset. This indicator is designed to
measure the strength of a trend, defined within a range from -1.0 to +1.0
where the larger magnitude indicates a larger upward or downward trend.
Recently, Lu & Yi (2022) shows the ER can be utilized to reduce overestimation
and underestimation in time series forecasting. Given the window size $M$ and
a series $\\{h_{1},h_{2},\ldots,h_{T}\\}$, it is calculated with a simple
formula:

$\displaystyle e_{t}$
$\displaystyle=\frac{s_{t}}{n_{t}}=\frac{h_{t}-h_{t-M}}{\sum_{i=0}^{M-1}|h_{t-i}-h_{t-1-i}|}$
(14) $\displaystyle=\frac{\text{Total move for a period}}{\text{Sum of
absolute move for each bar}},$

where $e_{t}$ is the ER of the series at time $t$. At a strong trend (i.e.,
the input series is moving in a certain direction, either up or down) the ER
will tend to 1 in absolute value; if there is no directed movement, it will be
a little more than 0.

Instead of calculating the ER of the closing price of asset, we want to
calculate the ER of the moving direction in the update methods for each
parameter. And in the descent methods, we care more about how much each
parameter moves apart from its initial point in each period, either move
positively or negatively. So here we only consider the absolute value of the
ER. To be specific, the ER for the parameters in the proposed method is
calculated as follows:

$\displaystyle\bm{e}_{t}=\frac{\bm{s}_{t}}{\bm{n}_{t}}$
$\displaystyle=\frac{|\bm{x}_{t}-\bm{x}_{t-M}|}{\sum_{i=0}^{M-1}|\bm{x}_{t-i}-\bm{x}_{t-1-i}|}$
(15)
$\displaystyle=\frac{|\sum_{i=0}^{M-1}\Delta\bm{x}_{t-1-i}|}{\sum_{i=0}^{M-1}|\Delta\bm{x}_{t-1-i}|},$

where $\bm{e}_{t}\in{}^{d}$ whose $i$-th element $e_{t,i}$ is in the range of
$[0,1]$ for all $i$ in $[1,2,\ldots,d]$. A larger value of $e_{t,i}$ indicates
the descent method in the $i$-th dimension is moving in a certain direction;
while a smaller value approaching 0 means the parameter in the $i$-th
dimension is moving in a zigzag pattern, interleaved by positive and negative
movement. In practice, and in all of our experiments, the $M$ is selected to
be the batch index for each epoch. That is, $M=1$ if the training is in the
first batch of each epoch; and $M=M_{\text{max}}$ if the training is in the
last batch of the epoch where $M_{\text{max}}$ is the maximal number of
batches per epoch. In other words, $M$ ranges in $[1,M_{\text{max}}]$ for each
epoch. Therefore, the value of $e_{t,i}$ indicates the movement of the $i$-th
parameter in the most recent epoch. Or even more aggressively, the window can
range from 0 to the total number of batches seen during the whole training
progress. The adoption of the adaptive window size $M$ rather than a fixed one
has a benefit that we do not need to keep the past $M+1$ steps
$\\{\bm{x}_{t-M},\bm{x}_{t-M+1},\ldots,\bm{x}_{t}\\}$ to calculate the signal
and noise vectors $\\{\bm{s}_{t},\bm{n}_{t}\\}$ in Eq (15) since they can be
obtained in an accumulated fashion.

###  3.2 AdaSmooth

If the ER in magnitude of each parameter is small (approaching 0), the
movement in this dimension is zigzag, the proposed AdaSmooth method tends to
use a long period average as the scaling constant to slow down the movement in
that dimension. When the absolute ER per-dimension is large (tend to 1), the
path in that dimension is moving in a certain direction (not zigzag), and the
learning actually is happening and the descent is moving in a correct
direction where the learning rate should be assigned to a relatively large
value for that dimension. Thus the AdaSmooth tends to choose a small period
which leads to a small compensation in the denominator; since the gradients in
the closer periods are small when it’s near the (local) minima. A particular
example is shown in Figure 2, where the descent is moving in a certain
direction, and the gradient in the near periods is small; if we choose a
larger period to compensate for the denominator, the descent will be slower
due to the large factored denominator. In short, we want a smaller period to
calculate the exponential average of the squared gradients in Eq (6) if the
update is moving in a certain direction without a zigzag pattern; while when
the parameter is updated in a zigzag basis, the period for the exponential
average should be larger.

Figure 2: Demonstration of how the effective ratio works. Stochastic
optimization tends to move a large step when it is far from the (local)
minima; and a relatively small step when it is close to the (local) minima.

The obtained value of ER is used in the exponential smoothing formula. Now,
what we want to go further is to set the time period $N$ discussed in Eq (8)
to be a smaller value when the ER tends to 1 in absolute value; or a larger
value when the ER moves towards 0. When $N$ is small, SC is known as a “fast
SC”; otherwise, SC is known as a “slow SC”.

For example, let the small time period be $N_{1}=3$, and the large time period
be $N_{2}=199$. The smoothing ratio for the fast movement must be as for EMA
with period $N_{1}$ (“fast SC” = $\frac{2}{N_{1}+1}$ = 0.5), and for the
period of no trend EMA period must be equal to $N_{2}$ (“slow SC” =
$\frac{2}{N_{2}+1}$ = 0.01). Thus the new changing smoothing constant is
introduced, called the “scaled smoothing constant” (SSC), denoted by a vector
$\bm{c}_{t}\in{}^{d}$:

$\bm{c}_{t}=(\text{fast SC}-\text{slow SC})\times\bm{e}_{t}+\text{slow SC}.$
(16)

By Eq (8), we can define the fast decay constant
$\rho_{1}=1-\frac{2}{N_{1}+1}$, and the slow decay constant
$\rho_{2}=1-\frac{2}{N_{2}+1}$. Then the scaled smoothing constant vector can
be obtained by:

$\bm{c}_{t}=(\rho_{2}-\rho_{1})\times\bm{e}_{t}+(1-\rho_{2}),$ (17)

where the smaller $\bm{e}_{t}$, the smaller $\bm{c}_{t}$. For a more efficient
influence of the obtained smoothing constant on the averaging period, Kaufman
recommended squaring it. The final calculation formula then follows:

$E[\bm{g}^{2}]_{t}=\bm{c}_{t}^{2}\odot\bm{g}_{t}^{2}+\left(1-\bm{c}_{t}^{2}\right)\odot
E[\bm{g}^{2}]_{t-1}.$ (18)

We notice that $N_{1}=3$ is a small period to calculate the average (i.e.,
$\rho_{1}=1-\frac{2}{N_{1}+1}=0.5$) such that the EMA sequence will be noisy
if $N_{1}$ is less than 3. Therefore, the minimal value of $\rho_{1}$ in
practice is set to be larger than 0.5 by default. While $N_{2}=199$ is a large
period to compute the average (i.e., $\rho_{2}=1-\frac{2}{N_{2}+1}=0.99$) such
that the EMA sequence almost depends only on the previous value leading to the
default value of $\rho_{2}$ no larger than 0.99. Experiment study will show
that the AdaSmooth update will be insensitive to the hyper-parameters in the
sequel. We also carefully notice that when $\rho_{1}=\rho_{2}$, the AdaSmooth
algorithm recovers to the RMSProp algorithm with decay constant
$\rho=1-(1-\rho_{2})^{2}$ since we square it in Eq (18). After developing the
AdaSmooth method, we realize the main idea behind it is similar to that of SGD
with Momentum: to speed up (compensate less in the denominator) the learning
along dimensions where the gradient consistently points in the same direction;
and to slow the pace (compensate more in the denominator) along dimensions in
which the sign of the gradient continues to change.

Empirical evidence shows the ER used in simple moving average with a fixed
windows size $w$ can also reflect the trend of the series/movement in
quantitative strategies (Lu, 2022a). However, this again needs to store $w$
previous squared gradients in the AdaSmooth case, making it inefficient and we
shall not adopt this extension.

###  3.3 AdaSmoothDelta

Notice the ER can also be applied to the AdaDelta setting:

$\Delta\bm{x}_{t}=-\frac{\sqrt{E[\Delta\bm{x}^{2}]_{t}}}{\sqrt{E[\bm{g}^{2}]_{t}+\epsilon}}\odot\bm{g}_{t},$
(19)

where

$E[\bm{g}^{2}]_{t}=\bm{c}_{t}^{2}\odot\bm{g}_{t}^{2}+\left(1-\bm{c}_{t}^{2}\right)\odot
E[\bm{g}^{2}]_{t-1},$ (20)

and

$E[\Delta\bm{x}^{2}]_{t}=(1-\bm{c}_{t}^{2})\odot\Delta\bm{x}^{2}_{t}+\bm{c}_{t}^{2}\odot
E[\Delta\bm{x}^{2}]_{t-1},$ (21)

in which case the difference in $E[\Delta\bm{x}^{2}]_{t}$ is to choose a
larger period when the ER is small. This is reasonable in the sense that
$E[\Delta\bm{x}^{2}]_{t}$ appears in the numerator while $E[\bm{g}^{2}]_{t}$
is in the denominator of Eq (19) making their compensation towards different
directions. Or even, a fixed decay constant can be applied for
$E[\Delta\bm{x}^{2}]_{t}$:

$E[\Delta\bm{x}^{2}]_{t}=(1-\rho_{2})\Delta\bm{x}^{2}_{t}+\rho_{2}E[\Delta\bm{x}^{2}]_{t-1},$
(22)

The AdaSmoothDelta optimizer introduced above further alleviates the need for
a hand specified global learning rate which is set to $\eta=1$ from the
Hessian context. However, due to the adaptive smoothing constants in Eq (20)
and (21), the $E[\bm{g}^{2}]_{t}$ and $E[\Delta\bm{x}^{2}]_{t}$ are less
locally smooth making it less insensitive to the global learning rate than the
AdaDelta method. Therefore, a smaller global learning rate, e.g., $\eta=0.5$
is favored in AdaSmoothDelta. The full procedure for computing AdaSmooth is
then formulated in Algorithm 1.

Algorithm 1 Computing AdaSmooth at iteration $t$: the proposed AdaSmooth
algorithm. All operations on vectors are element-wise. Good default settings
for the tested tasks are
$\rho_{1}=0.5,\rho_{2}=0.99,\epsilon=1e-6,\eta=0.001$; see Section 3.2 or Eq
(8) for a detailed discussion on the explanation of the decay constants’
default values. Empirical study in Section 4 shows that the AdaSmooth
algorithm is not sensitive to the hypermarater $\rho_{2}$, while
$\rho_{1}=0.5$ is relatively a lower bound in this setting. The AdaSmoothDelta
iteration can be calculated in a similar way.

1: Input: initial parameter $\bm{x}_{1}$, Constant $\epsilon$;

2: Input: global learning rate $\eta$, by default $\eta=0.001$;

3: Input: fast decay constant $\rho_{1}$, slow decay constant $\rho_{2}$;

4: Input: assert $\rho_{2}>\rho_{1}$, by default $\rho_{1}=0.5$,
$\rho_{2}=0.99$;

5: for $t=1:T$ do

6: Compute gradient $\bm{g}_{t}=\nabla L(\bm{x}_{t})$;

7: Compute ER
$\bm{e}_{t}=\frac{|\bm{x}_{t}-\bm{x}_{t-M}|}{\sum_{i=0}^{M-1}|\Delta\bm{x}_{t-1-i}|}$
;

8: Compute smoothing
$\bm{c}_{t}=(\rho_{2}-\rho_{1})\times\bm{e}_{t}+(1-\rho_{2})$;

9: Compute normalization term:
$E[\bm{g}^{2}]_{t}=\bm{c}_{t}^{2}\odot\bm{g}_{t}^{2}+\left(1-\bm{c}_{t}^{2}\right)\odot
E[\bm{g}^{2}]_{t-1};$

10: Compute step
$\Delta\bm{x}_{t}=-\frac{\eta}{\sqrt{E[\bm{g}^{2}]_{t}+\epsilon}}\odot\bm{g}_{t}$;

11: Apply update $\bm{x}_{t}=\bm{x}_{t-1}+\Delta\bm{x}_{t}$;

12: end for

13: Return: resulting parameters $\bm{x}_{t}$, and the loss $L(\bm{x}_{t})$.

(a) MNIST training Loss

(b) Census Income training loss

Figure 3: MLP: Comparison of descent methods on MNIST digit and Census Income
data sets for 60 and 200 epochs with MLP.

##  4 Experiments

To evaluate the strategy and demonstrate the main advantages of the proposed
AdaSmooth method, we conduct experiments with different machine learning
models; and different data sets including real handwritten digit
classification task, MNIST (LeCun, 1998) 111It has a training set of 60,000
examples, and a test set of 10,000 examples., and Census Income 222Census
income data has 48842 number of samples where 70% of them are used as training
set in our case: https://archive.ics.uci.edu/ml/datasets/Census+Income. data
sets are used. In all scenarios, same parameter initialization is adopted when
training with different stochastic optimization algorithms. We compare the
results in terms of convergence speed and generalization. In a wide range of
scenarios across various models, AdaSmooth improves optimization rates, and
leads to out-of-sample performances that are as good or better than existing
stochastic optimization algorithms in terms of loss and accuracy.

###  4.1 Experiment: Multi-Layer Perceptron

Method | MNIST | Census
---|---|---
Momentum ($\rho=0.9$) | 98.64% | 85.65%
AdaGrad ($\eta$=0.01) | 98.55% | 86.02%
RMSProp ($\rho=0.99$) | 99.15% | 85.90%
AdaDelta ($\rho=0.99$) | 99.15% | 86.89%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.9$) |  99.34% |  86.94%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.95$) |  99.45% |  87.10%
AdaSmDel. ($\rho_{1}=0.5,\rho_{2}=0.9$) |  99.60% | 86.86%
Table 1: MLP: Best in-sample evaluation in training accuracy (%). AdaSmDel is short for AdaSmoothDelta. Method | MNIST | Census
---|---|---
Momentum ($\rho=0.9$) | 94.38% | 83.13%
AdaGrad ($\eta$=0.01) | 96.21% | 84.40%
RMSProp ($\rho=0.99$) | 97.14% | 84.43%
AdaDelta ($\rho=0.99$) | 97.06% | 84.41%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.9$) | 97.26% | 84.46%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.95$) | 97.34% | 84.48%
AdaSmDel. ($\rho_{1}=0.5,\rho_{2}=0.9$) | 97.24% |  84.51%
Table 2: MLP: Best out-of-sample evaluation in test accuracy for the first 5
epochs. AdaSmDel is short for AdaSmoothDelta.

Multi-layer perceptrons (MLP, a.k.a., multi-layer neural networks) are
powerful tools for solving machine learning tasks finding internal linear and
nonlinear features behind the model inputs and outputs. We adopt the simplest
MLP structure: an input layer, a hidden layer, and an output layer. We notice
that rectified linear unit (Relu) outperforms Tanh, Sigmoid, and other
nonlinear units in practice making it the default nonlinear function in our
structures. Since dropout has become a core tool in training neural networks
(Srivastava et al., 2014), we adopt 50% dropout noise to the network
architecture during training to prevent overfitting. To be more concrete, the
detailed architecture for each fully connected layer is described by
F$(\langle\textit{num outputs}\rangle:\langle\textit{activation
function}\rangle)$; and for a dropout layer is described by
DP$(\langle\textit{rate}\rangle)$. Then the network structure we use can be
described as follows:

$\text{F(128:Relu)DP(0.5)F(\text{num of classes}:Softmax)}.$ (23)

All methods are trained on mini-batches of 64 images per batch for 60 or 200
epochs through the training set. Setting the hyper-parameter to
$\epsilon=1e-6$. If not especially mentioned, the global learning rates are
set to $\eta=0.001$ in all scenarios. While a relatively large learning rate
($\eta=0.01$) is used for AdaGrad method since its accumulated decaying
effect; learning rate for the AdaDelta method is set to 1 as suggested by
Zeiler (2012) and for the AdaSmoothDelta method is set to 0.5 as discussed in
Section 3.3 while we will show in the sequel the learning rate for
AdaSmoothDelta will not influence the result significantly. In Figure 3(a) and
3(b) we compare SGD with Momentum, AdaGrad, RMSProp, AdaDelta, AdaSmooth and
AdaSmoothDelta in optimizing the training set losses for MNIST and Census
Income data sets respectively. The SGD with Momentum method does the worst in
this case. AdaSmooth performs slightly better than AdaGrad and RMSProp in the
MNIST case and much better than the latters in the Census Income case.
AdaSmooth shows fast convergence from the initial epochs while continuing to
reduce the training losses in both the two experiments. We here show two sets
of slow decay constant for AdaSmooth, i.e., “$\rho_{2}=0.9$” and
“$\rho_{2}=0.95$”. Since we square the scaled smoothing constant in Eq (18),
when $\rho_{1}=\rho_{2}=0.9$, the AdaSmooth recovers to RMSProp with
$\rho=0.99$ (so as the AdaSmoothDelta and AdaDelta case). In all cases, the
AdaSmooth results perform better while there is almost no difference between
the results of AdaSmooth with various hyper-parameters in the MLP model. Table
1 shows the best training set accuracy for different algorithms, indicating
the superiority of AdaSmooth. While we notice the best test set accuracy for
various algorithms are very close; we only report the best ones for the first
5 epochs in Table 2. In all scenarios, the AdaSmooth method converges slightly
faster than other optimization methods in terms of the test accuracy.

(a) Train Loss

(b) Test Loss

(c) Test Accuracy

Figure 4: CNN: Comparison of descent methods on MNIST digit data set for 50
epochs with CNN. Though not significantly better than RMSProp or SGD with
Momentum, AdaSmooth converges slightly faster and obtains better out-of-sample
accuracy.

(a) Epoch 1

(b) Epoch 5

(c) Epoch 20

(d) Epoch 50

Figure 5: CNN: The distribution of ERs for all parameters in different epochs
when training MNIST data for AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.99$) model.

(a) Threshold=0.02

(b) Threshold=0.04

(c) Threshold=0.06

(d) Threshold=0.1

Figure 6: CNN: Percentage of ERs when they are larger than a threshold for
different epochs in AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.99$) model.

(a) Epoch 1

(b) Epoch 5

(c) Epoch 20

(d) Epoch 50

Figure 7: CNN: The distribution of step $\Delta\bm{x}_{t}$ in magnitude for
different epochs in AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.99$) model.

###  4.2 Experiment: Convolutional Neural Networks

Convolutional neural networks (CNN) are powerful models with non-convex
objective functions. CNN with several layers of convolution, pooling and
nonlinear units have mostly demonstrated remarkable success in computer vision
tasks, e.g., face identification, traffic sign detection, and medical picture
segmentation (Krizhevsky et al., 2012) and speech recognition (Hinton et al.,
2012a; Graves et al., 2013); which is partly from the local connectivity of
the convolutional layers, and the rotational and shift invariance due to the
pooling layers. To evaluate the strategy and demonstrate the main advantages
of the proposed AdaSmooth method, a real handwritten digit classification
task, MNIST is used. For comparison with Zeiler (2012)’s method, we train with
Relu nonlinearities and 2 convolutional layers in the front of the structure,
followed by two fully connected layers. Again, the dropout with 50% noise is
adopted in the network to prevent from overfitting.

To be more concrete, the detailed architecture for each convolutional layer is
described by C$(\langle\textit{kernel size}\rangle:\langle\textit{num
outputs}\rangle:\langle\textit{activation function}\rangle)$; for each fully
connected layer is described by F$(\langle\textit{num
outputs}\rangle:\langle\textit{activation function}\rangle)$; for a max
pooling layer is described by MP$(\langle\textit{kernel
size}\rangle:\langle\textit{stride number}\rangle)$; and for a dropout layer
is described by DP$(\langle\textit{rate}\rangle)$. Then the network structure
we use can be described as follows:

C(5:10:Relu)MP(2:2) C(5:20:Relu)MP(2:2) (24) $\displaystyle\text{- DP(0.5)
F(50:Relu)DP(0.5)F(10:Softmax)}.$

All methods are trained on mini-batches of 64 images per batch for 50 epochs
through the training set. Setting the hyper-parameters to $\epsilon=1e-6$ and
$\rho_{1}=0.5,\rho_{2}=0.9\text{ or }0.99$, i.e., the number of periods chosen
find the exponential moving average are between $3$ and 19, or between 3 and
199 iterations as discussed in Section 2.3.1 and 3. The AdaSmooth with
$\rho_{2}=0.9$ or $0.99$ is a wide range of upper bound on the decay constant
from this context; and we shall see the AdaSmooth results are not sensitive to
these choices. If not specially described, a small learning rate $\eta=0.001$
for the convolutional networks is used in our experiments when applying
stochastic descent. While again, the learning rate for the AdaDelta method is
set to 1 as suggested by Zeiler (2012) and for the AdaSmoothDelta method is
set to 0.5 as discussed in Section 3.3.

In Figure 4(a), we compare SGD with Momentum, AdaGrad, RMSProp, AdaDelta,
AdaSmooth, and AdaSmoothDelta in optimizing the training set loss (negative
log likelihood). The RMSProp ($\rho=0.9$) does the worst in this case, whereas
tuning the decay constant to $\rho=0.99$ can significantly improve performance
making the RMSProp sensitive to the hyper-parameter (hence the learning rates
per-dimension). Further, though the training loss of AdaDelta ($\rho=0.9$)
decreases fastest in the first 3 epochs, the performance becomes poor at the
end of the training, with an average loss larger than 0.15; while the overall
performance of AdaDelta ($\rho=0.99$) works better than the former, however,
its overall accuracy is still worse than AdaSmooth. This also reveals the same
drawback of AdaGrad for the AdaDelta method, i.e., they are sensitive to the
choices of hyper-parameters.

In order to evaluate whether the AdaSmooth actually can find the compensation
we want, we also explore a random selection for the number of
periods/iterations $N$, termed as AdaSmooth(Random) in the sequel, which
selects the decay constant between $\rho_{1}$ and $\rho_{2}$ randomly during
different batches. The result shows the AdaSmooth(Random) cannot find the
adaptive learning rates per-dimension in the right way showing our method
actually finds the correct choices between the lower and upper bound of the
decay constant.

The analysis of different methods in test loss and test accuracy are
consistent with that of training loss as shown in Figure 4(b) and Figure 4(c).
We further save the effective ratios in magnitude as stated in Eq (15) during
each epoch; the distributions of them are shown in Figure 5. In the first few
epochs, the ERs in magnitude have more weights in large values, while in the
last few epochs, the probability decreases into a stationary distribution; in
other words, the ER distributions in the last few epochs are similar. In
Figure 6, we put a threshold on the ER values, the percentages of ERs larger
than the threshold are plotted; when the training is in progress, the ERs tend
to approach smaller values indicating the algorithm goes into stationary
points, the (local) minima. While Figure 7 shows the absolute values of update
step $\Delta\bm{x}_{t}$ for different epochs; when the training is in the
later stage, the parameters are closer to the (local) minima, indicating a
smaller update step and favoring a relatively smaller period (resulting from
both the step and the direction of the movement) of exponential moving average
in Eq (18), which in turn compensates relatively less for the learning rate
per-dimension and makes the update faster. In Table 3, we further show test
accuracy for AdaSmooth and AdaSmoothDelta with various hyper-parameters after
10 epochs in which case different choices of the parameters do not
significantly alter performance.

Method | MNIST
---|---
AdaGrad ($\eta$=0.01) | 96.82%
AdaGrad ($\eta$=0.001) | 89.11%
RMSProp ($\rho=0.99$) | 97.82%
RMSProp ($\rho=0.9$) | 97.88%
AdaDelta ($\rho=0.99$) | 97.83%
AdaDelta ($\rho=0.9$) | 98.20%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.9$) | 98.13%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.95$) | 98.12%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.99$) | 98.12%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.9,\eta=0.5$) | 98.86%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.95,\eta=0.5$) | 98.91%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.99,\eta=0.5$) | 98.78%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.99,\eta=0.6$) | 98.66%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.99,\eta=0.7$) | 98.66%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.99,\eta=0.8$) | 98.58%
Table 3: CNN: Best out-of-sample evaluation in test accuracy for AdaSmooth and
AdaSmoothDelta with various hyper-parameters after 10 epochs. AdaSmDel is
short for AdaSmoothDelta.

(a) MNIST training loss

(b) Census Income training loss

Figure 8: Logistic regression: Comparison of descent methods on MNIST digit
and Census Income data sets for 50 and 70 epochs with logistic regression.
Notice that the training curve of AdaSmooth $(\rho_{1}=0.5,\rho_{2}=0.9)$ is
close to $(\rho_{1}=0.5,\rho_{2}=0.95)$ in this case for both data sets.

###  4.3 Experiment: Logistic Regression

Logistic regression is an analytic technique for multivariate modeling of
categorical dependent variables and has a well-studied convex objective,
making it suitable for comparison of different optimizers without worrying
about going into local minima (Menard, 2002). To evaluate, again if not
mentioned explicitly, the global learning rates are set to $\eta=0.001$ in all
scenarios. While a relatively large learning rate ($\eta=0.01$) is used for
AdaGrad method since its accumulated decaying effect; learning rate for the
AdaDelta method is set to 1 as suggested by Zeiler (2012) and for the
AdaSmoothDelta method is set to 0.5 as discussed in Section 3.3. The loss
curves for training processes are shown in Figure 8(a) and 8(b), where we
compare SGD, SGD with Momentum, RMSProp, AdaDelta, AdaSmooth, and
AdaSmoothDelta in optimizing the training set losses for MNIST and Census
Income data sets respectively. The unaltered SGD method does the worst in this
case. AdaSmooth performs slightly better than RMSProp in the MNIST case and
much better than the latter in the Census Income case. AdaSmooth matches the
fast convergence of AdaDelta (in which case AdaSmooth converges slightly
slower than AdaDelta in the first few epochs), while AdaSmooth continues to
reduce the training loss, converging to the best performance in these models.
As aforementioned, when $\rho_{1}=\rho_{2}=0.9$, the AdaSmooth recovers to
RMSProp with $\rho=0.99$ (so as the AdaSmoothDelta and AdaDelta case). In all
cases, the AdaSmooth results perform better while the difference between
various hyper-parameters for AdaSmooth is not significant; there are almost no
differences in AdaSmooth results with different hyper-parameter settings,
indicating its insensitivity to hyper-parameters. In Table 4, we compare the
training set accuracy of various algorithms; AdaSmooth works best in the MNIST
case, while the superiority in Census Income data is not significant. Similar
results as the MLP scenario can be observed in the test accuracy and we shall
not give the details for simplicity.

Method | MNIST | Census
---|---|---
SGD ($\eta$=0.01) | 93.29% | 84.84%
Momentum ($\rho=0.9$) | 93.39% | 84.94%
RMSProp ($\rho=0.99$) | 93.70% | 84.94%
AdaDelta ($\rho=0.99$) | 93.48% | 84.94%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.9$) |  93.74% | 84.92%
AdaSmooth ($\rho_{1}=0.5,\rho_{2}=0.95$) | 93.71% | 84.94%
AdaSmoDel. ($\rho_{1}=0.5,\rho_{2}=0.9$) | 93.66% |  84.97%
Table 4: Logistic regression: Best in-sample evaluation in training accuracy.
AdaSmoDel is short for AdaSmoothDelta.

##  5 Conclusion

The aim of this paper is to solve the hyper-parameter tuning in the gradient-
based optimization methods for machine learning problems with large data sets
and high-dimensional parameter spaces. We propose a simple and computationally
efficient algorithm that requires little memory and is easy to implement for
gradient-based optimization of stochastic objective functions. Overall, we
show that AdaSmooth is a versatile algorithm that scales to large-scale high-
dimensional machine learning problems and AdaSmoothDelta is an algorithm
insensitive to both the global learning rate and the hyper-parameters with
caution to set the global learning rate smaller than 1.

## References

  * Becker & Le Cun (1988) Becker, Sue and Le Cun, Yann.  Improving the convergence of back-propagation learning with.  1988\.
  * Dauphin et al. (2014) Dauphin, Yann N, Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, Ganguli, Surya, and Bengio, Yoshua.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.  _Advances in neural information processing systems_ , 27, 2014.
  * Dozat (2016) Dozat, Timothy.  Incorporating nesterov momentum into adam.  2016\.
  * Duchi et al. (2011) Duchi, John, Hazan, Elad, and Singer, Yoram.  Adaptive subgradient methods for online learning and stochastic optimization.  _Journal of machine learning research_ , 12(7), 2011.
  * Graves et al. (2013) Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey.  Speech recognition with deep recurrent neural networks.  In _2013 IEEE international conference on acoustics, speech and signal processing_ , pp. 6645–6649. Ieee, 2013.
  * Hinton et al. (2012a) Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  _IEEE Signal processing magazine_ , 29(6):82–97, 2012a.
  * Hinton et al. (2012b) Hinton, Geoffrey, Srivastava, Nitish, and Swersky, Kevin.  Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.  _Cited on_ , 14(8):2, 2012b.
  * Kaufman (1995) Kaufman, Perry J.  Smarter trading, 1995.
  * Kaufman (2013) Kaufman, Perry J.  _Trading Systems and Methods,+ Website_ , volume 591.  John Wiley & Sons, 2013.
  * Kingma & Ba (2014) Kingma, Diederik P and Ba, Jimmy.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Krizhevsky et al. (2012) Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Le Roux & Fitzgibbon (2010) Le Roux, Nicolas and Fitzgibbon, Andrew W.  A fast natural newton method.  In _ICML_ , 2010.
  * LeCun (1998) LeCun, Yann.  The MNIST database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * Lu (2022a) Lu, Jun.  Exploring classic quantitative strategies.  _arXiv preprint arXiv:2202.11309_ , 2022a.
  * Lu (2022b) Lu, Jun.  Matrix decomposition and applications.  _arXiv preprint arXiv:2201.00145_ , 2022b.
  * Lu & Yi (2022) Lu, Jun and Yi, Shao.  Reducing overestimating and underestimating volatility via the augmented blending-ARCH model.  _arXiv preprint arXiv:2203.12456_ , 2022.
  * Menard (2002) Menard, Scott.  _Applied logistic regression analysis_ , volume 106.  Sage, 2002.
  * Moulines & Bach (2011) Moulines, Eric and Bach, Francis.  Non-asymptotic analysis of stochastic approximation algorithms for machine learning.  _Advances in neural information processing systems_ , 24, 2011.
  * Polyak & Juditsky (1992) Polyak, Boris T and Juditsky, Anatoli B.  Acceleration of stochastic approximation by averaging.  _SIAM journal on control and optimization_ , 30(4):838–855, 1992.
  * Qian (1999) Qian, Ning.  On the momentum term in gradient descent learning algorithms.  _Neural networks_ , 12(1):145–151, 1999.
  * Robbins & Monro (1951) Robbins, Herbert and Monro, Sutton.  A stochastic approximation method.  _The annals of mathematical statistics_ , pp. 400–407, 1951.
  * Ruder (2016) Ruder, Sebastian.  An overview of gradient descent optimization algorithms.  _arXiv preprint arXiv:1609.04747_ , 2016.
  * Rumelhart et al. (1986) Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J.  Learning representations by back-propagating errors.  _nature_ , 323(6088):533–536, 1986.
  * Ruppert (1988) Ruppert, David.  Efficient estimations from a slowly convergent robbins-monro process.  Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
  * Schaul et al. (2013) Schaul, Tom, Zhang, Sixin, and LeCun, Yann.  No more pesky learning rates.  In _International conference on machine learning_ , pp. 343–351. PMLR, 2013.
  * Smith (2017) Smith, Leslie N.  Cyclical learning rates for training neural networks.  In _2017 IEEE winter conference on applications of computer vision (WACV)_ , pp. 464–472. IEEE, 2017.
  * Srivastava et al. (2014) Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Sutskever et al. (2013) Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey.  On the importance of initialization and momentum in deep learning.  In _International conference on machine learning_ , pp. 1139–1147. PMLR, 2013.
  * You et al. (2019) You, Yang, Li, Jing, Reddi, Sashank, Hseu, Jonathan, Kumar, Sanjiv, Bhojanapalli, Srinadh, Song, Xiaodan, Demmel, James, Keutzer, Kurt, and Hsieh, Cho-Jui.  Large batch optimization for deep learning: Training bert in 76 minutes.  _arXiv preprint arXiv:1904.00962_ , 2019.
  * Zeiler (2012) Zeiler, Matthew D.  Adadelta: an adaptive learning rate method.  _arXiv preprint arXiv:1212.5701_ , 2012.