<p>Reinforcement learning is very different from traditional supervised learning because the training data distribution <em>changes</em> as the policy improves. In optimization terms, the objective function can be said to be <em>non-stationary</em>. For this reason, I suspect your intuition is likely correct -- that a "one-cycle" optimizer would perform poorly after a while in your application.</p>

<p>My question is, what is wrong with Adam? Typically, the choice of optimizer is a minor detail for deep reinforcement learning; other factors like the exploration policy, algorithmic hyperparameters, or network architecture tend to have a much greater impact on performance.</p>

<p>Nevertheless, if you really want to try other optimizers, you could experiment with RMSProp, Adadelta, or Nesterov Momentum. However, my guess is that you will see incremental improvements, if any. Perhaps searching for better hyperparameters to use with Adam would be a more effective use of time.</p>

<hr>

<p><strong>EDIT:</strong> In my original answer, I made the claim that the choice of a particular optimizer is not primarily important for reinforcement learning speed, and neither is generalization. I want to add some discussion that helps illustrate these points.</p>

<p>Consider how most deep policy gradient methods operate: they sample a trajectory of experience from the environment, estimate returns, and then conduct one or more gradient steps to improve the parameterized policy (<em>e.g.</em> a neural network). This process repeats until convergence (to a locally optimal policy).</p>

<p>Why must we continuously sample new experience from the environment? Because our current data can only provide a reasonable first-order approximation within a small <em>trust region</em> around the policy parameters that were used to collect that data. Hence, whenever we update the policy, we need to sample more data.</p>

<p>A good way to visualize this is to consider an <a href="https://en.wikipedia.org/wiki/MM_algorithm" rel="nofollow noreferrer">MM algorithm</a>. At each iteration, a surrogate objective is constructed based on the data we have now and then maximized. Each time, we will get closer to the true optimum, but the speed at which we approach it is determined only by the number of surrogates we construct -- <strong><em>not by the specific optimizer we use to maximize each surrogate</em></strong>. Adam might maximize each surrogate in fewer gradient steps than, say, RMSProp does, but this does not affect the learning speed of the agent (with respect to environment samples). It just reduces the number of minibatch updates you need to conduct.</p>

<p><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/c/cc/Mmalgorithm.jpg/440px-Mmalgorithm.jpg" rel="nofollow noreferrer"><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/cc/Mmalgorithm.jpg/440px-Mmalgorithm.jpg" alt="MM Algorithm"></a></p>

<p>SAC is a little more complicated than this, as it learns Q-values in an off-policy  manner and conducts updates using experience replay, but the general idea holds. The best attainable policy is subject to whatever the current data in our replay memory are; regardless of the optimizer we use, we will need to sample roughly the same amount of data from the environment to converge to the optimal policy.</p>

<p>So, how do you make a faster (more sample-efficient) policy gradient method? You need to fundamentally change the RL algorithm itself. For example, <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">PPO</a> almost always learns faster than <a href="https://arxiv.org/abs/1502.05477" rel="nofollow noreferrer">TRPO</a>, because John Schulman and co-authors found a different and empirically better way to generate policy gradient steps.</p>

<p>Finally, notice that there is no notion of generalization here. We have an objective function that we want to optimize, and once we do optimize it, we have solved the task as well as we can. This is why I suspect that the <a href="https://arxiv.org/abs/1705.08292" rel="nofollow noreferrer">"Adam-generalizes-worse-than-SGD"</a> issue is actually irrelevant for RL.</p>
Best Reinforcement Learner Optimizer
<p>I'm running a SAC reinforcement learner for a robotics application with some pretty decent results. One of the reasons I opted for reinforcement learning is for the ability for learning in the field, e.g. to adjust to a mechanical change, such as worn tires or a wheel going a little out of alignment.</p>

<p>My reinforcement learner restores it's last saved weights and replay buffer upon startup, so it doesn't need to retrain every time I turn it on. However, one concern I have is with respect to the optimizer.</p>

<p>Optimizers have come a long way since ADAM, but everything I read and all the RL code samples I see still seem to use ADAM with a fixed learning rate. I'd like to take advantage of some of the advances in optimizers, e.g. one cycle AdamW. However, a one-cycle optimizer seems inappropriate for a continuous real-world reinforcement learning problem: I imagine it's pretty good for the initial training/calibration, but I expect the low final learning rate would react too slowly to mechanical changes.</p>

<p>One thought I had was perhaps to do a one-cycle approach for initial training, and triggering a smaller one-cycle restart if a change in error that indicates something has changed (perhaps the size of the restart could be based on the size of the change in error).</p>

<p>Has anyone experimented with optimizers other than ADAM for reinforcement learning or have any suggestions for dealing with this sort of problem?</p>