Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention  thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement on the task performance, and the improvement is typically more significant when the task is more difficult. We also compare with the widely used entropy regularization and find $L_2$ regularization is generally better. Our findings are further confirmed to be robust against the choice of training hyperparameters. We also study the effects of regularizing different components and find that only regularizing the policy network is typically enough. We hope our study provides guidance for future practices in regularizing policy optimization algorithms. Regularization, typically referring to methods for preventing overfitting, is a key technique in successfully training a neural network. Perhaps the most widely recognized regularization methods in deep learning are L 2 regularization (also known as weight decay) and dropout (Srivastava et al., 2014) . Those techniques are standard practices in supervised learning tasks from many domains. Major tasks in computer vision, e.g., image classification (He et al., 2016; Huang et al., 2017) , object detection (Ren et al., 2015; Redmon et al., 2016) , all use L 2 regularization as a default option. In natural language processing, for example, the Transformer model (Vaswani et al., 2017) uses dropout. and the recently popular BERT model (Devlin et al., 2018) uses L 2 regularization. In fact, it is very rare to see state-of-the-art neural models trained without any regularization in a supervised setting. However, in deep reinforcement learning (RL), those conventional regularization methods are largely absent or underutilized in past research, possibly because in most cases we are maximizing the return on exactly the same task as in training. In other words, there is a lack of generalization gap from the training environment to the test environment (Cobbe et al., 2018) . Moreover, researchers in deep RL focus more on high-level algorithm designs, which is more closely related to the field of reinforcement learning, and focus less on network training techniques such as regularization. For popular policy optimization algorithms like Asynchronous Advantage Actor-Crtic (A3C) (Mnih et al., 2016) , Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) , Proximal Policy Optimization (PPO) , and Soft Actor Critic (SAC) (Haarnoja et al., 2018) , conventional regularization methods were not considered. Even in popular codebases such as the OpenAI Baseline , L 2 regularization and dropout were not incorporated. Instead, the most commonly used regularization in the RL community, is an "entropy regularization" term that penalizes the high-certainty output from the policy network, to encourage more exploration during the training process and prevent the agent from overfitting to certain actions. The entropy regularization was first introduced by Williams & Peng (1991) and now used by many contemporary algorithms (Mnih et al., 2016; Teh et al., 2017; Farebrother et al., 2018) . In this work, we take an empirical approach to questioning the conventional wisdom of not using common regularizations. We study agent's performance on the current task (the environment which the agent is trained on), rather than its generalization ability to different environments as many recent works (Zhang et al., 2018a; Zhao et al., 2019; Farebrother et al., 2018; Cobbe et al., 2018) . We specifically focus our study on policy optimization methods, which are becoming increasingly popular and have achieved top performance on various tasks. We evaluate four popular policy optimization algorithms, namely SAC, PPO, TRPO, and the synchronous version of Advantage Actor Critic (A2C), on multiple continuous control tasks. A variety of conventional regularization techniques are considered, including L 2 /L 1 weight regularization, dropout, weight clipping (Arjovsky et al., 2017) and Batch Normalization (BN) (Ioffe & Szegedy, 2015) . We compare the performance of these regularization techniques to that without regularization, as well as the entropy regularization. Surprisingly, even though the training and testing environments are the same, we find that many of the conventional regularization techniques, when imposed to the policy networks, can still bring up the performance, sometimes significantly. Among those regularizers, L 2 regularization, perhaps the most simple one, tends to be the most effective for all algorithms and generally outperforms entropy regularization. L 1 regularization and weight clipping can boost performance in many cases. Dropout and Batch Normalization tend to bring improvements only on off-policy algorithms. Additionally, all regularization methods tend to be more effective on more difficult tasks. We also verify our findings with a wide range of training hyperparameters and network sizes, and the result suggests find that imposing proper regularization can sometimes save the effort of tuning other training hyperparameters. Finally, we study which part of the policy optimization system should be regularized, and conclude that generally only regularizing the policy network suffices, as imposing regularization on value networks usually does not help. Our results also show that neural network training techniques such as regularization, can be as important as high-level reinforcement learning algorithms in terms of boosting performance. Our main contributions can be summarized as follows: • We provide the first comprehensive study of common regularization methods in policy optimization algorithms, which have been largely ignored in the RL literature. • We find conventional regularizations can often be very effective in improving the performance on continuous control tasks, espcially on harder ones. Remarkably, the most simple L 2 regularization generally performs better than the more widely used entropy regularization. BN and dropout can only help in off-policy algorithms. • We experiment with multiple randomly sampled training hyperparameters for each algorithm and confirm our findings still hold. The result also suggests that proper regularization can sometimes ease the hyperparameter tuning process. • We study which part of the network(s) should be regularized. The key lesson is to regularize the policy network but not the value network. Why does regularization benefit policy optimization? In RL, we are typically training and evaluating on the same environment, i.e., there is no generalization gap across different environments. However, there is still generalization between samples: the agents is only trained on the limited trajectories it has experienced, which cannot cover the whole state-action space of the environment. A successful policy needs to generalize from seen samples to unseen ones, which potentially makes regularization necessary in RL. This might also explain why regularization could be more helpful on harder tasks, which have larger state space. In this case, the portion of the space that have appeared in training tends to be smaller, and overfitting to this smaller portion of space would cause more serious issues, in which case regularizations may help. Some detailed analysis are provided in Appendix G. Why do BN and dropout work only with off-policy algorithms? One major finding in our experiments is BN and dropout can sometimes improve on the off-policy algorithm SAC, but mostly would hurt on-policy algorithms. There are two possible reasons for this: 1) for both BN and dropout, training mode is used to train the network, and testing mode is used to sample actions during interaction with the environment, leading to a discrepancy between the sampling policy and optimization policy (the same holds if we always use training mode). For on-policy algorithms, if such discrepancy is large, it can cause severe off-policy issues, which hurts the optimization process or even crashes it. For off-policy algorithms, this discrepancy is not an issue since they naturally accept off-policy data. 2) Batch Normalization layers can be sensitive to input distribution shifts, since the mean and std statistics depend heavily on the input, and if the input distribution changes too quickly in training, the mapping functions of BN layers can change quickly too, and it can possibly destabilize training. One evidence for this is that in supervised learning, when transferring a ImageNet pretrained model to other vision datasets, sometimes the BN layers are fixed (Yang et al., 2017) and only other layers are trained. In on-policy algorithms, we always use the samples generated from the latest policy; in off-policy algorithms, the sample distributions are relatively slow-changing since we always draw from the whole replay buffer which holds cumulative data. The faster-changing input distribution for on-policy algorithms could be harmful to BN. Previously, BN has also been shown to be effective in Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) , an off-policy algorithm. In summary, we conducted the first comprehensive study of regularization methods with multiple policy optimization algorithms on continuous control benchmarks. We found that L 2 regularization, despite being largely ignored in prior literature, is effective in improving performance, even more than the widely used entropy regularization. BN and dropout could also be useful but only on off-policy algorithms. Our findings were confirmed with multiple hyperparameters. Further experiments have shown that generally the best practice is to regularize the policy network alone but not the value network or both.
We show that conventional regularization methods (e.g., $L_2$, dropout), which have been largely ignored in RL methods, can be very effective in policy optimization. The authors study a set of existing direct policy optimization methods in the field of reinforcement learning and provide a detailed investigation on the effect of regulations on the performance and behavior of agents following these methods. This paper provides a study on the effect of regularization on performance in training environments in policy optimization methods in multiple continuous control tasks.