<p>Seems that someone have sorted out (2018) the question (2017).</p>

<p>Vanilla adaptive gradients (RMSProp, Adagrad, Adam, etc) do not match well with L2 regularization.</p>

<p>Link to the paper [<a href="https://arxiv.org/pdf/1711.05101.pdf]" rel="noreferrer">https://arxiv.org/pdf/1711.05101.pdf]</a> and some intro:</p>

<blockquote>
  <p>In this paper, we show that a
  major factor of the poor generalization of the most popular
  adaptive gradient method, Adam, is due to the fact that L2
  regularization is not nearly as effective for it as for SGD.</p>
  
  <p>L2 regularization and weight decay are not identical.
  Contrary to common belief, the two techniques are not
  equivalent. For SGD, they can be made equivalent by
  a reparameterization of the weight decay factor based
  on the learning rate; this is not the case for Adam. <strong>In
  particular, when combined with adaptive gradients, L2
  regularization leads to weights with large gradients
  being regularized less than they would be when using
  weight decay.</strong></p>
</blockquote>
Should I avoid to use L2 regularization in conjuntion with RMSProp?
<p>Should I avoid to use L2 regularization in conjuntion with RMSprop and NAG?</p>

<p>The L2 regularization term interferes with the gradient algorithm (RMSprop)?</p>

<p>Best reggards,</p>