title: Why limit the weight size can prevent overfitting in machine learning
body: <p>The most popular way to prevent over-fitting is weight decay(L2, L1) in machine learning(Like <code>logistic regression</code>, <code>Neural network</code>, <code>linear regression</code> etc). The purpose of weight decay is preventing the weight get big.<br>
  <strong>My question is why small weight can prevent over-fitting</strong>.<br>
  <strong>what if I do weight normalization after each iteration</strong>.</p>