<p>I am wondering about this as well. The best thing I have found is <a href="https://arxiv.org/pdf/1603.02754.pdf" rel="nofollow noreferrer">the original XGBoost paper</a>. Section 2.1 makes it sound as if XGBoost uses regression tree as a main building block for both regression and classification. If this is correct, then Alpha and Lambda probably work in the same way as they do in the linear regression.</p>
<p>Gamma controls how deep trees will be. Large gamma means large hurdle to add another tree level. So larger gamma regularizes the model by growing shallower trees. E. g., depth-2 tree has smaller range of predicted values than depth-10 tree, so such model will have lower variance.</p>
What is the difference between alpha, lambda and gamma regularization parameters for xgboost?
<p>I have a question to ask:</p>
<p>How exactly are different L1 and L2 regularization terms on weights in xgboost algorithm.</p>
<p>As I understand, L1 is used by LASSO and L2 is used by RIDGE regression and L1 can shrink to 0, L2 can't. I understand the mechanics when using simple linear regression, but I have no clue how it works in tree based models.</p>
<p>Further more, gamma is another parameter, that makes the model more conservative. How should I notice the difference between L1/L2 and gamma parameter.</p>
<p>I have found in documentation very little to this problem:</p>
<p><strong>lambda [default=1, alias: reg_lambda]</strong></p>
<ul>
<li>L2 regularization term on weights. Increasing this value will make
model more conservative.</li>
</ul>
<p><strong>alpha [default=0, alias: reg_alpha]</strong></p>
<ul>
<li>L1 regularization term on weights. Increasing this value will make
model more conservative.</li>
</ul>
<p><strong>gamma [default=0, alias: min_split_loss]</strong></p>
<ul>
<li>Minimum loss reduction required to make a further partition on a leaf
node of the tree. The larger gamma is, the more conservative the
algorithm will be.</li>
</ul>
<p>All of them ranging from 0 to inf.</p>
<p>Thanks in advance for any answer/comment!</p>