<h3>CLOSED FORM (TIKHONOV) VERSUS GRADIENT DESCENT</h3>
<p>Hi! nice explanations for the intuitive and top-notch mathematical approaches there. I just wanted to add some specificities that, where not &quot;problem-solving&quot;, may definitely help to speed up and give some consistency to the process of finding a good regularization hyperparameter.</p>
<p>I assume that you are talking about the <strong>L2</strong> (a.k. &quot;weight decay&quot;) regularization, linearly weighted by the <em>lambda</em> term, and that you are optimizing the weights of your model either with the <strong>closed-form <a href="https://en.wikipedia.org/wiki/Tikhonov_regularization" rel="nofollow noreferrer">Tikhonov</a> equation</strong> (highly recommended for low-dimensional  linear regression models), or with some variant of <strong>gradient descent with backpropagation</strong>. And that in this context, you want to choose the value for <em>lambda</em> that provides best generalization ability.</p>
<hr />
<h3>CLOSED FORM (TIKHONOV)</h3>
<p>If you are able to go the Tikhonov way with your model (<a href="https://www.youtube.com/watch?v=NN7mBupK-8o#t=12m45s" rel="nofollow noreferrer">Andrew Ng</a> says under 10k dimensions, but this suggestion is at least 5 years old) <a href="https://en.wikipedia.org/wiki/Tikhonov_regularization#Determination_of_the_Tikhonov_factor" rel="nofollow noreferrer">Wikipedia - determination of the Tikhonov factor</a> offers an interesting <strong>closed-form solution, which has been proven to provide the optimal value</strong>. But this solution probably raises some kind of implementation issues (time complexity/numerical stability) I'm not aware of, because there is no mainstream algorithm to perform it. This <a href="https://arxiv.org/abs/1610.01952" rel="nofollow noreferrer">2016 paper</a> looks very promising though and may be worth a try if you really have to optimize your linear model to its best.</p>
<ul>
<li>For a quicker prototype implementation, this <a href="https://pypi.python.org/pypi/InverseProblem/1.0" rel="nofollow noreferrer">2015</a> Python package seems to deal with it iteratively, you could let it optimize and then extract the final value for the lambda:</li>
</ul>
<blockquote>
<p>In this new innovative method, we have derived an iterative approach to solving the general Tikhonov regularization problem, which converges to the noiseless solution, does not depend strongly on the choice of lambda, and yet still avoids the inversion problem.</p>
</blockquote>
<p>And from the <a href="https://github.com/kathrynthegreat/InverseProblem" rel="nofollow noreferrer">GitHub README</a> of the project:
<code>InverseProblem.invert(A, be, k, l) #this will invert your A matrix, where be is noisy be, k is the no. of iterations, and lambda is your dampening effect (best set to 1)</code></p>
<hr />
<h3>GRADIENT DESCENT</h3>
<p><em>All links of this part are from Michael Nielsen's amazing online book &quot;Neural Networks and Deep Learning&quot;, recommended reading!</em></p>
<p>For this approach it seems to be even less to be said: the cost function is usually non-convex, the optimization is performed numerically and the performance of the model is measured by some form of cross validation (see <a href="http://neuralnetworksanddeeplearning.com/chap3.html#overfitting_and_regularization" rel="nofollow noreferrer">Overfitting and Regularization</a> and <a href="http://neuralnetworksanddeeplearning.com/chap3.html#why_does_regularization_help_reduce_overfitting" rel="nofollow noreferrer">why does regularization help reduce overfitting</a> if you haven't had enough of that). But even when cross-validating, Nielsen suggests something: you may want to take a look at <a href="http://neuralnetworksanddeeplearning.com/chap3.html#regularization" rel="nofollow noreferrer">this detailed explanation</a>  on how does the L2 regularization provide a weight decaying effect, but the summary is that it is <strong>inversely proportional to the number of samples <code>n</code></strong>, so when calculating the gradient descent equation with the L2 term,</p>
<blockquote>
<p>just use backpropagation, as usual, and then add <code>(λ/n)*w</code> to the partial derivative of all the weight terms.</p>
</blockquote>
<p>And his conclusion is that, when wanting a similar regularization effect with a different number of samples, lambda has to be changed proportionally:</p>
<blockquote>
<p>we need to modify the regularization parameter. The reason is because the size <code>n</code> of the training set has changed from <code>n=1000</code> to <code>n=50000</code>, and this changes the weight decay factor <code>1−learning_rate*(λ/n)</code>. If we continued to use <code>λ=0.1</code> that would mean much less weight decay, and thus much less of a regularization effect. We compensate by changing to <code>λ=5.0</code>.</p>
</blockquote>
<p>This is only useful when applying the same model to different amounts of the same data, but I think it opens up the door for some intuition on how it should work, and, more importantly, speed up the hyperparametrization process by allowing you to finetune lambda in smaller subsets and then scale up.</p>
<p>For choosing the exact values, he suggests in his conclusions on <a href="http://neuralnetworksanddeeplearning.com/chap3.html#how_to_choose_a_neural_network%27s_hyper-parameters" rel="nofollow noreferrer">how to choose a neural network's hyperparameters</a> the purely empirical approach: start with 1 and then progressively multiply&amp;divide by 10 until you find the proper order of magnitude, and then do a local search within that region. In the comments of <a href="https://scicomp.stackexchange.com/a/10673">this SE related question</a>, the user Brian Borchers suggests also a very well known method that may be useful for that local search:</p>
<ol>
<li>Take small subsets of the training and validation sets (to be able to make many of them in a reasonable amount of time)</li>
<li>Starting with <code>λ=0</code> and increasing by small amounts within some region, perform a quick training&amp;validation of the model and plot both loss functions</li>
<li>You will observe three things:</li>
<li>The CV loss function will be consistently higher than the training one, since your model is optimized for the training data exclusively (<em>EDIT: After some time I've seen a MNIST case where adding L2 helped the CV loss decrease faster than the training one until convergence. Probably due to the ridiculous consistency of the data and a suboptimal hyperparametrization though</em>).</li>
<li>The training loss function will have its minimum for <code>λ=0</code>, and then increase with the regularization, since preventing the model from optimally fitting the training data is exactly what regularization does.</li>
<li>The CV loss function will start high at <code>λ=0</code>, then decrease, and then start increasing again at some point (<em>EDIT: this assuming that the setup is able to overfit for <code>λ=0</code>, i.e. the model has enough power and no other regularization means are heavily applied</em>).</li>
<li>The optimal value for <code>λ</code> will be probably somewhere around the minimum of the CV loss function, it also may depend a little on how does the training loss function look like. See the picture for a possible (but not the only one) representation of this: instead of &quot;model complexity&quot; you should interpret the x axis <strong>as <code>λ</code> being zero at the right and increasing towards the left</strong>.</li>
</ol>
<p><a href="https://i.stack.imgur.com/ZTQSP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZTQSP.png" alt="L2 diagnostics: instead of &quot;model complexity&quot; one should interpret the x axis **as λ being zero at the right and increasing towards the left" /></a></p>
<p>Hope this helps! Cheers,<br />
Andres</p>
How to calculate the regularization parameter in linear regression
<p>When we have a high degree linear polynomial that is used to fit a set of points in a linear regression setup, to prevent overfitting, we use regularization, and we include a lambda parameter in the cost function. This lambda is then used to update the theta parameters in the gradient descent algorithm.</p>

<p>My question is how do we calculate this lambda regularization parameter?</p>