<p>Here are some papers describing gradient-based hyperparameter optimization:</p>

<ul>
<li><a href="https://arxiv.org/abs/1502.03492" rel="nofollow noreferrer">Gradient-based hyperparameter optimization through reversible learning</a> (2015):</li>
</ul>

<blockquote>
  <p>We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum. </p>
</blockquote>

<ul>
<li><a href="https://arxiv.org/abs/1703.01785" rel="nofollow noreferrer">Forward and reverse gradient-based hyperparameter optimization</a> (2017):</li>
</ul>

<blockquote>
  <p>We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. [2015] but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets.</p>
</blockquote>

<ul>
<li><a href="https://arxiv.org/abs/1909.13371" rel="nofollow noreferrer">Gradient descent: the ultimate optimizer</a> (2019):</li>
</ul>

<blockquote>
  <p>Working with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer's hyperparameters, such as the learning rate. There exist many techniques for automated hyperparameter optimization, but they typically introduce even more hyperparameters to control the hyperparameter optimization process. We propose to instead learn the hyperparameters themselves by gradient descent, and furthermore to learn the hyper-hyperparameters by gradient descent as well, and so on ad infinitum. As these towers of gradient-based optimizers grow, they become significantly less sensitive to the choice of top-level hyperparameters, hence decreasing the burden on the user to search for optimal values.</p>
</blockquote>
sklearn: Hyperparameter tuning by gradient descent?
<p>Is there a way to perform hyperparameter tuning in scikit-learn by gradient descent? While a formula for the gradient of hyperparameters might be difficult to compute, numerical computation of the hyperparameter gradient by evaluating two close points in hyperparameter space should be pretty easy. Is there an existing implementation of this approach? Why is or isn't this approach a good idea?</p>