We propose an efficient online hyperparameter optimization method which uses a joint dynamical system to evaluate the gradient with respect to the hyperparameters. While similar methods are usually limited to hyperparameters with a smooth impact on the model, we show how to apply it to the probability of dropout in neural networks. Finally, we show its effectiveness on two distinct tasks.
An algorithm for optimizing regularization hyper-parameters during training The paper proposes a way to re-initialize y at each update of lambda and a clipping procedure of y to maintain the stability of the dynamical system. Proposes an algorithm for hyperparameter optimization that can be seen as an extension of Franceschi 2017 were some estimates are warm restarted to increase the stability of the method. Proposes an extension to an existing method to optimize regularization hyperparameters.