and Byung-Woo Hong\* [^1] [^2]'
bibliography:
- 'image.bib'
title: Adaptive Weight Decay for Deep Neural Networks
---

[Shell : Adaptive Weight Decay for Deep Neural Networks]{}

Adaptive Regularization, Deep Learning, Neural Networks, Stochastic Gradient Descent, Weight-decay

Introduction {#sec:intro}
============

The deep neural network model consists of a nested architecture of layers where the number of parameters is in millions[@bottou2010large; @simonyan2014very; @szegedy2015going; @he2016deep; @he2016identity; @huang2017densely; @bottou2018optimization]. Due to its high degrees of freedom, the deep model can approximate linear and nonlinear functions; however it is always at risk of over-fitting to training data. Thus the deep neural network requires regularization techniques in training process in order to achieve generalization, resulting in a good prediction for unknown data.

Since the number of example data is also huge, the deep model is trained using the stochastic gradient descent (SGD) [@robbins1951stochastic; @rumelhart1988learning; @zhang2004solving; @bottou2010large; @bottou2018optimization] that updates parameters using a small subset of data (mini-batch) in combination with regularization techniques. A simple yet major regularization technique is, so-called, early stopping [@prechelt1998early; @zhang2016understanding] where the training process is terminated manually at a certain epoch before the validation loss increases. The noise injection techniques, e.g., the mini-batch procedure that induces noise to gradients [@zhu2019anisotropic] and the 