The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results. Since its conception, variants of the backpropagation algorithm have emerged. More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements. Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm. In this work, we show that function changes in the  backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network. Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks. Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures. Credit assignment BID10 is the task of identifying neurons and weights that are responsible for a desired prediction. Currently, the backpropagation (BP) algorithm BID9 ) is the de-facto standard for credit assignment in artificial neural networks. The backpropagation algorithm assigns credit by computing partial derivatives for weights and neurons with respect to the networks cost function.Variants of the backpropagation procedure have emerged since its conception. More specifically variants that exploit function changes in the backpropagation procedure. Feedback Alignment BID5 is considered a biologically plausible alternative to vanilla backpropagation. Feedback alignment is a variant of the backpropagation algorithm that uses a random weight matrix instead of the weight transpose matrix in the backpropagation equation. Despite not scaling to the ImageNet dataset BID1 , the algorithm relaxes BP weight symmetry requirements and demonstrate comparable learning capabilities to that of BP on small datasets Similarly, BID0 produced unique backpropagation equations to train an artificial neural network by learning parts of the backpropagation equations. BID0 report early convergence on unique backpropagation equations for CIFAR10. In this work, we demonstrate that function changes in the backpropagation equations particularly activation function derivatives is equivalent to adding an implicit learning rate in stochastic gradient descent.
We demonstrate that function changes in the backpropagation is equivalent to an implicit learning rate