We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons.   Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information.   A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones.   Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns.   In this manner, we enable the slope of the activation function to be context dependent.   This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks. Artificial neural networks (ANNs), such as convolutional neural networks (CNNs) BID24 and long short-term memory (LSTM) cells BID14 , have incredible capabilities and are applied in a variety of applications including computer vision, natural language analysis, and speech recognition among others. Historically, the development of ANNs (e.g., network architectures and learning algorithms) has benefited significantly from collaborations with Psych-Neuro communities BID5 BID12 BID13 BID15 BID28 Turing, 1950; BID10 BID8 BID16 BID18 BID10 . The information processing capabilities of traditional ANN nodes are rather rigid when compared to the plasticity of real neurons. A typical traditional ANN node linearly integrate its input signals and run the integration through a transformation called an activation function, which simply takes in a scalar value and outputs another. Of the most popular Activation Functions are sigmoid BID32 , tanh BID19 and ReLU BID35 .Researchers have shown that it could be beneficial to deploy layer-/node-specific activation functions in a deep ANN BID4 BID40 BID9 BID11 BID0 . However, each ANN node is traditionally stuck with a fixed activation function once trained. Therefore, the same input integration will always produce the same output. This fails to replicate the amazing capability of individual biological neurons to conduct complex nonlinear mappings from inputs to outputs BID1 BID10 BID25 . In this study , we propose one new modification to ANN architectures by adding a new type of node, termed modulators, to modulate the activation sensitivity of the ANN nodes targeted by modulators (see FIG0 for examples). In one possible setting, a modulator and its target ANN nodes share the same inputs. The modulator maps the input into a modulation signal, which is fed into each target node. Each target node multiples its input integration by the modulator signal prior to transformation by its traditional activation function. Examples of neuronal principles that may be captured by our new modification include intrinsic excitability, diverse firing modes, type 1 and type 2 forms of firing rate integration, activity dependent facilitation and depression and, most notably, neuromodulation BID27 BID38 BID45 BID37 ).Our modulator is relevant to the attention mechanism BID23 BID34 , which dynamically restricts information pathways and has been found to be very useful in practice. Attention mechanisms apply the attention weights, which are calculated in run-time, to the outputs of ANN nodes or LSTM cells. Notably, the gating mechanism in a Simple LSTM cell can also be viewed as a dynamical information modifier. A gate takes the input of the LSTM cell and outputs gating signals for filtering the outputs of its target ANN nodes in the same LSTM cell. A similar gating mechanism was proposed in the Gated Linear Unit BID6 for CNNs. Different from the attention and gating mechanisms , which are applied to the outputs of the target nodes, our modulation mechanism adjusts the sensitivities of the target ANN nodes in run-time by changing the slopes of the corresponding activation functions. Hence, the modulator can also be used as a complement to the attention and gate mechanisms.Below we will explain our modulator mechanism in detail. Experimentation shows that the modulation mechanism can help achieve better test stability and higher test performance using easy to implement and significantly simpler models. Finally, we conclude the paper with discussions on the relevance to the properties of actual neurons. We propose a modulation mechanism addition to traditional ANNs so that the shape of the activation function can be context dependent. Experimental results show that the modulated models consistently outperform their original versions. Our experiment also implied adding modulator can reduce overfitting. We demonstrated even with fewer parameters, the modulated model can still perform on par with it vanilla version of a bigger size. This modulation idea can also be expanded to other setting, such as, different modulator activation or different structure inside the modulator. It was frequently observed in preliminary testing that arbitrarily increasing model parameters actually hurt network performance, so future studies will be aimed at investigating the relationship between the number of model parameters and the performance of the network. Additionally, it will be important to determine the interaction between specific network implementations and the ideal Activation Function wrapping for slope-determining neurons. Lastly, it may be useful to investigate layer-wide single-node modulation on models with parallel LSTM's.Epigenetics refers to the activation and inactivation of genes BID46 , often as a result of environmental factors. These changes in gene-expression result in modifications to the generation and regulation of cellular proteins, such as ion channels, that regulate how the cell controls the flow of current through the cell membrane BID29 . The modulation of these proteins will strongly influence the tendency of a neuron to fire and hence affect the neurons function as a single computational node. These proteins, in turn, can influence epigenetic expression in the form of dynamic control BID20 .Regarding the effects of these signals, we can compare the output of neurons and nodes from a variety of perspectives. First and foremost, intrinsic excitability refers to the ease with which a neurons electrical potential can increase, and this feature has been found to impact plasticity itself BID7 . From this view, the output of a node in an artificial neural network would correspond to a neurons firing rate, which Intrinsic Excitability is a large contributor to, and our extra gate would be setting the node's intrinsic excitability. With the analogy of firing rate, another phenomenon can be considered. Neurons may experience various modes of information integration, typically labeled Type 1 and Type 2. Type 1 refers to continuous firing rate integration, while Type 2 refers to discontinuous information BID42 . This is computationally explained as a function of interneuron communication resulting in neuron-activity nullclines with either heavy overlap or discontinuous saddle points BID33 . In biology , a neuron may switch between Type 1 and Type 2 depending on the presence of neuromodulator BID41 . Controlling the degree to which the tanh function encodes to a binary space, our modification may be conceived as determining the form of information integration. The final possible firing rate equivalence refers to the ability of real neurons to switch between different firing modes. While the common mode of firing, Tonic firing, generally encodes information in rate frequency, neurons in a Bursting mode (though there are many types of bursts) tend to encode information in a binary mode -either firing bursts or not BID42 . Here too, our modification encompasses a biological phenomenon by enabling the switch between binary and continuous information.Another analogy to an ANN nodes output would be the neurotransmitter released. With this view, our modification is best expressed as an analogy to Activity Dependent Facilitation and Depression, phenomena which cause neurons to release either more or less neurotransmitter. Facilitation and depression occur in response to the same input: past activity BID36 . Our modification enables a network to use previous activity to determine its current sensitivity to input, allowing for both Facilitation and Depression. On the topic of neurotransmitter release , neuromodulation is the most relevant topic to the previously shown experiments. Once again, BID25 explains the situation perfectly, expressing that research BID2 BID3 has shown "the same neuron or circuit can exhibit different input-output responses depending on a global circuit state, as reflected by the concentrations of various neuromodulators". Relating to our modification, the slope of the activation function may be conceptualized as the mechanism of neuromodulation, with the new gate acting analogously to a source of neuromodulator for all nodes in the network.Returning to a Machine Learning approach, the ability to adjust the slope of an Activation Function has an immediate benefit in making the back-propagation gradient dynamic. For example, for Activations near 0, where the tanh Function gradient is largest, the effect of our modification on node output is minimal. However, at this point, our modification has the ability to decrease the gradient, perhaps acting as pseudo-learning-rate. On the other hand, at activations near 1 and -1, where the tanh Function gradient reaches 0, our modification causes the gradient to reappear, allowing for information to be extracted from inputs outside of the standard range. Additionally, by implementing a slope that is conditional on node input, the node has the ability to generate a wide range of functional Activation Functions, including asymmetric functions. Lastly, injecting noise has been found to help deep neural networks with noisy datasets BID47 , which is noteworthy since noise may act as a stabilizer for neuronal firing rates, BID43 . With this in mind, TAB2 .2 demonstrates increased clustering in two-dimensional node-Activation space, when the Activation Function slope is made to be dynamic. This indicates that noise may be a mediator of our modification, improving network performance through stabilization, induced by increasing the variability of the input-output relationship.In summary, we have shown evidence that nodes in LSTMs and CNNs benefit from added complexity to their input-output dynamic. Specifically, having a node that adjusts the slope of the main layer's nodes' activation functions mimics the functionality of neuromodulators and is shown to benefit the network. The exact mechanism by which this modification improves network performance remains unknown, yet it is possible to support this approach from both a neuroscientific and machine-learning perspective. We believe this demonstrates the need for further research into discovering novel non-computationally-demanding methods of applying principles of neuroscience to artificial networks. 6 APPENDIX 6.1 SUPPLEMENTARY DATA METHODOLOGY Additionally we tested our modulator gate, with τ l (·) set to sigmoid, on a much more computationally demanding three-layered LSTM network with weight drop method named awd-lstm-lm BID30 BID31 . This model was equipped to handle the Penn-Treebank dataset BID26 and was trained to minimize word perplexity. The network was trained for 500 epochs, however, the sample size was limited due to extremely long training times. On the Penn-Treebank dataset with the awd-lstm-lm implementation , sample size was restricted to 2 per condition, due to long training times and limited resources. However on the data collected, our model outperformed template perplexity, achieving an average of 58.4730 compared to the template average 58.7115. Due to the lack of a control for model parameters, interpretation of these results rests on the assumption that the author fine-tuned network parameters such that the template parameters maximized performance.
We propose a modification to traditional Artificial Neural Networks motivated by the biology of neurons to enable the shape of the activation function to be context dependent. A method to scale the activations of a layer of neurons in an ANN depending on inputs to that layer that reports improvements above the baselines. Introduction of an architectural change for basic neurons in a neural network, and the idea to multiply neuron linear combination output by a modulator prior to feeding it into the activation function.