title: What do non-linear activation functions do at a fundamental level in neural networks?
body: <p>I've been trying to find out what exactly non-linear activation functions do when implemented in a neural network. </p>
  
  <p>I know they modify the output of a neuron, but how and for what purpose?  </p>
  
  <p>I know they add non-linearity to otherwise linear neural networks, but for what purpose?  </p>
  
  <p>What exactly do they do to the output of each layer? Is it some form of classification?</p>
  
  <p>I want to know what exactly their purpose is within neural networks.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Activation_function" rel="nofollow noreferrer">Wikipedia</a> says that "the activation function of a node defines the output of that node given an input or set of inputs."  <a href="https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0" rel="nofollow noreferrer">This article</a> states that the activation function checks whether a neuron has "fired" or not.  I've looked at a bunch more articles and other questions on Stack Overflow as well, but none of them gave a satisfying answer as to what is occurring.</p>