title: Neural Activation Functions - Difference between Logistic / Tanh / etc
body: <p>I'm writing some basic neural network methods - specifically the activation functions - and have hit the limits of my rubbish knowledge of math. I understand the respective ranges (-1/1) (0/1) etc, but the varying descriptions and implementations have me confused.</p>
  
  <p>Specifically <em>sigmoid</em>, <em>logistic</em>, <em>bipolar sigmoid</em>, <em>tanh</em>, etc.</p>
  
  <p>Does <em>sigmoid</em> simply describe the shape of the function irrespective of range? If so, then is tanh a 'sigmoid function'?</p>
  
  <p>I have seen 'bipolar sigmoid' compared against 'tanh' in a paper, however I have seen both functions implemented (in various libraries) with the same code:</p>
  
  <p>(( 2/ (1 + Exp(-2 * n))) - 1). Are they exactly the same thing?</p>
  
  <p>Likewise, I have seen <em>logistic</em> and <em>sigmoid</em> activations implemented with the same code:</p>
  
  <p>( 1/ (1 + Exp(-1 * n))). Are these also equivalent?</p>
  
  <p>Lastly, does it even matter that much in practise? I see on wiki a <a href="http://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Gjl-t%28x%29.svg/1000px-Gjl-t%28x%29.svg.png" rel="nofollow noreferrer">plot of very similar sigmoid functions</a> - could any of these be used? Some look like they may be considerably faster to compute than others.</p>